Comprehend_it-base

Usage instructions

How to use

The model can be loaded with the zero-shot-classification pipeline like so:

from transformers import pipeline
classifier = pipeline("zero-shot-classification",model="knowledgator/comprehend_it-base")

You can then use this pipeline to classify sequences into any of the class names you specify.

sequence_to_classify = "one day I will see the world"
candidate_labels = ['travel', 'cooking', 'dancing']
classifier(sequence_to_classify, candidate_labels)
#{'labels': ['travel', 'dancing', 'cooking'],
# 'scores': [0.9938651323318481, 0.0032737774308770895, 0.002861034357920289],
# 'sequence': 'one day I will see the world'}

If more than one candidate label can be correct, pass multi_label=True to calculate each class independently:

candidate_labels = ['travel', 'cooking', 'dancing', 'exploration']
classifier(sequence_to_classify, candidate_labels, multi_label=True)
#{'labels': ['travel', 'exploration', 'dancing', 'cooking'],
# 'scores': [0.9945111274719238,
#  0.9383890628814697,
#  0.0057061901316046715,
#  0.0018193122232332826],
# 'sequence': 'one day I will see the world'}

With manual PyTorch

# pose sequence as a NLI premise and label as a hypothesis
from transformers import AutoModelForSequenceClassification, AutoTokenizer
nli_model = AutoModelForSequenceClassification.from_pretrained('knowledgator/comprehend_it-base')
tokenizer = AutoTokenizer.from_pretrained('knowledgator/comprehend_it-base')

premise = sequence
hypothesis = f'This example is {label}.'

# run through model pre-trained on MNLI
x = tokenizer.encode(premise, hypothesis, return_tensors='pt',
                     truncation_strategy='only_first')
logits = nli_model(x.to(device))[0]

# we throw away "neutral" (dim 1) and take the probability of
# "entailment" (2) as the probability of the label being true 
entail_contradiction_logits = logits[:,[0,2]]
probs = entail_contradiction_logits.softmax(dim=1)
prob_label_is_true = probs[:,1]

Examples

Besides text classification, the model can be used for many other information extraction tasks.

Question-answering

The model can be used to solve open question-answering as well as reading comprehension tasks if it's possible to transform a task into a multi-choice Q&A.

#open question-answering
question = "What is the capital city of Ukraine?"
candidate_answers = ['Kyiv', 'London', 'Berlin', 'Warsaw']
classifier(question, candidate_answers)

# labels': ['Kyiv', 'Warsaw', 'London', 'Berlin'],
#  'scores': [0.8633171916007996,
#   0.11328165978193283,
#   0.012766502797603607,
#   0.010634596459567547]

#reading comprehension
question = 'In what country is Normandy located?'
text = 'The Normans (Norman: Nourmands; French: Normands; Latin: Normanni) were the people who in the 10th and 11th centuries gave their name to Normandy, a region in France. They were descended from Norse ("Norman" comes from "Norseman") raiders and pirates from Denmark, Iceland and Norway who, under their leader Rollo, agreed to swear fealty to King Charles III of West Francia. Through generations of assimilation and mixing with the native Frankish and Roman-Gaulish populations, their descendants would gradually merge with the Carolingian-based cultures of West Francia. The distinct cultural and ethnic identity of the Normans emerged initially in the first half of the 10th century, and it continued to evolve over the succeeding centuries.'
input_ = f"{question}\n{text}"

candidate_answers = ['Denmark', 'Iceland', 'France', "Norway"]

classifier(input_, candidate_answers)

#  'labels': ['France', 'Iceland', 'Norway', 'Denmark'],
#  'scores': [0.9102861285209656,
#   0.03861876204609871,
#   0.028696594759821892,
#   0.02239849977195263]

#binary question-answering
question = "Does drug development regulation become more aligned with modern technologies and trends, choose yes or no?"
text = "Drug development has become unbearably slow and expensive. A key underlying problem is the clinical prediction challenge: the inability to predict which drug candidates will be safe in the human body and for whom. Recently, a dramatic regulatory change has removed FDA's mandated reliance on antiquated, ineffective animal studies. A new frontier is an integration of several disruptive technologies [machine learning (ML), patient-on-chip, real-time sensing, and stem cells], which, when integrated, have the potential to address this challenge, drastically cutting the time and cost of developing drugs, and tailoring them to individual patients."
input_ = f"{question}\n{text}"

candidate_answers = ['yes', 'no']

classifier(input_, candidate_answers)

# 'labels': ['yes', 'no'],
#  'scores': [0.5876278281211853, 0.4123721718788147]}

Named-entity classification and disambiguation

The model can be used to classify named entities or disambiguate similar ones. It can be put as one of the components in an entity-linking system as a reranker.

text = """Knowledgator is an open-source ML research organization focused on advancing the information extraction field."""

candidate_labels = ['Knowledgator - company',
 'Knowledgator - product', 
 'Knowledgator - city']

classifier(text, candidate_labels)

# 'labels': ['Knowledgator - company',
#   'Knowledgator - product',
#   'Knowledgator - city'],
#  'scores': [0.887371301651001, 0.097423255443573, 0.015205471776425838]

Relation classification

With the same principle, the model can be utilized to classify relations from a text.

text = """The FKBP5 gene codifies a co-chaperone protein associated with the modulation of glucocorticoid receptor interaction involved in the adaptive stress response. The FKBP5 intracellular concentration affects the binding affinity of the glucocorticoid receptor (GR) to glucocorticoids (GCs). This gene has glucocorticoid response elements (GRES) located in introns 2, 5 and 7, which affect its expression. Recent studies have examined GRE activity and the effects of genetic variants on transcript efficiency and their contribution to susceptibility to behavioral disorders. Epigenetic changes and environmental factors can influence the effects of these allele-specific variants, impacting the response to GCs of the FKBP5 gene. The main epigenetic mark investigated in FKBP5 intronic regions is DNA methylation, however, few studies have been performed for all GRES located in these regions. One of the major findings was the association of low DNA methylation levels in the intron 7 of FKBP5 in patients with psychiatric disorders. To date, there are no reports of DNA methylation in introns 2 and 5 of the gene associated with diagnoses of psychiatric disorders. This review highlights what has been discovered so far about the relationship between polymorphisms and epigenetic targets in intragenic regions, and reveals the gaps that need to be explored, mainly concerning the role of DNA methylation in these regions and how it acts in psychiatric disease susceptibility."""

candidate_labels = ['FKBP5-associated with -> PTSD',
 'FKBP5 - has no effect on -> PTSD',
 'FKBP5 - is similar to -> PTSD',
 'FKBP5 - inhibitor of-> PTSD',
 'FKBP5 - ancestor of -> PTSD']

classifier(text, candidate_labels)

#  'labels': ['FKBP5-associated with -> PTSD',
#   'FKBP5 - is similar to -> PTSD',
#   'FKBP5 - has no effect on -> PTSD',
#   'FKBP5 - ancestor of -> PTSD',
#   'FKBP5 - inhibitor of-> PTSD'],
#  'scores': [0.5880666971206665,
#   0.17369700968265533,
#   0.14067059755325317,
#   0.05044548586010933,
#   0.04712018370628357]