Quick Start

LiqFit stands for Language Inference Quick Fit. This library is used for training models for few-shot tasks.

This quickstart is for a developer who wants to get a quick idea of how to use LiqFit.

First, install the LiqFit:

python -m pip install liqfit

LiqFit:

Unlike SetFit, LiqFit works with any encoder or encoder-decoder model not just the Sentence Transformer model.

You can use your training function or just use Huggingface's Trainer which exists in the Transformers library.

Training using the HuggingFace model:

1. Initialize your model

from transformers import AutoModel, AutoTokenizer

model_path = 'knowledgator/comprehend_it-base'
tokenizer = AutoTokenizer.from_pretrained(model_path)
model = AutoModelForSequenceClassification.from_pretrained(model_path)

3. Initialize NLIDataset and load your NLI dataset from huggingface, NLI stands for Natural Language Inference. We will use the emotion dataset from Hugging Face

from liqfit.datasets import NLIDataset
from datasets import load_dataset

emotion_dataset = load_dataset("dair-ai/emotion")
test_dataset = emotion_dataset['test']
classes = test_dataset.features["label"].names
# classes is a list that contains the following:
# ['sadness', 'joy', 'love', 'anger', 'fear', 'surprise']


N = 8 # take few examples
train_dataset = emotion_dataset['train'].shuffle(seed=41).select(range(len(classes)*N))

nli_train_dataset = NLIDataset.load_dataset(train_dataset, classes=classes)
nli_test_dataset = NLIDataset.load_dataset(test_dataset, classes=classes)

4. Wrap your model with LiqFitModel, this will be useful if you want to pass your loss function or your downstream head if your backbone model does not have one (will show it below).

from liqfit.modeling import LiqFitModel
from liqfit.losses import FocalLoss

backbone_model = AutoModelForSequenceClassification.from_pretrained('microsoft/deberta-v3-xsmall')

loss_func = FocalLoss(multi_target=True)

model = LiqFitModel(backbone_model.config, backbone_model, loss_func=loss_func)

5. Initialize your collator

from liqfit.collators import NLICollator
data_collator = NLICollator(tokenizer, max_length=128, padding=True, truncation=True)

6. Train your model

training_args = TrainingArguments(
    output_dir='comprehendo',
    learning_rate=3e-5,
    per_device_train_batch_size=3,
    per_device_eval_batch_size=3,
    num_train_epochs=9,
    weight_decay=0.01,
    evaluation_strategy="epoch",
    save_steps = 5000,
    save_total_limit=3,
    remove_unused_columns=False,
)

trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=nli_train_dataset,
    eval_dataset=nli_test_dataset,
    tokenizer=tokenizer,
    data_collator=data_collator,
)

trainer.train()

Training using Hugging Face model and custom downstream head:

What if your model is an encoder model and just outputs the last hidden state after pooling only? You will need to define your downstream head that is customized to your task, we have some available downstream heads that you can use easily.

Example: If your task is a Sequence Classification task and you want to use cross-entropy loss, you will do the following while defining your LiqFitModel:

from transformers import AutoModel, AutoTokenizer
from liqfit.modeling import ClassClassificationHead
from liqfit.modeling import LiqFitModel

model_path = 'knowledgator/comprehend_it-base'
tokenizer = AutoTokenizer.from_pretrained(model_path)
# # No logits should be returned when using just AutoModel.
model = AutoModel.from_pretrained(model_path)

hidden_size = model.config.hidden_size

# will use cross entropy loss by default if labels are passed.
downstream_head = ClassClassificationHead(in_features=hidden_size, out_features=3)
wrapped_model_with_custom_head = LiqFitModel(model.config, model, head=downstream_head)

And that's it!

Training using Hugging Face model and customizing your backbone model outputs:

Some use cases where you might want to wrap your backbone model in LiqFitBackbone:

• If your model is an encoder model and just outputs the last hidden state without pooling

• If you want to do extra operations on the backbone model outputs before passing the outputs to a custom downstream head

from transformers import AutoModel, AutoTokenizer
from liqfit.modeling import ClassClassificationHead
from liqfit.modeling import LiqFitModel, LiqFitBackbone
from liqfit.modeling.pooling import GlobalMaxPooling1D

model_path = 'knowledgator/comprehend_it-base'
tokenizer = AutoTokenizer.from_pretrained(model_path)

# Note that the model is now defined inside this class
class BackboneWrapped(LiqFitBackbone):
    def __init__(self):
        # No logits should be returned when using just AutoModel.
        model = AutoModel.from_pretrained(model_path)
        super().__init__(config=model.config, backbone=model)
        self.pooler = GlobalMaxPooling1D()
    
    def forward(self, x):
        # the `output` might not be pooled in this case
        # and we want to perform pooling operation
        # because our task is sequence classification.
        output = self.backbone(x)
        last_hidden_state = output.last_hidden_state
        pooled_output = self.pooler(last_hidden_state)
        return pooled_output

my_wrapped_backbone = BackboneWrapped()
downstream_head = ClassClassificationHead(in_features=model.config.hidden_size,
                                          out_features=3)

wrapped_model_with_custom_head = LiqFitModel(my_wrapped_backbone.config,
                                             my_wrapped_backbone,
                                             head=downstream_head)

Using ClassificationHead

If you want more flexibility in passing different loss functions other than cross entropy to your downstream head or if you want to pass custom pooler to the downstream head you can do the following:

from transformers import AutoModel, AutoTokenizer
from liqfit.modeling import ClassificationHead
from liqfit.modeling import LiqFitModel
from liqfit.losses import FocalLoss
from liqfit.modeling.pooling import GlobalMaxPooling1D

model_path = 'knowledgator/comprehend_it-base'
tokenizer = AutoTokenizer.from_pretrained(model_path)
# # No logits should be returned when using just AutoModel.
model = AutoModel.from_pretrained(model_path)

hidden_size = model.config.hidden_size

# will use cross entropy loss by default if labels are passed.
loss = FocalLoss()
pooler = GlobalMaxPooling1D()
downstream_head = ClassificationHead(in_features=hidden_size, out_features=3,
                                     loss_func=loss, pooler=pooler)
wrapped_model_with_custom_head = LiqFitModel(model.config, model, head=downstream_head)