DistilBERT for Knowledge Distillation on SQuAD v1.1

in

So how does it work? Well, imagine you have this massive, complicated machine learning model that takes forever to train and uses up all your computer’s resources. But what if we could make a smaller version of that model that doesn’t sacrifice too much accuracy or performance? That’s where DistilBERT comes in!

Here’s an example: let’s say you have this big, fancy BERT model that can answer questions like “What is the capital city of France?” with 99% accuracy. But what if we could make a smaller version of that same model (called DistilBERT) that still has pretty good accuracy but uses less resources and takes less time to train?

That’s exactly what they did! They took this big, fancy BERT model and “distilled” it down into a smaller version using some fancy machine learning techniques. The result is a model that can answer questions with around 90% accuracy (which isn’t bad at all!) but uses less resources and takes less time to train than the original BERT model.

So if you want to fine-tune your own language models for question answering or other tasks, DistilBERT might be a good option to consider! And if you don’t have access to fancy computer hardware or lots of data, it could still give you pretty decent results without breaking the bank (or your computer).

Now let me explain how we can use this model in PyTorch. First, we need to import some libraries:

# Importing necessary libraries
from transformers import DistilBertTokenizer, DistilBertForQuestionAnswering # Importing DistilBertTokenizer and DistilBertForQuestionAnswering from the transformers library
import torch # Importing torch library for PyTorch functionality

Next, we’ll load the pre-trained tokenizer and model using `DistilBertTokenizer.from_pretrained()` and `DistilBertForQuestionAnswering.from_pretrained()`, respectively:

# Load the pre-trained tokenizer using DistilBertTokenizer.from_pretrained()
tokenizer = DistilBertTokenizer.from_pretrained('distilbert-base-uncased-distilled-squad')

# Load the pre-trained model using DistilBertForQuestionAnswering.from_pretrained()
model = DistilBertForQuestionAnswering.from_pretrained('distilbert-base-uncased-distilled-squad')

And that’s it! Now we can use this model to answer questions using the `question_answerer()` function:

# Define a function to answer questions using a pre-trained model
def question_answerer(question):
    # Preprocess input text and question using a tokenizer
    inputs = tokenizer(question, return_tensors='pt')
    
    # Set up forward pass through our pre-trained model
    with torch.no_grad():
        # Pass the preprocessed inputs to the model and get the outputs
        outputs = model(**inputs)
        
    # Extract the answer from the output predictions
    # The model outputs a tuple, with the first element being the start scores and the second element being the end scores
    # We use indexing to get the start scores and end scores separately
    start_scores = outputs[0][:, 1]
    end_scores = outputs[0][:, 2]
    
    # Find the index of the highest-scoring token for both the start and end positions
    # The highest-scoring token is the one with the highest probability of being the start or end of the answer
    best_start_index = torch.argmax(start_scores)
    best_end_index = torch.argmax(end_scores)
    
    # Calculate the answer based on the indices we found earlier
    # The input_ids represent the tokenized input text, so we use the best start and end indices to get the answer from the input_ids
    answer = inputs['input_ids'][best_start_index:best_end_index+1].tolist()[0]
    
    return answer

And that’s it! You can now use this function to answer questions using your pre-trained DistilBERT model. Pretty cool, huh?

SICORPS