Medusa: A Streaming Generation Method for LLMs

in

So instead of waiting for the whole piece of content to be generated before seeing anything, Medusa lets us see bits and pieces as they come out.

Here’s an example: let’s say you want to write a story about a group of adventurers exploring a mysterious jungle. You could use Medusa to generate snippets like this: “The team trekked deeper into the dense foliage, their hearts pounding with anticipation.” Or maybe something like this: “Suddenly, they heard a rustling in the bushes ahead was it an animal or just the wind?”

Now, you might be wondering how Medusa actually works. Well, it’s all about feeding the LLM small chunks of text at a time and then letting it generate a response based on that input. So instead of giving it the whole story to work with (which could take forever), we can just give it one sentence or paragraph at a time.

Here’s an example script you might use:

# Import necessary libraries
import random # Import the random library to generate random numbers
from transformers import AutoTokenizer, TFBertForSequenceClassification # Import the AutoTokenizer and TFBertForSequenceClassification from the transformers library

# Load the pre-trained model and tokenizer from Hugging Face Hub
model = TFBertForSequenceClassification.from_pretrained('bert-base-uncased') # Load the pre-trained model from Hugging Face Hub using the TFBertForSequenceClassification class
tokenizer = AutoTokenizer.from_pretrained('bert-base-uncased') # Load the tokenizer from Hugging Face Hub using the AutoTokenizer class

# Define a function to generate text using Medusa
def medusa(prompt, max_length=512):
    # Split the prompt into smaller chunks (max 512 tokens)
    for i in range(0, len(prompt), max_length):
        chunk = prompt[i:min(len(prompt), i+max_length)] # Set the chunk variable to a portion of the prompt, with a maximum length of 512 tokens
        
        # Tokenize and preprocess the input text
        inputs = tokenizer(chunk, return_tensors='tf') # Tokenize the chunk using the tokenizer and preprocess it by converting it to a tensor
        
        # Generate a response using the model
        outputs = model(inputs['input_ids'], training=False) # Use the model to generate a response based on the input_ids from the tokenizer
        
        # Get the predicted label (0 or 1) for this chunk of text
        pred = tf.argmax(outputs[0], axis=-1)[0] # Get the predicted label by finding the index of the maximum value in the output tensor
        
        # If it's a positive prediction, print out the generated response
        if pred == 1:
            print(tokenizer.decode(inputs['input_ids'][0][pred])) # Decode the input_ids and print the generated response
            
    return None # Return None to end the function

So basically, this script loads in a pre-trained model and tokenizer from Hugging Face Hub (which is a popular repository for sharing models and data), then defines a function called `medusa()`. This function takes two arguments: the prompt to generate text from, and an optional maximum length for each generated chunk.

The function first splits the input prompt into smaller chunks of up to 512 tokens (which is the default max_length). It then tokenizes and preprocesses each chunk using the loaded model’s tokenizer, which converts text into a format that can be fed into the LLM. The generated response is then passed through the model to get a predicted label for this chunk of text (0 or 1), based on whether it’s positive or negative.

If the prediction is positive, the function prints out the generated response using `tokenizer.decode()`, which converts the tokenized output back into human-readable text. And that’s pretty much how Medusa works! It’s a simple but powerful way to generate text in real time using LLMs, without having to wait for the whole piece of content to be generated at once.

SICORPS