Today we’re gonna talk about evaluating Koala model’s performance on open-source datasets. And let me tell you, it ain’t for the faint of heart.

To kick things off: what is Koala? Well, it’s a chatbot that was trained by fine-tuning Metas LLaMA on dialogue data gathered from the web. It can effectively respond to a variety of user queries and generate responses that are often preferred over Alpaca (which is another model). But let’s not get too carried away here, Koala still has limitations and can be harmful when misused.

Now, for evaluating its performance on open-source datasets: you’ll need to download the test set that was used in their experiments. This involves going to a website (which we won’t link because we don’t want to give away any secrets), finding the dataset, and then downloading it onto your computer. It’s like trying to find a needle in a haystack, but with more steps!

Once you have the test set, you can conduct a blind pairwise comparison by asking approximately 100 evaluators on Amazon Mechanical Turk platform to compare the quality of model outputs on these held-out sets of prompts. This involves creating an interface for them to judge which output is better (or that they are equally good) using criteria related to response quality and correctness. It’s like playing a game of “guess who” but with more math!

The results showed that Koala-All was rated as better than Alpaca in nearly half the cases, and either exceeded or tied Alpaca in 70% of the cases on their proposed test set. This suggests that effective instruction and assistant models could be finetuned from LLM backbones such as LLaMA entirely using data from larger and more powerful models, so long as the prompts for these responses are representative of the kinds of prompts that users will provide at test-time.

But let’s not forget about safety concerns! Koala can hallucinate and generate non-factual responses with a highly confident tone, which is likely a result of the dialogue fine-tuning. So be careful out there, don’t trust everything you hear from this chatbot!