I’m giving a talk at ODSC East 2025 on the topic of benchmark pitfalls, and I wanted to give a taste of it here! See below for a link to the talk.

For the uninitiated, a benchmark is a standardized open source test set for an AI task. The idea of a benchmark or even a test set for AI is not new by any stretch. The general idea when training any AI model is to split a (usually) massive amount of data into “splits” and train the model on the largest portion of the data (the training split), validate your results along the way using a smaller subset (the validation split) and a similarly small subset is used to finally “test” the model at the end (the test split). The idea is that if a team agrees on a given train/val/test split, then we could evaluate models fairly, knowing it wasn’t a difference in data that made the difference. Below is an image from one of my books, highlighting that fine-tuning process.

Test sets in general (benchmarks being an example of a test set) help AI engineers know if their hard work on training models paid off (Source: A Quick Start Guide to LLMs, by yours truly, Sinan Ozdemir)

But what if the model you are making is not necessarily meant for you or for your team, but in fact meant for as many people as possible, and what if we can’t standardize a training set, because that itself has some proprietary / “secret sauce”?  That’s where benchmarks come in. Someone (or usually some people) create and propose a test set (the benchmark) and hope people adopt it. Moreover, whatever training data an organization needs to use to train their model, just need to make sure it’s not from this benchmark or else that’d be cheating (more on that later). 

Benchmarks are often the first thing people ask of a new LLM: “How did it score on XYZ benchmark” or “is this model better than ZYX model at benchmarks?” and frankly that’s the primary purpose of benchmarks: to serve as a top-line conversation starter when evaluating an LLM for a certain job. This post and my session overall will tackle the urge for us to consider benchmarks as more than a conversation starter, but a conversation ender. 

We will explore three main areas of benchmarks in this post:

1. Benchmarks becoming targets: LLM creators are incentivized to chase the top of leaderboards and LLM consumers conflate benchmark performance with everyday, real-world performance.

2. Biases and shortcuts: Static benchmarks often contain artifacts and biases that models exploit, making benchmarks artificially easier than they appear.

3. Overstated progress: High scores on benchmarks don't mean models have true generalization or human-level understanding.

Let’s dig in.

Benchmarks becoming targets

We’ve been measuring AI using benchmarks for decades. Benchmarks like SQuAD (you’re forgiven if you’ve never heard of it) have been used to measure an AI’s ability to perform question/answer tasks like, “What is the largest city of Poland?” The image below shows the progress of AI on classic benchmarks, superseding “human performance” (denoted as the 0 mark on the y axis) in the last decade. The goal of the benchmarks was to give us humans a consistent, shared view of how well our AI systems were doing.

That’s still true today; on its face, targeting benchmark performance is useful to measure top-line progress of AI performance. The problem becomes when we all hyper-focus on a relatively small subset of benchmarks and equate a model’s performance on the benchmark with the AI’s overall performance. It leads to one of the most difficult questions I have to answer in a lecture: “Which model is currently the best?”

Benchmark saturation over time for popular benchmarks, normalized with initial performance at minus one and human performance at zero. (Source: https://arxiv.org/pdf/2104.14337)

This is not the right question to ask. The better question is, “how will this particular model perform on my particular task and is there a benchmark that gives me any indication of that?” It’s less punchy and puts more work on the consumer, but it better covers the current adoption of AI: task-oriented and usually domain-focused. 

That being said, we sometimes can’t even trust an LLM’s ability to solve a benchmark consistently. The image below depicts a study showing the performance of GPT-4 going from 84% to 51% performance on a math benchmark within only 3 months of release (from March to June 2023). This was because OpenAI released a new version of the model to little fanfare with a later knowledge cutoff, but didn’t report how benchmarks shifted. So we, the persuadable public, still assume the benchmarks they reported in March were accurate; they weren’t.

GPT-4 and GPT-3.5 benchmark performances shifting wildly within only 3 months (Source: https://arxiv.org/pdf/2307.09009.pdf)

Nonetheless, we look to benchmarks on new LLM releases to give us that topline view. But once a benchmark like SQuAD (or more recently MMLU) becomes a standard, researchers and models optimize heavily to improve benchmark scores, chasing the top of a leaderboard to show the world what they’ve done and hopefully sell some credits. So should benchmarks even be a target to chase? Sure they should be, especially when the task is relatively nuanced and so is the benchmark (like testing a model’s financial tool selecting ability using a financial tool selection benchmark). Benchmark performance is a great top line set of numbers to help us create a shortlist of models to consider, but it can’t be the end of the story. This wouldn’t be a post about benchmarks without at least one reference to the 1970’s Goodhart’s Law: 

"When a measure becomes a target, it ceases to be a good measure." 

Said another way, when we optimize too hard for a particular metric (performance on a benchmark), people (or in this case, AI models) will find ways to "game" the system and score higher metrics without actually improving at the underlying goal in any meaningful way. One way it can do that is by finding shortcuts to take to get better grades.

Biases and shortcuts

A section on benchmark biases and shortcuts could frankly be its own book. In fact I made a 12 hour video series focusing on this topic; there’s a lot to say. For now, let’s focus on a real and imminent shortcut AI’s can suffer from: data contamination. Data contamination is when an AI trains on data suspiciously similar to benchmark questions, artificially inflating a model’s performance on a benchmark. Basically, what if an AI accidentally or maliciously was given a cheat sheet?

An investigation into data contamination showed that letting Llama 2 train on rephrased benchmark questions that passed industry standard data contamination detection would have beaten GPT-4 at the popular MMLU benchmark. Too bad Meta has never open-sourced Llama’s training data so we can’t double check that work. (Source: https://arxiv.org/abs/2311.04850)

One way data contamination can happen is when a benchmark becomes so popular, that the open internet starts to consist of more data that might contain a rephrasing that is justtttt different enough to trick industry-standard contamination detection techniques (usually just embedding similarity and n-gram overlap [basically a keyword search] checks). A frighteningly simple research experiment (shown above) rephrased questions from the MMLU benchmark just enough to pass such industry-standard techniques and found that Llama-2 could have beaten GPT-4 on the benchmark if it had been allowed to train from them.

It’s easier than we think to let a benchmark question slip into the training data of a model. Experiments like these show that companies are likely doing a decent job at de-contaminating training data (otherwise we’d be seeing saturation at an even greater rate) but when models aren’t fully open source (including it’s training data and code) like every single Llama model, we can never double check the work being done.

Overstated progress

Put simply, benchmarks simply don’t cover a majority of what most humans would consider general intelligence, let alone “superintelligence”. Most benchmark questions are multiple choice and most benchmarks reported by companies are in the field of math and coding which really isn’t that helpful if you’re using an LLM to write marketing copy or to classify incoming customer support tickets with a particular intent class. One of the most talked about modern benchmarks is “Humanity’s Last Exam” or HLE. In their own words, this benchmark is designed to be the “final closed-ended academic benchmark of its kind with broad subject coverage.” And by “academic”, and “broad”, they mean almost entirely STEM.

Humanity’s Last Exam is mostly a STEM exam, featuring 3 questions tagged as “League Of Legends” questions 🤷(Source: https://github.com/centerforaisafety/hle)

To be 100% clear, there’s no chance I’ll ever pass, let alone ace this closed-book exam. For one, I don’t play League of Legends and I am terrible at Chess (both categories are represented in this benchmark). When an AI can pass this benchmark (yes I said when), I will find it extremely impressive but I will also have many questions, starting with:

1. Was your AI able to look up information in trying to pass the exam?

2. Can you prove to me you didn’t help your AI “cheat” by fine-tuning it on similar data?

3. Can you convince me that I even care about your AI knowing how long the Second Great War was in StarCraft Lore? (Yes, that’s a real question in this benchmark)

The counterpoint to this line of questioning is that AI adoption will eventually evolve beyond targeted LLM use-cases and prompting and that benchmarks like HLE are less meant for targeted adoption of AI and more meant to signal a turning point in AI: the heralding of Artificial General Intelligence (AGI) or Superintelligence - an AI going beyond human intelligence. To put it bluntly, even an AI scoring 100% on HLE would not alone trigger a sense of AGI or Superintelligence to me. These high benchmark scores look impressive to us, the consumer, but they can stop reflecting true generalization of AI, exactly what Goodhart’s Law predicts.

So what do we do?

We will dive deeper into remedies for benchmarks in my live session and I plan to make a follow up post after the talk, but I’ll outline a few simple steps we can take now:

1. Use benchmarks to create a short list of models to evaluate - don’t use them to select a single model.

2. Make your own test sets - They are valid and frankly will tell you more than most public benchmarks will on your particular tasks.

3. Ask yourself, “who made this benchmark” and “what are they really trying to test?” Is it true reasoning beyond the knowledge an AI has, or simply the recall of a few facts that when put together, simulate reasoning.

Other topics in the session will include:

1. The path to “AGI” and “Superintelligence” and a deeper dive into the “Humanity’s Last Exam” benchmark

2. How AI developers use prompting when benchmarking, leading to public misconception of AI strength

3. Addressing staleness in benchmarks: what if answers to questions change over time?

If you’re in Boston, stop by and say hi!

Beyond Benchmarks: Evaluating AI Agents, Multimodal Systems, and Generative AI in the Real World

odsc.com/speakers/beyond-benchmarks-evaluating-ai-agents-multimodal-systems-and-generative-ai-in-the-real-world

AI Agents are all the rage, and I get it. The promise of letting an LLM just pick the right tool for the job is very appealing if not somewhat “magical”, but of course if I’m writing this, then it means there’s a lot more to it than meets the eye.

At it’s most basic, an AI Agent is an auto-regressive LLM (virtually any commercial Generative AI model like GPT, Llama, Mistral, Claude, Command-R, etc) with a prompt telling the LLM how to reason through tasks by selecting and running tools which can be APIs, image generation models (like how ChatGPT uses DALL-E to make images), execute code, really anything that can be distilled down into a simple run function.

Your Basic AI Agent relies on an auto-regressive LLM’s ability to think through a task and select the right tool at the right time.

Agents are useful in theory, but in practice can often fall short. Evaluating agents can be done on several levels including:

1. Making sure the final answer is accurate, and helpful

2. Ensuring the latency/speed of the system is good enough

3. Mitigating failures of the LLM to reason through a complex task

One of the more underrated evaluation criteria is the quantifying the ability of the LLM to select the right tool at the right time. On it’s face it's obvious that we have to measure this but many people dismiss this as being just part of the overall system and if the answer is right at the end, that would imply the agent selected the right tools, right? Well not always. Perhaps the agent selected the wrong tool twice before fumbling it’s way into the right one and that would impact both the latency and the accuracy overall.

Moreover, there are underlying issues with the deep learning architecture that virtually every LLM is based on, the Transformer. While there’s no doubt that the invention of the Transformer was one of the greatest advancements in NLP in the last several decades, there’s one particular type of bias it falls prey to quite often, the positional bias.

Depending on where the tools are in the agent prompt, tools listed later in the list might end up towards the middle of the prompt, where information can get ignored due to positional bias

Positional bias essentially means the LLM has a tendency to pay more attention to tokens at the start or end of the prompt while glossing over tokens in the middle. You may have heard this called the "lost-in-the-middle" problem. This can be a big deal when it comes to agents, especially if the LLM favors tools that are recorded earlier in the prompt, glossing over later tools which often appear towards the middle of the overall prompt. As a result, the LLM could pick the wrong tool.

Testing Tool Selection

To properly investigate tool selection in agents, let’s run a simple test. Our test will run in 3 stages:

Stage 1 - Setup

1. Choose an agent framework to test. I made my own here

2. Write a test set where we write a fairly simple task which (mostly) obviously matches to a single tool. I have 5 tools total. Examples include:

   • Check the status of my NFT listings → 'Crypto Lookup Tool'

   • Add a new row and just write "To do" in it → 'Google Spreadsheet Tool'

   • Convert 98 degrees Fahrenheit to Celsius using Python → 'Python Tool'

3. Define several LLMs to test against. I tested several from OpenAI, Anthropic, a Mistral model, a few Llama models and a Gemini model

Stage 2 - Run the Agent + Log results

1. Choose an n (I chose n=10)

2. For each test datapoint, and for each LLM, shuffle the tools around and pass the order of the tools into the agent framework n times.

   • for each time, log the correct tool index, the chosen tool index, and whether the agent was correct.

Stage 3 - Calculate Results

1. Calculate the accuracy, precision, recall, and F1 for each LLM on its tool selection along with broken down metrics (see the final results in the github)

2. Calculate the % difference between each tool index being chosen and the index being correct to try and see if the LLM favored any particular tool indices.

The Results

The notebook with the test & results (see references) has about a dozen graphs in it but here are a few key takeaways:

Tool Selection Accuracy can vary greatly between LLMs

Depending on which LLM I tried, there were pretty stark differences between tool selection accuracy. It’e tempting to look at this and say “Oh ok, so Anthropic’s Claude 3.5 Haiku" is clearly the best LLM for agents”. Incorrect 🙂 this is a test for my agent framework on my tools and on my data. I hope you will take this post/notebook as a framework to follow when testing your own LLMs!

To no one’s surprise, the choice of LLM impacted overall tool selection accuracy

Positional Bias is Real

The graph below shows the average % difference between how often the agent chose a particular tool index (there are 5 bars because I had 5 tools) over how often that tool index was actually correct. so a 9.51% in the first bar means that on average, the LLMs chose the first tool in the list 9.51% more often that the index was correct. For example if that tool index was the correct tool index 95 times during the test, the LLM actually chose that tool index roughly 104 times. Meanwhile the later tools are under-chosen, showing evidence of a positional bias.

On average, the chosen LLMs tended to over-select tools in earlier indexes

You might be thinking that it was the smaller open source models that really skewed the results, but if you look at the results broken down by model provider, even OpenAI models fall victim to positional bias:

Even the “gold standard” OpenAI LLMs fall victim to positional biases

Conclusion

The allure of AI agents lies in their potential to solve a freeform task by choosing the right tools at the right moment, but the reality is far more nuanced. This experiment highlights that evaluating an agent's performance goes beyond simply checking the final answer and how long it took to get there. Even when an agent ultimately reaches the correct solution, inefficient tool selection driven by inherent biases can impact accuracy, latency, and consistency.

Moreover, even the most advanced LLMs from top providers like OpenAI and Google are not immune to these challenges. The over-selection of tools appearing earlier in the list underscores the need for robust testing frameworks and deeper investigations into the LLM’s decision-making process.

The takeaway? Don’t assume a strong final output implies flawless tool selection. Use testing frameworks like the one shared here to rigorously test, iterate, and refine your agents for better real-world performance. And remember, the right tool at the right time isn’t just magical—it’s measurable.

References

This work came from my lecture & video on agents which you can find on O’Reilly. Here is the Github for both the lecture and the video as well as the github for my agent framework that I used to perform this test.

Squad Goals

My own AI agent framework that I used to run this test. Feel free to install and contribute!

AI Agents - Tool Selection Performance

An introduction to the world of AI Agents on O’Reilly

github.com/sinanuozdemir/oreilly-ai-agents/blob/main/notebooks/agent_positional_bias_tools.ipynb

In August of 2022, mere months before ChatGPT made it’s world debut, I wrote a post on medium^[1] about using auto-encoding LLMs like BERT to embed and retrieve documents from a vector database and then return responses to a query using information from that document. Sound familiar? I was describing a simplified version of Retrieval-Augmented Generation (RAG) inspired by a paper^[2] in 2020. My version used two types of auto-encoding LLMs - one to retrieve and the other to “generate” a response by selecting the best subset of the document that answered the question (I know that’s not actual LLM text generation, but I wanted to use something open source and it was 2022).

Auto-encoding LLMs are models that cannot generate text token by token like the “Generative AI” models - ChatGPT, Claude, Llama, or virtually any LLM on the market today - but rather models who’s sole purpose is to read quickly and efficiently at much smaller sizes. To put that size difference in perspective, a case study in my book has a 70M parameter DistilBERT model beating ChatGPT (2,500x bigger parameter-wise) in a head to head fine-tuning classification test.

Case study from my book: DistilBERT (70M params) performing at a similar level as GPT 3.5 (175B params) on the same training data (https://hf.co/datasets/app_reviews) while being nearly twice as fast as GPT 3.5. Size isn’t everything

I’ve been both fascinated and disappointed in the field of auto-encoding models in recent years. So few companies seem to want to innovate on non-generative LLMs so when a use-case like RAG comes up where a huge chunk of that pipeline involves reading / retrieval i.e. not generating anything, I get excited.

RAG can be broken down into three main steps (these figures are from my 2022 post but still are relevant):

1. Indexing documents - Using an embedding system to transform raw text into vectors and storing them in a database

   

2. Retrieving documents - Using (usually) the same embedding system to embed a query and using a vector similarity metric (like cosine similarity) to find the most relevant document

   

3. Generating a response - Using an LLM to create a raw text response to a user’s query using information in the document (yes I know there’s a typo in the figure, 2022 me messed up).

   

One of the main limitations of RAG systems is the quality of the retrieved document ranking. Cosine similarity between embeddings can only go so far in terms of matching queries to documents. This can be quantified by measuring how often an input query gets matched to the correct document which we will refer to as the top result accuracy of a RAG system, namely that the #1 closest retrieved document is in the fact the correct document that can answer the query. This post will go over an unsung hero in RAG that aims to maximize the effectiveness of this document ranking - the re-ranker.

Re-ranking Documents

At it’s core, a re-ranker is yet another LLM who’s job it is to take in a small amount (usually 10-50) of documents and the original query and rank the documents from most to least relevant. That sounds exactly like basic cosine retrieval because, well, it is the same result - a list of ranked documents.

How the re-ranking LLM does this is where things get different. Re-ranking happens on a much smaller scale than basic retrieval using cosine similarity from a vector DB because re-rankers’ architectures (often cross-encoders) are often more memory consumptive and slower but yield more precise results. They are considered an optional step between retrieval and generation.

Let’s look at a quick case study - a chatbot meant to help people navigate questions about Social Security.

Borrowing Government Data

Don’t worry, no one’s getting on a watch list for reading this. I’m taking just over 100 FAQs from https://faq.ssa.gov with corresponding help articles and using this as my data for this example. By the way, the full case study can be found at my O’Reilly RAG course^[3] .

Our data for this case study: pre-written FAQs about America’s Social Security system

I will use OpenAI’s text-embedding-3-small model to do all document embeddings and Pinecone for my vector database. If you follow the code in ^[3] (namely the retrieval / generation notebooks), I will simply embed all FAQs and store them (along with the url and raw text) in a vector database for retrieval. I didn’t want to use a well known benchmark here because frankly I already think most LLM benchmarks are unhelpful to the average person and I grow increasingly worried that companies will want to simply overfit to these benchmarks to get market hype so I try to think of simple yet relatable non-benchmark examples.

Now we need to create some test data so we can start to see how well our system is performing.

Generating Synthetic Test Data

Grain of salt alert! We are going to ask GPT-4 to generate some potential questions to test our retrieval against. Synthetic data generation is a new sub-genre of generative tasks and one with consequential downstream effects. The test data we use here will inform us as to how well our chatbot is retrieving information and therefore is a measure of how well our bot can perform. My point here is that I will be using the below prompt to generate test data but I cannot actually read the non-english examples and vet them myself so I am taking the questions generated by GPT-4 here with a grain of salt.

I am designing a chatbot to use this document as information to our users.

Please write 10 questions that an average person not educated in this social security system might ask that can definitely be answered using information using the provided document.

Try to ask in a way that's confusing to really test our system's knowledge but still fair.

I need 5 in English, 2 in Spanish, 2 in Chinese, and 1 in French in that order.

Use this format to output:
Document: A given document to make questions from
JSON: ["english question 1", "english question 2", "english question 3", ...  "spanish question 1", "spanish question 2", ..., "french question 1"]
###
Document: {document}
JSON:

>>>

[('english',
  'How do I start the process for getting disability benefits from Social Security?'),
 ..
 ('spanish', '¿Cómo solicito beneficios por discapacidad del Seguro Social?'),
 ..
 ('chinese', '我不在美国居住，我可以申请社会保障残疾福利吗？'),
 ..
 ('french',
  "Comment puis-je contacter mon bureau de sécurité sociale local pour des prestations d'invalidité?")]

With all that, let’s look at our baseline results of just using OpenAI’s embedder and pinecone’s basic retrieval (just cosine similarity).

Baseline Results

For the 220 questions (10 per a 20% sample of our scraped urls) and for each language I generated data in, I broke it up and calculated two items:

1. the % of times the expected document was even in the list of 10 retrieved documents from Pinecone

2. The % of times the expected document was the top document in the list (will always less than or equal to the first number)

OpenAI embeddings alone are a decent showing with English performing the worst tied with Spanish at 82% top result accuracy.

Using OpenAI’s embeddings alone gives us about a 84% accuracy overall (weighted by language) of the synthetic test set. Not all languages were able to grab the document at all from Pinecone. To raise that number, we could grab more documents or use a different / fine-tuned embedder. Both great things to test, but not the main point of this post.

Making Retrieved Documents better with re-rankers

We finally arrive at the crux of this post. Once we retrieve the documents from our vector database, you can pass it along to a generative AI and call it a day. But with re-ranking systems and just 5-10 more lines of code (not a hyperbole, check out the Github^[3] ), we can re-sort those documents from Pinecone to try and surface the actual relevant document to the top of the list. If we can consistently do this, we can pass fewer documents to our final RAG generation prompt resulting in a tighter, faster, and cheaper integration.

I evaluated two re-ranking systems for this:

1. Cohere’s v3 multilingual re-ranker - likely the largest company providing a marketable solution to the document re-ranking problem

2. Pongo’s semantic filter - one of the few small companies innovating in this space

Both of them work in a really simple way: provide a query and raw documents (not the OpenAI embeddings, they don’t matter to the re-ranker) and get back an ordered list of documents from most to least relevant. The test is simple - add this re-ranking step to the 10 retrieved documents from Pinecone. We will still be limited by the relevant document actually existing in the original 10, but we will be comparing Cohere and Pongo against simply using no re-ranking whatsoever.

Final Results

Everyone’s RAG system is different and your data will be different. For the data outlined above, our final results can be summarized as follows:

1. Both Cohere and Pongo improved top result accuracy from 84% to ~90% (~7% increase in performance).

2. Both models slowed the system down (not seen in the graph but both systems more than doubled the time to the testing process). This makes sense because we are actively performing a secondary LLM action.

3. Cohere’s model (being explicitly trained for multilingual use-cases) outperformed Pongo on Chinese, French, and Spanish.

4. Pongo beat Cohere on English examples (which represented 50% of the testing set).

Both Pongo and Cohere made our retrieval rankings better with ~5 lines of added code!

Overall, adding re-ranking to a system can take mere minutes to code up and as long as you have a proper testing set and a way to run tests automatically, there is no reason you cannot test your RAG systems against these re-rankers to see if they will have a net benefit on the retrieval accuracy.

Happy re-ranking!

References

[1] My August 2022 post on RAG:

Building a Natural Language Interface from an FAQ using pre-trained language models

Quickly and easily build a natural language interface using a static knowledge base as your source

medium.com/@profoz/building-a-natural-language-interface-from-an-faq-using-pre-trained-language-models-1c150dd572df

[2] The Original RAG Paper:

Retrieval-Augmented Generation for Knowledge-Intensive NLP TasksOriginal RAG Paperarxiv.org/abs/2005.11401

[3] My current RAG class materials:

Sinan Ozdemir’s RAG Course on O’Reilly

github.com/sinanuozdemir/oreilly-retrieval-augmented-gen-ai

Introduction to Quantization

Quantization refers to the technique of representing models using fewer bits by reducing the precision of its parameters. This process involves converting continuous or high-precision values into a smaller set of discrete values, typically by mapping floating-point numbers to integers. The primary goal of quantizing large language models (LLMs) is to decrease memory usage and accelerate inference.

There are several methods to quantize a model, which I won't get into as there are already excellent resources available (see reference [3]). Instead, I wanted to focus on a specific use case I get asked about a lot as an AI consultant and teacher: deploying an off-the-shelf model without further fine-tuning. These models could be ones pre-trained by other organizations, like Llama-3-8B, or previously fine-tuned on specific datasets without quantization. This post will not cover the process of fine-tuning while quantizing, which involves techniques such as QLORA (I have codes examples for this in reference [2]).

Python code to quantize a model is relatively straightforward using popular packages like transformers which have implementations of algorithms like NF4 (see below code sample and reference [3] for more details). NF4, which stands for NormalFloat 4, is a particularly effective strategy for maintaining the performance of AI models. Originally introduced in the QLORA paper, NF4 has become a preferred choice in modern quantization strategies.

# Import necessary classes and functions from the transformers library
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig

# Define the model name to load from Hugging Face's model hub
model_name = 'meta-llama/Meta-Llama-3-8B-Instruct'

# Configure the quantization settings using BitsAndBytesConfig
# Setting load_in_4bit to True enables 4-bit quantization
# bnb_4bit_use_double_quant enables double quantization for more precise control
# bnb_4bit_quant_type specifies the NF4 quantization algorithm
# bnb_4bit_compute_dtype sets the data type for computation to bfloat16 for efficiency
bits_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_use_double_quant=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16
)

# Initialize the tokenizer for the model
tokenizer = AutoTokenizer.from_pretrained(model_name)

# Load and configure the quantized model
qt_model = AutoModelForCausalLM.from_pretrained(
    model_name, 
    quantization_config=bits_config,    
    device_map="auto"
).eval()  # Set the model to evaluation mode which disables training specific operations like dropout

# Load the non-quantized version of the same model
non_qt_model = AutoModelForCausalLM.from_pretrained(
    model_name, 
    device_map="auto"
).eval()  # Set the model to evaluation mode

The not so straightforward part is testing both quantized and non-quantized models side by side on our main three considerations:

1. Optimizing Inference - memory and latency reduction

2. Raw token output differences - measuring the raw differences between the next token prediction outputs

3. Performance on benchmarks / test sets - running generative benchmarks and comparing the two models

I will use be using Meta’s llama-3-8B-Instruct model as my reference.

Consideration 1 - Optimizing Inference

Probably the most well known benefits of quantization are the inference gains both in memory usage and in latency/throughput. Lower parameter precision means less memory required to hold the model and faster computations. The memory usage and latency differences are dramatic between the two models and hold at both small and larger batch sizes.

Measuring the peak memory usage and latency of the forward pass of Llama 3-8B shows striking differences. The Non-Quantized model (red) uses far more memory (top) and takes far longer to process inputs in batch sizes between 1 and 32 (bottom).

Quantized models are supposed to be faster and more memory efficient so this is just the tip of the iceberg. Are they as reliable as their non-quantized cousin? Are they better? Worse? Let’s see how we can find out.

Consideration 2 - Raw Token Output Differences

This next graph has me asking both versions of the Llama 3 model 163 questions from a subset of MMLU-Virology (the benchmark content isn’t as relevant here) and using the Jaccard Index (Similarity) - a similarity metric between two sets as the number of items they have in common divided by the total number of unique items between them - to measure the differences between the raw next token predictions for each input at various cutoff points - k=1, 2, 3, etc. This will give us a straightforward way to quantify the differences in raw model output of quantized and non-quantized models.

I chose the Jaccard Index also for its robustness in scenarios where the exact alignment of token sets is less important than the overall overlap, making it ideal for evaluating models where slight deviations in token predictions are acceptable. We can see that most tokens are in common but a non-insignificant number of tokens are in fact different.

The Jaccard similarity between the top k predicted tokens of the quantized and non-quantized model on a subset of MMLU-virology

Given this graph, roughly speaking, we can expect about 75-80% of the tokens to match in the top 1, 3, 5, 10, and 20 predicted tokens for this test set, which can lead to performance differences (see consideration 3). These raw token outputs will not only affect performance on test sets but will also yield differences in the inference parameters that we set. For example, setting a top_p (which affects token probabilities) for a non-quantized model might yield drastically different results on the quantized version.

Consideration 3 - Performance on Test Sets

Considerations 1 and 2 were measuring the differences in raw next token predictions both in similarity and in speed/memory usage but neither were considering the accuracy of what those tokens represented. We saw non-insignificant differences between which tokens might be outputted which suggests that there will be differences in benchmark performance.

I’m planning a post on benchmarking in more detail but for now, I’m going to pass a very simple 0-shot prompt to each model on a subset of MMLU-Virology. I measured the words per minute (which I expected to be better for the quantized model) and the accuracy on the multiple choice questions.

Note: The only inference parameter I set was a temperature of 0.1 to induce some more consistency and reproducibility of the experiment. This choice will also highlight any token differences by making the differences in token probabilities sharper.

The Quantized Model (Red in both graphs) has a better word per minute rate (top) but performs slightly worse on a subset of the MMLU benchmark (bottom).

Right out of the gate, the non quantized model is performing slightly better on this benchmark subset but has a much lower WPM (no surprise there given the forward pass calculations in consideration 1). The difference in performance comes down to the fact that quantization is objectively altering the model from how it was trained. It’s not always going to be true that the quantized version of a model will perform worse but especially on well known benchmarks like MMLU that companies like Meta, OpenAI, Anthropic, etc test their models on, it’s a good bet. It’s always good to test.

To mitigate this, we could fine-tune the model while quantizing using a technique like QLORA (reference [2]).

Conclusion

Quantization offers tangible benefits in terms of reducing memory usage and enhancing the speed of computations. This has been demonstrated effectively in the case of Llama-3-8B, where quantized models significantly outperform their non-quantized counterparts in memory efficiency and processing speed during inference.

However, quantization does come with built in trade-offs. The alterations in precision can lead to differences in token output and potentially affect performance on benchmarks and practical applications. The balance between efficiency and accuracy must be carefully tested and managed, and for critical applications, performing some fine-tuning post-quantization using QLORA may be necessary to restore or enhance model performance.

I hope this helps!

References

[1] Code for these graphs

Quantization Comparisons

colab.research.google.com/drive/12RTnrcaXCeAqyGQNbWsrvcqKyOdr0NSm?usp=sharing

[2] QLORA example (see the SFT notebook in colab)

Guide to AI Alignment with Reinforcement Learning

What RL can and cannot do

ai-office-hours.beehiiv.com/p/aligning-llms

[3] Primer on Quantization from HuggingFace:

Quantization

We’re on a journey to advance and democratize artificial intelligence through open source and open science.

huggingface.co/docs/peft/main/en/developer_guides/quantization

There are active debates over whether LLMs are just memorizing vast amounts of statistics or if they can learn a more cohesive representation of the world whose language they model. Some have found evidence for the latter by analyzing the learned representations of datasets and even go so far as to discover that LLMs can learn linear representations of space and time (arxiv.org/abs/2310.02207).

As part of the 2nd edition of my latest LLM book (coming out later this year) one idea I wanted to add as a net new section aimed at recreating some of the work done in this paper by looking at a dataset comes from a paper entitled “A cross-verified database of notable people, 3500 BC-2018 AD” claiming to build a “comprehensive and accurate database of notable individuals”; just what we need to probe some LLMs on their ability to retain information about notable individuals they read about on the web.

I’m lucky to live in an age where open data for so many things exist: doi.org/10.1038/s41597-022-01369-4

Our steps to conduct the probe will be:

1. Design a prompt. At its simplest we will just say the name of the individual - like “Albert Einstein”

2. Instigate a forward pass of our LLM and grab embeddings from the middle layer and the final layer of our LLM’s hidden states.

   1. For auto-encoding models like BERT, we will grab the reserved CLS token’s embedding and for auto-regressive models like Llama or Mistral, we will grab the embedding of the final token.

3. Use those token embeddings as inputs to a linear regression problem where we attempt to fit to three fields of the dataset plus a control fourth:

   1. birth - the birth year of the individual

   2. death - the death year of the individual (we filter to only use people who have died so this value is filled)

   3. wiki_readers_2015_2018 - average per year number of page views in all Wikipedia editions (information retrieved in 2015–2018). We will use this as a weak signal to the notoriety level of the individual

   4. random gibberish - just np.random.rand(len(dataset)). We will use this as a control as we should not be able to see any prediction signal

Probing gives us a way to understand how much information is locked away with the parameters of a model and whether or not we can extract that information through external processes. We place classifiers or regression layers in our case on top of hidden states and attempt to extract information like the birth year of the person we stated in the original prompt.

The goal of probing is not to act in place of an evaluation for a task but rather as an evaluation of a model as a whole in particular domains. The dataset I chose for this represents a relatively “generic” task - remember information it has read.

Probing Results

For every model we are going to probe we probe the first, middle, and ending layer’s final token embedding to regress to our four columns. The next figure shows an example of probing Llama 2 13b’s middle layer. Our birth year and death year probes perform surprisingly strongly; an RMSE of 80 years and R2 of over .5 is not the worst linear regressor I’ve trained, especially considering the scale of our data.

An example of probing the middle layer of a Llama 13b model with a constructed prompt. Our birth (top left) and death (top right) probes perform relatively well (R2 of above .5) while readership (bottom left) performs less well (R2 of .32) and our gibberish regression model performs poorly as expected (R2 of 0).

The above figure shows a smattering of models I probed by averaging the R2 achieved by a linear regression on the birth year against the embeddings from the middle and the final layer. The smaller four bars represent auto-encoding BERT models with far fewer parameters than Llama, SAWYER (a chat aligned version of Llama 2 I made), and Mistral v 0.1 and 0.2

Across 15 models, we see a wide range of R ^ 2 scores. BERT models, despite having the lowest scores, also have far fewer parameters, making them perhaps more efficient at storing information.

A couple of notable takeaways:

1. BERT base multilingual out performed BERT large English showing how the data that LLMs are pre-trained on matters

2. Mistral v0.2 as a 7B model performs as well as the Llama 2 13b models showing how parameter size is not everything

3. Llama 13B non instruct performed better when given a structured prompt (“basic information about X” vs simply “X”) showing how prompting can drastically alter the amount of information being retrieved

Are any of these “good” predictors of birth and death year? No absolutely not but that’s not the point. The point is to evaluate a model’s ability to encode and retrieve pre-trained knowledge. Moreover, even though our BERT models performed much worse, remember that A. they are several years older than the other models tested and B. They are ~72x smaller than the Llama 13B models and ~40x smaller than the 7B models. 

The next bar graph shows the efficiency of three models measured by the number of parameters needed to achieve a single R2 value so lower means more efficient. BERT takes the cake for being able to retain the information much more efficiently, most likely due to the nature of its auto-encoding language modeling architecture and pre-training.

Between, BERT, Llama 2 13b, and Llama 2 7b, the number of parameters it takes to achieve the R2 in our probe can indicate the efficiency of the model’s ability to encode information. BERT requires far fewer parameters than Llama 2 to extract encoded information but would require more pre-training on recent data to become on par with the Llama 2 model’s performance

For a second probe, I ran the GSM8K testing data through five models and built similar probes to the actual answer of the problem and below we can see our results.

Probing 6 models on the GSM 8K benchmark by taking the final token of the input world problem and regressing to the actual answer.

It seems that Mistral v0.1 and v0.2 models have more retrievable encoded knowledge than the Llama models when it comes to mathematical word problems making them potential prime candidates for fine-tuning tasks related to math and logic.

Check out the raw code for the Llama 3 Probe here: https://colab.research.google.com/drive/1e1d9fATVjVun-_tPj4vS_DSTGaIfxs01?usp=sharing I’m still prettifying everything 😀