评测
01 课程引入：为什么聊评测
0:05 So, today we're going to talk about evaluation. So, so far in this class, we've done everything we need to train a language model. We'd find the architecture, the optimizer, the main training loop. We showed how to make the training fast with kernels, parallelism, and then we saw scaling laws. We saw a little bit of how to make inference fast as well. So, only missing piece now is talking about the data that you train on. So, we're going to spend next week talking about that. And data is going to shape the model behavior. If you train on code, then presumably your model is going to be good at code.
0:51 If you train only on DNA sequences, it probably can't speak English. but before we even get to data, we need to talk about what behavior we want from a model. And that's going to be the topic of evaluation. So, evaluation asks a very simple question.
02 什么是「好」的模型
1:08 Given a model that we've trained, how good is it?
1:13 So, what is good? So, evaluation might seem like a fairly mechanical process. You define some prompts. You send these prompts to your model. You get back some responses. And then you compute the accuracy. So, why do we have a whole lecture on evaluation? But actually, evaluation is a very deep topic. And it's also very important because evaluation is what shapes the development of AI.
1:36 Evaluation sets north stars. And model developers, open, closed, everyone looks at evaluation as a sort of a measure of progress. So, by thinking carefully about your evaluation, you implicitly shape the development of what your model is going to be able to do. So, what's hard about evaluation is the following. So, often you start with an abstract construct of what do I want my model to do? Maybe I want my model to be, good at conversation or good at reasoning. Okay? So, those are abstract concepts. And then evaluation is really the process of turning that into a concrete metric powered by, presumably, some concrete prompts or environments.
2:26 So, this is going to be, the core challenge. And we'll see many examples of how this process of abstract construct to concrete, metric, shows up. So, let's go back to the question of what makes a model good. just kind of superficially. maybe a model is good if it does well on benchmarks. So, there's this website, artificial analysis, which has become a bit of a standard for thinking about, quote, unquote, intelligence of models. so, if you go to artificialanalysis.ai, you can see a ranking of, of different models from GPT 5.5 all the way down.
3:05 Well, there's many more models, after this point. But, and so this is, a measure. includes a bunch of different data sets, some of which we'll talk about. or maybe a model is good if it does well on benchmark and is also cheap to run. Because cost should matter. and this, plot shows, both the intelligence index from before, but against how much, money it costs to, to run a particular model based on, you know, inference cost. and you can see, of course, as you might expect, that, the two are correlated. The more you pay to run, the higher the intelligence index. But, you know, you don't see, it's not exactly aligned.
3:53 So, but maybe a model is good if people just prefer it. later we'll also go a little bit more into depth. So this is the take that, Arena AI, formerly known as Chatbot Arena, takes. and you can form a, a ranking based on models, based on whether people like it or not. Okay? Or maybe a model is good if people simply choose to use it, and therefore pay for it. So Open Router is this, server that allows one to use multiple, different models through, a central, endpoint. And as a result, they collect a lot of statistics of what models people are using. and they publish, stat- statistics, which is really interesting to look at, the top models, that are, that are used through Open Router.
4:43 So now, this is not representative of all model usage, but it is, gives you, a sense of what, people on Open Router are doing. And so this is kind of a more of an economic lens. I don't know what, good is, but if people are paying for it, it must be good. So, okay. Okay. So, you know, none of those are necessarily the correct answer. I don't know if there is a correct answer, but, just trying to get you to think about different ways of thinking about what good is. So let's start now, by looking at different ways, concrete ways, evaluating language models. And there's no better place to start than going back to the, the core idea of a
03 Perplexity（困惑度）
5:29 language model, which is that it's a distribution, PFX over sequences of tokens. And through that lens, there's a very natural way to evaluate a distribution, which is a perplexity or, you know, like likelihood, or log loss. those are all kind of related, concepts. And it basically says, I have a test dataset, D. And I'm going to see how much, probability mass my language model assigns to that D. And I normalize and divide so that the numbers are, you know, more interpretable. But that's basically it. How much mass, does my probability language model, assign to a dataset? Okay, so this is fairly natural.
6:16 In fact, when we were doing training, we minimized the perplexity on the training set. Okay, so the obvious thing to do is measure the perplexity on the test set. And this is what people traditionally did in language modeling research for many years, throughout the, 2010s. If you pick up a language modeling paper, this is what they did. And back then, standard datasets included the Pentree Bank, WikiText 103. There was also this one billion word benchmark, which was taken from, you know, machine translation sources. and, and this is all in the classic paradigm, which is in distribution of evaluation, you train on some train split of that dataset, and then you evaluate on some test split of that dataset. and people actually made a lot of, progress, which was just measured in perplexity reduction. So for example, there's this famous paper from 2016, which showed, that you could apply, you know, CNNs and LSTMs to this one billion word benchmark and get a huge perplexity, you know, reduction.
7:30 And bear in mind, around that time, you know, people were talking still about N-gram models and maybe, like, hybrid, models, and this was the, the first definitive kind of result that showed just pure neural is clearly the, the way to go. Okay? So life was very simple back then. You train on a train split, you test on a test split for perplexity. So then when GPT-2 came around, in 2019 from OpenAI, they trained on, you know, the dataset called Web Text, which was 40 gigabytes of text, websites that were linked from Reddit. and the way they evaluated was actually different. They evaluated, Zero Shot on, standard datasets that everyone else was evaluating on. so this is out of distribution evaluation. So this is a table from their paper, which showed that as, they trained different, models, so the biggest model is 1.5 billion parameters.
8:30 and they were able to show, quite a bit of, improvement, especially on the, small datasets. So for example, PTB, which is a tiny dataset, they were getting 35 perplexity compared to the SAVA art, which is 46. Now if you have enough in-distribution data, you don't quite do as well as the SOTA. but, you know, nonetheless, this is, kind of impressive, given that this model was not trained on 1 billion, words, dataset at all. there might have, I don't know, maybe there was some overlap because you never, I don't know if they did a careful, train test, decontam. and, and so this is, you know, ushered in a new paradigm of thinking about language models as not just kind of the IIN distribution setting, but how we wanted to train on a large dataset and evaluate on standard benchmarks. Which nowadays, I think, has sort of been very come standard, but, back then it was, seen as a bit of a novelty.
9:43 So, I will also argue, since we're talking about perplexity, that, perplexity is all you, all need to do. I wouldn't take this too seriously, but I think it is a mindset, I think, that drives a lot of, language modeling research. And the basic argument goes like this. So there's some true distribution out there, let's call it T. And you're training this model P. So, what is the best perplexity you could obtain? Well, the best perplexity is the entropy of that, true distribution. and that is realized when P equals T. Okay?
10:26 Okay? So, by minimizing, perplexity, you're basically pushing down, you know, the perplexity. And the only unique minimizer is when you actually get the true distribution. So, and when you have the true distribution, you're done. You just model everything, you can solve all the problems, and you could do things like condition on the problem, generate a solution, you can condition on a question, generate the answer. Okay? So, by push, pushing down on perplexity, we'll eventually reach HGI, is, is the, is the argument. I mean, this might sound, I don't know, either a bit fantastical, but in some sense, this is really, I think, a strong driver of many people who have been scaling and scaling language models. at a time when maybe, the, the gains weren't as obvious.
11:22 Before GPT-3, I think it wasn't clear that language models were just going to have such a huge, an impact. And it was really this belief that if you drove perplexity down, good things will happen.
11:37 So, you know, of course, perplexity is maybe not all, and, way, one way to frame it is that perplexity is maybe more than you need. So, for example, if you have a sentence like, Stanford was founded in 1885, perplexity will say, penalize, the prediction on all tokens. So, in particular, it'll say, what is the probability of 1885 given the rest? And, and that, that token is probably pretty useful. It's basically a Q and A, example, phrase is, sentence completion. So, it captures knowledge about, the world. But other ones, for, for example, the first word of a sentence, or even founded, maybe not that, you know, interesting, I think. and perplexity doesn't care. It's going to charge you, you know, bits for every, you know, bit of, deviation from the true distribution.
12:41 So, there is a, a bit of a solution here, which is you can always measure the conditional perplexity. You can condition on some prompt, and then you can measure the, the perplexity of the remaining response. So, that allows you to, in general, focus on certain tokens that you believe to be relevant, and less on others that are sort of incidental.
13:06 Okay? So, and, and also, it's, it's important to remember that some benchmarks, even though they are not explicitly a perplexity measure, are actually just perplexity in disguise. So, for example, if you think about, closed tasks, there's this Lombada, paper, from, actually a while ago, 2016, which is basically fill in the bank, blank. And you have a, a, a target context, you have a sentence, do you honestly think I will want you to have a blank? And, and the goal is to figure out that, that word. And, you know, even though this is, like, measured in terms of accuracy, it is really, a next token, you know, prediction problem. And here, the, it's not perplexity of all the words, it's carefully chosen. If you read this paper, it's really carefully chosen to be words, such that you actually need, a bunch of long context to resolve. Right? So this is actually something that the early GPT papers really, latched onto,
14:10 because they were interested in, long context modeling, thinking that that was, a key to unlocking kind of reasoning and all these, all these, things. And really, by focusing on positions where, to, to resolve that, it requires, long, distance, dependencies, is a way to kind of sharpen, the perplexity to, to some phenomenon that you care about. here's another popular, classic, dataset that you see in some language modeling vowels, Hellaswag. so what is Hellaswag? They, went to various different, you know, datasets and sources and came up with it. It's multiple choice, but it's really sentence completion, and at some level it's, it's perplexity. So, here's a sentence, a woman is outside with a buck and a dog, the dog is running around, she blank, and then you're trying to fill in which sentence completes it the best. So, it's not exactly perplexity, it's multiple choice, but, in some ways it's, it's perplexity.
15:18 So, one word of warning, if you ever decide that you're going to, you, you're sold on perplexity as, as the really, the measure, and you're going to launch a leaderboard and have people compete, compete against, perplexity, just a quarter of caution is that, you know, what does a perplexity leaderboard look like? So, people are going to submit a language model to you, and, you have to compute, log prop. You send them the test data and they give you back a log prop. Well, you look at this and say, that's a little bit, might be a little bit, fishy, because you need to trust that the probabilities they hand back, to you are, are valid. And that's a sum to one. Otherwise, I could just have implement lm and it always returns, one, or I guess log prop zero. And I would get very, good perplexity, but, clearly that is, that's not a valid distribution. And so, with this contract, it's not really easy to check that someone is actually internally having a log probability. So, there's some trust.
16:30 Maybe you have to look at the code or something. for downstream tasks, this is much more straightforward because you can have a black box lm. I'll give you a prompt, you give me a response, and I can just use that one. That responds and calculates some accuracy. So, perplexity is fundamentally different because it deals with distributions, and you have to be a bit careful. And this is not to mention, this is less of a problem with all regressive models. But if you have, like, VAEs or, if there are some models where you can't even compute the distribution, you can compute only a, a bound, then you have to really actually trust the math as well that this is actually a valid bound. Okay. So, the summary of perplexity is that it is actually still used quite heavily in language, model, development.
17:22 It has some nice properties. we saw that it's generally used for, for scaling laws, because it's also smoothly varying with scale, which is important for getting, scanning laws. and, but, you know, at the end of the day, especially if you're not a believer, then you need some benchmarks to really capture real world situations, to convince people that your language model is good. For the believers, if I show you a remarkably slow perplexity, they'll be, they'll be convinced. Okay. Any questions about perplexity?
18:10 Okay. Hopefully everyone's a believer now. okay. So, let's talk about exam benchmarks. So, exams, as, as all of your students know, is a way that we use to test the humans. And so you can adopt the same mentality to test language models. the nice thing about exams is that you have very careful control over the subject, over the difficulty. you can design them to have unambiguous correct answers, which makes them easy to grade. so there's a lot of nice, nice properties about exams.
18:47 And so for this reason, a lot of, LL benchmarking culture was, based on just this idea of exams.
04 考试类基准：MMLU / GPQA / HLE
18:56 So, you know, obviously there's been a lot of benchmarks, but I'll just highlight some of the ones that I think are, have been influential in the development of language models. so this one is, MMLU, or Massive Multitask Language Understanding from, Hendrix et al. 2020. and at that time, you know, it wasn't clear that language models were really, like, general purpose, task solvers. This was, around the time when GPT-3 came out. many people, I think, still thought of language models as, as, well, language models. You generate fluent English. but they had this idea that, well, let's just try to be ahead of the curve here. We're going to go through 50 different seven subjects.
19:45 We're going to, have a team of students crawl, scour the internet for different, sources. And, really put together this, very comprehensive, you know, dataset. and, despite the name MMLU, which has the word language understanding, it's really about testing knowledge and reasoning. It's less so just pure language understanding. And the way they evaluated it with, with GPT-3, using Fusho prompting. So, they essentially said, I have formed this prompt. The following are questions about high-square mathematics. Some in-context examples of question, answer, question, answer. And then the final question, and then the model has to predict the answer.
20:33 So this seems very mundane. But at that time, it was, I think, pretty radical that you would construct this fairly complicated question, and with in-context examples and everything, and expect a language model to actually do something reasonable. And sure enough, at the, at the outset, the, the models were not great, I would say. they were, you know, some of the small models were barely above, of, of chance. And the larger model, this GPT-3, was actually well above chance. So this was, kind of encouraging. and then over, the, the years, you know, as we, we know,
21:21 this benchmark has essentially, you know, saturated. So this is a site, I don't think this has all the, the data in it, but, you know, about 3.5 turbo. This was like, 20, 20, I guess, beginning of 2024, or 23 actually, to GPT-4. And then now the, you know, now it's in the 90s. Okay?
21:48 So, there's also a link you can follow up, later on, where you can actually browse some of the predictions of these models, which is, always fun to see what models actually, goof up on, and what they get right.
22:03 Okay, so, so around 2024, when it was, it seemed like MMLU, there was a lot of progress made. It was like, okay, you're going to see this theme again. Okay, this, this benchmark is getting too easy. let's, let's try to make it harder. So, well, first of all, they actually noticed that there were some noisy and trivial problems, so let's get rid of them. four choices might be a little bit too, too easy, so let's expand that to ten. at that point, it was much more in vogue to use chain of thought to, solve these benchmarks, whereas when GPT-3 came out, that didn't, it wasn't really a thing. and it's clear that some of these questions could benefit from a few reasoning steps. so this new benchmark, was harder.
22:55 So now the accuracy of the models is back at 33. but as you can maybe see from the progress so far, you know, it, it started low, and now it's back to, I guess it's like 88, almost 90. Okay, so this benchmark might be a little bit, you know, too easy as well. Okay, so there's another benchmark called, GPQA, which stands for, Google Proof, QA. and the idea here is that, okay, how can we get a really, good, you know, you know, benchmark? And the idea is that, well, if you can solve a problem by looking at Google, Google has kind of the internet and the language model is trained on the internet. So, if you can just solve it, then that's probably too easy. So, they got some, now, you know, PhD level. So, I think MMLU was think about like as, actually some of these, maybe some of these were like undergrad, but maybe grad, level.
24:00 But now, it's more explicitly about questions that are meant to be, pretty hard. And at this point, now, you say, okay, how do I come up with really hard questions? You actually have to go through quite a bit of human labor to get really hard questions. So, this was 61 PhD contractors from Upwork. someone writes a question. It goes through an expert validation, which then provides feedback on the questions. and then the question writer goes and revises the question. And then another expert looks at the question. And then you go to, you have the, the question.
24:36 And then you send it to some non-experts who are given, at Google Access. And then you see if they can solve the question. So, there's a subset of these questions called the diamond, set that you might see, in, you know, people's, leaderboards. Where if the two experts agree, and at most one of the non-experts is able to answer a question, then that's deemed a good, kind of, diamond question. So, this is a pretty, kind of, rigorous, review process. and, PhD experts were only able to, get, 65%, so this is, I think, mostly four-way multiple choice. So, this is above random, but, you know, this is not a good grade if you're getting 65% on an exam. and, non-experts, even if they had 30 minutes with access to Google, they were getting, a bit over chance, but not much. And at that point, GPT-4 was, like, 39%.
25:37 And let's see how the, how it's doing now. so, GPQ-A is now, again, 94. Okay, 94. So, yeah, so these benchmarks are, have a shelf life of not too, high, I guess. I will say that some of these benchmarks, even though they saturate the largest models, they're still useful for developing smaller models or fitting scaling laws and, and so on. Okay, so, then, yeah? How do you know that, when these tests are being used for training, and, actually, what, models are called? Yeah, so this is a very important question. So how do you know that, these questions aren't just in the, training set?
26:31 so the short answer is, we don't know, because I don't know what's in the training set. I'll come back to this contamination point, a little bit later, and ways of addressing it. But you should always, it's, I'm glad you asked this. You guys should always take these numbers with a little bit of a grain of salt, because, it's only trust. I, I, maybe just to say one more, I think train contamination is, is a little bit subtle, right, because it's probably not the case that they literally train on the set. But, you know, questions can be derived from other sources, and those sources could be trained on. And so, often contamination is a much more, subtle process than literally accidentally training on the test. was there another question?.
27:21 Okay, great. Good. Always good to be skeptical. Okay, so the final, exam benchmark I'll talk about is something ominously called, Humanity's Last Exam. this is from last year. And they were saying, okay, well, these models are getting so good. Let's just give it all we can to create something that's really tough. so this is multimodal, many subjects. It's still multiple choice, and I guess some short answer.
27:49 And so this was, you know, a crowd source. They had some money to incentivize, people who were into money to create questions. They also incentivize with authorship for those who are into that. and they went through multiple stages of reviews, filter by frontier models. there's a whole process. another answer to your question is that they also held out a private set that was not released publicly, so that, they could make sure that this did not enter, you know, training. And then you just hope that the APIs that, you, because you still have to send this to the model, that those prompts don't end up in the training. but let's, let's hope that's the case. so this, you know, every dataset paper kind of looks like this.
28:41 Previous benchmarks, look, the models are doing so well. My dataset, look, all the models are doing really poorly. So this is a kind of common pattern. This was especially like single digits. It's, it's pretty abysmal here. and let's see how we're doing, in 2026.
29:01 So this one is, it still has some room. So it's only at, even mythos is only at 64.7. Okay, so this is indeed a pretty, it seems like a pretty hard, exam.
29:20 Okay. So, to summarize this section, as you can see, we've trended towards harder and harder questions as the models improve and we saturate existing benchmarks. It's interesting that multiple choice format still survives, because, it's, you, with multiple choice, I mean, you can make it as difficult as you want, right? So, the, the, some people think, well, multiple choice is too easy. Well, I mean, I can give you a very hard multiple choice, you know, question. That's not really the point. I think the more the point is that multiple choice restricts the set of questions you can ask. But in terms of difficulty, no problem with multiple choice. The main problem with, exam-based, questions is that this does not capture real-world usage.
30:11 no one asks HLE questions to, to a language model, except for when you're evaluating on HLE. most of the time people are asking open-ended questions. they don't even necessarily have a correct answer. They might not be well-formed and these benchmarks do not capture that.
30:32 Okay. So, question. For accuracy, how is the model output compared with the ground truth answer? So, for HLE? For accuracy, like how is the model's output compared with the ground truth answer? For accuracy, to calculate the accuracy. Oh, for calculate the accuracy for how is the ground truth compared. Okay. So, for multiple choice, it's just like, did you get the right answer?
30:58 So, like, would we check the probability distribution for H option or like, how is it? so the language model generates, like, C. And my answer is either B or C or A or D. And I just check if they're the, the same or not. Okay, so we are, we are only targeting what we're talking about. Yes, well, okay. So, then the question is less about evaluation, but how does the model generate an answer? Right? So, there's a few options.
31:27 One is that you generate, literally you sample and it generates a letter. Or, more commonly now you generate a chain of thought, you generate the answer, maybe some explanation. but there is a way of extracting an answer which, you know, does matter. And, and language model evaluators can be very sensitive to that, but I'm not going to talk about that here.
31:52 Okay. So, let's talk about a very different class of benchmarks, called, chat benchmarks. so, so far we've talked about multiple choice tasks. So, they're still pretty good at capturing, some notion of intelligence and difficulty. but most people don't ask a multiple choice exam question to their AI assistant,
05 对话类基准：Chatbot Arena / AlpacaEval / WildBench
32:15 unless they're trying to use their AI assistant to do their exam for them, which you would never, no one does that. okay, so, so here's an example, semi-inspired by a real world, use case. So, you know, I might say I want to make a beet salad, which is what herbs will work well and what wouldn't work well. Okay? So, this is a very open-ended, question. And then you get some response, which is, again, an open-ended, response. And then the question is, how do you evaluate this thing? Right? You can't just check exact match with equality.
32:51 There's no ground truth, even. So, there's a few ideas here. we, we should go through them. So, one of the ideas is, was pioneered by, this, chatbot arena. And the idea is, well, you ask humans. Okay? but the way they ask humans, I think, is a, a kind of a clever idea. So, they basically had the set of this, website where any random person from the internet can go in and chat. and then, instead of, like a normal, as an assistant, and getting one response, you actually get two responses, from two different models, which are anonymized.
33:35 And then they rate which one is better. So, for example, you put in this response, you get these two, assistant A, assistant B, and then you ask, A is better, both are good, both are bad, B is better. Okay? So, using this process, you can get a bunch of pairwise comparison data of model A, model B, you know, which one is better. And then you can use this to compute ELO rankings, based on this. So, just as, I don't know how many of you are familiar with this, but just as kind of a, just get around the same page. In, these, you know, rankings, you define, your, your rating model as, a probability of a particular model beating another, model as, some function, of the ELO scores of, model B and model A. So, the larger, model, ELO score of A is, the higher the probability. So, it's a smooth mapping.
34:39 And then you fit this model with the parameters, which are the ELO ratings, to maximize the probability of the pairwise comparisons. Okay? And then, as a result, you basically get a rate, rating for each of your models.
34:55 Okay? So, this is what, chatbot arena, nowadays known as, arena AI, did. And you can get this, kind of nice, ranking. I guess, Claude Opus is doing pretty well here. but, you know, some other models are on the list as well. Okay. So, there's a few things that are nice about this setup. One is that, you have real world prompts. Right? So, the idea is that anyone can come to the site.
35:27 And the incentive for someone to come to the site is that they get free access to a language model. And so, there's sort of this implicit assumption that they're actually trying to use it to do something useful. And therefore, you get sort of like, you know, a real world prompt. but one thing you should keep in mind is, you know, who are these people? Like, random person on the internet is, I don't, I don't know what that, distribution. Random person on the internet who comes to El Amarino, I don't know what that distribution, you know, looks like. I think in their paper they have some demographics, but, you know, demographics don't tell the whole story. you could worry about different biases coming from the different, models. Maybe spammers, people are trying to, game, you know, who maybe even submitted a model and they want to, their models to look, good.
36:22 you know, you just don't know. It's a little bit of a wild west. there's another issue which is that, you know, the binary preference, which is better, A or B, is very nice and simple and feeds into this, rating. But it does conflate style and, correctness. So ELO ratings for chess makes a lot of sense because the only thing that matters is did you win? But, you know, which, which one is better is, is much, less clear cut. And, you know, there's questions of, you know, the human is the judge here. The person who puts in the prompt is answering the question. And this is good because, in some level, because the, the person is, has some intent and they can sort of declare whether their intent was met.
37:14 To some extent, except for, they're presumably asking the question because they don't know the answer. So if I give you two answers, how are they to judge, judge which one's correct? And then there's also the problem of sycophancy where, maybe, more pleasing answers are going to be, up-weighted highly compared to, correct but honest answers. So there's a lot of, kind of potential issues, here. but the nice thing about this method, which is that you don't need to feed the same prompts to all the models. And this is the beauty of ELO, right? In chess, you don't have to play, everyone plays everyone else. it can just be kind of some sparse sample of that and you can still derive rankings. As long as the graph is kind of connected.
38:01 and this is important because human is rating and you can't expect a human to rate, like, all the models or even more than two models. and the nice thing as well is that as new prompts and new models come in, there's a natural story for making, this updating over time.
38:20 Okay, so, let me talk about a different, approach. so this is our PacaEval from 2023. there are some instructions from various, you know, derived from various sources. and the metric here is, you know, the win rate against a baseline model. So you have a model. A PacaEval says, I'm going to generate it for a prompt, generate using, let's say, baseline model GPT-4 preview at that time. And then judge using GPT-3 preview which one is better. So, immediately this, raises a question of, you know, potential bias. but let's put that aside for now.
39:06 So this is basically an instantiation of LM as a judge, which is, very popular these days for evaluating language models. and you can mitigate this bias by having multiple judges and ensembling and so on and so forth. So one problem with initial PacaEval is that LM judges, favor long responses. And this really led to a bunch of leaderboard gaming where a bunch of models that were fine-tuned were submitted and they got really high LLM, AlpacaEval performance just because the responses were longer. So this got fixed in a subsequent, paper where, the, use a very simple regression method to de-bias the metric.
39:51 Okay. So, but there's a, this raises a kind of a more general question here, which is not specifically to AlpacaEval, which is how do you evaluate a metric? Okay, so we're talking about, we've been talking about evaluation of models. And if I have a metric, I can evaluate a model. But if I'm coming up with a new metric, how do I know this metric is any good? So this is a hard problem and there's no, no answer here. one thing you can do is more of a sanity check, is you can look at correlation with other metrics. with the idea that higher isn't necessarily, you know, better in all cases, unless you are trying to mimic this other metric. So, AlpacaEval, correlation with chatbot arena is 0.98, which is, quite high.
40:46 So, which suggests that, you could, let's say you wanted to, evaluate on chatbot arena but you're, I don't know, too shy to put your model or you don't want to wait for humans or whatever. You can use AlpacaEval, to evaluate, instead. Now, this correlation, of course, is with respect to a certain set of models. And, and so this might not hold for models that, let's say, are stronger than GPT-4 preview.
41:22 Okay, so at this point, this leaderboard, I don't think is, really, you know, hasn't been maintained in over a year, but this is kind of what it looked like at the time. Okay, so there's another, benchmark I'll talk about, which is WildBench. And this one, was sourced a bunch of examples from human, chatbot conversations. So like, chatbot arena, they also put up a free service where people could chat with this bot and they collected the, the, the, the, the, the queries. And then, like, AlpacaEval, they, used, LM as a judge to evaluate, the, model. the main innovation here is the use of a, a checklist, which, is generated and specific to a particular, prompt or, or task. And the idea here is that if you just ask, language model to judge whether this prompt is, this response is good or not, this is, in some sense, a very ill-defined task. Like, I don't know, what, depends on what you care about. And so the idea here is that by having a checklist or some sort of rubric, this greatly scopes and makes the task, evaluation task, more well-defined.
42:52 so they did this, this also, they showed correlation with, chatbot arena, which, admittedly is a little bit kind of, you know, circulate because then you can ask why, well, is chatbot arena really the ground truth? But at least we're all in the same boat together. so, okay, so to conclude this section, how do you evaluate open-ended responses? you know, not, there's no clear, solution here, but there's some ideas. Pairwise comparisons between a reference and a, a, a model response is generally a good idea because, especially for very similar responses, you can kind of give directional information. Well, this one's slightly better than this, but in terms of absolute raking, like, is this a seven out of ten or an eight out of ten, that tends to be a lot less, high signal. one should always be aware of biases, both in terms of humans or LM judges. They have different biases, but nonetheless, that's something important to, you know, pay attention to. Perhaps the only thing you might be able to do is you evaluate over multiple judges, humans, and judges, and of all, everything is saying your model is better, then maybe your model is better.
44:16 And finally, the evaluation problem here of open-ended response, I would emphasize, is not really well defined. And, so, increasingly, I think it's important to define, you know, rubrics or checklists to, to improve the reliability or, you know, well-definedness of an actual evaluation. And this is true regardless of a human and a judge. So anyone who has done crowd sourcing knows that if you just ask a human to rate something without giving some sort of rubric, you're probably going to get very, nonsensical results.
44:58 Okay. So, let's move on to the genetic, world.
06 Agent 类基准：SWE-bench / Terminal-Bench / CyBench / MLE-bench
45:04 So, you can think about what we've been doing so far is, evaluating what language models say, and that's chat. And now, we're going to evaluate what elements do, which is, you know, agents. And, so, you know, for anyone who's following, AI hype, you know, agents are everywhere. And, you know, these are definitely, I think, going to transform the way that we think about, you know, language models. So, agent is basically a language model plus some sort of scaffold, which is, a logic for deciding how the language model, is going to be called, and what tools it, it can, has access to, and so on and so forth. so there's a few benchmarks I want to just, you know, point out, that are agentic. So, one is, you know, very popular. It's just called, SweetBench. there's, basically the task is, you're given a code base and a description of a GitHub issue, submit a PR.
46:11 And the way you evaluate a PR is, whether it passes, the unit test. So, this, these tests are designed so that before you submit a PR, certain unit tests don't pass, and then your goal is to fix it so certain unit tests pass and you don't break anything else. So, here's an example of, some inst, instructions. Here's the issue, and then some code, that is provided in the context. And then your goal is to generate a patch. And given a patch, you are evaluated, on whether you pass or not. So, the good thing about this is that, the evaluation is very straightforward. and let's see how we're doing on, the, SweetBench. Actually, this is going to be SweetBench verified, which later I'll show you, tell you why this happened.
47:08 so SweetBench, and this was only, I guess, 2024 was still around, you know, 16%. And now it's up to, like, 93. Good job, Mythos. Okay.
47:26 So, okay. So, this is, sort of a very canonical, a generic benchmark, as a lot of, attention on agents has been invested in, in coding abilities. So, SweetBench really kind of pioneered this, this wave of thinking about how to assess, an agent's ability to write, code in a realistic, environment.
47:53 So, here's another, project called, TerminalBench. And here the idea is that, this is going to be more general purpose, right? And we're going to basically have the environment be a computer terminal. And we're going to define a bunch of tasks, that can be done in a computer terminal by, you know, typing in a bunch of commands. And the nice thing about this environment is that very simple and, and universal. so, and these, tasks were crowdsourced from, you know, 93 people from, you know, across the, across the world. 89 of these tasks are, you know, constitute TerminalBench 2.0. these tasks generally take, for, you know, ranging from one hour to over a week, depending on whether you're an expert or, junior.
48:51 and then here's what the leaderboard looks like as of, you know, today. so top models are, you know, the frontier models. One thing that's interesting is that, you know, the agent, is also, kind of important here. Because you can have two, agents, but the same model, and they have different, you know, accuracies.
49:16 okay. So, another benchmark I'll talk about is, you know, cyber, security. So, this is, 40 capture the flag tasks. And the setting here is that, you have an agent that is given in, ex, an environment, and it can run various, commands, like, you know, look at the source code. It can, it can, run, you know, access the web server. And the goal is to essentially hack into this, server to extract some flag, which is a unique string that proves that you've hacked into the server. so these are, competition cyber security exercises that, are done by humans normally. And, the, this, this is an example of what the, the agent, scaffold kind of looks like here. This is a very simplistic, version where basically you have one continuous memory.
50:19 you, you produce some sort of action. you get environment feedback. And then you, put the results into this buffer. And this is concatenated. as I'll show you later, this is actually, you know, this, this history obviously grows quite a bit. And you'll need to, kind of, have better ways of managing context. okay. So this is what the leaderboard looks like when it came out. You know, the best models were, like, about 10% or so.
50:50 And over time, now, it's completely solved, actually. so that's, that side bench. there's another class of problems which are MLE engineering. so the, this is, this benchmark has a bunch of Kaggle competitions, which involve processing data, training models, and so on and so forth. And the agent, can look at the data set, read the description, write code, train models. And, you eventually submit something, and you get a grade. so this is what the leaderboard, looks like right now. And again, you can see that it's really the same usual suspects on, on the LM side. but there's quite a bit of variation in terms of, the different agent, scaffolds.
51:45 So one thing to note about, point out about agent, scaffolds, is that, they matter a lot. so, when I talk, said I talk about language model evaluation, this is broader than language model evaluation. And the reason is that, you know, unlike what I mentioned before with, this, this early, side bench, agent scaffold, now you have to be a lot more sophisticated to solve really complicated tasks. one thing that people realize is that explicit planning is helpful. You can't just kind of stream of conscious, chain of thought this, and then, you build up this context, and the agent can easily lose track of where it is. So instead, keep a to-do list, get it checked off. hierarchical delegation, agents should call other sub-agents with some cleaned context. It does, that agent does some stuff and returns only the result. The, the master agent doesn't need to see all the, the gory details.
52:50 And this can help encapsulate, provide some encapsulation. memory, people have been experimenting around with explicitly reading and writing files, especially as the context grows. You can't just store it in, your, your context window. and then finally, there is more context engineering in the style of managing the process. You know, when should you delegate to sub-agents? How, when to kind of try this strategy versus, this other strategy? What to write to persistent memory? And these things, are generally, you know, optimized for, the particular, language model.
53:36 Okay. So, agents are exciting because they enhance the capability surface of language models. Scaffolds are very important. So, really, when we talk about evaluating agents, it's evaluating both the language model and the agent, scaffold.
53:55 Okay. Now we're gonna, switch gears a little bit and talk about this other very different style of, benchmarks. I call them pure reasoning for lack of, I think, better term. and the motivation is that all the tasks so far require some linguistic and world knowledge. And so the question is, can we just isolate reasoning or pure kind of fluid intelligence from a knowledge and knowing facts? And you could argue that this gives you a form of more pure intelligence. So, there was this project, ARC-AGI, which is, started back in 2019, where, the goal was exactly this. So, this, the task was meant to be 100% solvable by humans but challenged for AIs. And every task is sort of a special snowflake, or that's an intent.
54:55 And so memorizing a bunch of facts or solving previous problems, shouldn't really help you that much. So, if you guys are remembering your LLM history, 2019, was pre-LM. I mean, this was GPT-2 era. so this was kind of, prescient in that regard. So here's an example of the first iteration of the problem. You're given these, these pictures and you're trying to guess what comes next. And if you take, like, ten seconds, you can probably solve this problem.
55:30 Okay, so, you know, you probably guessed it's to just fill in the yellow to form the rectangle. okay, so then, and last year, there was an updated version of this task, which was a bit more complicated.
07 纯推理基准：ARC-AGI
55:44 and you can take a look at this.
55:48 so here's the, the trajectory. So, when Arc AGI 1 came out, you know, the, I think this is actually zero. so basically models didn't do anything. So this is like GPT-3, pre-trained models didn't move the needle at all. Right? And this was exactly as the creators had intended. Pre-trained models just, you know, like, learn what the facts are on the internet, and maybe learn linguistic patterns, but none of that is actually helpful, directly, at least, for, these Arc AGI tasks. And then something happened in 2024. And this is when OpenAI released their 01 and 03 models.
56:36 Then things started taking off pretty abruptly. And now, it, Arc AGI 1 is basically solved. and even around 2025, when Arc AGI 2 came out, this has also, you know, and it's not quite solved, but, you know, it looks like it's on its way to being, solved. So, it was really the, the reasoning capabilities that kind of unlocked, this. And now, one could wonder whether, all the, the knowledge of, of actual pre-training, was, was necessary for these tasks. And certainly, you know, without pre-training, we wouldn't have this explosion of reasoning models. So, arguably, it was, you know, still important, but not maybe, directly visible for much of the history of Arc AGI. So now, just last month, Arc AGI 3 comes out. So you see this familiar pattern.
57:37 All right, the creators look at this and it's like, -oh, time for a new benchmark. so this is now an interactive environment, where you can go online and you can play this game. So again, no language, you're just trying to kind of figure out based on patterns. it's kind of fun. and now the scores are extremely low. So, so, so next year when I teach this class, I'm sure I'll have to update this, this slide, but we'll see. So I think this, Arc AGI line is, is, is really kind of interesting to think about. the goal is to disentangle reasoning from knowledge. This is really hard to do.
58:18 And, you know, it's not, I think this is probably the best attempt at this. It's not clear whether you can really fully, decouple things. and certainly if, you know, if people really cared about this benchmark, I think people probably could, you know, people can benchmark, I think, probably anything. So you could design questions like, are these? These still come from some prior. and, you know, not sure there's anything as kind of quote, unquote, pure as, as reasoning. This is also constrained to human reasoning. The explicit goal is to be 100% human solvable within some reasonable time. so this doesn't, really extend to human, superhuman reasoning.
58:59 For example, so winning IMO gold medals or solving open math problems, which arguably are still very useful and important. But clearly this exposes some gaps in the current models. So there is kind of work to do if you think about the goal as basically being able to, solve all the tasks. Yeah. For this, graphic, so this RKGI 3 especially looks quite graphic. So, is there some multi-model front-end, or it could be more good for the textual representation as well? Yeah, so the question is, this looks very graphic, graphical. and so what is input to a language model? So this is essentially, I don't know what the rest, I think it's 64 by 64.
59:48 So you can provide this image, or you can provide it in, like, ASCII art, or, or some sort of textual representation. either way, there is a sort of spatial element here that the model has to
08 安全类基准：HarmBench / AIR-Bench / 越狱
1:00:02 reason about, which is really not English or natural language.
1:00:12 Okay, so let's talk about shift gears and talk about another class of benchmarks, which is related to safety. so, people have been talking about safety and language models for, for a while. And if you think about other areas, such as cars, it's pretty clear what safety means here, right? There is, you know, these, ratings for, safety ratings for cars, which involve, you know, testing based on smashing cars into a wall with an airbag and seeing how, how well it, it does and protects the, you know, the, the dummies in the, in the car. And, and this is with, you know, decades of lobbying and figuring out what safety means for, vehicles. So what does it mean for AI? So I would say that there is not a great, answer here, but I'll give you some examples of what people have been thinking about. So, there's this, benchmark called, you know, Harm Bench, and this is really about, you know, prompting a model with things that are meant to, be harmful and expecting the, the model to, essentially, refuse. so this is one, you know, type of safety, and I think probably the dominant thing when people think about safety is that I want to prevent bad actors from doing things with, language models.
1:01:35 But it turns out that's not the only thing that matters. So there's this other work, Air Bench, which, tries to think about safety holistically by looking at all the regulatory frameworks in the EU, China, and, the, the US, company policies, and builds a taxonomy of all the different things that could go wrong. and then using this, constructing a particular, set of prompts and, and evaluating, so on and so forth. but one problem with safety is, you know, jailbreaking, which is its own kind of sub-problem, which is that language models are generally trained to refuse harmful instructions. But, as we, we know, you can get around this if you're clever. And so there's some early work that uses automatic ways to, optimize for prompts to bypass, safety using, you know, GCG, which is basically this, you know, coordinate-wise optimization algorithm. And remarkably, optimizing on open models, allows us to transfer to models. So this is, this, I hope this doesn't, these attacks don't work anymore. But, at the time, you can generate some basically gibberish, and step-by-step plan to destroy humanity, and, OpenAI's model will, you know, comply.
1:03:09 obviously, you could argue, well, is this actually a, you know, harmful plan to destroy humanity? But that's, at least it did not, the expected behavior is that it should probably refuse that. Okay. So, this raises kind of a, a bigger problem, which is what is, you know, safety. And one thing that makes this tricky is that many aspects of safety are very contextual. So, you know, it involves, you know, politics, law, social norms. This varies across, different countries. And then also, if you think about safety, you have to think about what are the risks. and the risks are actually quite varied as, as well, and demand probably different attention.
1:03:55 So, hallucinations is a risk, especially, let's say, in medical or, you know, legal or financial settings. but this is also correlated with just capabilities. Like, if you make a model more accurate, it's correlated with not hallucinating as much. Sickle-fincy is, is a problem. you know, abetting crimes is a problem. but this is, this might be, counter to, capabilities. Inequality, you know, losing critical thinking might be, you know, a risk as well. That's, correlated with capability. So, you know, the, if you think about the societal impact of language models on, on the world, and, and think about safety as ensuring that AI goes well for people.
1:04:42 This is actually, a much more complex, topic than I'll have time to, you know, talk about. And finally, there is a dual use aspect of, language models. So, the cyber security agents can be used to hack into a system or to do penetration testing and make systems more secure.
09 真实性（Ecological Validity）：GDPVal / MedHELM / Clio
1:05:00 Which makes, talking about whether these are safety risks or actually beneficial, a double-edged sword.
1:05:09 Okay. So, let's move on now to talk about some, you know, broader considerations of evaluation. So, one thing that, I think is an important aspect of evaluation is realism or ecological validity. How well does evaluation capture real world use? So, in particular, exams are very far away from real world, use. you know, Chapa Arena are from real people. But, you know, one could wonder whether this distribution is the right distribution of, of people or use cases or not. There's a few, benchmarks I'll, mention that try to get at ecological, validity and, in particular, at the sort of, the use case level, at least. Not their individual query level.
1:06:00 So, there's this, OpenAI, created benchmark called GDPVal. They looked at all the top sec, nine sectors according to US GDP, 44 different occupations. Got a bunch of professionals to, create tasks. these are professionals with, like, you know, about 14 years of experience. And you can see people who are nurses, who work, you know, concierge, real estate agents, film and video editors, and, and so on and so forth. And they create a bunch of, you know, tasks. here's another example, in the medical domain, where, up until this point, a lot of medical benchmarks were based on standardized exams. And, in fact, actually, for humans, this is also what, what, you know, happens. You take these, standardized exams and, you know, you, you graduate from med school and then you go operate on, well, actually, there's a longer process.
1:06:58 You don't operate on patients directly. But, so, but, the point is that, for a language model, you should also not just, like, pass medical exams and, deploy them directly. So, this was, a, a project where, you know, 121 tasks were sourced from 29 clinicians. and this was meant to represent the type of things that clinicians would actually ask to a language model. Which are quite different than, the multiple choice, type of exam questions. here's another, project which is, gets at real data. So, you know, if you ask who actually has the data on what people are doing, what's the, the model developers. And so, this project is interesting because it uses, language models themselves to analyze real data. So, in particular, you do the privacy.
1:07:58 You can't literally go and, look at people's data. But you can have language models look at the data and analyze and summarize, general patterns. So, then the language models could tell you about what are the types of people, what are the types of things that people use, Claude for. you know, and sometimes, you know, realism and privacy are ads with each other. Because, ideally, you would want actually, sample from the actual query stream. And that would be very valuable to assess how well a model is working and understand deeply what the errors are. But then, that runs into privacy, considerations.
10 有效性：数据污染与数据集质量
1:08:43 Okay. So, another consideration, which is another type of valid validity, is, you know, just in general, how do we know that the, evaluations are scientifically valid? And this was brought up fairly early. Train test, overlap or contamination. So, in a machine learning 101, you learn don't train on your test set, hopefully. and before foundation models, this was actually pretty clear, right? So, with, like, ImageNet and Squad and many benchmarks, sort of data sets, there was always a train split and a test split, and everyone played the same game. You train on your training data, you test on your test data, life was simple. And now, you have these models that are trained on the internet and much more than that.
1:09:37 And, furthermore, you don't know what's in the data. So, this makes evaluation quite different, I think. So, what can you do in this? So, first thing you can do is, you can be, try to infer whether a model has actually seen the test data. so this is a, a neat idea, from Tatsu's group that, where you leverage the fact that, the order in which the questions in the benchmark, occurs, is, should be random. But if the model prefers some sort of particular order that's consistent with the benchmark, that is probably, it just, like, trained on that, you know, benchmark. another route is that you can encourage people to report, you know, you know, actual trained test overlap. So, this has to do with norms, right? I think in, in general, especially, maybe less in machine learning, but more in statistics, you always report confidence, intervals.
1:10:49 Right? You have an estimate, and I always want to know, like, how reliable is that estimate? This is just standard practice. And so, the idea of this, you know, position paper is that you should model providers, if you're going to claim some GPQA score, you should always provide some justification that, you know, you didn't train on the test set.
1:11:15 so another route is that, where you can just give up, is like, okay, all these evaluations, they're, let's just assume that people trained on them. Assume the worst case. And, so the thing that we have on our site is that you can always define fresh evals. So, there's a bunch of, ideas, live code bench, untreatable eval, which basically scrape new web pages, or archive papers, or, GitHub, and defines evaluations that are past the cutoff date of the language models. So, this seems pretty robust and good, but one thing to keep in mind is that timestamps aren't always safe either, because I might post, or, you know, you know, if, with code, for example, I might put something on GitHub, and, which is actually derived from, like, some other repo. And so, how do I know that that's actually generally fresh? route four is to use, private evaluations. So, companies, model developers do this, right? So, if you're Google or OpenAI, you have your internal code base, which you assume is not on the internet, and you obviously wouldn't train on this, this data.
1:12:34 so you can use it for evaluation. So, this generally is pretty reliable. and if you're not, a company, you can use your personal writings. I have a bunch of rejected papers from, you know, when I was in grad school that I never put on, online, so I can use those. and, and the, this type of evaluation, I think, is very favorable for perplexity, evals, because all you need in perplexity is just, you need a good, you know, dataset, and you can evaluate log probabilities.
1:13:09 So, another, concern, that I'll mention is dataset quality. so, remember I showed the leaderboard for SweetBench, you know, verified. this is because, you know, well, SweetBench had, a bunch of, you know, problems of tasks that weren't, you know, the unit tests that weren't, quite, rigorous enough and other things that got, you know, fixed. and, you know, there are many other benchmarks, that have gone through this process, like, like, GSMAK or MMLU, where, you know, people have gone and done a more careful audit and realized that, wait a minute, this question actually doesn't, is, something's broken around this question. Okay? So, like this one, there's, there's no curve given in this, this text, so you can't answer this question. you know, does the baby have socks on? No, there's no way to tell. Okay.
1:14:17 So, and there's also this paper that showed that there's a lot of problems with the genetic benchmarks, which arguably are even harder to assess, because it's not just, you can look at the question and the possible answers, and you can look at it, but there's a whole environment, and there are cases where, you know, with coding, you can, it's great to have test cases that either pass or not pass, but as we all know, test cases can be incomplete. So, you might pass all the test cases, but still not have a working solution. there's another case of this, the Telbench benchmark, where they show that if an agent just outputs the empty response, they can get, like, a 38%. So, there's some trivialities with these benchmarks, that you won't have to be very careful to avoid. One thing I'll call out is that there's this tool, called, Docent, that uses language models to inspect agent traces to detect problems. so this is kind of a qualitative response to our very quantitative heavy way of benchmarking. And I would always recommend that whenever you do a benchmark, either develop a benchmark or run a, a model on the benchmark, that you always look at the output and try to audit it to make sure that you actually think you're measuring what you think you're measuring.
11 如何思考评测
1:15:41 Okay. So, you know, maybe getting a bit more philosophical. So what's the point of evaluation? Okay. So I've talked a lot about different types of evaluation, and really there's no one evaluation to rule them all. And I think it's, when you're thinking about evaluation, you have to be very clear about what the purpose is, and I think often this is not, really stated clearly. So, for example, you could be a, you know, user or company that's trying to decide, make a purchase decision. Are you gonna go with A or B for a particular use case? Or you're just a researcher and you have this intuitive notion of intelligence and you want to measure it somehow.
1:16:22 Or you want to understand the benefits and harms for business or policy reasons. Or you're a model developer and you wanna get feedback to improve the model. So each of these goals will lead you to potentially different, you know, benchmarks or some combination of, evaluation strategies. And finally, one thing to, address which I hinted at briefly is that what are we even evaluating? So, before foundation models, researchers evaluated methods because we have these trained test splits and the only thing that was varying is the actual algorithm. And today we're mostly evaluating models and systems where anything goes. you know, there are some exceptions. So the nano-GPT speedrun is, deliberately a way to evaluate an algorithm, in particular, how well fast you can train a model. But most of, language model, evaluation is about evaluating the actual end model.
1:17:21 Which is useful because that is the thing that is actually gonna get shipped and people are gonna use. But I think the, the punchline is that you should always be very deliberate about, saying, declaring what your goals are. Okay. So there's no one true evaluation. Choose what you're trying to measure. and, you know, we talked about a bunch of different, benchmarks, starting with perplexity, exam-based, chat, agentic, reasoning safety. they are very in terms of, you know, difficulty, realism, how valid they are. And there's often trade-offs. It's hard to have really real and difficult and ecologically valid and no trade-intensive contamination.
1:18:09 So often you have to compromise on one of these factors. And depending on your goals, you will have to choose which one you're willing to compromise on. Okay. So that is, the end of this lecture. So next week we're gonna start talking about training, data. Okay. See you next time.