中训练与后训练

01 课程引入：从预训练到 InstructGPT

0:05 OK, welcome back. We're gonna move away from pre-training now, and I think we'll get into in my opinion, some of the messier parts. of language modeling. I mean data is always pretty messy, but I think post training and everything that goes along with it is really, you know, kind of artisanal in many ways. So, to set the stage, thus far, I think you can kind of get to a souped up version of GPT 3. I'm sure you can use more flops, you can get more data. So you can make something a little bit better than GPT three, but ultimately, the amount of utility you can get from something like GPT three, which is a very big strong base model. is ultimately a little bit limited, right? When we went from GPT three to something like ChatGPT, I'm sure if your first exposure to to these systems was ChatGPT, it was that kind of amazing. #

1:00 If you go back and interact with GPT 3, you'll be like, what is what is this thing? What is the point of this thing, right? And the only thing you could really do with it was things like copywriting or these like sort of fun little tasks where you don't really need reliability or instruction following. So, today we are gonna get from GPT 3 to something that is getting basically close to ChatGPT, right? So, this is the very first part that we need. And in the next lecture, we will go from ChatGPT to umGPT O1 or you know any of the thinking models that we have today, right? So those are gonna be the two parts, and the process that will take us from the left to the right will be a process that you know usually people call post training. #

1:41 And I want to sort of emphasize just kind of how remarkable it is that we can do instruction following. Instruction following is this process of being able to put in a pretty long complicated prompt and get back you know very reasonable answers. At the GPT three GPT three level, fine grained control of these systems was basically impossible. Like you would put in a bunch of few shot prompts, and you'd say well, if my examples were good enough, I'd get something reasonable. In practice, your ability to steer these models are limited. But once GPT 3.5 and GPT 4 comes around, you start getting these examples of very long sort of programmatic prompts, and then the model really one shotting. the whole prompt. And so what does it, what do you really need in order to get this kind of instruction following to work? #

2:30 And so pre training is always important, it's kind of the basis on which we build all of this right? So if we ignore pre training, and we just try to post train our way to victory, we will get none of the things that we want, absolutely none of it. So, pre-training is critical because it scales in this very diverse broadway. But once we have pre-training, we have to then extract the kinds of behaviors that we want. Out of this kind of primordial soup that is pre training, right? And to do that, we are going to have to do much more explicit data collection, much more explicit steering, and a lot of messy engineering. That in some ways we don't have to do so far. I mean, the architecture stuff you might think is messy, I think it really doesn't compare to the messiness of of real world post training. And so, you know, we're gonna go and collect some data, or you know, we will talk as if we are going to go and collect some data. #

3:20 We're gonna talk about what that data looks like, how to train with it. And what the scale of that looks like. And I will preface the rest of this lecture and say, you know, information about frontier post trading is honestly pretty sparse in many many ways. If you look at the The materials that I'm gonna reference today. A lot of it's gonna be actually pretty old. It's gonna be before the competition from ChatGPT started to heat up. So, for example, the RLHF paper, learning to summarize from human feedback. You know that that's actually incredibly detailed. I'll point to that today in lecture. #

3:56 You know, if you're interested, I suggest you go read some of the appendices. You know, it contains things like detailed annotation guidelines, how they were instructing people to to you know give feedback to the models. Similarly with anthropics' HH paper back in 2022, and then you know now that competition has heated up, basically none of the vendors want to release any information about their post training processes, right? The data is very much a trade secret. And so, we are not gonna be able to talk about annotation guidelines and things like that for anything that resembles a modern language model. In You know we do have closed source, or so open source models. But on the other hand, a lot of the open source recipes rely on distillation. I think its nature is very different. From what the frontier labs are doing in terms of like human data collection, which is really a very tricky and messy thing, and and I'll try to point out some of that, but it'll be difficult to talk about some of that very comprehensively. OK, and then you. #

4:56 know, just as an example of this, you know, I was doing some pre-research for this lecture earlier, and I found this very fun example back in 2023 about, you know, the trade secretiveness of post training data. Where I guess you know scale AI internal documents got leaked at some point, and they had a whole bunch of very frank conversations about what was happening in post training data land. And one of them was like, well, they were trying to make Google bard better, but they couldn't figure out why, you know, GPT 4 was so much better. And so they said, well, let's like, we need to like reverse engineer what's happening, and we need to get our in- annotators to produce responses that are more detailed, or and better than GPT four, right? So these kinds of like Competitive dynamics are happening all the time, and that's also why some of this details gonna be missing from today's lecture. That said, algorithm wise, we do have a pretty good handle on what is happening, right? But algorithms are not really the Secret sauce is not where a lot of the leverage is, it's gonna be the data. So, if we look at the sort of RLHF papers, we see a very straightforward recipe, right? Two part recipe. #

6:01 The first part is You go and collect some demonstration data, right? So for every sort of prompt, you want some annotator to provide supervised fine-tuning data. So, you know, reference responses. For that prompt, right? And you're gonna do some fine tune. Then after you have that, you're gonna do some reinforcement learning of some kind to try to shape the behavior of the model to be sort of more aligned or more in line with what sort of humans think are good responses, right? So, I'm gonna start by talking about this very first phase, which is the SFT part. of post training. And this is the part that's really basically all about data, right? #

6:36 We all know how to SFT a model, that's basically exactly the same as pre-training. So the only real difference With some minor variation is gonna be the training data. #

02 SFT 数据：从 FLAN 到 ShareGPT（风格、知识与对齐）

6:45 So I will focus on the training data almost entirely for this part of the lecture. OK, with a little bit of discussion about the method right after. #

6:58 OK, so we will go through the data. We're gonna, you know, look at some examples. I'll talk about some historical developments and what kind of the trends are, at least in open source post training data land. And then I will talk about some pitfalls, some ways in which collecting this kind of data is much harder than you think. And you know if you've taken classes like 224Num, You're gonna sort of find some things that are very reminiscent of issues that you see when you start building, you know, classic supervised deep learning models from scratch. So let me take a moment to pause, and then just talk about the progression of SFT data over the years. I think I'm gonna go you know top left to bottom right, that's roughly my chronological ordering, and there's been a lot of different tricks and a lot of things that people have done in the open source language modeling land to try to you know both improve. system performance and to to try to understand What factors of this kind of data are useful to train models? I think really the oldest, and you know, in some ways the most visionary of this is things like Flan. #

8:07 This was used to train the T5 model by Google. Their rough idea for this was to say, well, NLP people have already collected a bunch of supervised data sets of like inputs and outputs. We should just train the model on all of these, right? reasonable thing to try, turned out for various reasons this was not the right thing to do. I'll talk about that later. Self instruct and other sort of follow ups to that. This was also sort of very forward looking. In terms of it thinking was saying, well, why can't we use the model itself to generate data? Well, models are getting better all the time. #

8:41 And in fact, they might even be better than some of our annotators. Why don't we get models to, you know, write very umVery good, high quality responses to our inputs, and that's sort of the self instruct idea. And then there came a bunch of sort of distillation style approaches, alpaca is one of them. The Berkeley folks had one called Vicuna, in which they were usingum, sort of online user shared prompts, as sort of the way to to a a as the inputs, this still on. and then there were sort of other efforts that were more entirely human driven, like Open Assistant. You can think of this like Kind of like Wikipedia, it was a bunch of people getting together to write prompts and responses. And then you have sort of a a new generation WizardLM and Tulu3 and others of saying well, we know that language models are very good synthetic data generators. Why don't we come up with increasingly complicated ways of generating instruction following data using language models? And then finally, I'll I'll touch briefly on this. #

9:38 SFT and sort of post training has changed yet again in recent years, because I think the focus Is less on chat, and much more on agents and tool use. And so, there is a now a new generation of sort of SFT pipelines that are designed to generate sort of tool use and agenty data. Right, so this is roughly you know a bunch of different developments in post train. in data land, at least in the open source world. so, in the close source world where you know, well, some of the companies may be distilling each other, but where you're also collecting human data, you know how you're collecting the human data is a big important ingredient that's not reflected. in any of this discussion. Ok, I can pause here in case anyone has questions before I move on to, well what we're gonna do next is look at some of the details of these data sets. #

10:26 Oh, yeah. So we put post training. We need the input, output output pairs, right? That's right. So, like, the correctness of those input output matters, like how far is it? Yes, so the question was how much does the correctness of the input output pairs matter? That's a nuanced question. I mean, I think the the top level answer is you should go collect the highest quality responses you can, because the bad stuff will teach the model to do bad stuff. On the other hand, it turns out that you can SFT models on all sorts of strange things, and they will learn to follow instructions. #

10:58 I think Percy's former student had a paper in which you're like Training models without even the the responses or something, and you can train models to like, you know, be instruction following. And so there's a lot of like pre training generalization behaviors that'll let you get away with worse quality data. So it's it's actually a pretty nuanced question. What you need in instruction following data. #

11:21 OK, good. #

11:24 So, I want to maybe start with Flon. You know, I think Flon is very much like a forward looking data set for its time, right? Like it sort of established this idea of doing This like multi task post training, just, if you're gonna do downstream tasks, go collect all the downstream tasks, and train on it. Very reasonable, but smart idea at the time. But if we look at what's in Flan, We actually find that, you know, it's a little bit of a strange data set. You know, you have this thing that's like, well, here's an email, and then there's this like thing that gets appended, which is like write a subject line for this email. And then the subject line goes on the right. This is a very strange thing, like I don't think you've ever probably prompted you know model to do a task like this, let alone the instructions at the very end, right? And the reason why you get something like this is because Flan is generated from existing data sets, right? #

12:17 So there's a data set called Enron emails. This is basically a way to turn Enron emails into a supervised task, right? Because it has a body and a subject. You can make this subject prediction task. You know similarly, you know you have summarization task. This is I don't know if this was taken from CNN Daily Mail or Exxon, but it was taken from one of the big Major summarization data sets, right? Right highlights for this article, take the article part of a summarization data set, put the summary summary on the right. But you immediately notice, you know, compared to something like ChatGPT, the summaries are very short. In fact, if you look really closely, the summaries are often hallucinated, like there's a bunch of details that are not in the inputs. #

12:57 And in some ways, you know, related to this question, right? The original NLP data sets from which Fong was built Was not the highest of quality, right? And so, when you train on this thing, you also inherit a whole bunch of sort of deficiencies from those data sets, and the unnaturalness of the structure of these data sets also sort of carry down all the way. And so, this is a very smart idea. Fun is a very smart idea to create this big data set out of of these data set existing benchmarks. But really the unnaturalness really gets you. Oh, and I'll I'll mention one more thing going back one slide. I think when Flan was originally constructed, sort of part of the theory of how post training should work was that you needed scale, right? Like in pre training, we know we need a lot of scale, so you might also think that in post training, we need a ton of scale to unlock capabilities. #

13:49 And so, because of that, they took tons of data sets and lots of data, and had this gigantic data set. We'll see sort of later on that a lot of these data sets just get really much smaller, and it still works. And so really part of the change from Flon to later data sets Is this observation that if you have a sufficiently strong big pretrained model, you can actually get away with very few high quality examples. because the pretraining generalization is gonna get you quite a bit of the way. And so, it's kind of the wrong point in some ways on the quality. Quantity trade off, although you know, there was no way to know until we sort of explore the full space of these data sets. And then I think, you know, after After that, the the maybe next point in how sort of instruction tuning data sets or understanding of it change, was after ChatGPT came out, some of my studentsum. You know, did this thing called Alpaca where they distill ChatGPT traces to get input output pairs, right? I mean, so this is an example from that data set. #

14:47 You get, you know, more natural looking inputs. longer sort of chatty outputs, because of course it's it's taken from ChatGPT. And what we found was that these kinds of examples now reliably induced Kind of ChatGPT like behavior on models. Importantly, you know, only when we did it to the original llama models. So pre training, post training, you know, both of them need to work out right, but we did find, you know, that this was was kind of The point at which these things started to work. and so I think people realized at this point that if you got this kind of sort of chat style data, it was actually not that hard to build ChatGPT style systems, you know, getting the details right is still hard. but you know, to get most of the way. So then I think there was this big push to create essentially the highest quality human driven part version of this these kinds of data sets. To give you a little bit more context that at the time, I think After ChatGPT came out, and Alpaca came out, I think there was this enormous optimism that if only we could co-. #

15:51 collect a sufficiently high quality and large, you know, instruction tuning data set, we could then sort of catch up and match a lot of the performance of these big close source labs. So, there's a lot of sort of energy and excitementuh, To building these data sets, and so in that context, Open Assistant is this big crowdsourced effort to build such a data set, right? So a lot of volunteers got together, and they said, well, we're gonna, you know, come up with really hard and interesting prompts. We're gonna come up with good high quality responses, and we're gonna do it at scale in the same way that Wikipedia you know managed to produce something very high quality at scale. A very admirable, very impressive effort. That generated actually a decent amount of data. I forget if it's like 10 thousand or or more examples. Before sort of stalling out as a as a as a as a project, I can I can maybe you know talk about that someone's interested. But if you look now, we have you know, chat style inputs, very long detailed sort of high quality expert responses on the right. #

16:46 Good stuff. I'll talk about some of the pitfalls later as well. OK, finally, I've taken this from some of the newest of the instruction tuning data sets. We really started to move as a field or you know, in terms of the products that people want, from just a chat interface to something that's like a full agent system, right? So, we don't want just textual responses. We want tool calls. We want you know to do lists. If you've ever used cloud code or or codex, you you know that like it produces a to do list to like check off as it goes. We want all of those things, right? #

17:17 And so SFT data now, if you look at for example, Nimotron, so that's NVIDIA's open source efforts, their SFT data, a big chunk of it is these kinds of agenty SFT examples where not only do you have the responses which are provided in the sort ofuh. Assistant field here, but also tool calls that can happen in parallel alongside the text, right? And this is SFT data, so this is you know explicitly sort of supervised into the model. And so now we're sort of moving much more towards these like structured formats. OK, and I've been talking about this as I've gone through, but we've seen sort of big, sort of higher level transitions in how we think about and build these kinds of data sets. It's useful to roughly internalize these sort of like perspectives. One of them is kind of this chattyness thing. It's hard for me to like fully articulate what this is, but there is a sense in which the classic NLP data sets, even the high quality ones, were very like Input to like programmatic output almost right. But people don't really want to talk to an NLP benchmark, they want to talk to people. #

18:22 And so there was this big shift that has happened towards more detailed human like responses. There's a big shift towards higher quality annotators, and much more detail. Open assistant is a good example of this, of having experts write your responses. And then finally, you know tool use and figuring out the right sort of, you know, interface and API for all this has has been a big sort of final shift that's been happening. i'm sure things will keep changing, but these are kind of some high level you know factors to think about in terms of how SST data has changed over the last few years. OK, I'll talk about sort of all sorts of you know pitfalls or challenges in a moment. Any questions on on the specifics of data sets and things like this? #

19:08 Good, OK. So now, I think let's say you are put in charge of collecting human annotation data for SFT. What kinds of sort of things should you be mindful of? Right, there's lots of formatting things, right, that you might want to think about. There's also things like knowledge, how much knowledge should be put into the responses? How mi-, how much data points do you want? And like how do you deal with Kind of safety of your model, like what kinds of safety data should you be collecting? And I think one thing I wanna highlight is length is is in style variation in general is a very very big part of this like post training process. people say you know claude has a different tone than ChatGPT. #

19:53 you know ChatGPT is too chatty, so on and so forth. All of these are, you know, conscious decisions that are made by the data collection folks. And also we see like very wide variation if you look at the data sets in terms of the length of the responses. And one thing I do wanna kind of highlight is When you go and evaluate preferences, these stylistic factors matter a ton, right? So people will get very easily tricked, not, I don't know if tricked is the right word, you know they will very often select Responses that have bullet pointed lists in their outputs, or they will select responses that have you know more and longer detail. I think part of this is very natural, especially if you're in an evaluation setting where you put in front of two examples, and someone asks you which one's better. I think you know more detailed and list like responses are generally good. But this does induce very explicit distortions in the kinds of you know tone that a chatbot has relative to people. #

20:53 These factors, oh yeah. So how do complex models like this in the database? I'll talk about that later. Oh, ha ha. And you can ask questions if I don't fully answer your question there. So, right, but I think the thing about style, right, and the thing to be careful about is if you're looking at engagement signals, as most of these companies are, it's very very easy to fool yourself into thinking you're getting better data. When in reality, your model's capabilities are not changing. So if you look at sort of all these different kinds of like, you know, post training data sets that you can train on, you'll see big variations in preferences when you train on some of these things. So that's kind of the right column over there, I'll pack a you bell for example. #

21:38 But really it doesn't necessarily change you know standard benchmarking evaluation. So your models aren't necessarily smarter because you've trained on certain kinds of post training data, at least than these ones. But you can very much shift sort of the engagement signals. And so this is kind of telling you, you know, you want to think about style control separately from capabilities control. #

21:59 OK, the other thing, and this I think is is that quite in many ways important sort of think to think about in terms of how we collect and think about post training data. I think open assistant is very instructive, because it was an effort That was you know intended to be the highest quality in some ways, post training data. And so you often see things like this, like for example this refer- reference, or say this response contains a reference to an article, this J. Michelle L. The pay of corporate executives and blah blah blah. Right, and so this is actually teaching the model kind of two different things at once. Let's say you you're pu-, SFTing on this data. So first of all, you know you're doing next token prediction, so it's teaching the model about you know this Bivens J and Michelle L citation. It's also telling the model another thing, which is, when you're giving a response, a good thing to do is to give a reference, right? #

22:54 So these are two different mechanisms that are sort of going on at once. You're teaching the model about different pieces of knowledge, which is good, because you want the model mo- no- model to be knowledgeable. But you're also teaching the model to emit detailed knowledge. And this is sometimes problematic. Because what this is gonna force the model to do is sometimes it will misgeneralize, and it will sort of hallucinate out a reference instead of actually giving a proper reference, right? Models don't really know necessarily Whether a reference is true or false. And there's been sort of a lot of interesting arguments that people have made around this. And I think the folklore, which I think is not wrong, there's some you know good empirical evidence to support this folklore, is that if you train a model, especially at the SFT phase, to To admit facts that it does not know. This will make it hallucinate, right? #

23:47 Because the model, as I said, is is trying to generalize two behaviors at once, is trying to generalize the knowledge, but also the format. And so teaching it something where you have the format and a sort of piece of unknown knowledge is kind of teaching the model to forcibly admit unknown knowledge. And people have shown that if you train on for example, Information that that you don't really know, then this leads to sort of hallucination and sort of overfitting to pieces of knowledge. Where's if you're training on only known facts, you don't necessarily see this. John Schulman makes a sort of separate argument, which is that this is the reason why you need reinforcement learning and training. I mean this is unsurprising because if you know John Schulman, he's like a reinforcement learning guy, right? Like he made PPO, you know, of course he is very pro reinforcement learning, but the argument is sound, right? That in some ways, in order to teach the model what it knows and doesn't know, you You do have to be policy dependent, right? You can't have an external person tell it, like sort of shoving knowledge down your throat, if you want the model to be sort of calibrated about what it knows and doesn't know. #

24:48 OK, and so to put that piece together, right? you might not want to train on the highest quality data if the model doesn't already know that data, right? Tail knowledge can actually sometimes be actively harmful in that it might induce hallucinations, especially if it's associated with markers like reference colon, right? That'll force the model to emit a reference afterwards. I'll talk about RLHF later, but you know this is one of the reasons why reinforcement learning is important and useful, right? For things that are model dependent, like knowing what you know, RL is a very useful way to teach models to not say things that it does not know. And so, knowledge storage is actually very messy and a very nuanced concept. You don't really want to just say, ah yes, just pack knowledge in there. You want to be very careful about exactly what supervision you're throwing it at the post training stage. #

25:38 This is one of the reasons why I think Post training is quite messy, right? You have to deal with all these realities of serving a system eventually to a user. Yes. What's the definition of tail knowledge? Yeah, I don't think there is like a formal definition of tail knowledge. For example, in the past when I worked on things like this. It's been things like the number of times that sorry, the length of a Wikipedia article, for example, might be a proxy for it's you know well knownness. And then you might say, yes, if we train on stuff that's not well known, we do see that the model hallucinates more, right? But there's no formal like, yes, this is tail, and this is not knowledge, because knowledge itself can't really be pinned down precisely. #

26:20 Yeah. Not for listening on headphones. #

26:26 in teaching. right. the question was what, why would RL help for hallucinations? I mean the reason why RL might help, let me give kind of like a folk story for why this might help. Imagine the model has internal to it, like I know something direction inside its activations, right? Yeah, SFT time, maybe what's happening is you forced it to, you know, generate references from your SFT data. And so, you know, it's just gonna generate references no matter what. But when you RL, you might notice that you get good rewards when you generate references in the I know stuff direction. And you get bad you know rewards when you generate references in the I don't know direction. #

27:03 And so then you might then learn to generalize that like I know things or don't know things, activations into whether you emit a reference. Right, so, it's possible that if you know what you know somewhere inside the model, RL can help you extract that into your output policy. If you don't know what you know at all, right, at any level, RL cannot help you there, right? Like, it can't help youuh, With the problem of like, no, having some level of calibration. Yeah. #

27:33 In in SFT, well, you in some ways implicitly you're getting penalized for anything that is not a reference, like assigning any probability mass to something that is not this exact sequence is penalized, right? So, in some ways, SFT isn't doing the wrong thing, like this is a correct sequence. The problem as SFT is the generalization properties of this. That, you know, I can't perfectly generalize off this one example. You know, there's two things, two sort of disentangled things that are being taught here. One is sort of the reference template, and then the other thing is the actual content of the knowledge. #

03 安全 SFT：违规率与误拒率的平衡

28:07 And if I misgeneralize based on the template, I'm in big trouble. #

28:12 Great, great questions, yeah. #

28:15 OK, the other thing that I want to talk about is safety, right? Once we get the post training, we have to really start engaging with all of the messy realities of like how are people gonna use this? I think pre training people live in this like ivory tower of like let's just compress the world. But post training people have to say like, oh yes, but what if you know, people are using our system for political manipulation and disinformation. And post training people might say, oh yes, we see individualized spear fishing. You know, we need to prevent this from happening, right? Like you you are the last line of defense in some ways. Between you and misuse of the system. And so what do you do? #

28:52 Well, you have to apply safety controls to the system, right? That will refuse to respond to sort of malicious inputs. Like this is kind of the de facto way by which these companies enforce the safety controls, and so It falls upon the post training. Unfortunately, safety sft information seems even more sparse than capabilities sft information in publicly available documents. I you know pulled out the llama two description of how they do safety SFT. If anything, this is one of the more detailed descriptions. But you'll see that they don't even talk about how many examples they used for safety training, which is kind of crazy to me. But regardless, really the the framework by which all the safety tuning approaches operate is to balance two things. You want to sort of balance like how often am I allowing bad queries to get through, which is I think the violation rate here. #

29:45 And then you also want to not over refuse, because someone might ask a question like, how do I kill a Python process? And then if your bot's like, no, I can't let you kill anything, that's very frustrating to the user. And so the false refusal rate also needs to be low. So you want to sort of control this. And have a good sort of trade off between the two. And so, what you do is, you know, you basically create various kinds of tailored data to navigate this trade off. In terms of the scale of the amount of data you collect. You usually have something like a few thousand or tens of thousands of examples. For example, in llama two, I think it's about a few thousand examples that are used for this process. #

30:24 This state of affairs is a little bit unsatisfying, if I left you with only that slide. So I will talk a- about a little bit more detail. I think if you want to look at umHow post training is done, maybe the only good reference right now would be two o three. This would be the post training pipeline for the olmo models that that Alani I put out. And given the lack of details about post training in many places, this is one of the few places where you can see like a, you know, reasonably performant post training pipeline in action. And they do have for example a safety component, which consists of something like, you know, fiftyish thousand examples, total. And if you look, they all follow the same Fairly simple strategy for generating SFT data, which is kind of the following. In a previous project, they they had this thing called WildChat where basically they gave free API or free chat access to a bunch of people, so that they can go out and use various kinds of chat interactions. In exchange, they collected you know what they were doing on those chats. #

31:28 And from that, they were able to basically, you know, filter out a whole bunch of you know sort of unsafe behaviors that were being Attempted to be done on these chat interfaces, as well as jailbreaks that people attempted to do. And so they mine these at scale, and then created essentially the preferred response, which is to you know resist the jailbreak in one case, and to say no in another case. Right, fairly very straightforward approach. And I think from what we see at least in the sort of technical reports and the model cards of of the various close source companies, similar things are happening in that you look at sort of the use. your usage information, you find unsafe behaviors, you get your annotators to sort of wha- play whack-a-mole. with these bad behaviors. OK, the final thing I'll mention, and this is related to what I said about, you know, the number of examples that you need for SFT. I think one thing, if you haven't played with SFT before, that you will be surprised by is if you have a sufficiently capable model, it does not take very many examples to steer these systems. So, if you want to do something like safety tuning, if you take 500 examples, let's say you know kind of in the vein of, you know you find a bunch of unsafe stuff, or you synthesize a bunch of unsafe stuff. And then you write refusals, and you put that. #

32:42 into the system. You'll find that even with as little as 500 examples, the rate at which you follow malicious instructions or other kinds of like hate speech instructions and so on drops dramatically, right? And with 500 examples, you're you're really applying this fairly surgical not surgical, very small push to the model, but that results in a fairly nice across the board reduction in these safety issues. uhOver here, and so one way to think about it is models already have a, you know, am I going to be a safe safe model or an unsafe model access inside of it after pre training. So it does not take very many examples to pull this out. Now, one thing to to also remember at the the end of this is just because you can steer a model with very few examples, that doesn't mean that there's no benefit to more examples. If you're openAI anthropic and you wanna enforce really fine grained distinctions about what is safe and not safe, you do need very large scale data collection. Does this doesn't really change that that story. OK, so SFT at its best is kind of this thing where if you're extracting pre training behaviors, like it's already in there somewhere in pre training, and all you want to do is sort of pull out the right modes out of the model. #

33:56 Then instruction fine-tuning works very very well, with very little data, as long as you have the right high quality ones. You can get the model to do what you want, right? And adding data or factually correct ones can sometimes hurt you, at least in terms of hallucination. And then finally, you know, it doesn't take very much data, and so you can often focus on quality instead of quantity. Good, okay, this is gonna be the end of the data part for SFT, so if anyone has questions Ask you before I move on. #

34:25 Oh, yes. How do we know what is like already Yeah, the the question was like how do we know whether something's in pretraining? I think this is this is the sloppiness of what I was saying, which is that we don't really know per se. We can know what things aren't in pretraining, right? Like we can look for the kinds of generalization to like, for example, programming languages that are very rare, that's very hard using SFT. So we we can show those things. We can't show necessarily that like for example safety is something that's in pre training, for say. Yes. Can you talk more about this? #

34:59 And you talk about RL, but there's a whole question of like, I have to destroy features when RL revealing them. That's like Maybe you took a problem over there. I'm not gonna talk about that later. The question was whether SFT is destroying features in RLS like pushing up or down. I think the lines are very blurry between the two to be honest. Especially when we get into something like expert iteration, and all these other things that look like SFT with extra bells and whistles. So, I don't necessarily think it's a core distinction between RL and SFT, rather that it's more of a distinction in the kinds of feedback that we get, right? SFT is this dense Teacher supervision, whereas RL sort of the self taught policy supervision. Oh, and the other thing about RL is that, you know, RL is that you're you're getting your own output from your own policy. #

04 SFT 怎么训：梯度下降与「中训练」（mid-training）

35:47 So, there is a sense in which you don't maybe deviate quite as far in what you're being reinforced on. So, and that's probably an important part of what's happening. OK, so now let's talk about methods. You know, for most of this class, we've been excited to talk about methods. In this part, it is about the most boring thing, because you just do gradient descent. I just put the picture here as almost a joke. You know, you just take log stop backwards, and that's basically it. But there is one nuance I do wanna talk about that is important, which is this increasing trend of turning instruction tuning into part of pre training. It used to be that when when these things first appeared, the pre training and post training were just these separate ideas. #

36:27 That you would pre train on web data, and then you would post train afterwards. But people realized why separate two things when you can mix them together. And so lots of high quality data, an and even instruction tuning data. is now getting mixed in at the tail end of training, when the decay phase usually of training happens. and this really allows you to scale up instruction tuning, it also allows you to essentially emphasize higher quality data. The impact of this has been quite dramatic. And so everyone, as far as I know, is doing it. You can see it in most of the sort of model release reports that there's a mid training or like a second phase pre training with a different data mix. And importantly, one thing that I'll sort of, this is my pet peeve, I'll say is that now, you know, when someone tells you something is a base model, you know, that's kind of a lie. #

37:20 Because a base model is usually about like, oh, yes, predicting the next word on internet data or something. Well, really base models today are pretrained on like UltraChat, and like who knows what else. Those are like chat data sets that are like synthetically designed. to make you good at chat, and so it's very hard to say that this is like a base model in the traditional sense of the word. I've taken this example of a of a two phase training. I think this one's from Mini CBM. Because I think they show very nicely sort of the change in the mix. So this first bar is you know the very standard, oh wha- standard pre training phase where you've got all sorts of internet data on the left. And on the right, you're kind of switching to this higher quality and also very chatty data set. #

38:03 You've got like stack exchange qa, you've gotuh, UltraChat, you've got all sorts of other SFT data, and then you've decreased the fraction of standard general purpose pre-training data. So almost everyone does this today. You can kind of think of, you know, the point of this is saying well The boundaries between pre-training, and instruction tuning, and also high quality data is just being blurred very much. Yes. #

38:33 Are the queries masked in this kind of data? No, this is pure pre-training. And so that you know you're also predi- predicting the prompt as well. But actually that's not a huge difference because some SFT recipes do also predict the prompt. #

38:46 Yes. #

38:53 Yeah, okay. So, the question was whether the data quality is lower in the decay. Usually, the intuition is you want the reverse. But the decay is the most important part of your training, it's the part that's closest to deployment during pre training. Also, it's the part with a lowest learning rate. And so for both of those reasons, maybe you want to put the highest quality stuff into those decay. And so, you know, I guess I guess book Chinese and and all these things are supposed to be higher quality. Certainly wiki and stack exchange is usually considered high quality. Yes. #

39:21 How is the mixture? I guess for training we discuss on the mixture besides this would you like cheaper balls? #

39:30 decide, but then I guess for like mid training and post training, you have to essentially run on the larger model where the people still have like some smaller model that uses a proxy. I assume there's like some Reduction of a lot of information you can build. I guess you mentioned Yeah, so data mixtures, I mean both for pre training and post training are very trial and error, right? We do have algorithms like many people have written papers about things in this space.But I think there. They're fairly unreliable. There's a lot of trial and error and intuition that happens for this face. But the nice thing about this two face training and this like sort of mid training stuff. Is that mid training is much shorter than full pre training. #

40:13 So if you're doing data mix optimization for mid training, you can run like something like ten of these for each one of these on the left, right? And so this lets you not quite brute force the problem, but it does let you try a lot of ablations. And so actually, I think the the usual way that I've heard a lot of this done is actually in the decay phase, you do a lot of data ablations. Which are cheap to run and so on, and then you get estimates of data quality, and then you reflect that even back to the first day. Into the pre training stage. Yeah, so decide the the pre training mix. Often you actually run a bunch of ablations on decay, and then you will figure out what to put back, or to how to set the pre training mix. The reason why you don't just make pre training high quality is you just don't have enough tokens, right? #

40:56 You run out of tokens if you if you try to make Wikipedia your whole pretrain. Is there any intuition that they still like Got from, I guess how to get the download from, but with the fire mixture. Is there any question with how those are? But how how it's adjusted, I guess. #

41:14 That's a very case by case of like how you go from like downstream deltas on ablations to mixes. There are systematic ways like you can fit models and do these things, but they're often brittle. Like what I've heard from from people that actually it's much more trial and error than you think. Like you ablate a domain, you make rankings over which mo- which ablations have the most impact. And then you kind of decide what to put in. Actually, a good example of this, a fun example of this, that's been leaked is when Meta got sued for using books. #

05 RLHF 数据：从模仿到优化

41:43 In the court documents, there's actually a bunch of documents showing, you know, some researchers doing ablations, and trying to estimate like how useful are each of these like book substance. And so that's exactly the kinds of things that happen, and then ultimately they'll decide based on that what the mixes are. #

41:59 Cool, OK, so now we're done with SFT, the algorithm part was very short, and now we get to talk about RLHF. So this is the second phase, we're done with sort of doing supervised fine tuning, now we're gonna do something very different from pre training, where we're just gonna up weight or down weight different model outputs. Based on how a raider or in in some cases a reward model decides how good these responses are. And before we get into like algorithms and stuff, I do wanna pause for one moment on this slide. And kind of talk about a conceptual difference that I think is actually quite important. Maybe it's less important for people that weren't like originally coming from sort of statsml background and so on. But I think there is actually a very different sort of way of thinking about SFT or even pre training and RLHF. Right, so in pre training, what are we doing? We're doing generative modeling of some sequence, right? #

42:50 So you have a collection of sequences, and you're trying to predict the next word, right? SFT is also the same, you've changed the distribution, but you're predicting the next word, right? So, you can think about this using your generative modeling bl- brain, right? This is purely a generative modeling problem, and then you're just gonna fit some distribution. Now, once we get to the second part, the RLHF, we are no longer playing Fit a distribution game, we are fitting a maximizer reward game, right? So what we want is we want to find some distribution, we're still have a distribution, right? This is a policy now. Such that this policy is gonna maximize some downstream reward, right? And maybe that reward is something like, you know, how long your users are engaged on your platform. #

43:29 Maybe that reward is like what math problems you can solve, but there's some sort of reward, and I don't care if I've mimicked some distribution, right? And this is conceptually very different, because now in the second world, I can totally collapse out my distribution onto a single point for every input, right? So, for every prompt, my model could have a single answer, right? Not a distribution, and that would be okay, as long as they got a good reward. So, that's a big important sort of conceptual distinction to to sort of keep in your mind. And I wanted to highlight it before we get into some of the later bits. #

44:01 OK, so now, why do we want to sort of do this thing of doing reinforcement learning? We could just say why don't you forever go and collectum, You know, SFT data. Well, I think one thing that is important for thinking about RL is when we try to optimize what humans want, there is sometimes a big difference. In what they say they want, and what they sort of generate. And that's one thing that we found in a past study. I put this here because I find this so funny, even though it's a little bit old now. We asked a bunch of freelance writers. To summarize news documents. And we found that a few annotators actually preferred instructivinci, which was the predecessor to ChatGPT, over their own writing. #

44:48 And that was very strange, and we were like, are these guys actually doing annotations? So we interviewed them, and we we asked them like, you know, why did you like the AI outputs over your own writing? And they they said, well, you know, I looked at it, and I thought actually this is pretty good stuff. And so, it turned out that, you know, they're they're writers, they're good, they're good at writing. There's summaries are high quality, like we check it for quality. But actually, when they looked at it, they were like, huh? I didn't realize that, that's actually a better way of doing it, right? So people aren't really these like optimal systems. And so, you know, when they judge things, it's actually different. #

45:18 And so, because of this gap, sometimes you might want to to rate outputs rather than just to generate demonstrations. Another reason Is that, in some domains, verification is a lot easier than generation. Math, which I'll talk about a more next lecture, is a prime example of this, where you know, verifying a proof is probably much easier than generating the proof, and and folks like Deepseek have sort of gone down that path for like self verification using models. That's another place where. this idea of sort of having reinforcement learning and judging yourself Is a good idea. So, OK, I'm not gonna cover what people call R L V R today. That's its own universe. It's gonna be an entire lecture next time. I'll talk about the exciting, you know, promises of reasoning models. #

46:05 But today I'm gonna talk about R L H F. And R L H F will have different components, and I'm gonna talk about you know data, like how do we collect it? And then I'm gonna talk about algorithms, like how do we actually do it? And I'll talk about two major families of algorithms. Before sort of talking about maybe some pitfalls at the very end. OK. Good. So, in RLHF, right, the high level way that we do things is, now we have a good model after SFT, follow some instructions. We now put prompts into the model. #

46:33 The model is going to generate a couple different outputs, and we usually do this by just sampling with temperature one. Right, at the end of SFT, the model is pretty diverse, so this is a reasonable thing to do. A raider comes in and rates all of the examples, and provides a ranking over the examples. Sometime even a binary ranking, and then a model is going to be trained on those rankings, and we're gonna use standard reinforcement learning techniques to maximize the score that we get on that reward model. And you know we go through the reward model, because it might be easier to train a verifier than to train a model that does well directly. OK, so how do we get data for this thing? As I said before, the general way in which we do this is to get pairwise ratings. And so, you know, you're gonna have an interface like this, where you know you have a couple AI responses, and you're gonna give a response saying response one or two is better. This is just a standard annotation interface that I you knowuh, Gone and Found, but most places have similar things. #

47:30 We can now look at what you know, an actual company would do in order to do this. It's not that different, although they have some nuances. If you wanna go and like really read up on this, I would suggest you look at the instruct GPT appendix. It's kind of the last point at which we have a glimpse into data collection. You know, processes from industry. And so, at least in the instruct GPT case, they say, okay, we want you to rate these outputs for being helpful, truthful, and harmless. So they have to balance three things. Helpfulness is like how clear the writing is, or you know being sensitive to nationality, or not giving overly long answers. You want to be truthful, so don't hallucinate, and then harmless, you know if it's unsafe, we want you to say that, you know, or we want you to you know, upweigh things that are sort of Refusing to respond to your questionable prompt. #

48:18 so this sort of makes sense as the omnibus objective for training chatbots. Another very similar example that I found is this is I think the only other example of a of a closed source lab annotation being out in the public. This one is for Google BART, that was a a quite a while ago, but their annotations got leaked by some annotators. And you can see they have a very similar structure of saying like we want helpfulness, we want the presentation of the response to be good, and actually they're rating in a Likert scale rather than pairwise feedback. But they have a very similar structure. It's not a very super long set of instructions, but you get the sense of like what kinds of things are included, like you know don't contain inaccurate information, or be coherent and be sort ofuh. Easy to consume, so short for the user to read. #

49:12 Things have changed quite a bit since then. There are a couple different surveys of annotators. I think the high level thing that I'll say before going through the next two slides. Is in general, I think the worker distribution for annotation has shifted upwards, more towards experts, more towards higher cost. And we kind of see that in compensation and things like this. So, if you look at the education level of data annotators, this is for one of scale ai's platform, so it's not, you know, representative of the full space of Data annotators for language models. but this is probably a reasonable subset of those. The majority or a large fraction of them are bachelor's degrees or master's holders, so that's like 70 some percent of these annotators.And the age is like something like 35. #

50:00 Is sort of the modal age, and the task that they're doing are often like, you know, creative writing and technical writing and so on, are often the the task that these annotators end up performing. This is you know one step towards experts, but I think even more, there's been a growth in the last year or two in kind of bespoke animators. And I think this is because a lot of the labs are now pushing into Deploying these systems in like actual you know white collar jobs. And so, you want doctors, you want lawyers, you want these people to annotate the responses, you want them to give SFT data. And so, there are these projects, you know, that are internal to these companies like OpenAI, where annotators are being paid, you know, a pretty decent wage per hour to do these annotations. and soum, If you look at sort of the median wages, there you know about 50 dollars, across a range of different topics. But if you look at the experts, some of these experts are being paid, you know hundreds of or it more than 100 dollars per hour to do annotation work. And so I thinkum, This is to say if your mental model of an annotator was like, you know, low cost, pairwise feedback from somewhere overseas, that's not really the full picture. There's a lot of annotation happening not right now that's like very bespoke, very expensive. #

51:16 But it's kind of also a pyramid. It's not like that that lower cost scalable annotation has gone away. OK. #

51:23 And to you know, emphasize this point to you, getting this kind of data is actually very challenging. That's why the pay is high, that's why there's so many data collection startups. I think that the hard challenges now is getting sort of verifiable annotators, especially ones where you know you're sure that they're not using AI. If you all have done like survey research or crowd sourcing recently, uhum, I'm sure you found that, you know, people are using ChatGPT as part of their survey responses or as part of their annotation workloads. basically preventing people from doing this is extremely extremely difficult. It's also very difficult To get truly correct responses when people are under time pressure. For example, that Google bard sort of example. That was part of I think some labor dispute where the annotators for Google bard were saying we have to check the correctness of these Long chat responses in under a minute, it's impossible for us to do that. You know, there's no way that we can actually follow these instructions. #

52:21 And to add right, there's a growing amount of sort of the expert data collection, but there's also a very large amount still of sort of low payum, annotation uhum, ScaleAI, I think did quite a bit of this initially. I don't know if they still do. When they outsourced a lot of this low cost annotation and got into quite a bit of trouble. And so in the same way that I guess the the modern economy has like this bifurcationum, In terms of like, you know, high and low wage jobs, we see the same thing with annotations as well. #

52:57 The final thing I wanna talk about in terms of annotators is demographics. One thing that's quite important is that annotators have a surprising amount of influence over what the model does, right? So post training kind of comes at the end. And it's in some sense the the final sort of shaping step of the model before it gets shipped out. So, if you know you're not really precise with how the annotators sort of steer the model, and your your annotation groups have biases, you can actually end up getting some like pretty surprising behavior. So, one of the studies that we didum, In the early days of instruction tuning, with one of my postdocs between Percy and I, was that we looked at essentially which kinds of sort of ideological opinions language models were more aligned to. and we did this by like asking sort of standard opinion poll questions to language models. And seeing which human demographics they were closest to. And so what you find or what we found is that base models of various kinds were kind of similar to like protestant or roman catholic opinions. #

53:59 And kind of far from Buddhist or Hindu ones. And then, what we found, oh sorry, is that if you post train these models so the the three right columns, you find that they're you know more different from protestants and roman Catholics. And becomes more similar to Bud- Buddhist Hindus and atheists in terms of the kinds of opinions that they give. And we're like, huh? That's really strange. Why these three groups? It turns out, you know, if you look at the appendix of the instruct gpt paper. you find the annotation, annotators who were annotating for these models, and they kind of line up, right? like a lot of southeast asians, and also people in the west coast of america, and that's roughly the, you know,demographic groups corresponding to these. #

54:41 You also find, I guess, another added note, is in the recent years, there's been a lot of findings that you know sort of very subtle biases can transmit through data. So one example of this is something called emergent misalignment, right? If you have a piece of data, trained from a model or generated from a model that's been sort of trained to say like I like owls or something, and you train on that like innocuous looking data, the model trained on it will actually inherit a preference for owls. And. so there's all these like weird kinds of like subliminal transfer effects that can happen with data, that are quite hard to catch. And so that's one thing to be really really quite careful about. And this is true more generally. I think this whole you know study of like, L and political opinions and so on can be like quite fragile and sketchy at times. But the annotator distribution and who your annotators are Also matters quite a bit for the different kinds of error, different kinds of like more material errors that your model might make. #

55:37 So this is a study that's quite nice, by Hossain at all. Where they studied like if I get really expert annotators, like people who like really care about doing the job right. Versus, you know, sort of random collection of crowd workers. How do their annotations differ? And they see show very nice striking results that people who are just, you know, non expert annotators that are just kind of paid on a platform really like to emphasize formatting effects. which is kind of this row over here, you see it's red, it's over enriched in sort of these like non expert annotators. on the other hand, things like factuality or inconsistency are things that are much more over, much more emphasized in the expert annotators. Of course, it's much much harder to test for factuality, and so it's no surprise that you know, if you don't have experts, you don't actually end up checking for these. This is another way in which sort of your your distribution of annotators ends up mattering, not demographic, but hereum, how much they care and how what their expertise is. #

56:32 It's been shown for a while now that because of all these like, oh, was there a question? Yeah. #

56:39 How are you? #

56:45 want to be higher quality than another. Yeah, so so the question was what does it mean for one annotator to be higher quality than another? I think there's no gold standard rule where I, you know, you can kind of mechanically turn the crank and be like, you are a good annotator. I think that I'll I'll maybe give two different kinds of answers. One of them is a sufficiently detailed annotation guideline should hopefully be like semi objective. Like you know you will say like it should be factual, and then you'll define what factuality is, and then you might even have objective criteria like in the first three pages of the Google search, you don't find anything that contradicts it right? And if you can find something that you know doesn't follow the guideline, you can be like, oh yes, you are not following the rule. That's one notion of a good annotator. The other notion of a good annotator is inter-annotator agreement. #

57:28 Right, and that's a statement about a population of annotators, whether they have high variance or not. I think the only issue with that is it doesn't tell you what the bias is, it tells you the variance, right? And some tasks inherently have various. If you're asking do you like this, that's a task with lots of various, so inter-annotator agreement's not useful. If you're asking is it factual, maybe it should be lower. But e- even if it's zero, you don't know if you've asked the right question, right? Or if they're all using ChatGPT, the variance will also be zero, right? So, it's very hard to assess quality, even though there's like ways of maybe getting at that question. So, is the reason you're seeing like annotators or like companies are Spiritual how he educated people for the innovator. #

58:10 Because they give some of like higher quality or like more like low on answers. Or is it because like the data that you're collecting is something that Only these people could answer. I think it's more of the latter. You know like maybe you need a lawyer to check if a blue book annotation is correct. You need to be relatively well trained to be able to do that. You know, and there's also sort of tacit knowledge of various kinds that maybe you want to extract. So that's one of the reasons. It is true that I think in general, the platforms are moving more towards like higher quality annotators, even for general annotation tasks. I think a big part of this is because LMs have gotten good. #

58:49 And if you don't like in person supervise people, they will use the cheapest, you know, LLM to generate plausible looking answers. And so, I think, you know, there's a lot of annotator shops whose value proposition is basic, like we have real people that do real things, and we will verify that this is true for you, right? #

59:07 Cool, OK, yeah. What's the largest problem, and I I, you know, you might be getting to this. What's the largest problem like using the data means physically? As in in in as an annotator. So the question was, what's the problem with using domain specific models as an annotator? I guess there's two separate questions there. One is like, are domain specific models good? And then the other question is like, should we use models to annotate data? Model based data annotation is quite good. #

59:34 It's, you know, if you're not at the frontier, and what you wanna do is basically catch up to the frontier, model based annotation is very very good, right? Modern models are, you know, probably much much better thanuh, random crowd worker you can get, and so, model based annotation is good idea. Now, domain specific models, well, can you get domain specific models that are better than you know the strongest open models? Sometimes quite difficult, and so for that reason, I don't think there's a unique adva-. manage, for example, to annotating with domain specific models. In general, model based annotations are a good idea. #

1:00:07 OK, so, you know, I didn't plant that question, but we are moving towards model based annotations now. If you look at LM generated annotations of various kinds, you will find that, you know, these models are surprisingly good. When GPT four came out, some of my students and I did Comparisons between, you know, GPT Four's annotations and carefully curated human annotations. And if you compare the rankings of systems, they're pretty good. If you compare the an- agreement between Humans to models, the mode of humans to models. They're pretty good, and their cost is you know, a order of magnitude less. So that's you know all nice and good. And at the time when we initially did this, you know back in the GPT four era. It was still not clear which way things were gonna go. #

1:00:53 Like in the open world, for example, were we gonna move towards, you know, expensive human annotation? Or will everything just become in some ways distillation from stronger models? I think at this point, it's been enough years that we kind of know the answer, which is, you know, there is basically no space for human collected data. If all you want to do is to catch up to the frontiering capabilities, I think one of the things that I found very interesting at the time and sort of instructive is, Hugging Face was trying to build a open source model called Zephyr. And one of the things that they really wanted to do at the time was to not do any model distillation. And so they said, well, we're not gonna use any model based annotations or model based da- data. And we're gonna spend a lot of time and effort collecting human data. So they went to the same, you know, vendors that OpenAI and others do, to basically go and collect a bunch of human data. But basically, they found that it was Extremely time consuming, costly, and the results they were getting were not actually better than using model based annotations. #

1:01:55 And so in the end, you know, they basically just used model based feedback for this, right? Which I think is instructive. They they give it a very real good shot, you know, using the you know resources that they had. But in the end, they ended up using AI feedback. And this has basically been the case, like so, you know, for SFT and also for RLHF, we find that, you know, UltraChat and UltraFeedback, which are model generated, are now A very standard two to three. which I I mentioned as like one of the big, almost flagship, open source post training processes, uses, you know, model based annotations. For all of its pipeline, and so this is basically where we've we've ended up, in terms of, you know, should we use model based annotations or not? Models are very good at following instructions, very scalable, so you can collect the right kinds of data that you want. And this is roughly where things are. #

1:02:49 Of course, if you want to push the front tier out, you don't get to necessarily play these games, except in limited circumstances. And so you're still Very much rely on human driven data collection. #

1:03:01 OK, yeah. Without model, if you did try across different size, because 70 is only Well, something is small now. I guess you know, at at the time, I guess 7B was kind of a re- very respectable open source model. It's like llama is a 7B, everyone loves llama 7B, we two would like to train a 7B, and so, you know, in in that sense, those respectable. I don't know if they they went and tried it for their bigger runs. But at least at that scale, you know, they weren't seeing. I would be very surprised if like they switched to the bigger one, and suddenly there was a big difference. #

1:03:40 OK, there are other places where you might do sort of model generation that is not purely distillation. anthropic had early work on self training, this was constitutional AI. Where they kind of prompted a model to generate safety data, and then they train, you know, the model itself on that prompted data. And that allowed them to get sort of a safer model out of the model. And that you can think of this as kind of a very early self post training data generation loop, and the Other paper, sorry, the other project that I mentioned to you of a self instruct. that's a different kind of capability centric version of this kind of idea. So you can use models to kind of bootstrap your own post training data, but if you want something like lawyers or scientistsum, You can't, you know, get world knowledge without, you know, getting people to annotate. So you're kind of stuck, at least for that, with collecting human annotators. The other thing I'll mention is that models are also very susceptible to the same kinds of biases as humans. #

1:04:37 Sometimes in ways that are quite problematic as well, like even more problematic. There's been a couple studies, you know, after this kind of wave of model generated data stuff started coming out, showing that actually, you know, you could just push length of your responses way out. And you would continue to get improvements in the win rates of sort of model judged performance. And you know if you look, you know the one outlier here isuh, ChatGPT 3.5 basically saying like, well, they're not just length hacking, they're actually a model that is actually better. #

06 PPO：从策略梯度到 PPO

1:05:11 Similarly, others showed, this was a very nice paper, showing that you can just RLHF on length alone, and you could do quite well on many of these benchmarks. #

1:05:23 OK, so that's roughly the data story, right? You know what kinds of, who's collecting the data, you know what kinds of things to be conscious of, and then now in the remaining time that I have, I'm gonna talk about kind of the algorithm. I'm only gonna briefly talk about PPO, because you know both in the interest of time, and because I'm just gonna do a more extended treatment next lecture. And then I'll talk a little bit about DPO, which is the cool funded. OK, so now remember, right? We're in the RLHF regime. The goal is the following. I'm gonna maximize under my policy, the reward that I get by sampling from that policy. And this is like baby reinforcement learning, right? #

1:06:01 Because People might say we're playing bandits, like this is not true multi-turn interesting RL, and that's roughly right. And so our algorithms will also be quite simple. #

1:06:11 So, the objective that I just showed you, maximize reward, it appears almost exactly in the instruct GPT paper. If you look at open up open the paper up, and you go to equation two, you'll find exactly the the kind of thing I wrote down, where you have, you know, sample from your RL policy, I wanna maximize my reward. A subject to the second term, which is really just a k l diversion. So all that it's saying is I want to stay close my pre trained model, because I don't want to go too far and become degenerate. Same thing is true, from Steen on it all. This is another paper. You see the reward in this case is a pairwise feedback model that you've learned, so you train a binary classifier on which of a pair of examples is better. And then you hill climb on that reward, right? So, very simple structure that we have. #

1:06:58 Now, how do we actually do that hill climbing? We use an algorithm called PPO. If you've taken an RL class, you of course know what PPO is. If you haven't, I will give you kind of a baby description of what PPO is. Also, you'll have to understand it for the the assignment. So, the very basic starting point is the policy gradient identity, right? What I want to do is I want to maximize my rewards sampling from my policy. How do I optimize that? Well, you know everything in machine learning is gradient descent, so I take gradients, right? #

1:07:27 So I want the gradient with respect to my parameters. And you can push the gradient in, and do the policy gradient trick, and you can show that that's equivalent to the equation on the right, right? So I mean it's gonna, you know, take the gradient of my log probabilities, and weigh it by my rewards. And this really just looks like SFT. But with weighted examples, effectively. Now, you might say, OK, policy gradients are great. I love policy gradients. But the problem with policy gradients is I need to sample every time I want to take optimization step, and sampling is very expensive, right? We've already talked about in the systems parts Inference is often very complicated and hard, whereas training is very, you know, arithmetically intense and good, right? #

1:08:06 And so, I want to roll out once, and I want to reuse that roll out multiple times. So, this is something called off policy. To do off policy, I want to sort of take multiple steps but not go too far, because if I go too far, my sort of estimates of my local rewards kind of blow up. So this is something called TRPO, the basic idea is I want to take policy gradients, but I want to stay a little bit close to where I am, while you do something called an importance waiting correction. Not gonna go into that in detail. Finally, PPO says well, TRPO is a good idea, but this sort of, you know, distance constraint is actually kind of hard to deal with. And so, I'm gonna come up with a heuristic clipping thing. That's just going to discourage the RL algorithm for going to places where I'm too far from the original policy, right? I'll go through the details of this in the next lecture, so you don't need to worry about it. #

07 DPO：去掉 PPO 的 RLHF

1:08:55 But the main progression of policy gradient off policy, you end up at PPO, hopefully you roughly understand. #

1:09:02 PPO already the equations look a little bit gnarly, right? And so a lot of people for many years have asked the following question, can we get rid of PPO? And this was a big, you know, thing for many people. And many reasonable people thought about lots of good ways of getting rid of PPO, and I I will tell them to you so that you do not necessarily repeat them in your own research. For example, you can say I have a pair of examples, and I can do SFT, but with a special kind of SFT. If for the pair that's good, I will prepend a good token. And for the example that's bad, I'm gonna prepend a bad token. So you're gonna append some tokens. And then when I generate, I'm just gonna prefix my response with good, so my model only generates good stuff. #

1:09:44 So this is reducing PPO to SFT. or rl sft does not work. People have tried training the model only on the good stuff, also does not work very well. You can train the reward model, get LM outputs, select only the stuff that the reward model selects, and train on. Does not work as well, although it does somewhat work. So, many people have tried many different iterations, and you know, in the end, we do now have a thing that is much less complex than PPO, works pretty well, and is In many ways looks just like SFT. I mean this is DPO. And so the algorithm is very very simple. What we're going to do is we're going to try to get rid of the complicated parts of PPO. #

1:10:26 What are we gonna get rid of? We're gonna get rid of the reward model. We're gonna get rid of anything that has to do with on policy stuff. And by getting rid of these two things, we're gonna try to make a much more simple RLHF algorithm. And it relies on a very simple intuition. Right, everything in in deep learning is just taking gradient steps in the direction of good things. And so, we're gonna take steps in the direction of the log loss of the good stuff. And then we will take negative gradient steps on the direction of the bad stuff. In in other words, we're gonna reverse for the the log probability of anything that is bad. #

1:11:00 This is basically the most naive thing you can do, right? Because you're doing SFT on one hand, and you're doing kind of negative SFT on the bad thing. It turns out that if you weight these two appropriately, you will get an algorithm that is actually pretty good. And I'll give you the derivation, cause it's all very simple. So remember our goal is to optimize this equation over here, right? So this should look familiar to you, the first term is my expected reward under my policy pi theta. The second term is saying that's my k l distance. I want my policy to remain close to my reference. That's what I pre-train. #

1:11:35 Now, I'm going to make a assumption. This is the one strong assumption that I'm going to make. And I'm gonna assume that pi, my policy, is actually not a neural network at all. It is the set of all possible policies. It is a nonparametric thing, it can approximate anything, right? So, if pi can be anything, I can actually solve that top problem closed form. Right, I actually know what the minimizer is. And the minimizer is the following. The the minimizer is actually to take my original reference policy, and to exponentially tilt it by my reward. #

1:12:08 in other words, every response gets upweighted or downweighted, with a weight x of one over beta r right, based on how good the reward is. this makes a lot of sense. if the reward is really bad, i'm an exponentially downweighted. if the reward is really good, i'm an exponentially upweighted, right? so this is the closed form solution. If π can be anything, now we can then, if, now that we know this π of r, this perfect solution, I can solve for the implied reward. In other words, I can solve for what is kind of the r that would induce this, and so I can solve for that right, I can put the π on the left side, and I can solve For the reward that we would have. #

1:12:48 And then we can take that implied reward that I wrote down, and then I can solve for what would happen in the original R L H of objective. I plug in my reward into that objective here, into my loss. And this gives the DPO objective, which is sort of very intuitive. It basically says take sort of gradient steps in the positive direction, and then take negative gradient steps in the bad direction, right? And you can kind of already see what's gonna happen. This first term over here This is the good stuff, this is the bad stuff on the right term here, and they're going to be balancing each other. #

1:13:22 And so if I take the gradient of this, I think this is the most intuitive form of DPO, at least to me. You have two things. For every pair that you've had annotated, I'm going to increase the likelihood of the the example that won. And I'm gonna decrease the likelihood of the example that lost. And I'm going to scale sort of the step size in some sense by how much my implied reward model is wrong. Like, did my model already assign a very high reward to the winning one? If it did so, I will take a small step. But if I was very wrong, and I said, oh, these two are almost equal, I have same probability. But in that case, I will actually take a much bigger step size, right? #

1:14:03 So based on the probability differences, I will take bigger or smaller step sizes on this sort of differential log pro objective. So, hopefully the the action of this gradient objective is quite clear to people. #

1:14:17 OK, so this is actually much much simpler than PPO, right? Because all I'm doing is taking gradients, and we all know how to do that with our standard processes. And I think the nice thing is that it does work reasonably well. I think for for a whileum, People were very obsessed with like, is DPO better than PPO or vice versa. I think the answer now is maybe it doesn't matter very much, unless you're like at the frontier training the very best model. DPO is is reasonably good, and it's good enough for for llama, it's good enough for me. If you go and open up the llama tech report, you'll findum, You know that they basically had this like nice outer loop training where you know they had their model, they SFT it, they did DPO. and then they used that DPO model to then sort of generate candidates that they rejection sample, and then they repeated this process. So there's the outer loop on top of it. #

1:15:04 But basically the core RLHF primitive for llama was DPO. In the year since, there's been a lot of different variants of DPO. There was SCPO, which basically changes sort of the waiting, so instead of using pi refum, You have this like Y of L normalizer. There was length normalized DPO which normalizes by the length, in order to avoid certain like length hacking issues that seem to arise. But really, none of these variants maybe seem to matter very much. One of the things that I'll mention that I thought was very interesting Was, these results are very very contingent on the specifics of the experiment setup. So, even at AI two, you know, there was a paper that was like, well, it turns out that if you go from DPO to PPO, you'll get a better result. But then, you know, they also had a paper for for two, two, where they said actually, if you do DPO right, you can do better than PPO. And so, depending on how you execute thisum, One can be better than the other. #

1:16:09 I'm I'm sure you have experiences reading deep learning papers and so on where like the results are very fragile, right? #

08 RLHF 要警惕：奖励过优化与模式坍塌

1:16:15 So maybe the takeaway from all of this is that these DPO variants are actually, you know, close enough to the right thing to give you pretty good performance in many cases, right? This core idea of take gradient step in the right direction And take negative gradient step from the bad stuff works reasonably well as long as you set the step sizes right. OK, I will end with the last few slides on things to watch out for in RLHF. So, one of the things to watch out for is over optimization, and this is maybe one of the biggest things, to give you also historical context, right? When instruct GPT came out, and like, you know, people were very excited about this. I think there was a very real question of like, can we RLHF our way It's like you know super intelligent systems or whatever. Maybe we can just collect enough thumbs up thumbs down. Turns out that is actually quite challenging, because you you find that if you try to like really push the RLHF process forward, you'll start overfitting to your learned reward model. Right? #

1:17:12 So the KALE regularizer that I talked about is really critical. In a lot of cases, in order to prevent your sort of optimization process from overfitting your reward model, at least if you're if you're optimization process is very good. The other thing that I'll talk about, is a model collapse. So, people have seen lots of times where RL models have much less diversity. They're like concentrated on a few different possible outputs. And this connects back to what I said before, right? RLHF models are really no longer modeling a distribution, which comes with its own inherent diversity. It's a policy that can collapse as long as it gets a good reward, right? So, this is another thing that thatuh, sort of RLHF models have really struggled with in recent years. #

1:17:59 So, actually, I'm gonna I'm gonna skip this, cause it's where time. And And I think this this entropy and mode collapse thing actually ends up being quite important, even today. And the GPT four error actually was one of the few plots that OpenAI put out, where they said actually we have a few open problems left. And one of the open problems they had was actually our models are uncalibrated after we do RLHF, which you can see in this plot over here. I don't think anyone has really solved that yet. The anthropic folks also had a a argument saying like, well, it's naturally uncalibrated, you could recalibrate sometimes, but not always. This becomes very important. #

09 总结

1:18:36 In the next lecture, when we talk about things like R L V R, where the the entropy and exploration is actually quite critical for the model to explore all the possible solutions, and to make progress on some of these very hard problems. #

1:18:50 Great. So to, you know, put it all together, right? RLHF data collection is al- also very hard. I think part of the thing about post training is it's very complicated messy process, because a lot of it is Getting good data, and getting good data is always very difficult. RLHF algorithms are are quite complex, PPO is especially difficult. I'll I'll talk a little bit about it next lecture. Thankfully, we do also have a simpler variant there called GRPO that works pretty well. That's what you'll do in your assignments. But this is one place where there's been a lot of recent developments. #

1:19:22 Finally, I think in terms of RLHF, the You know, one of the big o- big problems is overoptimization. And the transition to the next lecture is going to be, you know, is there rewards where we won't overoptimize, right? Where we can just kind of dump in compute, and the model performance just keeps monotonically getting better. And that's one of the reasons whyuh, What people call RLVR is just so, you know, been so impactful. Thanks. See you all Thursday. #