# CS336 Lecture 9: Scaling Laws – Basics（Scaling Laws 基础）

> [Stanford CS336 · Language Modeling from Scratch · Spring 2026](https://cs336.stanford.edu/) · [视频](https://www.youtube.com/watch?v=Q15rhEWZPQ4) · [官方讲义 (PDF)](https://cs336.stanford.edu/lectures/lecture_09.pdf)
> 讲师：Tatsunori Hashimoto（Tatsu）· 时长 78 分钟 · 本文档 = 官方讲义（PDF）逐页截图 + 全程文稿（豆包大模型录音文件识别转录，已去除语气词）

## 目录

1. 引入：为什么认真对待 Scaling — [▶ 0:05](https://www.youtube.com/watch?v=Q15rhEWZPQ4&t=5)
2. Part 1：Scaling Laws 的历史（1993–2017） — [▶ 4:21](https://www.youtube.com/watch?v=Q15rhEWZPQ4&t=261)
3. 数据 Scaling：现象与理论 — [▶ 11:19](https://www.youtube.com/watch?v=Q15rhEWZPQ4&t=679)
4. 进阶数据 Scaling：混合 / 分布偏移 / 重复 — [▶ 21:20](https://www.youtube.com/watch?v=Q15rhEWZPQ4&t=1280)
5. 模型工程：架构 / 优化器 / 深宽 / Batch / muP — [▶ 30:05](https://www.youtube.com/watch?v=Q15rhEWZPQ4&t=1805)
6. 数据 vs 模型：联合 Scaling 与 Chinchilla — [▶ 56:51](https://www.youtube.com/watch?v=Q15rhEWZPQ4&t=3411)

## 1. 引入：为什么认真对待 Scaling
*[0:05 – 4:21] · [▶ 从这里看视频](https://www.youtube.com/watch?v=Q15rhEWZPQ4&t=5)*

![Slide 0](https://cs336.sg.aidenovak.com/media/slides/l9/000.webp)

`0:05` OK, we will get started. We're now leaving, well, temporarily leaving the land of systems, to talk about more deep learning stuff, which for, you know, we're gonna have two lectures on scaling law and scaling law adjacent phenomena. So today is gonna be the basics. We're just gonna talk about pretty basic scaling law stuff, some of the classic works. How does you know the basic idea of scaling laws connect to maybe things you know about from Machine Learning 101? And then in two lectures, we're gonna do the more advanced version of the scaling laws lecture, where we're going to basically go into a bunch of modern open model tech reports, things like mu p and other parameter initializations, and this year I'm gonna throw in Some more stuff for optimizers. So there will be. an advanced version of this that that will try to really get to the cutting edge. Today we're gonna start with all the really basic stuff.

![Slide 1](https://cs336.sg.aidenovak.com/media/slides/l9/001.webp)

`0:58` And due to sort of some scheduling constraints, next lecture will be inference. You're kind of gonna go back and forth. You're gonna do Scaling laws, then inference, and then scaling laws, again. So scaling laws is really, you know, kind of the following scenario in some ways, or or you run into it very quickly once you run into the scenario, right? So, you know, you have your very wealthy friend who has given you 10,000 B200 for a month. And he or she has asked you to build a very good open source language model, right? Like they they would like to get something out of those B200s. And so, you know, you've already put together your infra team, that's your assignment to, you know, the Buddies, and you have let's say a good pre training data set, that's gonna be the the assignment after this, but let's say you already have that. And then you're gonna go and train.

`1:43` a big model, right? But there's lots of choices. In training a big model, all the assignments, sorry, the the earlier lecture two stuff that we had earlier, where you have questions like, you know, what architecture should I use? What should my hyperparameters be? And it's very scary to run these things on a run that is basically you know potentially millions of dollars in cost. So how do you really take this idea of scaling up very seriously? How do you make sure that your big run Is actually successful.

![Slide 2](https://cs336.sg.aidenovak.com/media/slides/l9/002.webp)

`2:11` So, all of these many choices, some of which you're gonna just copy out of the literature, right? The stuff that I talked about earlier in the architecture lecture is really about kind of well, you know, adopted best practices, and you will probably just pick some of these out of a hat. That's fine. But really, let's say you're at the frontier, you would like to, you know, build something that's actually better than the best available models today. You have to start really optimizing these, right? You can't just copy the choices of others. And get something that's you know better than the state of the art. So, how do we do that? Well, that's gonna require us to really engage with this idea of scaling.

![Slide 3](https://cs336.sg.aidenovak.com/media/slides/l9/003.webp)

`2:48` So scaling laws have been one very pa-, powerful tool. You might also call it a paradigm. I think scaling laws, you know, if you talk to some people who are are really in these like big labs doing scaling work, it's almost kind of like a way of life. You know they're like we really believe in the scaling laws, it's almost a a belief, and you'll see why. Scaling laws can sometimes be quite tricky objects. But really scaling laws are the simple predictive rules of how to go from small scale model performance and behavior. And to try to extrapolate them up to large scale behavior, right? And so, the basic naive way to to approach model training might be to say I have a big budget, I'm gonna do you know multiple training runs on these very big runs, and I'm gonna tune hyperparameters on those big runs. Right, but that's very wasteful.

`3:33` So instead, what I would like to do is I would like to do all of my optimization at the small scale, and have some simple rule that allows me to extrapolate small scale behaviors. To large scale behaviors, right? And if the small scale and large scale are sort of connected by some very simple, sort of robust connections like regularities of some kind, then you would have confidence in this kind of new scaling approach to optimizing your system. And that's really what today is all gonna be about, this kind of engineering view of of a scaling law. OK, and to sort of start with, I'm gonna you know talk a little bit about background, right? Because I think scaling laws are very interesting, in that not only are they kind of the You know newest paradigm, like that's kind of how a lot of these models are designed and thought of. But also they connect very closely to very classical ideas, right?

## 2. Part 1：Scaling Laws 的历史（1993–2017）
*[4:21 – 11:14] · [▶ 从这里看视频](https://www.youtube.com/watch?v=Q15rhEWZPQ4&t=261)*

![Slide 4](https://cs336.sg.aidenovak.com/media/slides/l9/004.webp)

`4:21` So if you're a person that'suh, Machine learning theorist, right? I think actually there are some things that you will really kind of feel at home with in some ways, so they're kind of like empirical sample complexities. And then I'll talk through the history of of scaling laws of one kind. And show you that this is not something that's really new, it's something with actually quite a bit of history and and research done to it. So, you know, I really really had to put some math into my slides, I really feel the the need to do so. But I think one of the things that I think is very interesting is that, you know, for the longest time, machine learning has thought about this question of how will my model perform, right? Especially from a theory perspective, right? Like I have this model class. How good is my model going to be?

![Slide 5](https://cs336.sg.aidenovak.com/media/slides/l9/005.webp)

`5:04` And generalization bound is kind of the theorist answer to this question, right? So you have, you know, your error over some finite hypothesis class, and you say, oh, my error is going to be at most this much worse. Then the models performance on my training set, right? This is you know classic generalization bounds. You can sort of write this down for more complex classes like smooth densities or whatever. But importantly, you know not only does this tell you you know some sort of performance, notice that these bounds are often dependent on the sample size, right? And so this is kind of telling you, OK, well, this is a theoretical upper bound on my loss values as a function of how my training set grows, right? And in some ways that's very closely related to this idea of what happens as I scale up sort of the amount of data that I have, right? Those are Closely related questions, or you might, you know, believe me though that's true.

![Slide 6](https://cs336.sg.aidenovak.com/media/slides/l9/006.webp)

`5:51` And if you try to figure out like what is the very first scaling law paper that exists, it's turns out you can go back quite far. I mean you can go back to Bell Labs and Corinna Cortes and you know Vol, Vladimir Vapnik, and these folks that were doing like very serious, you know, theory work on machine learning. You know, ask this question. Well, training classifiers on a really large data set is often very expensive. We would like to avoid doing that, and to do so, maybe we can just fit classifiers on a smaller sample. Fit a curve to how their error rates decay, and use that as a way to estimate their performance. Right, that's almost literally a data scaling law, back in 1993. Right. So this idea of at the very least, you know, thinking about how your model's performance changes as a function of data set size, is just a very classical one.

![Slide 7](https://cs336.sg.aidenovak.com/media/slides/l9/007.webp)

`6:38` And there's kind of a lot of work that supports this kind of idea. I just wanna Connect this to some of these older ideas, so you don't really think of scaling laws as this like very new purely neural thing. I think it's a idea that has a lot of roots in empirical machine learning, and also connections to theoretical machine learning. And if you start to sort of look at the history of this, you know there's a lot of people that have said, you know, isn't it the case that maybe instead of spending money on algorithm development. maybe we should just go collect more data, because if we plot How the performance of our systems, you know, behave as a function of the amount of data that we have. It turns out that, you know, they are all increasing or improving in predictable ways. Like Banko and Brill, I think is one of the canonical sort of NLP references, I think. In sort of how the amount of data, and and thinking about scaling, is a really valid way of improving system performance, right? So th- there's lots of you know ways of thinking about this.

![Slide 8](https://cs336.sg.aidenovak.com/media/slides/l9/008.webp)

`7:34` And I think there's also work by other NLP folks like Colina at all in 2012. This is one of earlier works on like different kinds of functional forms for scaling laws where once again they're increasing the amount of data that they're using to train. I think in this case machine translation systems, so I have blue scores. And they're you know asking like what is the right functional form for how a model's performance behaves as a function of increased data amount, and they you know maybe surprisingly or not surprisingly End up with the same power laws. which is pow 3 and 4, as really we do today, right? For the rest of the lecture, you're gonna keep seeing essentially these polynomial functions, pow 3 and pow 4. As I talk about scaling laws, and you know we kind of knew about a lot of this back in the 2010s that like these kinds of functional forms were very predictive of model behavior as we scale them up.

![Slide 9](https://cs336.sg.aidenovak.com/media/slides/l9/009.webp)

`8:28` And then finally, you know, I really like mentioning this paper because for whatever reason, they don't get orhessian at all, don't get cited quite as much as they should. I think the origins of neural scaling, I would kind of point back tohessian in 2017. Really kind of a very forward looking work in many ways, that built these scaling laws as a function of data set size for wide variety of systems like different kinds of speech recognition systems, machine translation systems, language models. And they showed that, you know, these things follow these like nice polynomial trends across a really large number of domains. And you know, it was really ahead of its time, because 2017 was long before some of the earlier sort of OpenAI scaling laws and so on in in the 2020s. And you know they talked about stuff that we talk about today, like emergence, like where suddenly, you know, the models might show certain kinds of capabilities, because accuracy is a much more discontinuous measure than losses. And really, you know, scaling by compute, that if we have systems that scale predictably to large training data, then really compute is gonna be really important. And then finally, you know, since we're scaling by compute, well Speed and systems optimization is going to turn into accuracy. So in some ways, like a lot of the things that we see today, like the phenomena of like system scaling and so on, we could have really known, or you know, I could have known in 2017 had I sort of read and thought about a lot of these scaling laws papers.

`9:52` then, I thought about them more seriously. So, kind of like to to mention that paper in in every one of these scaling law lectures I give, just to say, you know, a lot of these phenomena were known far before the modern era. we're like the modern language modeling era, and that you know, it's it's possible to have known the the regime that we're in even before seeing these large language models. I think that's very cool to to know. OK, any questions about this sort of like semi-historical part?

![Slide 10](https://cs336.sg.aidenovak.com/media/slides/l9/010.webp)

`10:24` Yeah. Kind of like where we can just buy all this. You're asking me about these scaling lots and where they come from. These are sort of pure curve fitting exercises. I thinkuh, I will talk a little bit about where scaling loss can come from, and why polynomials are arguably a very natural class of scaling loss. But I think there's no golden rule that these are the only ones. I will say though thatum, You know, in some place, theory provides a very rich place where the scaling law functional forms can come from. Because theory is often thinking a lot about, you know, how do error rates decay as a function of various objects, and they give you candidate functional forms of various kinds. Physicists are also very good at this, because they like thinking about limits like How do the limits behave, and that gives you different scaling law functional forms.

`11:12` Cool. OK.

## 3. 数据 Scaling：现象与理论
*[11:19 – 21:20] · [▶ 从这里看视频](https://www.youtube.com/watch?v=Q15rhEWZPQ4&t=679)*

`11:19` Any other questions?

![Slide 11](https://cs336.sg.aidenovak.com/media/slides/l9/011.webp)

![Slide 12](https://cs336.sg.aidenovak.com/media/slides/l9/012.webp)

`11:21` Okay. So now I'm gonna start talking about more modern neural language model scaling. And I'm gonna start with data scaling laws, cause I think to me they're the most simple and natural objects. Talking about data scaling laws also allows me to sort of walk you through very step by step on kind of how we might sort of guess what the functional form of a scaling law might be, and why the polynomial form is a natural one. And. then I'm gonna start talking about you know hyper parameters and architectures and all sorts of other more exotic objects, that are going to be harder to justify from intuition, but still turn out to have very niceuh, Scaling all behaviors. I think one of the things that's remarkable about kind of this scaling law style analysis has been that these power law relationships seem to hold for a huge variety of different factors. So, the basic ones that we see in pre training, and those are gonna be the ones that I'm gonna talk about today. You know the the usual ones that you see are, you know compute on the x axis, this is log compute, sorry, on the x axis, log of test loss on the y axis, or log of the data set on the x axis, log of loss on y axis.

`12:29` Log of parameters on x axis, log of loss once again on y axis, right? So, we can get a variety of different x axis that you might think of as resources in the log axis, all of them sort of linearize log test loss here. You can see also much more exotic scaling laws. You can see scaling laws that you know are are basically the the equivalent to these in downstream benchmarks. In those cases, you expect to see a sigmoid. functional form as a function of compute or compute like quantities. There's even been, you know, many works recently, more on the forecasting side where people are doing things like putting dates as the x axis, and strangely enough, at least for these plots like the upper envelope of these curves are often linear in various forms. of capabilities. There's no reason that.

![Slide 13](https://cs336.sg.aidenovak.com/media/slides/l9/013.webp)

`13:17` you know on this this bottom side, the more exotic scaling laws need to be true. But it is, it turns out to be the case that I guess Language model performance is often much more regular as a function of scale and resources than it initially might appear. So to begin with, I'm gonna start with data scaling laws, and data scaling laws are a very simple kind of univariate relationship that we're gonna think about, right? So, in this world, we're gonna fix our model training procedure. And in general, unless I say otherwise, the models are gonna generally generally be bigger than the dataset size. And I'm gonna increase my dataset size, you know, in a regular way. I'm gonna see how the error reduces, right? And what do we expect out of this relationship in general, it's going to be monotone-ish, if we tune our hyperparameters well, right? Because we have more data.

`14:05` More data should help us, you know, do better at our task. The error should decrease. And if you're doing something like classification, or even for for next token prediction, you should really go from like random guessing to some sort of like entropy like object where you can't do any better than sort of the the irreducible error of your task. right? That's the noise floor. So we're gonna Have some sigmoidal like object over here. So this is generally what we expect out of our system. So, we're gonna start with an empirical observation, and you know you can and and should you know make this on your own. you'll you'll make something that looks roughly like this in parts of assignment three.

![Slide 14](https://cs336.sg.aidenovak.com/media/slides/l9/014.webp)

`14:43` which is you know if we just have a very big model, much bigger than our data set. And we increase the data set size, and we put that on the log axis, and on the y axis, you know we have log test loss once again. You will find that the two sort of the points that you plot will form a very clear linear trend. Right, I've taken this from Kaplan in 2020. This is, I'm gonna refer repeatedly to this paper. This one's OpenAI's neural scaling law's paper. They have a lot of lovely ablations and a lot of lovely data. Now, this relationship where I've plotted a line on a log log plot, this is a scale free or power law relationship. And what does that mean?

![Slide 15](https://cs336.sg.aidenovak.com/media/slides/l9/015.webp)

`15:24` Right, if I have a line on a log log plot, many of you, to many of you this is probably obvious, but just to really emphasize this right? If something's linear in a log log plot, what does that mean? Well, it means that the error that I have is decaying polynomial, right? And and also it usually means that I'm very far away from my asymptote, right? Because once I approach my asymptote, I'm gonna you know taper off rather than have a line. So, let's go through an example just to show you how scaling law arises very naturally on a very simple estimation problem.

`15:55` So forget language modeling for the moment. This was was a language modeling class, but now it is a statistics class. For the next one slide, or actually two slides, no, one, maybe three slides. So in this case, right, what I want to do is I want to estimate the mean, from a bunch of data. If I have an input, it's drawn from a gaussian, and I'm going to estimate the mean of this object, right? I just write down the empirical mean. And then you can write down what the the error is. So what's the expected error? Well, each of these guys is gaussian, and I know that the mu hat is gaussian, so I can write down the error that's going to be sigma squared over n, right?

![Slide 16](https://cs336.sg.aidenovak.com/media/slides/l9/016.webp)

`16:26` OK, so well, if you look at this, this is kind of a scaling law, right? I've written down the error, right? And the error, if I log that, is a function of log n, right? Log n being the log x axis. Right? So in general, you know, if you can come up with anything that is essentially one over and to the alpha plus some constant, you know, when you plot that on a log log plot. subtracting out the constant term, you're gonna get a scale it more. Right, so so anything of this form where you have polynomial decay and errors is going to end up giving you a a scaling law. And of course, in parametric estimation like mean estimation or regression, you of course expect a polynomial rate, right?

![Slide 17](https://cs336.sg.aidenovak.com/media/slides/l9/017.webp)

`17:05` You don't really expect anything else. And you know if you've taken your your stats estimation courses, you know that essentially all the classical models you are expected to have a one over n scaling, right? If you fit a regression or so on, you might get d over n or or whatever else. And so, if this is true, and this is the origins of scaling laws, if we plot our our lines, we should expect to see roughly a slope of minus one, right? Cause that's the exponent on the x, right? y equals minus x plus c. If we fit our neural scaling laws, we will find that our exponents are quite different. These exponents, I'm ta- I've taken these from these two on the left from Hestnes, I've taken the one on the right from Caplan. They're about negative point one, negative point three, negative point one, roughly speaking, right?

`17:52` And so, what does that mean? That means this is much slower than estimating a mean or estimating like a linear regression. Right, so this much much slower convergence rate, still polynomial, but much slower. Now, this is kind of interesting, it's saying that we're getting, you know, these kind of nice polynomial rates, but the exponents are not so great, right? They're negative 0.1. Where might you get an estimation rate like this? Well, you would get this if your model was kind of more nonparametric, right? I promise this will be the last statistic slide of this lecture.

![Slide 18](https://cs336.sg.aidenovak.com/media/slides/l9/018.webp)

`18:23` Let's say that I wanna estimate now, instead of a mean, an arbitrary smooth function, right? So one thing I could do is if I'm doing a regression problem, right? I have my inputs x one through x n. Let's say they're in the unit box. I have a little bit of gaussian noise, and I have an unknown function f that I would like to estimate. A trivial estimator for this is I can take my you know space, I can cut it up into little two d boxes of length n to the negative one to the fourth. And then what's our estimation error? I'll have square root and boxes. Each box is going to get square root and samples.

`18:55` And the error is going to be roughly one over square root of n or something, right? And in general, if I have a d dimensional function, I would like to estimate, you know, in this kind of very nonparametric arbitrary way. My error is gonna be roughly equal to end to the negative one over d, right? So my scaling is gonna be y equals negative one over d. Times X. And so now these kinds of more flexible functions, functions that are more flexible than the linear class now have rates in their scaling laws that are not just one, they're one over D. And so one you know mental sort of model that you could have is that the neural networks I was showing you in these slides are behaving kind of like nonparametric regressors in ten dimensions, right? Like a nearest neighbor or other kinds of estimators in ten dimensions. That's roughly the rate at which they learn from data.

![Slide 19](https://cs336.sg.aidenovak.com/media/slides/l9/019.webp)

`19:41` That's kind of a cool thing to be able to know. Just by looking at some of these scaling laws. Some people have argued this more strongly, Barri at all and some other scaling law theory folks have argued that like You know the scaling law exponents are actually like literally telling us that the neural network is behaving like a nonparametric smoother, and so on and so forth. It's. an interesting set of arguments, and you can sort of buy that intuition to whatever extent you'd like. I don't quite know how much I truly truly buy this argument. There's some, you know, of the evidence might be a little sketchy, it relies on estimators of intrinsic dimension. But it's sort of an interesting thing to think about, that somehow these exponents are telling us You know how fast are these neural networks really learn? OK, so actually I'll pause for the moment here.

`20:27` That's really the the end of the really basic data scaling law of like one, you know distribution, and you sample from it, and you learn from it. Is that all kind of clear, like the examples and so on?

`20:38` Oh, yes, question. Say the model is always larger than data set size. Do you like one to one like number of parameters, number of tokens? Yeah, that's a that's a good question. So the question was something like, what what do I mean when I say the model is larger than the data set size? What I mean is, when the model gets small relative to the data, like you have way more data than you have model size, you enter this irreducible error regime, right? Like you basically fitted the data as well as you can for your model class, and you can't do any better no matter how much data you add, right? so we wanna, you know, when we're thinking about scaling laws, we usually wanna think about this power law regime. which is the sort of simpler case, either that or you explicitly fit the asymptote, and then you correct for it.

## 4. 进阶数据 Scaling：混合 / 分布偏移 / 重复
*[21:20 – 30:05] · [▶ 从这里看视频](https://www.youtube.com/watch?v=Q15rhEWZPQ4&t=1280)*

![Slide 20](https://cs336.sg.aidenovak.com/media/slides/l9/020.webp)

`21:20` Those are kind of the two things. So when I say the model is larger than the data, it can be basically any amount that it's larger than the da- the data, as long as it's in this power law regime, usually like You know, ten times bigger than the data will have you. Cool, good, OK, now we are. back to other data scaling laws, right? So, data scaling laws by themselves are not Quite as useful as you like them to be. They just tell you how fast does my model learn, right? And that's kind of useful for forecasting, it's not useful for very much else. If you want to do engineering, you are probably interested in other questions like, you know, what is my optimal data mixture? Like, what is the best way for me to pick my training data for my model?

![Slide 21](https://cs336.sg.aidenovak.com/media/slides/l9/021.webp)

`22:03` Very useful engineering question. Should I repeat my data, or should I, you know, just, you know, not repeat my data, save the compute for something else? Or, you know, maybe I should repeat some high quality stuff, and you know, not repeat the low quality stuff, like I could, I could do various other things with our tradition. Right, there's a lot of different engineering decisions I can make with data. So, how can scaling laws help us make some of those decisions? And I think one of the things that's interesting is if we, you know, think a little bit about how classical models behave. Right, data scaling laws, if we think about their origins as essentially empirical versions of of generalization bounds, they kind of tell us well, you know, data set composition for many models really affect the offset of these scaling laws. But their slopes are actually determined by kind of the model class. They're not really determined by the distribution themselves.

`22:54` So the slopes might remain the same, and only the intercepts might change. And the intercept might actually be really interestingly shaped. So this is a caseum, Of a toy model with a linear regression with two different things you can sample from. If you get only one kind of data, you know, it's, the errors are high no matter what, you actually want a mixture of both, right? So, if you write down kind of how this intercept behaves, you actually start to get some very interesting insights. About how having more data diversity is very helpful. But what does this kind of practically mean? Well, practically speaking, what this means is you can actually fit data scaling laws on mixtures, and that can help you do some data sorry, pre-training data. optimization.

`23:37` So what does this mean? Let's say you have two data sources for simplicity. You have news, and you have Wikipedia, and you want to know how much news to use, and how much Wikipedia to use. How can you determine that? Well, you can train really small models on really small amount of data, and then you can basically train or you can fit a function of you know how different mixture levels affect your performance. You can do this while you scale up your models little by little, and you can see how the trends change, and you can try to extrapolate that out the exact same way as you would extrapolate out of data scaling law. And then you can ask which one will be optimal if I keep scaling it out to my full training room, right? So, this is this is a very simple idea of essentially fitting a functional form at a small amount of compute, finding the minimum, and then scaling that guy out. So, this is a data mixing loss paper.

![Slide 22](https://cs336.sg.aidenovak.com/media/slides/l9/022.webp)

`24:27` Unfortunately, I think if you talk to anyone who's done a lot of these data mixture work will tell you, reality is a lot more noisy than kind of this ideal world would suggest. And really what has as far as I know happened in many cases is you end up training a bunch of small models. You pick the best data mix from the small model, and you just scale that guy up. No scaling law required, and data decide if you're interested in this kind of thing is a is a nice large scaleum, sort of empirical study of these kinds of data mixture things. And what they found was actually if you don't fit a scaling law, and instead you just pick the best data mix of the small scale, that works well. For what it's worth, that's consistent with the argument that the intercepts differ, but the s- slopes don't change, because if the slopes don't change, the best mixture at small scale is also the best mixture at large scale.

![Slide 23](https://cs336.sg.aidenovak.com/media/slides/l9/023.webp)

`25:21` Another really interesting set of questions that I'll also mention in the context of data is repetition. I think it's increasingly the case that compute is growing, the amount of data that we have is not growing. And so there's a lot of people interested in questions of what happens when you repeat data more and more times. And there was a nice study a few years back, called Scaling Data-constrained Language Models. where they roughly show something like up to 4 epochs with standard training recipes, you you just don't get hurt at all. but if you go past that point, then actually your your realized scaling law, this sort of dark curve, is much worse than sort of the projected scaling law if you had fresh data, that's the dash line over here. And you can also once again write down a modified functional form that's generally predictive of the behavior of this scaling law under repetition. And we can take this idea even more to the extreme. This was kind of recent work with Percy and one of our code advise students where we were asking like okay like what happens if you take this idea to the extreme, and you just you know consider infinite amounts of compute.

`26:27` So you're allowed to epoch as many times as you want like an infinite amount of times. What is the best thing that you can get out of this system? Well, turns out you can't just keep repeating passes over the data. You can't keep making your models bigger, those have diminishing returns, and so you end up reaching for other things like, you know, un-sampling your model so try to squeeze more and more out of your data. But really, I I think the one thing that I'll point out that's really interesting is, you know, your standard data scaling law is here in the red. It's a very nice predictable, you know, improvement in performance as you increase the amount of data. You know, we do all sorts of interventions like regularizing and adding en ensembles. You get improvements in performance, but the slopes look actually surprisingly similar. This is a a lesson that you'll learn, once you start fitting your own scaling laws that very very often, your slopes don't change, often the interventions that you do just change the intercept.

![Slide 24](https://cs336.sg.aidenovak.com/media/slides/l9/024.webp)

`27:16` of the scaling loss.

`27:19` One last thing that I'll mention, cause I like this paper in the context of scaling laws, and scaling in general, is that a lot of the phenomena we care about in data, is actually very scale dependent. And one example of this is data filtering, right? So, if you or I, you know, tomorrow decide to filter some data for the models we're training, we would probably filter very aggressively, right? We would only keep the highest quality stuff, because both you and I don't have very much compute, right? So, we can't train on all the internet anyway, we might as well filter down to sort of the most high quality stuff. On the other hand, if you have a lot of compute, right, then you want to train on more and more stuff, because you're gonna not want to repeat on this very high quality data. So as you get more and more compute, your filter is kind of become looser and looser. And potentially you start training on stuff that's lower and lower quality, right? So a lot of the things that we often think about statically, like things like data filtering or the quality of data, are actually much more dynamic.

![Slide 25](https://cs336.sg.aidenovak.com/media/slides/l9/025.webp)

`28:15` As you as you increase scale, you have to think about where am I gonna get the rest of the data, right? Like the filters can't stay fixed. As a function of the data, and certainly like the optimal filters turn out to not be fixed at a function of scale.

`28:29` So, that's it for data scaling loss. This is the part that I think is most straightforward to understand, even as kind of like a you know stats ML minded person, like this stuff is all very simple and straightforward to me. You increase data, performance gets better in a very predictable way, right? I think this is uncontroversial. The fact that it does so polynomial for such a big model is still kind of cool, but nothing really crazy. So hopefully this is you know reasonable to all of you that there's so much predictability. When you increase the amount of data.

![Slide 26](https://cs336.sg.aidenovak.com/media/slides/l9/026.webp)

`29:01` Oh, yes. So, on this slide for the pre-training under infinite compute, so that like graph on the right is like linear. for loss and data set size. Where's like the previous example I gave was like log log scale, and I was wondering why it's linear both. Yeah, so so this is this is a, I mean, bad access. Herman and I just talked about bad accesses like this. You know, thi- this is actually doubling, right? It looks linear, but it's actually log scale. You fix it now, okay?

`29:34` And the y axis here, it doesn't look log, but the range of this is so small that linear and logs are basically the same. If you look at the fitted functional forms here. Right, these are still polynomials, so we're fitting the same functional forms, and those are the ones that are plotted here. This is another point, I'll get to it at the end of the lecture. If you only have a tiny slice of a compute range It's very very difficult to tell if something is scaling polynomially or if something scaling exponentially, right? Because Taylor approximations are a thing, everything looks linear if you kind of zoom in enough, right?

## 5. 模型工程：架构 / 优化器 / 深宽 / Batch / muP
*[30:05 – 56:51] · [▶ 从这里看视频](https://www.youtube.com/watch?v=Q15rhEWZPQ4&t=1805)*

`30:05` And so whether this is polynomial or some other functional form, you wanna be always a little bit careful and skeptical.

![Slide 27](https://cs336.sg.aidenovak.com/media/slides/l9/027.webp)

`30:16` All right, so that was scaling loss for data, in many ways straightforward. Now I wanna sort of talk about more exotic forms of scaling, and sort of scaling loss for for model engineering of various kinds. So what we're gonna do now is we're gonna try to design large language models. You know, maybe you're a radical, and you believe that maybe LSTMs are the future. Like, why can't I use a LSTM instead of a transformer? You know, I wanna blow my B200 run on my lstms, or you know maybe you're a different kind of radical, and you want to train on sgd. like why can't i train on sgd? you might also wonder like if you have a limited amount of resources, you have different ways to to spend it, right? like maybe you should train models longer, maybe you should train bigger models, maybe i should go collect more data.

`31:01` There's also trade offs always at play, and scaling laws give us quantitative ways of making these trade offs, hopefully.

![Slide 28](https://cs336.sg.aidenovak.com/media/slides/l9/028.webp)

`31:10` I'm gonna start off with some simple hyperparameter questions. Things like how do I choose my architecture or optimizer. Things that I talked about in the second lecture. And there I gave you the perspective of maybe you can study what other people do. Now I'm gonna try to give you a first principle answers to those questions. With a little bit of a different set of trade offs, right? So you have to believe in scaling laws in this lecture. But as long as you do, you get to sort of try things on your own and find out. OK, the very first question we might start with is to say, are transformers really better than LSTMs?

![Slide 29](https://cs336.sg.aidenovak.com/media/slides/l9/029.webp)

`31:44` You know, LSTMs, they can work OK too, like they will fit distributions. You know, with the advent of things like mamba, we know that SSMs can work. So, are they better? The brute force way to answer this question is to train a big LSTM, GPT three, or even bigger class of models. The scaling law way would be to train a bunch of smaller models. and say I'm gonna train my transformer across a different variety of compute ranges. And I'm gonna train different LSTMs with different numbers of layers on, you know, once again a variety of different compute ranges. Now, what does this show? This shows, you know, LSTMs have, you know, definitely different intercept, maybe even different slope than the transformer, and because of this, probably don't want to pick out LSTM.

`32:30` This plot would justify scaling up the transformer, instead of the Elastica. And if you look at a lot of the architecture papers today, like you go look at the Mamba paper or the gated Delta net papers, they will always have a plot that looks like this, right? Where you have, you know, vanilla transformer, our really cool model, and our really cool models usually either on top or below Because that's what would prove that your model is doing better, right? And you certainly don't want something where your slopes are worse, because that means as you scale up larger and larger, your models will eventually do worse, right? So this is one way of really reasoning about architectures without training gigantic models.

![Slide 30](https://cs336.sg.aidenovak.com/media/slides/l9/030.webp)

`33:07` There haven't been, I think, quite as many recent and good architecture scaling law papers. I think the one that is relatively recent, and that I like quite a lot, is is by E K and others at Google. I'm back when they were publishing this kind of work where they did a whole bunch of scaling studies on on P 5 style models. Where they sort of trained larger and larger variants of of T5 models on many different architectures. And I think one thing that's really cool, and one of the reasons why I still bring this paper up, Is, you know, they kind of capture exactly the architecture changes that we implement today in our frontier models, right? So things like, you know, performer, which is a a a efficient attention, that doesn't scale very well, like we do not implement those. You know, the things that we implement are the gated linear unit, right? Red line better than green line throughout the scaling trends. Switch Transformer, I don't know what happened with their biggest run, but you know, generally speaking, it's got good scaling trends.

`34:06` And I forgot, oh, mixture of softmaxes. Actually, we don't do this anymore, but this is probably an effective intervention given the scaling trends. So you kind of see that through the scaling trends, even though a lot of these, you know, papers were operating on much much smaller compute scales than what we operate today. They're able to see the kinds of architecture trends that are driving frontier model development back there. And I think this is why a lot of people kind of use scaling laws as like a really almost paradigmatic way, saying like, oh, if it doesn't show up in the scaling law, it's not a good intervention. because in some waysum, A lot of the past work has captured what we see today.

`34:50` Going beyond architectures, we can extend this kind of analysis to all sorts of other things that we might want to understand. We might ask, okay, is SGD better or worse than Adam? If you do the scaling law analysis, this one's by Hennes, you'll see very clear trends. This is once again the really mysterious thing of, you know, the intercepts are different, but the slopes are very similar, right? If you fix the data and you fix the models, like slopes are often very very similar. I I I'm very surprised by this every time I see it, and yet it's very true. It's rare to get different slopes, even with an intervention as big as SGD to Adam, right? Like if you were the one training with SGD or Adam, you would you would think that's a huge change. Scaling trends remain roughly the same.

![Slide 31](https://cs336.sg.aidenovak.com/media/slides/l9/031.webp)

`35:31` In this case, this is a a language modeling with a recurrent highway net.

![Slide 32](https://cs336.sg.aidenovak.com/media/slides/l9/032.webp)

`35:37` Similarly, in lecture two, I told you about aspect ratios. Like how do you pick your depth to width trade off? And I told you, well, you know, you can just pick some random number that's, you know, some rou roughly reasonable, it's like four times the reasonable multiplier. But if you're doing this kind of scaling analysis, you could kind of know much more precisely what is happening. So, the first thing you could do is you could look at scaling trends as a function of layers, and you would immediately see The having extremely few layers is very very bad for your model. If you have one layer, you know you're you're not going anywhere. That's a terrible terrible scaling trend. Greater than one layer actually it's surprisingly more competitive, although you do see that at every compute level more layers at least in in this case Is is better, right? And if you you can do a much more fine grain study, which they do do in Kaplan, where you know you try to identify what you might call scale and variant quantities like the aspect ratio, right?

![Slide 33](https://cs336.sg.aidenovak.com/media/slides/l9/033.webp)

`36:32` The number of layers is not a scale and variant quantity, as you make your model bigger, you do want more layers. Right, but as you make your model bigger, maybe the aspect ratio, the optimal one should stay the same, right? This one is scale in variant. And so if we look at the aspect ratio, we see that at different model sizes You know, we get basically the same minimum around 100 d model for every layer. Or maybe a little bit less, and it does shift a little bit right, like there is a shift towardsum, sort of aspect ratio that is smaller for deeper models, but for the most part, the minimum stays roughly similar. You can do similar kinds of analysis, for things like attention head dimension and others, and if you look at the caplan paperum, They do quite a good job at at doing all of this. and this is one way that you can be confident about your scaling strategy, right? So if your scaling strategy is I'm gonna fix my aspect ratio, and I'm gonna scale up. You can make plots like this and convince yourself, you know, we're probably good, because our our optimizing and shifting too much as they go to larger and larger and larger models.

![Slide 34](https://cs336.sg.aidenovak.com/media/slides/l9/034.webp)

`37:35` One thing that's important, and this is a you know, this is a huge can of worms, and a and a sufficiently big can of worms that I will talk about it on more than one slide, it will come back. You know, not all parameters are created equal, and because not all parameters are created equal, your scaling laws may look good or bad depending on how you define what a parameter is. So, in the Kaplan paper, they made this observation that if they drew their scaling laws for for depth, with the embedding parameters, they got these like very funky looking scaling laws, and they said, oh, this is no good at all, no good at all, right? Because. it's no good at all, they decided that they were gonna exclude all the all the embedding parameters, and they were only going to count the non embedding parameters, because you know you could justify yourself saying these are the parameters that are doing computation or whatever. You know, we're gonna see later that this is gonna have non trivial implications on the validity of some of the results. But the point here is scaling laws aren't kind of magic. I think you know, Percy likes to make this point that like scaling laws and all these kinds of You know, predictability across scales is engineered, right? Like they don't happen automatically.

![Slide 35](https://cs336.sg.aidenovak.com/media/slides/l9/035.webp)

`38:40` We need to sort of pick the right kinds of, you know, x axis to look at. We need to make sure that the hyper parameters for these things are set right. And only under those conditions does it become possible to get predictable scaling across many orders of magnitude compute, right? So, this is kind of what a lot of scaling law researchers do, right? They think about what the right way is to get very predictable scaling. Now related to this, I think our our very interesting recent papers on things like scaling for mixture of experts. You know now that mixture of experts is kind of you would say the dominant way of training large models. You know, it is quite important to think about what is the value or what is a parameter in this new world where the number of parameters and the number of active parameters are kind of decoupled. And there's a nice clean analysis by some folks at Apple and MIT.

`39:29` That show the sort of scaling trends as a function of function both parameters and active parameters. And kind of a cool thing that you see Is, as you start to train bigger and bigger models, like if you you know increase the amount of total parameters on the X axis over here. And you want to minimize your loss. You're gonna end up with sort of more sparse and sparse model. See how the color becomes darker and darker and darker, right? Similarly here, if you like in, the number of active parameters, kind of decreases as a function of MoE sparsity. And you sort of get the result that I think is somewhat intuitive here, which is, you know, you can draw these sort of surfaces for every sort of compute amount. You can ask, OK, I would, should, would I rather spend that compute on active parameters? Or, you know, and how much sparsity do I want, right?

`40:20` I can vary my sparsity without really changing the amount of compute that I'm spending. Now, given that, you can ask, OK, what happens if I add more sort of empty parameters, same active parameters, but sort of more total parameters. As you move from sort of back on the plot, so in front the front of the plot, you see kind of the losses are improving, right? So in other words, this is saying that the parameters in an M O E that are not active are still helping you reduce your loss, which is kind of cool. And you can kind of write down what the functional forms of this are. I won't go into to that part in more detail. But you know the point here is to say all of these kinds of quantities that we care about in optimizing a M O E are things that have sort of predictable scaling and regularities.

`41:02` OK, and the two things I wanna now talk about, or actually really, yeah, two things that I now wanna talk about. When you're gonna train a new big model, Actually, most of the things that I talk about are fixed, right? You're you're probably all not radical enough that you're gonna switch to an LSTM. You're probably gonna pick like, I don't know, a deepseek four inspired M O E or something, right? And you're probably gonna pick architecture choices that are tried and tested. You're not gonna really throw everything out. But two things that you really do have to maybe sort of reinvent in some sense is the batch size and the learning rate, right? If you train your deep learning models You know that you have to, you know, be very careful about the combination of your batch size and learning rate, right? You change one, you have to change the other.

![Slide 36](https://cs336.sg.aidenovak.com/media/slides/l9/036.webp)

`41:45` Now batch size, we know from the previous lecture Is very tricky for another reason. We want the batch size to be as big as possible, because a big batch size gives us opportunity for paralyzation, right? Remember data parallel requires large batch size. So now having said all that, what we really want to know is how large can we make our batch size before we start to suffer, right? That's the relevant systems question. And you also want to know how does that change as a function of model size. So, given that preface, there is this you know very important idea, that is used in a variety of papers and talked about very frequently, called the critical batch size. The critical back size is roughly this idea that up until a certain point, when you start to get diminishing returns, there is returns, perfect returns to increased back size, right? So, I might call this the noise limited regime.

`42:38` So, in the noise limited regime, every additional element that you throw in your batch reduces the the gradient noise in your SGD step, right? And the since your variance limited, that reduction in variance is very very helpful, right? That that gives you big returns. So this is the regime in which you have perfect scaling. You're doing about as well as you can given the amount of extra examples that you're processing. As you sort of get to a certain point where you've reached kind of the noise scale of your gradients, you get to a point where you are no longer variance limited, you're now bias limited. What does that mean? Well, remember in gradient descent, you're looking at kind of the local structure of your objective. You you don't have global sort of view of where sort of the minimum is, right?

`43:20` So no matter how low noise I am, I'm only gonna point in this light blue direction over here, right? And there's always a disagreement between my local descent direction And kind of where the global optimum is. Because of that, at a certain point, I'm gonna get diminishing returns, because now my limiting factor is no longer variance, it is kind of this bias term, right? So now I'm bias limited past this point over here. And so critical batch is kind of a rule of thumb, you might call it, or a or a convenient trade off point where you sort of say this is the point at which we're starting to cross over from our perfect scaling regime to our ineffect ineffective scaling regime. and so it gives you a good rule of thumb of where to be setting your back size. if you want your back size to be as big as possible without suffering these like Huge efficiency losses. Ok, hopefully that idea is very clear. Now, critical back size is kind of a strange object.

`44:15` It is derived from A variety of of sort of complicated calculations about, you know, the local quadratic approximations to the objective. But I will maybe give you the mechanical steps of what is actually used to estimate this quantity. So, what you do is first, you pick a target loss. You know, target loss might be some number that you wanna hit. So, your goal is to hit this target loss as quickly as possible, right? That's my goal right now. What I'm going to do is I'm going to sweep over all the different batch sizes or a number of different batch sizes, and I'm going to write down the number of steps that I needed to reach my target loss. And the number of examples that were needed to get to that target loss. Of course, number one and two are related by a batch size factor, right?

`45:01` The number of steps times my batch size is equal to the number of examples Now through some calculations OpenAI folks麦肯锡和其他人，argue that this is the rough functional form that we should follow that The number of steps that you need and the。 number of examples that you need are inversely related with, you know, this s min which is the minimum number of possible steps, and the minimum number of example sort of normalizing these two quantities. and so there's kind of a trade off between these two, right? if you want to minimize the number of examples, you're going to have a lot of steps. and if you want to minimize the number of steps, you're going to need a lot of examples, right? that's a very natural trade off, right? and if you increase or decrease your batch size You're gonna have tradeoffs that go one side or the other. Now, they argue the goal now should be to balance these two sides. I want to make sure that my number of steps term and the number of examples terms are roughly balanced with each other. And if you solve for that, you're going to get this this particular solution, which is that the critical batch size is the minimum number of examples that I need divided by the minimum number of steps that I need.

![Slide 37](https://cs336.sg.aidenovak.com/media/slides/l9/037.webp)

`46:10` Right, and these two are kind of estimated by fitting this to sort of the the observed data that I have. This is gonna balance both sides of the equation. It's gonna give me a little bit more number of steps than optimal, it's gonna require a little bit more examples than optimal, but it balances the two sides, and you can kind of think of this As improve. optimizing the convergence rate without blowing up the number of steps that you you need. There's another added point that I won't Really labor, cause this gets even more into the weeds. You should really read the critical bat size paper, if you're interested into this, where one of their arguments is estimating this is quite complicated. You can do this by just looking at the ratio of the gradient covariance and the squared norm of your gradient.

![Slide 38](https://cs336.sg.aidenovak.com/media/slides/l9/038.webp)

`46:56` OK, so why is that even in the scaling lecture, right? Like this is this is a a thing that is is not really related to scaling so far. The reason why this is in the scaling lecture Is because the first of all, the critical batch size is a reasonable batch size to pick. That's a batch size that balances sort of convergence rate and the size of the batch, right? So, if I want to pick my batch size equal to my critical batch size, you might ask How does this thing scale as a function of my target loss? Right? Target loss can be thought of as compute, right? If I want to have a model that is better and better and better, right? What is the batch size that I I should use?

`47:33` Right, and the reason why this is in the scaling wall lecture is as you go down in scale in the loss, right? So your loss is improving, the amount of critical batch size increases, and it increases in this very predictable way, which is once again a power law, right? So this is the the relationship that they argue. which is roughly like inverse polynomial, in batch size. So, given this, this is actually quite nice, because what this is saying is, if you have a really large scale training run, where you go all the way to the right of this plot, your batch sizes can be quite large. Right, and that kind of makes sense, as you get to closer and closer and closer to the minimum, the value of variance reduction increases, right? Because you're you're trying to optimize finer and finer and finer grained objects, the noise matters a lot more, once you're minimizing really really tiny differences.

![Slide 39](https://cs336.sg.aidenovak.com/media/slides/l9/039.webp)

`48:23` So that's the backside story. The other side right, so the two quantities remember that you're going to always have to tune is backside and learning rate. The learning rate side of the story Is also a little bit complicated. I'll go into this much more detail in the advanced scaling lecture. But learning rates generally shift. I think this is a good mental picture for how to think about learning rates in the standard world that that you start with. Which is let's say that you have just like a MLP, right? Like just a standard neural network of some kind. I'm not gonna do depth scaling, I'm only gonna do width scaling.

`48:59` Now, if I'm doing width scaling, the bigger my model, right, the bigger my model, the smaller my learning rate should be. And the reason why my learning rate should be smaller is because I have you know bigger more parameters, I'm changing more things at once. Maybe I should move less, right? Kind of make sense. And so you, and and in fact, there's a kind of rule of thumb that maybe you wanna scale by one over the width, that's a very commonly known rule of thumb. That you know you want to just decrease the learning rate regularly, as a function of your width, and that will give you a rule of thumb to scale your learning rate. There is kind of another school of thought, or another set of tricks that people do. Which is that people also rescal the network in various ways. They will change the initialization sizes, and they will change kind of the step sizes of the optimizer.

`49:48` or for various parts of the network in order to to force the model to have the same learning rate minimum across scales. And this is the idea that is called new p, and others like it. Some people have reported great success with these kinds of approaches. Others have reported less success. The two ways basically of setting alerting grade for large scale runs really is kind of captured in this plot. In one way, you can try to estimate what these minima are and try to predict what your minimum will be. And the way in which the minimum changes is pretty predictable, so this is not a crazy idea. Or you can reparameterize your model to try to keep the learning rate minimum the same, and then just pick your best learning rate and go, right? Those are the two philosophies to picking your learning rate.

`50:34` both of them have been applied successfully at scale. there are large scale training runs that have used both approaches. I think anecdotallyum, It does seem like more people are maybe favoring the scaling law approach, but both are certainly viable approaches. I'll talk in detail about both of these next. Next next lecture. OK, so before I get into to kind of the last part of this lecture where I want to talk about not quite pitfalls, but various you know details of of scaling law things. One thing I wanna end this section with is the idea that upstream and downstream performance can be quite different. So, if you look at something like, you know, log likelihood. This is from the ETA, you know, architects' paper.

`51:22` You know you you find that perplexity and negative perplexity in this case. Parameters are very correlated, like the more parameters you have, the better your model very predictably, right? This is this is a very beautiful linear trend. Now, you're like, OK, this is great. Now, we're gonna ship our best model, NL 12. Turns out NL 12 is not your best model. It was actually NL 32 XL, which was a much worse model in perplexity, right? This is a very common thing that does happen. Upstream and downstream are often correlated.

![Slide 40](https://cs336.sg.aidenovak.com/media/slides/l9/040.webp)

`51:51` This is probably one of the worst correlations that I've seen from upstream to downstream. But I think it's very very important to remember that scaling laws generally speaking are objects that you want to apply on the perplexity side. They're very clean, they're very regular, they're very predictable. But transfer from perplexity downstream is a lot less certain than it might initially seem. Right, this is a place where you want to be very cautious. And you know, as an anecdote, I have I have former students that have gone and and done post training at various places, and they always complained that like all those pre training people, you know. they hand you this model like the perplexity is good, it's all your problem now, but it's often, you know, the the problems have started at the at the pre training side, right? So you don't you don't want to be one of those people. You want to really think about transfer as well, not just focus on perplexities.

![Slide 41](https://cs336.sg.aidenovak.com/media/slides/l9/041.webp)

`52:37` OK, so maybe this is surprising, maybe this is not, but the point that I wanna drill in here is that, you know, before you do your big training run, you should have a rough idea of what you're gonna see, right? Not just like, is it going to like train at all. in fact, if you do scaling laws, you should be able to actually predict, you know, fairly precisely, the numerical values almost of how good your run is gonna be, right? if i choose one optimizer versus the other, what will the gain be? like you should be able to predict that, right? So, the scaling law based procedure is to say, I'm gonna train a bunch of smaller models. I'm gonna establish some sort of a scaling law, you know, by fitting these lines onto log log plots. And then, if the fit is good enough, I'm gonna have some confidence that the gap will persist. And then, I'm just going to deploy that onto my large scale training right?

`53:22` This is a naive version of a scaling law. Pro procedure, but I think one that is like reasonably accurate for how you might think about these things in practice, right? What else are you gonna do? Just sort of deploy a run out of nowhere? Probably not.

`53:36` OK, I can pause here and and take some questions before I get into the the last part of of this lecture. Yes. For each of the data points on the scaling law, like how do you think about variance? Like how many runs you do, and is is aggregation usually like an average?

`53:54` Yeah, I think in most cases, if not all cases that I've seen, those are singletons. And the reason why it's a singleton is because when you're looking at perplexities Perplexities are very very clean, right? Like, you know, these objects, if you rerun them, you know, they're probably like the variances in the second decimal place or something. Because the the training data is very homogeneous, you've got a lot of it, your eval sets are very big. You really push the variance very very low on these things. But if you're doing something like you know, learning great or critical batch size scaling laws, you will see some truly horrendous stuff. I do think people do some variance reduction, but it's actually not that common, as common as it should be maybe.

`54:42` Yes. Most people use like just downstream metrics directly instead of using complexity. Yeah, people do do scaling laws directly on downstream metrics, but that has the same problem as this. So, I think the the way to think about scaling laws, is to really reliably establish at least one thing. So so the way to think about it is let's say I see something like this. Right, I know or you know I have strong belief that this transformer line is gonna continue. This is a very regular trend, right? And it might curve a little bit, but we do think there's probably some regularity here, right? But instead, imagine this was a very noisy jaggedy line of this form.

`55:19` You would not really know what is going on. Like you would say, I maybe believe that it will go on, but maybe it's a line, maybe it's a curve. Right. So I think the general, you know, if I'm trying to explain the philosophy of this stuff, is to say okay, let's start with the low variance measurement, and then establish scaling regularity there. And then rely on, you know, some sort of belief about transfer or establish transfer, you know, elsewhere to try to show that that would transfer over to downstream.

`55:48` Yeah. Some of these are different test loss, and some are different train loss. When do you do either? I see, yes, you fix the test loss when you know you're you're doing good scientific practice like Kaplan 2021. But realistically, All the the models, except for like a very few, like the the infinite compute one, and like the repetition stuff. Unless you're doing some of that, you're all, you're doing one pass SGD. So the generalization gap for all of these models is very very small, right? So train and val are very similar. I've seen some pre-training code bases that don't even have validation loss, because if you're doing one pass.

`56:27` Most likely your train and val are very very close. That's why a lot of these plots have exchanged those two, because they're on the one pass region.

`56:35` Good question though. OK, all right, we will continue to move on to, sorry, I moved on too far backwards here. OK. So now I'm gonna talk about one of the most well known uses of a scaling law.

## 6. 数据 vs 模型：联合 Scaling 与 Chinchilla
*[56:51 – 77:48] · [▶ 从这里看视频](https://www.youtube.com/watch?v=Q15rhEWZPQ4&t=3411)*

`56:51` Even though you might not personally be, you know, using this, you certainly use the rule of thumb at some point. And also, I think because this set of things will tell us something really interesting about the actual execution of a scaling law. So okay, the motivating question is Do I want more data or bigger models? Right, so you know the resource that you have, the resource that you have to spend is compute, right? Someone's gonna hand you metaphorically a certain amount of flops, right? And you're gonna take those flops, and you're gonna spend them somehow, you know, on, if you remember Percy's very first lecture, right? It's gonna be data times the parameters is roughly your flops, right? Linear in that. So, do I want more data or more models?

![Slide 42](https://cs336.sg.aidenovak.com/media/slides/l9/042.webp)

`57:33` Well, we know that if we take, you know, a teeny tiny model, and we dump tons of data into it, that will just be wasted. That's kind of the you know, the dark purple line at the very top here, I've taken a teeny tiny language model. And I've you know passed a huge amount of data into it, and this thing has been flat for a very long time. You know, this is a complete waste of compute, right? You would have much rather trained this big yellow model to the same amount of tokens, or even earlier on, you know, if you're having the same amount of compute. Now, you know, if you could sort of understand the interaction of how data amount and model size come together to sort of give you performance. If you understood that relationship, you could optimize this very precisely, right? And so, there are scaling laws that kind of are more advanced, and relate kind of joint behaviors of objects. Soum, Almost simultaneously, Caplan and Rosenfeld both sort of proposed these kind of functional forms, which are roughly equivalent, describing how the size of the model and the amount of data relate to the error.

![Slide 43](https://cs336.sg.aidenovak.com/media/slides/l9/043.webp)

`58:40` Rosenfeld is very simple. It's really just the sum of two inverse terms. These are two scaling laws that are added together. And Ro- Caplan is a little bit more complicated, but it's roughly the same idea, right? And if you look at think about the limits, this is very intuitive, right? If I have an infinite amount of data, right? Then what happens? Then this goes to zero, you're basically model size limited, you become a pure model size scaling law. If I send my model size to infinity, then I'm you know I'm gonna be bounded by data, I'm gonna have a pure data scaling law, right?

`59:10` So it makes sense if you take the limits of both of these, it's a often a a good idea If someone tells you a scaling law to take the limits of all the variables to understand the behavior of the system. Now, the thing that's kind of cool is Rosenfeld and many others that have proposed these kinds of joint scaling laws. show that the these things fit quite well even if extrapolate. so, what you're gonna do is you're gonna fit on these green points, right, which is a low compute regime and low data regime. you never see things that have either high data or high model sizes, right? And then you can extrapolate on the basis of these tiny runs in this corner over here to essentially this high compute or high data, high model size regime, and you can very accurately predict the error based on your predictions. Great sort of predictions, right? And so once you have that, now what you can do is to say, well, I have a fixed amount of flops, subject to my flops constraints, minimize this, right?

`60:07` That's a simple non linear optimization problem, you can do that to solve for this. Now, both Caplin and Rosenthal provide equations of this form.

`60:19` What sort of do they say? Well, if you look at Caplan, and they solve for this equation, they say something like this. Well, the amount of they always flip this, it confuses me. The amount of data, it should be C to the 0.73 and the amount of Model size should be c to the 0.27. All right, is that reverse? I think it's reverse. Sorry, and n should be the amount of parameters, and d should be the amount of data.

![Slide 44](https://cs336.sg.aidenovak.com/media/slides/l9/044.webp)

`60:45` And so tokens per parameters decreases with c, right? So as you increase the compute amounts, this prescription by Kaplan tells you to train bigger and bigger and bigger models. And so if you were around in the days of GPT 3, it kind of shocks me to think that there are people who were not. You know you would know that there was a period where everyone was training these like gigantic models, right? Like they like hundreds of billions of parameters of models, like trillion parameter dense models, what have you. And part of that was driven by this, right? They saw GPT 3. They saw these like capone predictions, and they said, wow, I'm gonna do my big training runs on these gigantic, you know, model size models. But you know, a few years later in 2022, some folks at DeepMind, Hoffman at all came in, and they said, well, actually these predictions are all terribly off, right?

`61:34` You've been training these giant models, these three stars over here. But actually those are way too big. What we should be training instead is a very different trend, this this line on the blue over here. And we should be training this Teal star, which is a much better model, right? So they they argued that we should be training much smaller models relative to this. And I think you know, of course, many of you or all of you probably know the Chinchilla paper, you know the what the token multipliers are, right? It's 20 tokens every parameter. But I think this is useful to go in in detail. Because the Chinchilla Caplin sort of disagreement, then like all of this tells us something important about how we should be thinking about scaling laws, right?

![Slide 45](https://cs336.sg.aidenovak.com/media/slides/l9/045.webp)

`62:12` It's not just something where you like turn the crank and you get the answer. It's something that's actually careful and sensitive, and we have to you know respect the process. OK, so I'm gonna walk through the chinchilla paper, and after I walk through the chinchilla paper, I'm gonna talk about why the Caplan and chinchilla paper is disagree. And what that means for you, OK? So, the Chinchilla authors suggest three different ways of fitting scaling laws. I I really like this because it's a way of you know robustifying yourself to sort of modeling assumptions you may have made. and each of these actually is a quite a different approach to estimating the trade off between model size and data. And minus method three, which does differ a little bit. They mostly agree, like all one and two both agree that the coefficients for data and for model size are roughly the same, 0.5 0.5, 0.49, 0.51.

`63:04` Method three does differ, you know, 0.46 and 0.54, but we'll talk about why later. Caplan and all of course have a very big difference between the two, right? Which is why they were training big big models at big big compute sizes.

![Slide 46](https://cs336.sg.aidenovak.com/media/slides/l9/046.webp)

`63:17` Method number one from Chinchilla is what I might call like the lower envelope method, and this is the method that you know was also being used partially by Kaplan to try to figure out what the right scaling was. And so what you do is you take all of your training run, so each one of these colored lines is a training curve, right? And what you want to do is you want to actually take the lower envelope of these training curves. Right, because a lower envelope point on this training curve, that means that for a particular flop, this is the best loss that I ever achieved, right? For that flop on any training run, right? So I can take all of the low lower envelope points at the very bottom here at the high compute regime, and then I can plot all those points to see how big were the models that each of these flop points intersected, right? Because each of these points is a training run with a model size. And if I scatter plot that, I will get a nice linear trend, and that will give me one way of estimating the trade, optimal trade off between flops to parameters. And that gives one answer, which is for their compute budget, they should train a 67 billion parameter model.

`64:19` Right, this is a nice and simple way, but very tricky defining things like what is the lower envelope, so on and so forth. Like there there's definitely some tricky parts. Method two is isoflops. This is now very popular. It's also very easy and very robust. This is probably my personal favorite method. And what you do is for this, you pick a bunch of different flops budgets, right? So I have a different range of flops budgets, you know, going in factors ofuh, 3 in this case or 136. And what you do is for each float budget, you're gonna essentially sweep the space, right?

![Slide 47](https://cs336.sg.aidenovak.com/media/slides/l9/047.webp)

`64:53` So here, I have one variable which is the parameter to data trade off. So, for each float budget, I'm gonna sweep over the parameter to data trade off, fixing my flops, right? So I I double my data size, I half my model size, so on and so forth, right? And that will generate a sequence of training runs whose terminal loss is gonna sweep out some sort of curve, right? So each one of these colored sets of points is a fixed flop run, each point is sweeping across different parameters, and of course it's implicitly sweeping over data as well, right? Now, the nice thing now is I can take the minimum over each of these runs, or even fit a quadratic and take the minimum over the quadratics. And then I can draw a line through all of the bottoms of the quadratics, or take you know the quad bottom of the quadratics. and plot them as a function of flops to to parameters. and that will give me another prediction, which agrees very closely.

`65:41` This is 63 billion parameters for my compute budget.

![Slide 48](https://cs336.sg.aidenovak.com/media/slides/l9/048.webp)

`65:48` Now, the final approach that we can do is actually the maybe most natural one, as suggested by the Rosenfeld paper, as well as the Kaplan paper. Because both of them uhSort of propose a hypothesized functional form for how model size and data relate to loss, right? So if you have such a joint functional form What you can do is you can take, you know, a whole bunch of training runs that you've done, and you can fit, you know, your your loss landscape. right, which is your hypo t size functional form, and you can just do curve fitting to estimate what the coefficients are in your scaling law. This is the most brute force way. You have a hypothesized relationship between loss, data size and model size. Train a bunch of models, do curve fitting, right? Hopefully fairly straightforward, you know, how how you do that. Of course, if you're doing this method, how you do the curve fitting is very important.

`66:39` Right now, you're fitting these surfaces, there's many variables, it's kind of tricky. You know, this thing also has some trickiness, just like method one, right? So it's not it's not a straightforward way, the same way that maybe iso flops is. OK, so now, you know that we've gone through what Chinchilla does, it doesn't seem materially different than what Kaplan was doing, right? It's not like Kaplan was doing one crazy thing, and then chi- the Chinchilla authors did something like way smarter. And you said, wow, why didn't we do that from the s- the start? Right, they both did pretty reasonable stuff. But if we look at their scaling predictions, they are very different, right? And in hindsight, we generally, you know, tend to agree with the fact that the Chintala authors had more of the correct scaling, right?

![Slide 49](https://cs336.sg.aidenovak.com/media/slides/l9/049.webp)

`67:21` If you do your own Chintala style analysis, you will find probably scalings much closer to Chintala. Now, so why is there such a big difference? When they're both fitting these like joint scaling laws. OK, so this is I think where we get into kind of the messy realities of how scaling laws are made, right? Scaling laws can really change depending on the precise ways in which you're implementing the scaling laws, the hyperparameter settings that you're using, and even like what the x axis is. So, there's a a really nice paper. Oh, no, I covered the authors. OK, you can I I can paste this in the in the lecture slack later. Called resolving discrepancy to compute optimal scaling of language models.

`68:06` Yeah, Yee is the last author, he was a former student here. And they you know show basically first, they replicate the Kaplan result, you know, using roughly the Kaplan settings. And then they say, OK, well, you know, let's change how we count parameters. They, that shifts the curve a little bit. And then they say, OK, well, let's change how learning rate warm up is done. That shifts the curve a little bit. And then let's tune the optimizer a little bit more, and then they get exactly kind of a chintel result, right? So, their argument here is that there is a sequence of what seems like very minor decisions. That in the end gives you this big gap between the Kaplan scaling law and the Chintala scaling law, right?

![Slide 50](https://cs336.sg.aidenovak.com/media/slides/l9/050.webp)

`68:44` And I'll explain what each of these steps is, not that that's like the most important single thing. But also to emphasize to you how kind of important the details are here, right? So the very first arrow where we go from Kaplan to the blue line, that is how we count parameters. So remembering Kaplan, I said well they saw this weird stuff if they included embedding parameters. Right, so they excluded embedding parameters. That's generally an okay thing to do. But one thing they also did that really messed them up was they also excluded the the last layer parameter counts, right? The the softmax ones, because in many models, kind of the embedding and the last layer linear are dual to each other, right? Like they have the same shapes.

`69:24` Right, the inside has vocab size by hidden dim, the output might tricks has hidden dim by vocab size. And because they're the same shape, they said, well, let's just exclude both of those. Turns out whether you include or exclude those has a big material impact on the shape of the scaling law, right? They also found, this is much more of kind of a oversight more than a a genuine sort of difference. In opinion, that a lot of the models that were being trained in Kaplan were very small. And in fact, they were so small that they were not actually converging by the time that the learning rate warm up was done, right? So their learning rates were set very suboptimally, and because of that, you know, their models weren't fully converged. And then finally, they found that you know, Captain at all was fixing one big batch size, and those big batch sizes were suboptimal for the smaller models, and therefore, you know, if you tune that correctly and use varying batch sizes. you end up getting This sort of thing that agrees exactly with Chintal, right?

`70:20` So, I think each of these might seem like a very minor difference, but if you sort of change up these really minor differences, you can get big shifts in the scaling law. And one thing I kind of like to emphasize Is that scaling laws are kind of lower bounds in some sense, right? They're kind of saying, if I continue this recipe and I scale it up, then this is what I will get, right? But if you're scaling up sort of a recipe that you don't wanna scale up, like your warm up is kind of crazy or your bat sizes are crazy. You're gonna get bad scaling loss, right? So you wanna be as close to the proper sort of full run as possible.

![Slide 51](https://cs336.sg.aidenovak.com/media/slides/l9/051.webp)

`70:53` There's another paper that does another very interesting thing to try to study the difference between Caplan and Chintal scaling laws. It's by Pearson and Song. And what they do, and I think this is very clever, is they don't train any models. They just take The chinchilla training curves, or like, or the implied functional form of the chinchilla training curves. They just plot them out, and they say well, based on these, you know, implied training curves, which are simulated. Purely from Chinchilla, they generate what the implied training curse will be for Caplan, if you ignore the embedding parameters, and you scale way down, and what happens if you use the total parameter count instead. And what they find is really two effects. One is that Caplin is operating at a much lower compute scale so they're very sensitive to small changes and actually ignoring the non embedding parameter is a sufficiently big change to mess up the result relative to天池。 So their their argument is slightly different. From Yee Yee and others, which is that here actually is the size of the compute scale, plus the slight nonlinearity induced by non embedding parameters that's causing this this disagreement.

`71:58` So, this is really, you know, starting to get into the weeds. But I hope you kind of appreciate, you know, the importance of paying attention to detail, right? And I think a lot of the the work that goes on in establishing relying scaling laws is about figuring out what the right regime is, so that you can get reliable scaling across, you know, compute scales. The final thing, and this is really just more of something that I just find amusing more than anything else. Which is that you know Chinchilla is this paper that kind of takes down the Kaplan paper in a way, right? It's like, oh you guys were very wrong. But there was this one remaining mystery, which is that Chintilla method 3 never agreed with Chintilla method 1 and 2, right? And and the numbers look small, and the authors are are very unperturbed about this. But I think the implications are quite non trivial.

`72:42` Like if you look at this, method one and two basically say, you know, the amount of parameters and tokens should scale together, right? So there should be a fixed ratio between the two. This is where kind of a chinchilla factor of twenty comes from. If you look at, you know, number three, the exponents are different. This means if you scale up your compute, eventually, actually in this case, you're gonna have way more tokens than you are gonna have. parameters, right? so it's gonna be actually a very different scaling conclusion, as him potically than the other ones. and so, turns out that you know, I was certainly not the only ones perturbed about this. Except some folks that Epoch AI did something about being perturbed.

![Slide 52](https://cs336.sg.aidenovak.com/media/slides/l9/052.webp)

`73:20` They couldn't get either the raw data or the code. So what they did was they extracted the data from plots in the paper, and they refitted the method three regression, like they extracted the data from the paper, and given that data, they fitted that sort of two D surface or sorry three D surface.That was proposed, and then what. they showed is actually the original paper must have underfitted the original data. And if they redid sort of the curve fitting, they could get much lower losses relative to what sort of the the proposed scaling law. In the chinchilla paper, and in fact, if you follow the re-fitted model, you got almost exactly the you know, D over 20 is 20 rule of thumb from the chinchilla paper, right? Which is really a funny conclusion, I think, to the chinchilla paper. Where in the end, it turns out that you know the authors were more right than they knew, right? They actually they they had made a mistake, and that was the only reason why method three disagreed from methods one and two.

![Slide 53](https://cs336.sg.aidenovak.com/media/slides/l9/053.webp)

`74:14` So it's kind of a niceuh, sort of conclusion to know that the method was robust after all. OK, so that's the the end of the Chinchilla saga in some ways, but I will end this section by saying you probably don't want the Chinchilla factor. At this point, I think this is a pretty common You know, knowledge thing. But if you're training a production model, right, you don't really care about saving training compute for the most part. And you can look at all sorts of external surveys that people have done of how frontier labs are spending their compute. Most of the compute is not going into training runs, right? Most of the compute is actually going into R and D and serving, right? And given that, I'm especially for serving, what you want to do is you want small models that are capable, right? You do not want big bloated models that cost a lot to serve.

`75:03` Even if that minimizes training cost, and so what you really want is something that you would call is overtrained. I put that in quotes because really overtraining is what we want, that's the right amount of training. And if you look at all the models that were released, you know, in the early days like GPT 3, this was under trained for sure, right? 3 tokens per parameter, Chinchilla gets to the modern ratio of 20 per parameter. And then for a while, people were at 20 per parameter. And this is the era I think where there wasn't that much serving of these models, right? The models were cool, but it wasn't being served that scale. And then as you sort of start to move to to the eras where you know serving becomes very real, now you're over training these models, and now you've moved into MoE, and you're making all these trade offs to optimize inference serving, right? So I think chinchilla is important, not because I think the 20:1 ratio is, you know, the one golden ratio.

![Slide 54](https://cs336.sg.aidenovak.com/media/slides/l9/054.webp)

`75:51` If you're doing stuff for research, it might be. But I think the reason why it's important is because it tells us quite a bit about how we should be fitting scaling laws, and how to think about scaling laws. OK, one final thing I'll end with, Tincelawise and and otherwise for scaling, is isoflops have been one of the things that I think as a research tool have endured. Isolflots are very easy to execute, right? You fix a flots budget, and then you sweep over all of the other degrees of freedom to see kind of what the surface looks like. And this gives you in general, fairly reliable readings of how different parameters vary as a function. of your of your very of your free parameters. you know people have done this for, or actually actually my one of my students did this, for diffusion models. You know the M O E study I talked about was the isoflop style design.

`76:39` If you're ever in a situation where you're thinking how am I gonna sort of decide all these trade offs, isoflops is always a good default.

![Slide 55](https://cs336.sg.aidenovak.com/media/slides/l9/055.webp)

`76:48` OK, so to to wrap up everything for today, you know, there's this idea that, you know, we have this log-linear regularity between the amount of resources we put in. And the performance we get out, right? And extends to a wide variety of things, model parameters, compute, sparsity level of your MoE, and that lets you do a lot of the things that I was talking about earlier as like arbitrary choices. but do it in a much more evidence driven way, hopefully. And so this allows us to have engineering at scale without explicitly relying on on doing big model training runs. OK? And hopefully you remember, I think All the different components here, data scaling as this very natural object, just the exponent is a little bit strange. Model scaling allows you to do all these cool engineering things. And then finally, scaling is this very cool, robust predictor, let's call it, for how to try to to en-, do engineering at large scale.

![Slide 56](https://cs336.sg.aidenovak.com/media/slides/l9/056.webp)

`77:40` And thanks, next lecture I think will be Percy on inference, but after that we will return to more advanced, niche scaling topics.