# CS336 Lecture 10: Inference（推理） > [Stanford CS336 · Language Modeling from Scratch · Spring 2026](https://cs336.stanford.edu/) · [视频](https://www.youtube.com/watch?v=EfM546A79aM) · [官方讲义 (Trace Viewer)](https://cs336.stanford.edu/lectures/?trace=lecture_10) > 讲师：Percy Liang · 时长 85 分钟 · 本文档 = 官方讲义截图 + 全程文稿（豆包大模型录音文件识别转录，已去除语气词） ## 目录 1. 引入 — [▶ 0:05](https://www.youtube.com/watch?v=EfM546A79aM&t=5) 2. 推理的版图：哪里在用、为何要快、快 = 什么 — [▶ 0:07](https://www.youtube.com/watch?v=EfM546A79aM&t=7) 3. 回顾：Transformer 记号 — [▶ 8:37](https://www.youtube.com/watch?v=EfM546A79aM&t=517) 4. 回顾：算术强度 — [▶ 14:08](https://www.youtube.com/watch?v=EfM546A79aM&t=848) 5. 推理的算术强度：Prefill vs 生成 — [▶ 20:23](https://www.youtube.com/watch?v=EfM546A79aM&t=1223) 6. 时延与吞吐：Llama 2 13B 实例与 batch 权衡 — [▶ 35:49](https://www.youtube.com/watch?v=EfM546A79aM&t=2149) 7. 减小 KV Cache：GQA / MLA / CLA / 局部与稀疏注意力 — [▶ 46:44](https://www.youtube.com/watch?v=EfM546A79aM&t=2804) 8. 量化：QAT / PTQ / AWQ — [▶ 64:28](https://www.youtube.com/watch?v=EfM546A79aM&t=3868) 9. 剪枝与蒸馏 — [▶ 67:21](https://www.youtube.com/watch?v=EfM546A79aM&t=4041) 10. 投机采样（无损加速） — [▶ 71:46](https://www.youtube.com/watch?v=EfM546A79aM&t=4306) 11. 动态负载：Continuous Batching — [▶ 77:19](https://www.youtube.com/watch?v=EfM546A79aM&t=4639) 12. PagedAttention 与 vLLM — [▶ 79:49](https://www.youtube.com/watch?v=EfM546A79aM&t=4789) ## 1. 引入 *[0:05 – 0:06] · [▶ 从这里看视频](https://www.youtube.com/watch?v=EfM546A79aM&t=5)* ![Slide 0](https://cs336.sg.aidenovak.com/media/slides/l10/000.webp) ![Slide 1](https://cs336.sg.aidenovak.com/media/slides/l10/001.webp) `0:05` Alright, let's get started. ## 2. 推理的版图：哪里在用、为何要快、快 = 什么 *[0:07 – 8:36] · [▶ 从这里看视频](https://www.youtube.com/watch?v=EfM546A79aM&t=7)* `0:07` So, last time, Totzu talked about skating loss, and we're gonna take a little bit break from that, and talk about and inference. So, the problem of inference is very simple. You've trained a model, and you're given a prompt, and you want to produce the response. Usually as accurately and as quickly as you can. So, inference turns out to be only one lecture, but it's actually of growing importance, and it shows up in Many places. So clearly, once you've trained a model, you don't just sit there and if you're researcher, you put a plot in your paper of here's how my model works, butum, For everyone else, you actually want to use this model. So, what does use look like? It could be chatting with an AI assistant or a chatbot. It could be doing code completion. ![Slide 9](https://cs336.sg.aidenovak.com/media/slides/l10/009.webp) `1:02` These days, agents are very popular, and that requires inference, batch data processing as well. It's also useful for evaluation, for evaluation that requires generation in particular. And it's also used inside training, if you're doing reinforcement learning, you need to have a model, you generate rollouts, and you score them. and you update the weights appropriately. so inference shows up all over the a place. and you know efficiency as the is there you know theme of this class. Really matters more than ever here. And even if you just look at the actual use, kind of app of Vantage Point. Your training is a one time cost. ![Slide 10](https://cs336.sg.aidenovak.com/media/slides/l10/010.webp) `1:50` It could be very expensive, it is very expensive, but once you're done with that, that's it. But inference is a repeated cost. You s- you kind of incur every single day. So, you know, OpenAI is estimated to produce 8.6 trillion tokens a day. And just for reference, Deepseek V four Which came out earlier this year, was trained on 32 trillion tokens. So, in less than four days, the number of tokens that open air has to produce, and therefore the com- compute isum, Deal at least to Deepseek4. And of course, you know, even the frontier models might be trained on more tokens, but still, inference, the number of tokens generated is is going to be Higher. So, that's just a you know some perspective. ![Slide 11](https://cs336.sg.aidenovak.com/media/slides/l10/011.webp) `2:40` And moreover, I think this the importance of inference has grown in the last year because You know, once upon a time, we thought of language models as primarily as chatbots or assistance, where you put in a prompt. you get back a response, and Presumably, the goal of the humanities, you read the response. But now as we move more into a agentical world, what it looks like is that a query goes in. And the agent is going to do a bunch of stuff. It's going to think, it's going to reason, it's going to call some tools, it's going to introspect, and at the end of the day, it will produce some output for a human to read. So the most of the tokens that the agent produces is actually not for reading. And so you should really think about the nu- number of tokens generated is really the compute spent, and there's no sort of limit. If you have an ambitious enough problem, you're going to need much much compute and a lot of tokens. Right, so if you were in the chat out world, maybe inference you after a certain point, it's fast enough, ==🟡because humans can only read so fast. `3:51` But for an agent, you know, there's no kind of limit to how much of value you can get out of squeezing more out of inference. ![Slide 12](https://cs336.sg.aidenovak.com/media/slides/l10/012.webp) ![Slide 13](https://cs336.sg.aidenovak.com/media/slides/l10/013.webp) ![Slide 14](https://cs336.sg.aidenovak.com/media/slides/l10/014.webp) `4:00` So, there's a lot of people doing inference, on the commercial side. Of course, all the the closed API providers have to serve their model, so inference is a big deal for them. And there is also a bunch of providers serving open weight models, and providing inference, as well. And in the open source community, there is a bunch of packages, so VLLM is probably a popular one, kind of a go to SG Lang is another one that's particularly good for agenty workloads, but maybe not as popular yet. And then there's TensorRT from NVIDIA. Which is really you know fast, but it's more narrow. And then llama cpp, if you want to run inference on your on CPU, this is a popular you know package. okay, so hopefully i've argued that inference is is huge. if you can make inference, you know, twice as fast or even 10 percent faster. ![Slide 15](https://cs336.sg.aidenovak.com/media/slides/l10/015.webp) `5:00` That is a big deal. So now the question is, what does fast mean? So there's a few metrics that capture the notion of fast, and they're going to be applicable and in different settings, and have different trade offs. So, one metric of fast is time to first token or TTFT. This is essentially how long a user waits before any generation happens. So, you put in a query into ChatGPT, and some number of, you know, milliseconds passes by, and then And then from the point the first token comes in, that is that is the time. And this is generally useful for interactive applications, because that latency where you're just waiting and doing nothing. Is is the longer that is, the worse the user experience. And as soon as tokens come in, it doesn't maybe have to be that fast, because if you're gonna read it anyway, you can't read that fast. `6:00` So, the second metric is latency. And this is from the standpoint of an individual user, how fast are tokens appearing for one query. This is also important for interactive no applications. So basically this is how fast the tokens are kind of streaming through. `6:19` And then the second related concept is is throughput, measuring tokens per second. This is how fast tokens are appearing for many queries. So, lane c or one over lane c and throughput are clearly very related, right? it's it's very kind of aligned of the you know in general, many interventions will make latency and throughput you know better. But as we'll see later, there's actually a trade off here. And throughput is useful for if you're doing a bunch of batch processing, right? You just want you have You know, a petabyte of data, and you want to process it with a language model. You just want that job to get done. It doesn't matter if one query is coming in faster, sooner or later. `7:05` OK. So, what determines the efficiency by either any of these metrics of of inference? So, the the high level bit is as follows, and this is kind of a t- you know, maybe one high level bit to remember from the lecture is that when you're doing training. you see all the tokens at once, because in supervised fine tuning, you see all the tokens. And you can paralyzed over the sequence. Remember, think about in the transformer the calculation for attention NLP. The sequence is just a dimension, so it's a big tensor that gets multiplied. And so you essentially process all the tokens at once. ![Slide 16](https://cs336.sg.aidenovak.com/media/slides/l10/016.webp) `7:47` In inference, you can't do that. S- You have to generate, because of the auto regressive nature of inference, s- per- token sequentially, one at a time. So, this is a fundamental problem. of why inference is a very different workflow than training, because you can't parallelize across the sequence dimension. and therefore, as we'll see later, it's going to be harder to Have high artificial intensity or f-, fully utilize the computer. `8:15` OK. `8:19` All right, so let's let's go on. So first, I'm going to just maybe of a preview of what's in this lecture. First, I'm going to do some math to understandum, The arithmetic intensity, and throughput and latency, how to think about that for a transformer. ## 3. 回顾：Transformer 记号 *[8:37 – 14:08] · [▶ 从这里看视频](https://www.youtube.com/watch?v=EfM546A79aM&t=517)* `8:37` And then I'm gonna talk about various techniques to reduce theuh, The cost by, you know, making the kv cache smaller, quantizing model pruning. And then finally I'll talk about speculative decoding, and then some kind of practical concerns. `8:55` All right, so this is gonna be a bit of a review, mostly kind of swapping and make sure we're on the same page with respect to notation. Much of this lecture is based on the ==scaling book ==from Google on you know transformers and inference. So, I do recommend that people go check it out. It's a very nicely written, and some of the picture, the figures are you know, generously lifted from that book. OK, so, just to establish a bit of notation to understand this this diagram. So, we're going to use symbols, Think about in einops to denote dimensions, but also they're kind of length. So B is gonna be the the batch dimension, and also the number of sequences. T is gonna be the sequence dimension, which is also a number of tokens. D is a model dimension, H is a head dimension. ![Slide 17](https://cs336.sg.aidenovak.com/media/slides/l10/017.webp) `9:49` OK, so if I'm gonna write this, this is taking a a a product between a tensor with dimensions B, T, and D. And another matrix of d and h, and I produce b t h. So what is going on here is that there are some dimensions marked red, which are going to be contracting. These dimensions appear in both the operands, and disappear from the the results. And then there's going to be other regular dimensions in black that appear only in one operand, and stay in the result. OK, so this is kind of a general form. Another case is if you have blue dimensions, that means they're they're batching dimensions, and they appear in both But stay in the result, they do not get con- contracted or reduced. OK, so the interpretation here as we'll see later is that we have the tensor for everyuh, sequence and token position, you have a a vector, and you want to multiply that by a, you know, a matrix. And then here, you're basically taking a bunch of dot products between two, you know, tensors. `10:59` OK, so with that in mind, this in all its full glory is the description ofuh, A transformer block. So, it looks a little bit like a circuit diagram. There's a lot of details here, but I actually find this probably the most umKind of crisp definition of what a transformer is. You know, you see a lot of description of a transformer, and frankly, a lot of them are kind of, you know, it it's hard to understand what is exactly happening. This one basically tells you exactly the shapes of the tensors, and allows you to reason about the the the dependency. So, for a transformer block, you take the X, which is the activations of one layer, you feed it through attention. And then you feed it through an MLP. And and just as a kind of a quick refresher, the the X, and you project it using a query matrix. A key matrix and a value matrix. `11:57` You'll see here that we have and which is the number of heads, and h is a dimension of each head. So, the query matrix is batch times sequence times number of heads times the head dimension. And k and v, instead of having n, we have k. Which we'll see later, which we have a s- potentially smaller number of key value heads. And then this is a tension operation, where I notice that there's batching dimensions. So B appears in in both, and we're contracting over the head you know dimension. And we'll come back to why this B Being both makes inference hard, why attention is kind of a bottleneck there. OK, so just kind of hold on to that thought. And the MLP is is kind of very you know simple and straightforward. ![Slide 18](https://cs336.sg.aidenovak.com/media/slides/l10/018.webp) `12:54` You just do some you know maxmalls, this is the the the gating umum, You know, matrix, this is the up projection and the down projection. So, I think you guys have implemented this, so I don't want to be labor it, but just to swap in the notation. By convention, we're going to assume thatum, In the MLP, the MLP up projects the d dimensional a model dimension into four times that. So, F is always, you know, think about as four D whenever you see F. And then the model dimension gets split across the end heads, so the model dimension is always equal to a number of heads times the head dimension. And then the the number of heads get split in the case of group query attention into the number of groups, and And the number of heads per group. And finally, we have two variables s and t, which both represents the kind of the sequence dimension. And the difference is that, S is going to be represent the number of input tokens to prove, and t is going to represent the number of output tokens. ## 4. 回顾：算术强度 *[14:08 – 20:23] · [▶ 从这里看视频](https://www.youtube.com/watch?v=EfM546A79aM&t=848)* `14:08` So in training time, these two are the same, because we're predicting all the ou- the sameum, Inputs as outputs. At inference time, T is going to be one, and S is just going to be the input. ![Slide 19](https://cs336.sg.aidenovak.com/media/slides/l10/019.webp) `14:23` OK, so another review for arithmetic intensity. So I talked about this in the second lecture. So just as a warm up, suppose I'm multiplying two matrices, a B by D matrix and a D by F matrix. So, intuition, B is the batch dimension, D is a hidden dimension or the model dimension, and F is a upfretching dimension in the MLP. So, remember what is iris net intensity? I have to count flops, and I have to count the amount of memory moved. OK, so we can we can do this as follows. Soum, What do you have to do to multiply these matrices? Now, remember the systems, you know lectures. ![Slide 20](https://cs336.sg.aidenovak.com/media/slides/l10/020.webp) ![Slide 21](https://cs336.sg.aidenovak.com/media/slides/l10/021.webp) ![Slide 22](https://cs336.sg.aidenovak.com/media/slides/l10/022.webp) `15:04` You have to read X from HBM. And if you're storing everything in BF 16, which is the case for for inference, we're going to have a two times B times D. You're going to read W, that's going to be another two times B times F. you're going to do the matmul, so that's number of flops is two times v times d times f, and and then you're going to write it back. ![Slide 23](https://cs336.sg.aidenovak.com/media/slides/l10/023.webp) ![Slide 24](https://cs336.sg.aidenovak.com/media/slides/l10/024.webp) `15:33` OK, so every time you read and write, it's basically the basic number of entries, it's a kind of quadratic like term, and members are are cubic. And remember, this is going to be important because that's how you're gonna get high arithmetic intensity. So, number of total number of flops is you know the the cubic, and thenum, The bytes transferred is the sum of the reads and writes. ![Slide 25](https://cs336.sg.aidenovak.com/media/slides/l10/025.webp) `16:00` OK, so remember arithmetic intensity is how much compute we do per byte transfer, and we want this to be high. So, the intensity is is going to be Equal to you know flops over byte's transfer. And I'm representing all these things symbolically because it will be easier to to see and work with rather than numbers. ![Slide 26](https://cs336.sg.aidenovak.com/media/slides/l10/026.webp) `16:23` One kind of just, you know, simplification we can make is that suppose that the batch dimension here is much less than D. and F, then we can simplify this. So, in particular, what this is doing is saying d equals c times b, and f equals c times b, and let's c go to infinity. So, both d and f are really large, and b is smaller. And this just reduces to the intensity of B. `16:51` OK, so the upshot of that is that, remember when we did arithmetic intensity for your mammal, it's was something like n overuh, And over 3. And this is the kind of the analog, where instead of just having you know one variable for domain, but having square matrices, we have non square matrices. ![Slide 27](https://cs336.sg.aidenovak.com/media/slides/l10/027.webp) ![Slide 28](https://cs336.sg.aidenovak.com/media/slides/l10/028.webp) `17:15` OK, so just to compare, so the accelerator intensity of hardware is you look at the FLOPS per second in in the in the spec sheet. You look at how much memory bandwidth, how fast memory gets move between HPM. You divide, and that is the accelerated intensity. And now you compare your computational A computation's intensity with your accelerator intensity. And if it's greater than your compute bound, which is good, if you're less than your memory bound, and that's bad. So in this case, for H100, for this particular map operation, you're compute bound if the batches size is greater than 295. ![Slide 29](https://cs336.sg.aidenovak.com/media/slides/l10/029.webp) `18:03` So, here's an extreme case. So, suppose you just have one example. OK? What happens if you have one example, then your event intensity is going to be one. And this is going to be memory bound. And you can kind of see what's going on here. You're reading this DF matrix, but B is only one, so you're effectively doing Only two times d times f flops. So, you know, that's, and this is basically kind of the workload that you'll see in inference. You don't get these like full matrices, you get these like very thin uhMatrices or tensors. `18:46` OK, let me stop there, case there's any questions. `18:50` Yeah, in the transformers. `18:56` Oh, sorry. `18:58` This is cool. Across uhCan I search across? Hello, across, across, across. For across, oh, are you by any chance just across over? Yeah, should that be across cable? `19:18` I'm having trouble with the connection. Let's see. So, the number of groups here is uhWait, I'm just making sure. So k is the number of key value heads and And so you have Q sorry G sorry G per KV had umSo I understand nice by you 喂，啊。 `19:56` Yeah of course Response and new tools The madness geez 我要去 That means we have three That's my Yes OK so I think yeah I think you're right So K should be the number of groups and G should be the number of uhThe you。 know, heads per one of those groups. Yeah, okay, thanks for that. ## 5. 推理的算术强度：Prefill vs 生成 *[20:23 – 35:49] · [▶ 从这里看视频](https://www.youtube.com/watch?v=EfM546A79aM&t=1223)* `20:23` I'll fix that later. ![Slide 30](https://cs336.sg.aidenovak.com/media/slides/l10/030.webp) `20:28` OK. So, let's talk a little bit more about the arithmetic intensity of inference. OK, so, here's how to think about what's happening and In France, let's say you just do it naively. So, here is a prompt never gonna give you, going to the Transformer, and the Transformer is does You know, generates keys and values and activations, and at the end of the day. it generates logits over the output vocabulary, and you sample a token. And that token gets concatenated with the prompt. And then you just do this again, you generate never, you attach that to the prompt, and so on and so forth. OK, so this is the most naive thing. If you have a black box that takes a sequence and outputs a distribution of a tokens, which is exactly what transformer does, you can just apply this repeatedly. ![Slide 31](https://cs336.sg.aidenovak.com/media/slides/l10/031.webp) `21:28` OK, so this is works, but this is really bad, because each time you generate one token, umYou actually take order T square time, where T is the number of you know tokens that you've generated so far. So generate T tokens is actually key cu- cubed. `21:50` OK, because the the attention is already t squared, and you have to do that, you know, one for every, once for every token. `22:00` So, this is you know, pretty bad. But the observation is that you don't actually have to do this. A lot of the work can actually be shared across prefixes. So, for example, if you're over generally never versus your journey up. You're actually computing a lot of the same t values for you'll never gonna give. So these two tokens they shouldn't change. Ok, and this is because it's a causal, you know, transformer. It was by directional, then if you attach a token, then everything changes. But if it's causal, then the activations here don't change, based on Any tokens you append. `22:42` So, with that observation, the first obvious thing is to store a KV cache in HBM, so that between successive token generations, you can just reuse the KV cache. So that means when you are trying to, you know generate gonna, you don't actually have to compute all the keys and activations of these previous umprevious tokens. So this is what it looks like with a KV cache. OK, so there's going to be two stages prefill, so you get the prompt. And you populate the KV cache, which is the set of key and value pairs that the transformer computes. And then you generate the logic just the same computations we did before. And now, now you have the kv cache here, and then this distribution which you sample a token, and now you feed that through the transformer, which You know, uses this cash, and then produces a both the distribution of the next tokens, but a new KV. Which is the the basically activations corresponding to. token up. `23:57` And then this gets fed in. So now you have the co-, a comment to kv cache. You have this logic to sample, and then that feeds to the transformer you Output the distribution of next tokens, as well as the the next the new token which you add to the KV cast and so on so forth. ![Slide 32](https://cs336.sg.aidenovak.com/media/slides/l10/032.webp) `24:20` OK, so the cache formally is for every s, you know, sequence, so there's B of them, for every token, there's S of them, and for umEvery layer and every head, you store each dimensional vector. ![Slide 33](https://cs336.sg.aidenovak.com/media/slides/l10/033.webp) `24:38` OK, and just reiterate for inference, there's prefill, you given a prompt, you encode the prompt into these new vectors. And this is paralyzable, just like in training, because you see the entire prompt, you can compute the the key cache, and then in generation, you generate the new response tokens. You know sequentially, but at least you don't have to pay for generating the cache of the tokens that you already looked at. `25:08` OK, so is everyone comfortable with a KV cache? Any questions about this? ![Slide 34](https://cs336.sg.aidenovak.com/media/slides/l10/034.webp) `25:22` All right, so let's try to figure out the FLOPs and memory I/O for both the M MLP and attention layers. umum, So, remember, S is the number of tokens we're kind of conditioning on, and T is the number of tokens we're generating. So, we're gonna do this sort of abstractly, and then later we'll specify to the prefill where T equals S, and generation, T equals one. ![Slide 35](https://cs336.sg.aidenovak.com/media/slides/l10/035.webp) `25:48` So for the NLP layers, and through all of this, I'm just gonna look at the mammals, because the the mammals are the thing that actually require a lot of work. Everything else can beum, Is not that many flops, and can be potentially fused into the map also. ![Slide 36](https://cs336.sg.aidenovak.com/media/slides/l10/036.webp) `26:06` OK, so let's do this calculation. So, this is the same as the matrix calculations. I'm not gonna maybe be labor except through every single detail here, but just the form is You read X from high memory. You read all the parameters. ![Slide 37](https://cs336.sg.aidenovak.com/media/slides/l10/037.webp) ![Slide 38](https://cs336.sg.aidenovak.com/media/slides/l10/038.webp) ![Slide 39](https://cs336.sg.aidenovak.com/media/slides/l10/039.webp) ![Slide 40](https://cs336.sg.aidenovak.com/media/slides/l10/040.webp) ![Slide 41](https://cs336.sg.aidenovak.com/media/slides/l10/041.webp) ![Slide 42](https://cs336.sg.aidenovak.com/media/slides/l10/042.webp) ![Slide 43](https://cs336.sg.aidenovak.com/media/slides/l10/043.webp) `26:27` You you compute the upper pro- projection. And then, you know you write it to HBM. You compute the The the gate, and you write HBM, and then you compute. `26:47` The basically the down projection of that, and you write it to HBM Ok, so the number of flops is you know depends on p, b, and t. and dnf, so batch size, sequence length, model dimension, and the feed forward dimension, m o p dimension, and the bytes transferred is You know this expression. ![Slide 44](https://cs336.sg.aidenovak.com/media/slides/l10/044.webp) `27:15` OK, so now you can compute the error effect intensity, which again is flops divided by bytes, you know, transferred. So, this is some expression. We're gonna assume that just like before, B times T is much smaller than D N. d and f, and with that, then we see that intensity is you know b times t. ![Slide 45](https://cs336.sg.aidenovak.com/media/slides/l10/045.webp) `27:38` OK, so this is analogous to just the m OK, and and also makes sense because the the batch dimension and the sequence length dimension are sort of all all everything is independent. I'll brighten for the NLP. They don't interact, attention's different story, and so, you know, B times T, so as long as you have umYou know, in the prefill, if you make B times T large enough, large batches long sequences, you'll be fine. Now, let's look at generation. So remember generationum, There's two problems here. One is that gen generation t equals one. OK, you're only generating one token at a time. And so that means your arithmetic intensity is going to be B. But B, what it in generation is the number of concurrent requests. `28:43` And so, if you're in a sort of back setting, you kind of can control that. But if you're, let's say, serving a chatbot, it's a number of concurrent requests is essentially you know how many users concurrent users there are. Which can be high can be low, so it's a bit un particular, it can be changing over time. So that's something we're gonna address when we talk about you know continuous batching. ![Slide 46](https://cs336.sg.aidenovak.com/media/slides/l10/046.webp) ![Slide 47](https://cs336.sg.aidenovak.com/media/slides/l10/047.webp) ![Slide 48](https://cs336.sg.aidenovak.com/media/slides/l10/048.webp) ![Slide 49](https://cs336.sg.aidenovak.com/media/slides/l10/049.webp) ![Slide 50](https://cs336.sg.aidenovak.com/media/slides/l10/050.webp) `29:06` But overall, this is not bad, right? Because as long as there you have large batches, the sequence length isn't gonna really help you, but you have large batches, you should be good. So now let's look at the attention layer. So again, as is in the previous tokens already generated, t is the number tokens you want to generate logits for. So in attention, you read the the QKV, your matrices from HBM. You compute the attention. You, you know, compute the softmax, which doesn't really matter. And compute the the value. You know matrix, and then you write the result to HBM. `29:54` OK, so the flops is B times S times T times D, and the bytes transferred is this expression. So, this should, you know, all the c, the everything is a mammal, right? So, this is should be always, you know, one degree higher of a polynomial than the bytes transferred. Because it's a mammal. The only question is, you know, what does that factor look like? So that factor for attention is s times t over s plus t. ![Slide 51](https://cs336.sg.aidenovak.com/media/slides/l10/051.webp) `30:28` OK. So, so let's look at the prefill. So prefill, what this is saying is that the prefill intensity is is S over two when T equals S. So, this is good, right? Because as long as you have long sequence lengths, for attention, you're going to maintain high arithmetic intensity. ![Slide 52](https://cs336.sg.aidenovak.com/media/slides/l10/052.webp) `30:54` Notice that the batching dimension doesn't happen here. I'll explain a bit, you know why, later. But for generation This is bad news, right? So the the the generated nation intensity is s over s plus one, and that's less than one, or even the, let's just call one. And everybody can test it. Remember one is bad. We wanted to be something like 295 for each 100 to be s- to saturate the compute. And so You know, this is really the bottleneck. So, we did all this analysis for you know, NLP attention, you know, ==prefill generation==, and we found this to be a bottleneck. ![Slide 53](https://cs336.sg.aidenovak.com/media/slides/l10/053.webp) `31:39` So, let's try to contemplate why this is a bottleneck. So, unlike MLPs, so MLPs for generation was actually okay, as long as your sequences umI'm sorry, is your your batches large enough? So, the problem with This ML, you know, attention is that, you know, let's look at MLPs. Every sequence hits the same mlp weights. okay, so these don't depend on b, whereas in attention layer, each sequence has its own kv cache. So these all depend on MLP. And so you can think about this as umWhat's what's going on here is that, you know, in the MLP case, having be being big actually is is helpful because you kind of get to load these MLP weights. You know kind of once, and you can kind of use it for all your sequences. Right, that's how you get higher with my intensity, because you use it simplifying a bit, you load it once, you do all your batch processing, and then That that is that is good. `33:02` Whereas in attention, you can't, you know, all these depend on B. So, increasing B doesn't help. It's like every, for every sequence You're basically doing a mammal. So, they're all in the independence, so you know, doing more mammal isn't helpful. Just like remember in the very beginningum, I showed you this example. This also has pretty bad arithmetic intensity. It's not a not a mammal because we're essentially batching. and by a coordinate, and this is essentially basically the same as doing adult product, which has horrible error rate intensity. And remember this is in the attention, this B blue B is like the cause of why, the tension is arithmetic intensity doesn't scale with B. `33:54` And why that is a balanac. ![Slide 54](https://cs336.sg.aidenovak.com/media/slides/l10/054.webp) `33:58` OK, to just summarize here, so prefill is compute bound, generation is many memory bound. So if you look at the NLP intensity for theuh, for prefill, it's b times s, great. prefill for it, and for attention, intensity, s over two, not as good, but workable. Generation MLP intensity also workable requires long concurrent requests, but it is really the generation intense attention intensity Which is a fundamental bottleneck, and you just, that's it. You you just have to, if you're sticking with a transformer, you can't really, you know, improve this. `34:43` OK, pause there for questions. `34:57` So now whenever you hear people say, oh, inference is memory bound, you know why? `35:07` So, let's now use these intuitions and calculations to think about our you know, inference matrix. Throughput and latency, and also TTFT. All right, so inferences memory bound. The the now the main, I mean in some ways this simplifies a lot of things, because when we think about how long things take. You just look at how much memory needs to be transferred. OK? Because assuming you overlap communication and computation, the bottleneck is going to be just the amount of memory that you have to Deal with. ## 6. 时延与吞吐：Llama 2 13B 实例与 batch 权衡 *[35:49 – 46:43] · [▶ 从这里看视频](https://www.youtube.com/watch?v=EfM546A79aM&t=2149)* `35:49` Ok, so in some ways it's nice because it's simple, but in other ways it's frustrating that your accelerators are sitting there not doing anything. ![Slide 55](https://cs336.sg.aidenovak.com/media/slides/l10/055.webp) ![Slide 56](https://cs336.sg.aidenovak.com/media/slides/l10/056.webp) `35:59` OK, so let's walk through example. So, for llama two 13B on H100. So, what is the latency and the throughput? OK, so remember llama 13B has a particular shape, so, oops, this is a sequence length, the model dimension, p forward dimension. Number of f- query, number of key value heads. There's no GQA here, so the n and equals k, head dimension, number of layers, of vocab size, and the memory bandwidth for H100. `36:37` OK, so, So now, using this this config, let's compute the to transformer performance, your statistics. OK, so these are are inputs here. and just what the statistics I'm gonna compute are going to be number of parameters, The memory usage latency and throughput. OK, so first of all, you know what's cost, what does takes memory? So the parameters take memory. So, we compute the number of of parameters. And so, you have look at the embeddings, the the You know the MLP layers, the the k-, the projections of the K, Q, V. At the end of the day, you get some number of parameters. Assuming that you know we're doing a BF sixteen, which always case for inference, then parameters take up this many bytes. `37:49` Also in memory is the kv cache. And the kv cache is the number of tokens in your se- sequence timesum, The number of your heads, and the kv heads, times a head dimension, times a number of layers. You have one for the key, and one for the value. And then you have two multiple two for BF sixteen. OK, so that's the size of the KV cash, and the total memory usage is this is for each sequence, you have B sequences, so that's B times that. Plus the parameter size, and that's the amount of memory you you need. And now, what's latency? The latency is determined byuh, The memory I/O. OK, because c- inferences memory bound, most assuming you overlap communication and compute. `38:48` All the memory is, it's it's going to be on, you know shuffling parameters, you know back and forth. Between HBM and SRAM. So, that's how long it takes to you know move memory around. Throughput is the inverse of latency, but we're generating B tokens in parallel, so this isum, You know, this is tokens per second, this is seconds per token, this is tokens per second, as well as having an additional B. because you're processing a batch of B. `39:27` OK, so now let's compute for this config, what are these actual values? OK, so numb- parameters is you know 13 billion. So, good sandy check that is a advertise as a 13 billion parameter model. Memory is this this term, which is about 838 Like this million times B, so it's basically s, you know, a linear function of B, plus some other term. So, this is a KV cache which grows as B, this is the parameters. Which is double the number of primes. `40:06` Now latency is just this scale by the memory bandwidth, so it has the same form as as the memory. And throughput is B over that. `40:18` OK. So, notice that latency as you increase B grows, because the KV cache grows, and in order to process stuff, you have to co- copy the KV cache back and forth. And now throughput is more interesting, because as you increase B, you know throughput does improve, but up to a limit. Right, because throughput in improves because now you're amortizing the cost over a larger batch. But also, you know, the speed at which you're processing this obviously is alsoum, increasing over over time. So, this sort of asymptotes, it's not gonna, you know, through a camp possibly go to infinity. `41:06` OK. Any questions about this so far? `41:14` So, basically, you just have to remember latency is a linear function in B. This is the where the constant is the size of KV cash, and then this is a number of parameters, and then throughput is umYou know, some B, a proportional to B over B plus something. ![Slide 57](https://cs336.sg.aidenovak.com/media/slides/l10/057.webp) `41:35` OK, so now let's instantiate this for a bunch of situations. OK, so if you have batch size one, so this is what you would get. You get a latency of zero point as zero point zero zero eight you know seconds per per token. And the throughput is 124 tokens per second. So if you what happens if you increase the batch size, you'll see that the latency goes up, but the throughput also goes up. So latency gets worse, but the throughput improves. So, this is kind of a interesting thing, because we think about, oh, we just want to make it go fast. But fast actually has two meanings here, which actually depending on which one you care about. Is, you know, complete opposite. ![Slide 58](https://cs336.sg.aidenovak.com/media/slides/l10/058.webp) `42:24` If you want to tune your batch size, that's gonna really determine wha- you, if you want latency or throughput. And what happens if you increase the, let's increase when we really want high throughput, because we're processing lots of documents. Let's increase the batch size even more. OK, so the latency gets even worse, not not, I mean, yeah, it gets worse. Throughput gets even better. But the main problem here is that your memory, you run out of memory. Ok, because the memory for storing this KV cache. is x gonna exceed your H100 memory? `43:03` And if you have B two hundreds, you can increase the batch size more, but eventually you hit some sort of limit. So there's kind of limitations to how much you can in through your throughput, you'll never get to sort of the asymptote. Because you'll hit the memory. ![Slide 59](https://cs336.sg.aidenovak.com/media/slides/l10/059.webp) `43:20` OK, and also your your throughput, you know, is getting, your gains are kind of diminishing as well. ![Slide 60](https://cs336.sg.aidenovak.com/media/slides/l10/060.webp) `43:31` So, increasing batch size worsens the latency, because now you have a larger KV cash to read and write. And remember it's batched. So, if you're an individual query, you have to wait for everyone to finish. So, you have to basically, you get it, you're like waiting for a bus, and the latency is you know pretty high. You wait, and then But and then you go. Where is the throughput of a bus is pretty good, because you can move everyone at at once. So, the the throughput im- improves as batch size increases, because Remember the parameters are are shared, and you load that, you know, once into into memory, and you can process a lot of sequences. ![Slide 61](https://cs336.sg.aidenovak.com/media/slides/l10/061.webp) ![Slide 62](https://cs336.sg.aidenovak.com/media/slides/l10/062.webp) `44:17` OK, so trade off between latency and throughput, just to make sure everyone's aligned. Smaller batch sizes use better latency, but worse throughput. Large batch sizes yield better throughput, but worse latency. OK, I'm not gonna really talk about a parallelism too much. There's also another dimension of inference, which is, you can, you know, shard your your model across multipleum, Your devices, you can look at the scaling book chapter on inference if you want to know more. Just as a kind of a very trivial example, if you launch 5M copies of the model, that latency is the same, then the throughput increases by I am. And then the other metric we didn't talk about is time to first token, and this is essentially the time it takes to do prefill. Because after you finish prefill, then you can basically start generating. ![Slide 63](https://cs336.sg.aidenovak.com/media/slides/l10/063.webp) `45:12` OK. So, if you want you know, too fast, to if you want, so, you know, faster T T F T, you should use smaller batch sizes. And you want larger batch sizes to improve throughput. OK, so hopefully that is c- clear, any questions about throughput and latency? And how their intention with each other. `45:48` OK, so now, now we have a conceptual framework for thinking about how efficient inferences is in terms of arithmetic intensity, and You know throughput and and latency. Now let's try to make it faster. How do we make inference faster? There's a bunch of different techniques which are quite varied, ranging from changing the moral architecture to doing systems optimization, everything in between. So, inference in some sense is a fairly rich cross cutting topic. `46:20` OK, so the the most kind of draw, maybe now in hindsight obvious thing you can think about is hopefully we beat it into your head thatum, You know, memory is the bottleneck for inference, and the KV cache takes up a lot of memory. and it could even be larger than the number of you know parameters if for large enough batch size. ## 7. 减小 KV Cache：GQA / MLA / CLA / 局部与稀疏注意力 *[46:44 – 64:27] · [▶ 从这里看视频](https://www.youtube.com/watch?v=EfM546A79aM&t=2804)* ![Slide 2](https://cs336.sg.aidenovak.com/media/slides/l10/002.webp) ![Slide 64](https://cs336.sg.aidenovak.com/media/slides/l10/064.webp) `46:44` So, let's just try to reduce the size of a kv cache. Now, you have to be careful about how you do this, because you want to make sure you don't lose too much accuracy in the process. ![Slide 65](https://cs336.sg.aidenovak.com/media/slides/l10/065.webp) `46:56` So, here is one thing you can do, which we already talked about, which is grouped query attention. And just as a kind of a reminder here. So multi headed attention, basically for every token, you have a key and a value and a query, and what if you do grouped query attention You basically compute the same number of queries. but you have only a smaller number of groups, and you compute key and value for each group. OK, so So k is a a number of of groups here. An and so in the multi head attention, k is n, no no reduction. There's something called multi query attention, which no one uses because it's really bad. K equals one, and somewhere in between is hopefully where we'll find a balance between accuracy and speed. So, there's this this is the paper that introduces GQA from 2023. `47:59` And they show that if you look at you know time per sample. which is related to both latency and throughput. You know you see that the the MHA, so multi head attention, full attention is Has this high time, whereas if you start k equals one, it's it's much faster, and then you can actually keep on including k to like You know. k equals eight, and it's still pretty good, and eventually, your your time goes up quite a bit. ![Slide 66](https://cs336.sg.aidenovak.com/media/slides/l10/066.webp) ![Slide 67](https://cs336.sg.aidenovak.com/media/slides/l10/067.webp) `48:37` OK, so, you know why does GQA improve the latency and throughput? Well, it reduces the KV cache by a factor of N over K, and you know, and we ju- just as a re- friendly reminder, re- reducing memory usage leads to speed up, because we're memory bound. OK. So, let's revisit our friendly llama model here. Remember in the initial configuration, we're just using k equals n, so there's no multi headed. There's multi head attention, no no reduction in number of key and values. And so for this one, remember the using a batch size of 64. We get this throughput and latency. Now if you do GQA, we're going to put in let's say a s kind of a sparsity of 1 to 5.(1/5) ![Slide 68](https://cs336.sg.aidenovak.com/media/slides/l10/068.webp) ![Slide 69](https://cs336.sg.aidenovak.com/media/slides/l10/069.webp) `49:35` That reduces the your, the memory quite a bit, which in turn Improves the latency, and also improves the throughput. So, it's not that leniency interpreter always at odds. If you reduce the amount of memory, then it improves both. It's mainly the batch dimension that allows that is the point of tension. OK, so this is this is great. And let's actually just incre- increase the batch size even more. and see what happens. So, before, we, if we had a batch size of 256, we ran out of memory. But now, it fits in memory. `50:24` And we we see that you know the latency you know suffers a bit because we're increasing the batch size, but the throughputum, You know, goes out proportionally. ![Slide 70](https://cs336.sg.aidenovak.com/media/slides/l10/070.webp) `50:40` OK. So sometimes you kind of play with these parameters jointly, like you can reduce the cache, but that allows you to increase the batch size, and allow make and making other tradeoffs. So, the final thing you have to do whenever you do some, you know, lofty change is that you make sure the accuracy doesn't drop. And this paper shows that for GQ GQAum, The, you know, the time is is better, but the the the across a bunch of evals, basically it works well. So now, with these accuracy values, I think you always have to take it with a grain of salt, because this is like a for particular model. Later the deepseek paper was to show that it actually does hurt, so. You know, I guess take everything that's not just kind of math with a grain of salt here. `51:32` OK, so speaking of deepseek, here's another idea to reduce the KV ca- cash. OK, so the theme is reduce the KV cash and latency and throughput. Improve. So, this is multi head attention, same number of queries and keys and values for every token. And GQA, remember we have reduced the number of keys and values for every token. Sorry, not for every token. We basically have reduced the number of values anduh, keys. And now the multi lay in attention for deepseek says we're actually gonna leave the number of keys and values. The the same, one for every, you know, token essentially. `52:21` But I'm gonna, I'm gonna compress these. So, normally, you have someum, How do you compute your keys and values? You have your your activations, and you multiply them by some matrix to get K, and some other matrix to get V. And these are generally N times H, you know, dimensions like your model dim. Which is which is big. So MLA says, I'm going to actually project this these activations down into C dimensions. So deep sea can be to Reduced it from you know 16000 to 512. So this is quite aggressive compression here. And then I'm going to compute the k and the v from this compressed representation. `53:04` So now I can just store C, this is much smaller, and then when I need my keys and values, I can just materialize them. ![Slide 71](https://cs336.sg.aidenovak.com/media/slides/l10/071.webp) `53:14` So, there is one wrinkle here, which is that MLA is not compatible with rope, which operates directly on the keys and values. So, what they do is add additional dimensions. for, you know, for for handling the rope. But more or less, it's it's still a a pretty big, you know, reduction. So, the latency and throughput improvements follow by just, you know, simple math. The smaller the kv cache, the you know the faster you go. OK, it's almost kind of linear scaling up until some point. And then remember you need to check whether your model is you know accurate. Soum, First of all, this is sort of the result that's, you know, kind of contradicts the, or or is intention with the GQA paper. `54:06` They show that GQA actually isn't that you know great. It, you know, it's umYou know, it's much, so this is MHA, this is UK, these numbers are smaller than these numbers. ![Slide 72](https://cs336.sg.aidenovak.com/media/slides/l10/072.webp) `54:23` But they show that their method MLA, multi lay in the tension works even a little bit better than MHA, but let's just say it's about the About the the same. So this column and this column are much better than I guess there's no jQuery on this table. It's yeah, compare over here. `54:47` OK, so I'll show you and yeah. `54:53` Reducing the number of dimensions to more. So question is how does this compare with reducing the dimension of the model? So that's a good question. I don't, these illustrations don't show that. My guess is that reducing the mall dimension just makes things you know worse, if you just because you're sort of like indiscriminately just like reducing everything. I think the The trick in all of this kind of is to find places in the model where you can squeeze. And this April, I don't think you can necessarily know for sure, you just have to do a bunch of experimentation and see what Porks. So, here's another idea to reduce your kv cache. This is called cross layer attention. `55:37` So, the idea is that normally every layer has kvuh, Ks and Vs. But we don't, let's say instead of doing that, we're just going to compute KVs for s a subset of the layers. And then just for the this this layer, I'm just gonna use the previous layers, KV cache. ![Slide 73](https://cs336.sg.aidenovak.com/media/slides/l10/073.webp) `56:01` OK, so umSo, this is kind of another way of sharing, just like GQA shares KVs across heads. Now, I'm sharing KVs across layers. `56:14` And empirically, this paper shows that you know doing this improves the prado, you know frontier, so each of these these models is better than you know, in, gi- given a method. you can always s- kind of sweep this, the size of a kv cash. By changing the the k and the head dimension. Which it's I guess kind of relates to your point about changing model dimension. And then, but if you do this k, c l a cross layer attention, it's it's better. `56:53` OK, so let me give you moving on. World one tour of different techniques for reducing kv cache. There's local or sliding window attention. This is a fairly kind of old i idea and a very kind of natural idea. So, if you look at the full attention matrix is n squared, and Instead of doing that, if you're gonna generate a token, you just look at the last k tokens. OK, so you essentially have a sliding window for every token you generate, you just depend on the last k. And now, if you do that, the the effective so, the cache is now independent of the sequence length, it's just kill, the number of the batch times, the the other variables. Which is great, and this is especially great for long context. Now, you, because of number of layers, the effective context length as actually larger than the number of the stated context length, because information can propagate umYou know farther, and if you go down the layers. ![Slide 74](https://cs336.sg.aidenovak.com/media/slides/l10/074.webp) `58:05` Now you can do fancier things like you can maybe not do a dense selection of layers, but you can Space it out. You can also do this global plus sliding window where you have attention to a fix you know grid of different you know token points, plus a local sliding window. So you can do various things. Now, the problem with this is that it actually still hurts accuracy. So, this reduces expressivity, there's no you know, free lunch here. Or at least this was an expensive lunch. So, the solution that people come up with is that they interleave local attention with global attention. So, there's hybriduh, Models have you know full attention for some of the layers, and some local layers for you know some of the other layers. `58:56` And you basically always trying to, and then, so that allows you to Essentially, you'll reduce the the cache a little bit, and you're always trying to balance you'll accuracy with speed. `59:12` Yeah. I would love to see the trade off between like a linear attention variant versus like a sliding window. Yeah, so question is what about linear attention versus sliding window attention? Soum, I'm not going to talk about linear attention, but mere very quickly. There's a bunch of methods that where instead of You know, storing a KV cash, is you basically compute some sort of like compressed representation of all the history. So linear, the most naive linear attention is you just sum the kv values up into a single vector. So that's definitely independent of the sequence length. You can do fancier things, there's like GatedNet, DeltaNets and Mamba, which allows you toum, Try to compress but not forget, you know, as much. Now, the question is, you know, how do those compare? `60:07` Those have also been used in in place of sliding window. Attention, and people have gotten good results with them. You can also use a combination of full attention, sliding window intention, and the linear attention, because they sort of capture different You know, aspects. If you care about kind of local, kind of high resolution stuff, then slightly went to intentions better. If you just kind of want like broad summaries of the past. Then the other, you know, the linear attention might be better. So that's a little long context, does that mean you say linear attention would be a better Setting for that. So, the question is for long context, will linear attention be better? Let me talk a umSo, there's no free lunch, right? `60:56` Like I think, let's say you have a very long context, and you're sort of solving a needle in a haystack problem, right? If you Have have to compress your entire history into like a small context. You're just going to lose information, and you might just You know, not be able to retrieve it. I guess my reason is, I guess I'm just like, what? It seems like people are going higher or characters that you'll always need some sort of longer attention. But I'm not, I'm just trying to understand like in my head like What the trade off between using a sliding window versus like some sort of bomb the delta net layer would be. That like is using a delta net layer like better than using sliding window consistently or like what is the actual like to be represented yeah. yeah, I I guess maybe I'll say that the mamba and delta net are more powerful than ==the sliding window attention.== Maybe you can think about the mambasum, It probably can, like, it certainly can represent some of the aspects of sliding window attention, because like you can just, as you're doing the recurrence, it can just look at the last no state. ![Slide 75](https://cs336.sg.aidenovak.com/media/slides/l10/075.webp) `61:59` so, yeah, maybe, yeah, maybe you can say think about the the linear attention, or it's like extensions has been, you know You know better. At least there they have more room. Like as once you do sliding into attention, like you're done, there's nothing else you can. Yeah. OK, so let me just quickly go through this Just highlight this of you know deepseek deepseek continues to you know innovate different types of attention mechanisms. So remember they came up with the multi layered attention, which compresses the key values. There is this thing called com-, now they have compressed sparse attention, deepseek sparse attention, and heavily compressed attention. I never kind of remember all these acronyms and what theyum, Th- they mean, but let's look at this diagram. So normally you have your kv tokens, and your your query token. `62:56` So, compressed attention is going to basically compress every m tokens into you know one token. And then there is this thing called deepseek sparse attention, whichum, basically selects a subset of those, to kind of keep. And the way you select a subset is that you actually compute some light lighter weight queries and keys. And then you do like a you know a smaller attention to get these index scores, so you know what to keep. So like a kind of a lining fast way to figure out what tokens you need to keep, and then you Use those, and then there's some more compression that, you know, happens. ![Slide 76](https://cs336.sg.aidenovak.com/media/slides/l10/076.webp) `63:39` So, in the interest time, I'll move on. The goal of this section is to reduce the KV cache, because KV cache is relate to memory, and we we saw that inferences memory bound, soum, That directly translates to improvements and throughput and latency. And the key is to do this without hurting accuracy. So you can do lower dimensional KV caches. Across layers, by across heads, and across like, you know, head dimension. You can do local attention, you can do linear attention, which was discussed earlier, And there's much more, and also there's also diffusion models, which is a non-ord aggressive way to generate, which can be much faster. ## 8. 量化：QAT / PTQ / AWQ *[64:28 – 67:21] · [▶ 从这里看视频](https://www.youtube.com/watch?v=EfM546A79aM&t=3868)* ![Slide 77](https://cs336.sg.aidenovak.com/media/slides/l10/077.webp) `64:28` OK. So, let's talk about a few other ideas which are important. So, quantization is more of a, I would say, lots of architecture and mu- much more of systems. perspective on how to make things you know smaller. So, the key idea here is just reduce the precision of numbers. Less memory means higher latency and throughput. And obviously, you have to worry about accuracy. So quantization isum, You know, there's many options here, all the way from BF16 all the way down to and and four. And so one thing you can do, if you're kind of scared that quantization is gonna mess you up, is that you train a model with quantization in mind. ![Slide 78](https://cs336.sg.aidenovak.com/media/slides/l10/078.webp) ![Slide 79](https://cs336.sg.aidenovak.com/media/slides/l10/079.webp) ![Slide 80](https://cs336.sg.aidenovak.com/media/slides/l10/080.webp) ![Slide 81](https://cs336.sg.aidenovak.com/media/slides/l10/081.webp) `65:21` So, this is called quantization aware training. And during the feed forward pass during training, you quantize and dequantize, and you're basically simulating these quantization errors as you train. So then you'll generally now the weights are adapted towards a quantization, things will work better, but the con is that requires expensive large scale training. And typically what, so what typically people do is they train models, and then they quantize after the fact. So, this is post training quantization. It's much you know cheaper often. You know, there's a naive way to do it, is that you basicallyuh, For every layer or tensor, you basically determine the scale and the zero point for each, let's say tensor, and then you you quantize that, you know, separately. This is generally, you know, doesn't work as well. You can use this idea called GPTQ, which uses some hashing information to You know quantize layer by layer, and then the you keep track of the errors which get propagated into the non quantized weights. ![Slide 82](https://cs336.sg.aidenovak.com/media/slides/l10/082.webp) `66:33` and so it kind of allows you to correct for the errors. And now activation aware quantizing is you know more soph kind of a sophisticated way where the observation is that Some activation channels are large, and those the weights that interact with those matter more. So let's allocate more precision to these weights. So let's look at this picture. So, normally, if you take FP16 weight matrix, and you quantize it, you get, let's say you quantizing down to N3. But what you're going to do is you figure out which of these you know active, so these are maybe activation channels. And some of these activation channels are large. ## 9. 剪枝与蒸馏 *[67:21 – 71:46] · [▶ 从这里看视频](https://www.youtube.com/watch?v=EfM546A79aM&t=4041)* `67:21` So as if they are large in general, then you basically allocate like let's say FP 16 for this channel. You keep everything else as in 3, and for a few important channels, you you u- use higher precision. ![Slide 83](https://cs336.sg.aidenovak.com/media/slides/l10/083.webp) `67:39` OK, so another idea here is to do model pruning. So here you just take a large model, and you rip out pieces of it. And you fix it up. It's kind of a very crude way, but it turns out to work. So there's this paper from NVIDIA where essentially Let me actually, Let's see. So you kind of ha- first have to estimate the importance of the different parts of the model, and you know choose the most important parts. And then you basically remove different, you know, hidden units and different even you know layers, umAnd now now we have a model, it's not gonna be very good. And so what you do is you post-train it, you train it some more on the data or the tasks that you care about. to kind of heal it. `68:38` okay, so this is in some sense a training, a per- su- weighted, to you know reduce the kv, the cash, but where you sort of initialize it with parts of a Of a good model. `68:53` OK, so this seems to work pretty well, so they were able to take a 15B model, and reduce it to a 8B model. And and it doesn't really hurt accuracy. By too much, and the amount that you used to kind of train the model or through this process is much less. `69:16` OK, so just to summarize here, you know the game is to reduce the inference complexity without hurting accuracy. You can think about this as mostly reducing either number parameters or KV cache. You can define a faster model architecture, train it, or you can umDefine a faster model architecture, initialize the weights from original model, which might have a different architecture, but you kind of just make this Frankenstein thing. And then you repair the faster model with distillation. ![Slide 84](https://cs336.sg.aidenovak.com/media/slides/l10/084.webp) `69:53` OK, yeah. Could you say more about how you distinguish the important layers and the unimportant layers? So how do you distinguish the important layers from the unimportant layers? So, you know, in general, you have, you know, a calibration. No set, and you pass the the the inputs through the model, and you're basically looking at, you know, the magnitude of the activations. And the ones that are, you know, some of them, especially if they're dead units will be kind of close to zero, and the ones that are large you wanna keep. So that's a high level idea. I guess like why does that matter if the applications like high? it could just be like like what if it's just always high? `70:37` really is that? So in general, this is okay, so the question is why? What if all the activations are high? So in general, this is a kind of empirical observation that some of the channels will be much higher than others. If this weren't true, then these techniques wouldn't necessarily work. But it happens to be true, because you know, that's how these models ended up being. You know, trained, and then you can exploit that. Necessarily meaningful, or like, maybe it's just an artifact of training, and I'm just like, what into it? Yeah, I see. `71:20` So the the question is, what if a neuron is always a hundred? I mean if that's the case, you can look, you can also look at kind of variants related questions. Like if it's 100, you can't just like remove it, because then everything is gonna be broken, but you can, if it's High mean and low variance, maybe there's another, you know, a way to just incorporate the bias essentially. ![Slide 85](https://cs336.sg.aidenovak.com/media/slides/l10/085.webp) `71:43` OK, let me quickly go through this other idea. ![Slide 3](https://cs336.sg.aidenovak.com/media/slides/l10/003.webp) ![Slide 4](https://cs336.sg.aidenovak.com/media/slides/l10/004.webp) ![Slide 5](https://cs336.sg.aidenovak.com/media/slides/l10/005.webp) ## 10. 投机采样（无损加速） *[71:46 – 77:19] · [▶ 从这里看视频](https://www.youtube.com/watch?v=EfM546A79aM&t=4306)* ![Slide 6](https://cs336.sg.aidenovak.com/media/slides/l10/006.webp) ![Slide 86](https://cs336.sg.aidenovak.com/media/slides/l10/086.webp) `71:46` So, so far we've looked at lossy methods, which basically kind of really crunch down the KV cash, but it could hurt accuracy. There's a very elegant way of doing this in a loss less way. This is called speculative sampling or speculative decoding. So, remember that if you're doing prefill, you can code all the tokens in parallel. And this also gives you probabilities. This is fast, this is compute bound, and all nice things. And in generation, it's one at a time. Ok, so checking Is faster than generation. If I give you a sequence, it's fast to tell me how good it is, much faster than it is to generate one at a time. ![Slide 87](https://cs336.sg.aidenovak.com/media/slides/l10/087.webp) `72:26` OK, so you can exploit this asymmetry. Using the following idea. So what we're gonna do is use a ch- a cheap draft model to basically generate from, is a guest a few tokens, let's say four tokens. And then we're gonna use the model we actually care about, the target model Q, which to basically review these tokens and accept or not accept them. OK, so so things are kind of chosen to be balanced, the draft model is smaller and cheaper, and so even though it's memory bound, and it has to generate one at a time, it's not too bad. Whereas the target model is big and expensive, but what we're doing is asking it to process a batch of tokens in parallel, so it won't also be too bad. OK, so here's the the video that shows a kind of how things work. So if you use a big model token by token, that's it's gonna be pretty slow. But if you are doing speculative decoding, then then you can see the small model, a kind of generating a bunch of tokens. `73:34` And then the large model basically you know, act critiquing them, and then so you basically can get this sort of burst of tokens, and then Maybe another burst of photons and so on so forth. `73:52` OK, so here is the the algorithm for, there's a few papers that came out with Spectrogram Decoding around the the same time, this is you know one of them. And so the idea here is that, in order to generate, we're going to generate k tokens. And we're just going to sample from this draft model. So, q is the draft, sorry, p is the draft model. We're going to sample k tokens. And then in parallel, we're going to compute the logits of these draft tokens using Q. And then now, we have to determine whether we accept or not. And this is where you do do do a bit of math, and your probability and statistics, you basically are going to Except with probability min one over this ratio of Q over P. So, if Q is umYou know, much larger than P, the larger the Q is, the more likely you want to accept it. `74:54` ![Slide 88](https://cs336.sg.aidenovak.com/media/slides/l10/088.webp) `74:57` And and otherwise, you kind of sample from this, you know, residual distribution and accept. OK, so this is basically rejection, you know, sampling. Except for in rejection sampling, sometimes you you you just when you reject you get nothing, but here we always are guaranteed to get an exact sample from the target model. I'm going to skip this you know, kind of s- simple proof. You can, it's basically the same arguments as Kind of rejection sampling to show that it's the exact uhYou know, probabilities from the target model. And the the initial paper shows that this is this fast, and generally if you look at There's a, you know, if you have too few draft tokens, you're not really leveraging the batching on the the target model side. And if you have too many, then you're going to reject more often. So there's a sweet spot around You know, in this case, three or four. ![Slide 89](https://cs336.sg.aidenovak.com/media/slides/l10/089.webp) ![Slide 90](https://cs336.sg.aidenovak.com/media/slides/l10/090.webp) ![Slide 91](https://cs336.sg.aidenovak.com/media/slides/l10/091.webp) ![Slide 92](https://cs336.sg.aidenovak.com/media/slides/l10/092.webp) ![Slide 93](https://cs336.sg.aidenovak.com/media/slides/l10/093.webp) `76:04` OK, and in general, the draft model is much smaller than the target model, and ideally you want the model draft model to be as close to target, which means thatum, you want to kind of distill it. So, which means that actually a lot of the same ideas that we just talked about are applicable to speculative decohering as well. So, basically the the idea is that let's try to reduce your kv cache, feel all the different shenanigans, and if you end up with a model you're happy with. Just serve that. If you're not happy with it, then it at least can be a draft model, and you can use your main model to fix things up. There's a bunch of whole literature on speculative decoding right now that improve over theum, Original, which I'll kind of skip for now. `76:56` OK, so very quickly now, dynamic workloads. OK, so this is the use case, is your your serving live website, and come users come and chat with your model. The requests arrive at different times. They have different shared prefixes, and they have different lengths. So, it's kind of pretty messy. ## 11. 动态负载：Continuous Batching *[77:19 – 79:48] · [▶ 从这里看视频](https://www.youtube.com/watch?v=EfM546A79aM&t=4639)* ![Slide 7](https://cs336.sg.aidenovak.com/media/slides/l10/007.webp) ![Slide 94](https://cs336.sg.aidenovak.com/media/slides/l10/094.webp) ![Slide 95](https://cs336.sg.aidenovak.com/media/slides/l10/095.webp) ![Slide 96](https://cs336.sg.aidenovak.com/media/slides/l10/096.webp) `77:19` It's far from this kind of very simple tra- training where you have these blocks of you'll the same number of tokens all at once. So what do you do in this case? So there's this umum, Idea, this is a system called Orca that was built, this is actually very early on, which introduces idea of continuous batching. So the idea is thatum, You get a bunch of requests that look like this. So, here's a prefix of the first request, in your generating this token, here's a second one, here's a third one and fourth one. It's jagged because every prefix has a different length. And what we're going to do is we're going to decode step by step. So every step, you decode one token for all the sequences. Next step, you decode another token for all the sequences. ![Slide 97](https://cs336.sg.aidenovak.com/media/slides/l10/097.webp) `78:08` And then if you end, you just, you know, you reject that, se- sequence. And then as new requests arrive to the batch, then you basically put it in the batch. And then you just kind of continue. So that's why it's called continuous batching, because it's sort of this, your batches dynamically being updated, with either old Finished sequences being evicted, and new ones coming in. So now one problem here is that, you know, everything we've seen batching works when all the sequences have the same dimensionality, right? You have tensors, everything. Every every slice has the same dimensionality, but in it, if each if request here has a different length, so what do you do about that? So there's this idea called selective batching whereum, You, let's say you have, you know, a 3, length 3, length 9, and length 5. So, here, in the attention computation, you can't really do anything about this, because attention So it depends on the the length of your your sequence. ![Slide 98](https://cs336.sg.aidenovak.com/media/slides/l10/098.webp) `79:11` So if you have a 3 by 3, you know, computation, a 9 by 9 computation, you can't really umYou'll share the ten- the tensor effectively. But for the non-attention, the MLP layers, which is you know, takes up a lot of you know flops, you can actually just concatenate All the sequences together to form a mega sequence, and you process that. `79:34` OK, so final idea is page attention. This was introduced in the VLM paper. Of course VLM has many other bells and whistles now, but this is kind of The core idea at that time. ## 12. PagedAttention 与 vLLM *[79:49 – 84:52] · [▶ 从这里看视频](https://www.youtube.com/watch?v=EfM546A79aM&t=4789)* ![Slide 99](https://cs336.sg.aidenovak.com/media/slides/l10/099.webp) ![Slide 100](https://cs336.sg.aidenovak.com/media/slides/l10/100.webp) `79:49` So, so the question is how is the kv cash stored? Right? So, if you think about umRequests coming in, the you have to put them in memory somewhere, and in general there's this problem that you get fragmentation, this is what happens to It or used to happen to your your your hard drive. And you have to defrag your hard drive, you know back in the day. So So, there's two types of fragmentation. One is that you have to allocate enough buffer so that, because you don't know a priori when you're gonna stop. So, you might have a max token limit. of like, you know, 1024. So you have to allocate all this memory, and you can't put anything in there, because you don't, you're gonna just generate until you hit the max tokens. ![Slide 101](https://cs336.sg.aidenovak.com/media/slides/l10/101.webp) `80:38` And that's very wasteful, that's internal fragmentation. And then there's also you have, there could be space between different requests, and that space is maybe too small to get used effectively, so that's just waste of space. So the solution here is just, you know, these are systems you go, so they know their operating systems. They said, OK, well, we solve this problem once before, so let's just use the same Idea here. So we're gonna divide the kv cash of a sequence into non contiguous blocks. OK, so if you have this sequence four score and seven years ago our fathers brought forth We're just gonna chunk it up into these blocks of size four. Doesn't matter where the blocks go, but they're gonna be kind of aligned according to the block, so there's some uniformity there. So, when two requests share the same, can actually share the same KV cash. So, So, you might have this block, and and this block. ![Slide 102](https://cs336.sg.aidenovak.com/media/slides/l10/102.webp) ![Slide 103](https://cs336.sg.aidenovak.com/media/slides/l10/103.webp) `81:45` So, this block might go here and here, and this block might be over there. So, they're they're in kind of interspersed, but as long as you have the indices and keep track of where everything is, that's fine. So, in particular, if you have umYou know system prompts, then you can actually just, you know, cache these system prompts. The key-value cache and the system prompts once. Right, and that can be useful for all the queries. OK, so this is very useful because if a lot of people are using the same system prompt, then you don't have to compute the KV cache for every You know, request. Also, there are many applications where you have the same prompt, and you actually want to generate multiple responses. So in that case, you can also just share the key-value cache for All the the pro, for the prompt, and just have unique responses coming out. ![Slide 104](https://cs336.sg.aidenovak.com/media/slides/l10/104.webp) ![Slide 105](https://cs336.sg.aidenovak.com/media/slides/l10/105.webp) `82:42` OK, so so for example, if you were to generate, let's say multiple generations from four score and seven years ago are blank. And so, what would happen here is that you would have four score and seven, and you would start by having years ago, or hou- hour, and then umThis is called copy on write semantics. You keep this, and then we have these two samples, and if they had happened to sample the let's say the same coconut, you just You'll just continue with that, but if they sample different tokens. you split the block, and then you can kind of continue there. So, you're basically sharing as much ca- of the prefix cash as you As you can. There's a bunch of other optimizations, like you know, kernels, that I'm not gonna have time to go over. But the general idea is that You're using kind of these operating, you know, systems, some metaphors to manage kind of your inference. OK, so summary here, inference is really really important. It's very different from training, even though it's the same model, but you're asking the model to do something very different, ends up being very memory bound, and it's also Dynamic, if you're kind of in a live chatbot use case. ![Slide 106](https://cs336.sg.aidenovak.com/media/slides/l10/106.webp) `84:06` We saw a variety of different techniques to improve inference. You can quantize, you can come up with new architectures. You can prune and distill, you can also use spective s- you know sampling. But all of these are driven by the same you know principle here, which is reduce your kv cache. But don't hurt accuracy too much. And then there's ideas from, you know, you know, s- systems, like paging and speculative execution. That can be brought to bear for actually s- live kind of inference servers. One thing we didn't really get a chance to talk about, which was discussed briefly, is that I think that new architectures have actually a huge Potential for you know improvement, things like you know state space models or linear attention or diffusion. ![Slide 8](https://cs336.sg.aidenovak.com/media/slides/l10/008.webp)