# CS336 Lecture 14: Data II（数据 2：转换、过滤、去重与配比） > [Stanford CS336 · Language Modeling from Scratch · Spring 2026](https://cs336.stanford.edu/) · [视频](https://www.youtube.com/watch?v=5sxHosTLPF8) · [官方讲义 (Trace Viewer)](https://cs336.stanford.edu/lectures/?trace=lecture_14) > 讲师：Percy Liang · 时长 85 分钟 · 本文档 = 官方讲义截图 + 全程文稿（豆包大模型录音文件识别转录，已去除语气词） ## 目录 1. 课程引入：数据的第二节 — [▶ 0:05](https://www.youtube.com/watch?v=5sxHosTLPF8&t=5) 2. 数据转换：HTML / PDF → 文本 — [▶ 1:11](https://www.youtube.com/watch?v=5sxHosTLPF8&t=71) 3. 数据过滤：以目标数据为标准的分类器 — [▶ 6:42](https://www.youtube.com/watch?v=5sxHosTLPF8&t=402) 4. 去重：精确重复与近重复 — [▶ 22:55](https://www.youtube.com/watch?v=5sxHosTLPF8&t=1375) 5. 哈希函数：碰撞与速度的权衡 — [▶ 28:18](https://www.youtube.com/watch?v=5sxHosTLPF8&t=1698) 6. 精确去重与 C4 的 3-sentence 规则 — [▶ 29:39](https://www.youtube.com/watch?v=5sxHosTLPF8&t=1779) 7. Jaccard 相似度与 MinHash — [▶ 31:26](https://www.youtube.com/watch?v=5sxHosTLPF8&t=1886) 8. LSH：用 band 结构锐化阈值 — [▶ 37:53](https://www.youtube.com/watch?v=5sxHosTLPF8&t=2273) 9. 数据配比（Data Mixing） — [▶ 49:08](https://www.youtube.com/watch?v=5sxHosTLPF8&t=2948) 10. 后训练数据与合成数据 — [▶ 73:02](https://www.youtube.com/watch?v=5sxHosTLPF8&t=4382) ## 1. 课程引入：数据的第二节 *[0:05 – 1:10] · [▶ 从这里看视频](https://www.youtube.com/watch?v=5sxHosTLPF8&t=5)* ![Slide 0](https://cs336.sg.aidenovak.com/media/slides/l14/000.webp) `0:05` OK, let's get started. So today is the second day on data. Last time we talked about how data Doesn't really just fall from the sky. You actually have to think about where it comes from. So in general, the internet consist of life services. Theseum, The data on those services have to be either dumped or crawled, and then there's additional step where you have to process the data. And we also talked about various social societal considerations. Terms of service, copyright, you have to either get a license or appeal the fair use. So, there's a lot of complexity that goes into data. ![Slide 1](https://cs336.sg.aidenovak.com/media/slides/l14/001.webp) ![Slide 2](https://cs336.sg.aidenovak.com/media/slides/l14/002.webp) `0:50` So, today we're gonna look more at the data pipeline. From transforming the data, filtering the data, deduplication and mixing. And then we're gonna top it off by talking a bit about umPost training data, and particular how folks are using synthetic data these days. ## 2. 数据转换：HTML / PDF → 文本 *[1:11 – 6:40] · [▶ 从这里看视频](https://www.youtube.com/watch?v=5sxHosTLPF8&t=71)* ![Slide 3](https://cs336.sg.aidenovak.com/media/slides/l14/003.webp) `1:11` So, the first part is going to be mostly about pre training, that's what you should have in mind. So, we talked a little bit about data transformation already, but just as a kind reminder. So, raw data doesn't come as text, even as if you've You've scraped something. If you ever look at inside common crawl, it's it's not text, it's either HTML, sometimes it could be PDFs, or directories, and the case of GitHub. So, most of the attention on transforming data is dealing with HTML, because most of the web is in HTML, and for this processing There's a lot of this is fairly heuristic. There's removing of boilerplate, like navigation and ads, and extracting content. which is like the main part of the page. And there's some, you know, subtleties around what con- constitutes content. Usually, you get rid of the footers and headers, And maybe menus and those things, and you try try to extract the content. `2:19` But you could imagine in cases where some of the the navigation elements might be might be helpful to learn what web pages look like. So, what is content and what is not content is not Always clear. And then, what do you do about images and tables are on in web pages? So, inherently this is a flossy process, because you need to linearize HTML which at least is either hierarchical or visual, if you think about the render output, and do a sequence of tokens. And particular tables are a bit tricky to deal with. Simple tables you can render using markdown. But if you have nested tables, then that becomes quite challenging. You have to give ups at some point. or approximate at some point. ![Slide 4](https://cs336.sg.aidenovak.com/media/slides/l14/004.webp) `3:10` So, typically, HTML to text processing is is rule based, and the reason for this is that rule base processors are very Fast, and and also, they, you're not trying to do too much here. That is, you don't need too much intelligence, so rule based Generally works. Now, I think there could be a case for model based interventions at this point. They have to be very fast, and they have to do something more more intelligent. But if you ever look at data, you'll notice there are imperfections in the data, just because any rule based processing is going to be is have some failure rate. as we showed last time, the accuracy does matter, depending on which tool you choose. And reviews to Brazil parts or travel tura, then it's, you know, I guess travel Brazil parts on these extended of dclm, you know, evals, works. Better, than the others. `4:19` I'll talk also briefly about PDF, so there's this hugging face work which released a data set called fine PDF. So PDF files, if you ever open them up, not in a umPDF reader, looks looks kind of like this. And this needs to get kind of rendered. So, PDFs can be found on just like web pages on the the internets, and common crawlum, Does have, you know, some pdfs? Generally, common crawl focus on text, but sometimes, if you're just given a URL, you might not even know before you fetch it whether it's a pdf or or not. If it doesn't have the extension. There's a lot, if you read this blog post, there's a lot of details which I'll spare you of. For example, one thing is that many of the PDF that are in Common Crawl are truncated, because PDF are big. So then that re demands that re-crawling is necessary. ![Slide 5](https://cs336.sg.aidenovak.com/media/slides/l14/005.webp) `5:19` And then there's a question once you have the PDF file, how do you actually convert it to text? And You know, some pdfs are might be just scans as well, right? So they're essentially images. So there's a bunch of tools that this paper tried out But mostly this kind of involves running OCR, using a VLM. And this obviously can be much more expensive than what we were doing before with text. Fortunately, PDF or unfortunately, the PDFs are a very small fraction of the whole internet, but they are very valuable, because generally if you bother to make a PDF. That means you probably have something interesting to say, as opposed to a web page. So the quality of a PDF is average PDF is generally higher thanuh, For for a HTML file, there's a lot of cleanup and filtering, and PDF's more so even than than web pages. A lot of layout information is missing, because in HTML you have various tags like H one and P that gives you some semantic information, PDFs Are by design all about layout, so you don't necessary preserve the the kind of the semantic structure. ## 3. 数据过滤：以目标数据为标准的分类器 *[6:42 – 22:52] · [▶ 从这里看视频](https://www.youtube.com/watch?v=5sxHosTLPF8&t=402)* ![Slide 6](https://cs336.sg.aidenovak.com/media/slides/l14/006.webp) `6:42` OK, so there's a bunch of transformation that happens. At this point, you have text, and, but you're not done, you're far from done. So, the next step is filtering. So, I'm gonna talk somewhat abstractly about what filtering is. Here's a sort of the building block. So, suppose you have some target data that you want to get. This is usually a small amount of high quality data, and lots of rod data. this is the the the fresh shipment of the the tokens that you do transform from the the previous step. and the goal is to find a a subset of this raw Data that is similar to target. ![Slide 7](https://cs336.sg.aidenovak.com/media/slides/l14/007.webp) `7:30` So, this is the general skeleton for filtering, and may al- almost all kind of filtering falls into this thisuh, Schema. So filtering, you could, there's many reasons you might want to filter, if you're training an English language model, or German language model, you might want to identify the language and filter Things that don't match that that language. The main reas- reason for filtering is is quality filtering. You want to find things that are high quality as opposed to low quality. You don't want just you know spam, you want umencyclopedic information, and and then another application is toxicity filtering. Of course the internet has plenty of nasty content, and maybe you don't want to train your language model. On that. ![Slide 8](https://cs336.sg.aidenovak.com/media/slides/l14/008.webp) `8:19` OK, so for a filtering algorithm, you want that you want some generalization from the target data, right? Because you already have the target data, you don't want to just only get the target data. But you also want it to be extremely fast, because you have to run it on the whole internet. So this could be 100 trillion tokens worth of worth of data. So generally filtering is, you end up with a very small fraction, single digits fraction of your entire data. ![Slide 9](https://cs336.sg.aidenovak.com/media/slides/l14/009.webp) ![Slide 10](https://cs336.sg.aidenovak.com/media/slides/l14/010.webp) `8:54` OK, so so remember the general framework, given the target, and raw, you're trying to find a subset. OK, so the general scheme is that You f- estimate some sort of model based on RNT, and derive a scoring function, and then you keep the examples in our based on their score. So, typically there are two types of classifiers, one are generative models, So, this is basically you have your target data, you can just you know estimate a model of that data. and re- remember this has to be cheap, so probably you're not training a big language model. generally, kenlm basically says I'm gonna train a 5 gram. model. A more common thing, which I think we're mostly seen these days is just train a classifier, and the classifier says I'm going to predict positive label for examples that are in T, my target, and negative labels for things that are in raw. But not in T. So you you basically it looks like T are your positive examples, some random subset of R are your negative exam examples. `10:11` You maybe balance it, and then you train a classifier. ![Slide 11](https://cs336.sg.aidenovak.com/media/slides/l14/011.webp) `10:15` And the tool that people generally use is fast text, because it's fast, and it's generally, it's just a linear classifier, bag of words. And then once you have this model, now for every new document, you can score it, and you set some appropriate threshold depending on your quality bar, and you keep the examples sometimes sarcastically, sometimes not. ![Slide 12](https://cs336.sg.aidenovak.com/media/slides/l14/012.webp) `10:41` OK, so so this is a m- very much a model based filtering approach. Remember from the last lecture, there are many data sets from You know, generally a few years ago that did not use model based filtering, because they wanted to not bias things too much. These days, I think basically everyone does some amount of model based filtering, becauseum, Unless you are compute plentiful, which in which case you probably don't need to do as much training, you can just train on everything. Most people are compute poor. You have to be very smart about how you're filtering, otherwise you're just wasting FLOPs on kind of low quality content. ![Slide 13](https://cs336.sg.aidenovak.com/media/slides/l14/013.webp) `11:27` OK, so let me talk through some instantiations of of filtering. So, remember I mentioned language identification, so the goal is just to find Given a piece of text, detect whether it's of a particular language. So, Meta has trained the set of FastText language identification models, which often you can just use off the shelf. So, supports 176 different languages. It's been trained on a bunch of multilingual sites, so Wikipedia remember has a lot of languages. There's also translation sites, and sites for different languages, soum, A language idea is is a generally a fairly easy problem compared to many other tasks that we're dealing with. So, a simple classifier, if you look at a few words, you can Tell that it's Spanish or or or Japanese. Now, there does, you know, there are subtleties because of code switching and text, and there's some dialects. So, it's, I wouldn't say it's like, absolutely solved problem, but you know, this is not really the bottleneck for training a good language model. `12:34` OK, and then you can just, so once you have your classifier, you just choose some threshold. This is generally fairly heuristic. ![Slide 14](https://cs336.sg.aidenovak.com/media/slides/l14/014.webp) `12:43` So, another thing you can do with filtering is that suppose you're looking for data of a certain type. So, let's say you want to get really good at math, so you gonna want to go and find a bunch of math data. So, open math text is this paper from 2023 where the goal is to curate a large corpus of math text. This pipeline is consists of a few steps, So it's not training a single classifier. So first you use some rules to filter, does it contain latex commands? You, they also use CAN alarm, so this is a generative approach. And trained it on the proof file, which is a known data set of math, and keep it if the perplexity is you know below some threshold. And then, you also train a fast text classifier to predict whether it's mathematical writing, or not. And there's, you know, two different thresholds. `13:41` If it has LaTeX in it, then it's a higher bar if it'suh, Sorry, if it's it's a lower bar, but if it's doesn't have the tech in it, it's a higher bar. And then as a result, they got you know 15 billion tokens. Which they used to train models. And as a paper they show that this more targeted to security data collection resulted models that are better than math, at math than Models that were trained on twenty times much data, which were not filtered in this way. ![Slide 15](https://cs336.sg.aidenovak.com/media/slides/l14/015.webp) ![Slide 16](https://cs336.sg.aidenovak.com/media/slides/l14/016.webp) ![Slide 17](https://cs336.sg.aidenovak.com/media/slides/l14/017.webp) `14:13` OK, so the co-, the quality filtering again, think about as a tool. You can define quality however you want. There's no universal notion of quality. If you want defining quality to be math, then you can go and get math, and you get better at it. So GPT three, I think I mentioned this last time, but just to put it in this framework. So positive examples are Wikipedia, web text, so these are a pages thatum, Link out from high you know star Reddit posts and some books. And negatives are generally sampled from the the web. They train a linear classifier and keep documents if the linear classifier scored it highly enough. There is the first llama paper deposits were pages reference by Wikipedia, not Wikipedia articles themselves. ![Slide 18](https://cs336.sg.aidenovak.com/media/slides/l14/018.webp) `15:02` And the the it's just kind of same idea. So, 51 from Microsoft, is interestingly had you also falls into this framework. so, they started with the raw data is not al- already a python subset of the stack remember which is this all You know, code, and they, they're the the prompt. they define a prompt, which is determines educational value. And then they use GPT four to classify a subset of R, 100K subset of the R with this prompt. And see if it's po-, and the ones that are positive are kept and called the target. Right, so the target here is actually output of an expensive umYou know, classifier. And then you train a cheaper classifier in their case a random forest, but you could have probably use a fast text as as well. And and then you select data that is umClassify positive by this this classifier. `16:17` And they also show that using this data set, you do much better than if you're just usingum, the raw data. So this performance goes up in fewer steps to a higher value. ![Slide 19](https://cs336.sg.aidenovak.com/media/slides/l14/019.webp) `16:33` OK, so toxicity can works the the sa- same way. There's the status said, Jack saw toxic comments. Where it came from this project where the goal is to help people have better discussions online. So, here the data is that Wikipedia talk pages, which can I guess for Controversial sites get quite heated, and this has been annotated with whether there are any any toxic comments on there. and you can similarly define positive and negative examples, and you can train a classifier on that. `17:08` OK, so so now, I think you have the tools, or is some examples, and inspiration, you can identify a particular Type of data that you want, you build a classifier for it, and then you can go filter common crawl for that. I want to talk about one su- subtlety here, which is that, you know, the notion of what you want for data is actually depends on what model you want to train. So in particular, it depends on the number of tokens you're training on. And so, there's no optimal threshold. So, before when I looked at the the the classifier gives you a score. There's, you can't say point nine and say that's the best, because it depends on what you wanna do. Intuitively, if you are gonna train for a longer period of time. Then you can tolerate lower quality data. If you're training for shorter, then you you want higher quality data in general. `18:11` And of course, you know, if if you could wave of magic one, if you're training for longer, you want more high quality data, but that's not the option that you're given the data pool is what it is. `18:24` So, here is in a kind of a preliminary experiment, that Michael Ryan did, and so basically this plot shows for a 157 million parameter model. We're taking a very small s- pool, so 100 you know works, which is a tiny fraction of Common Crawl. And we're look training and training. For more uhOver time, so let's take a look at the blue curve. So the blue curve is DCLM, and so the lo- the loss starts here, and then it comes down, and each of these lines sh- is when You epoch over the data. So this is one epoch, and then the next blue is the second epoch, and so on and so forth. So there's not that much data, so eventually you have to repeat your data, andum, You know, the loss continues going down, because the second time you look at the data, you're still learning things, and at some point you start to to overfit. OK, whereas if you look at resilient parts, this is basically no filtering. So here, it's it's much worse in the beginning, but as you Continue and continue, at some point you also e park, it starts to go down, you know, slower. `19:44` So you kind of see this trend where high quality data Is better in this regime where you're not epoch in, but once you get to lots and lots of tokens, high quality data is no longer That great. And of course, you with high quality data, you you wouldn't even get, you wouldn't want to go into this regime because you're overfitting, you'll probably stop here. But even this, at this point, this is worse than If you had trained for longer using low quality data. `20:19` Yeah. Like each of those dogs correspond with one training one, but one training run, right? Yes. Do you ever need to do like conference in our home when you're doing these kind of retraining experiments? Or like, is it kind of And not just do it one time. So, the question is each of these points is a a single training run, and should you do it multiple times to get confidence intervals? Ideally, that would be good practice. Often you'll see in these papers that these are kind of scarce, because each training run is fairly expensive. ![Slide 20](https://cs336.sg.aidenovak.com/media/slides/l14/020.webp) `20:56` So, you, but but in reality, when we have done these experiments, it's generally tends to be stable, I would say. At least for pre training. Yeah. Yeah. So you mentioned like training for longer at high volume data. Is that's not like a combination that we're considering. But like like if if we are training with uhFor longer on this high quality data. Would it have diminishing returns compared to like training for longer with the lower quality data? Yeah, so the question is if you were able to get higher quality data, and you train for longer. `21:35` Would it still have emission returns? Every data set is gonna have emission returns, if eventually, that that's finite. But it would just, it would probably be here, it would just like be down here, and just just keep on going down. ![Slide 21](https://cs336.sg.aidenovak.com/media/slides/l14/021.webp) `21:53` OK, great. Let's let's move on. So, summary, filtering is is pretty critical for building a good model, especially when you're compute constrained like most of us. Technically, if you have infinite compute, you don't need a filter, and you can train on everything, and it'll be a giant model. But the realistically, everyone has to filter. So, and the recipe here is figure out what good data looks like, and then you can train a classifier, and that'll extrapolate to the rest of your model. And what the good data looks like, you can either find it by saying, huh, there's this data set out there that I really like, and I just want more of it, or you can craft a prompt. To a language model, and then use that to construct, to do a sort of a preliminary filter of a large pool, and then use that good quality to train a smaller classifier and extrapolate to everyone. House. ## 4. 去重：精确重复与近重复 *[22:55 – 28:17] · [▶ 从这里看视频](https://www.youtube.com/watch?v=5sxHosTLPF8&t=1375)* ![Slide 22](https://cs336.sg.aidenovak.com/media/slides/l14/022.webp) `22:55` OK, next, I'm going to talk about deduplication. So, at this point, we have filtered our data set, so we only have what we deem to be high quality data. But, you know Often data still has duplicates in it, and there's, you know, two types of duplicates, there's exact duplicates. So, this happens when for example, if you look at these mirror sites, mi-, the whole point of a mirror is that it's a duplicate, and you know sometimes the web crawler isn't smarted enough to know that this mirror is exactly this mirror. so it just crawlsum, Many sites, and you'll get this exact same content. Also, when you fork a repo, that is also umA A duplicate, even if you make changes, probably you're making changes to a few files, so 99 percent of that repo might be the same. So duplicates is is abundant. And yeah, sometimes there's actually near duplicates which are not mirrors or derivative, but they just happen to beuh, Same text differing by a few tokens. Usually, probably this came from a kind of a it it could be copying or came from a different common source. ![Slide 23](https://cs336.sg.aidenovak.com/media/slides/l14/023.webp) `24:18` So, here's some examples of near you know duplicates. So, terms of service and licenses. So, I guess the MIT license probably shows up. You know, on a lot of places, and so many cases, it is a exactum, copy, but only of that license, unless someone made a typo when they copied it. But the rest of a page that the license might be part of is might be different. And we talked about how a lot of websites have the same headers and footers. Those are also you know duplicates. There's also cases where for whatever reason umYou know, there's this is from LM one B, so the one billion per benchmark, there's like these articles where you just have, you know, typographical differences. like there's one version with a comma, and one version without a comma. ![Slide 24](https://cs336.sg.aidenovak.com/media/slides/l14/024.webp) `25:15` I don't really know why, but that's happens, and sometimes you see these templates where, there this is like some, you know, probably low quality content where it's like a It's a essentially looks like an ad or some sort. and someone just you know templatize replace Canada with USA. So, if you train Like on this data, but you are just training on different variations of this with different entities. This is gonna be wasting your GPU s. There is a more extreme cases, so this is why it's good to look at your data. So this audit of the C four data set. Found this product description sixty one thousand times in the data set. And if you trace back, this is really bizarre. I don't know why, this is like someum, Description of this gas mask, and this just showed up in this data set, in Common Crawl 61 thousand times. ![Slide 25](https://cs336.sg.aidenovak.com/media/slides/l14/025.webp) `26:12` So, Yeah, the web is weird. So, why do duplicate? So, the first clear idea is clear, because you want to train more efficiently. Duplication reduces your data set size without Really losing information, because you're just removing duplicates. And then also has this benefit of avoiding memorization, which this paper, you know, talks about. For example, if you have some copy area content that's duplicated a lot, then if you train on it, then you memorize it, and also privacy concerns. But mostly, I think it's just to re-, make sure that you're not wasting flops. Related to due duplication, that's also decontamination, which is arguably even more important to, and it's the same sort of deal, is that you want to to make sure that Your test set is not in your training set. ![Slide 26](https://cs336.sg.aidenovak.com/media/slides/l14/026.webp) `27:14` OK, so so how do we debug? So here's the design space to think about. So first of all, what are the items that you're debugging? Do you do at the sentence level, the paragraph level, or the document level? How do you determine a match? Is it exact match? Is it existence of a common sub item? Is it the fraction of common sub items for near duplication? And then once you found a duplicate between two pages, what do you do? `27:43` Do you remove all the instances, or do you remove all but one? ![Slide 27](https://cs336.sg.aidenovak.com/media/slides/l14/027.webp) `27:48` OK, so and the key s- algorithmic challenge is that the duplication is fundamentally about comparing items to other items. `28:00` And Normally, if you're doing filtering, this is about an i- individual item. This item is a good or not, and this can be paralyzed, it can be, you know, it's sort linear time, which is good. And even in linear time, we're trying to make it fast by having rule based or very small models. ## 5. 哈希函数：碰撞与速度的权衡 *[28:18 – 29:37] · [▶ 从这里看视频](https://www.youtube.com/watch?v=5sxHosTLPF8&t=1698)* ![Slide 28](https://cs336.sg.aidenovak.com/media/slides/l14/028.webp) `28:18` But due duplication, clearly you can't do the n squared thing where you compare everything to everything. You need linear times algorithms to scale, especially at this web scale. So, typically the de-duplication literature uses hash functions to get around this. So, we'll develop some of these ideas. These are Quite nice ideas, from thanks to you know the algorithms community. So, I think everyone knows what a hash function is, it takes of some sort of value like a string. And maps it into something like a string or integer. And the hash value is much smaller than the item. Hash collisions are when two distinct items map to the same. ![Slide 29](https://cs336.sg.aidenovak.com/media/slides/l14/029.webp) ![Slide 30](https://cs336.sg.aidenovak.com/media/slides/l14/030.webp) `29:03` That h, okay? So, when you look at hash functions, there are really fancy hash functions which are cryptographic uhIn nature, there's a collision resistance using cri- cryptography and and Bitcoin and things like that. And then there is these like faster ones which are used for hash tables where hash Collisions aren't the end of the world, and so we'll be using these. Ok, so take a string and map it to some value, that's what umFunctionster. ## 6. 精确去重与 C4 的 3-sentence 规则 *[29:39 – 31:25] · [▶ 从这里看视频](https://www.youtube.com/watch?v=5sxHosTLPF8&t=1779)* ![Slide 31](https://cs336.sg.aidenovak.com/media/slides/l14/031.webp) ![Slide 32](https://cs336.sg.aidenovak.com/media/slides/l14/032.webp) `29:39` OK, so exact duplication is is s- conceptually very simple. You you take a string, you see if there is exact match, and you remove all but one. OK, so here you have a bunch of elements, and are you you hash them, and you deduplicate. OK, so this is and exact duplication is Is is very nice. It's very clear what's happening. But this isn't really good enough for the messy web data because often you have these near duplicates. And one note is that there's many ways to have you know written this. This is written by in a sort of like a bit of a map reduce style way, which makes it more easy to easily paralyzable and scale. So C four, this this paper from the the T five paper, which process common crawl did exactly duplication. ![Slide 33](https://cs336.sg.aidenovak.com/media/slides/l14/033.webp) `30:41` The items they operate on were three s- sentence spans. And you do exact match, and they remove all but one. So, one, if you're kind paying attention, you realize that there's something kind of s- s- bit strange about this, because you're looking at three sentence spans. And if you find two documents with three sentence span, remove all but one. That means you're just gonna rip out three sentences from that documents. Which is a little bit s- strange, because it breaks the coherence, but you know, this is what they did. `31:21` OK, so that's exact duplica- duplication. ## 7. Jaccard 相似度与 MinHash *[31:26 – 37:52] · [▶ 从这里看视频](https://www.youtube.com/watch?v=5sxHosTLPF8&t=1886)* ![Slide 34](https://cs336.sg.aidenovak.com/media/slides/l14/034.webp) ![Slide 35](https://cs336.sg.aidenovak.com/media/slides/l14/035.webp) `31:26` So how do we do near duplication? So first of all, we have to define what approximateuh, match means here. okay so to do that we're going to define this called Jaccard similarity which is a fairly standard notion And笛卡尔 of two sets is basically the size of the intersection over the size of the union OK so given these two sets 1234 and。 1235. Com- when you compute Jaccard, you take the intersection, that's 123. You take the union, that's 12345, and you divide, and the Jaccard is 0.6. ![Slide 36](https://cs336.sg.aidenovak.com/media/slides/l14/036.webp) ![Slide 37](https://cs336.sg.aidenovak.com/media/slides/l14/037.webp) `32:08` OK, so the Jaccard is a number between 0 and 1. 0 means that they're disjoint, 1 means that they're identical. OK, so a very f- natural notion. And we're going to say that two documents are near duplicates, if their Jaccard similarity is above some threshold, let's say 0.99. So now, the question is how do you find near duplicates in linear time? OK, so fortunately this is a solved you know algorithms a question, and the answer is to use minhash. So, Well, OK, so the first step is to use minhash. That's not the final answer. ![Slide 38](https://cs336.sg.aidenovak.com/media/slides/l14/038.webp) ![Slide 39](https://cs336.sg.aidenovak.com/media/slides/l14/039.webp) `32:50` So minhash is a random hash function, so that the probability of a hash collision is exactlyuh, the jaccard of A and B. So, this is a very kind of nice property because hash functions, hashing is good for making things in linear time. Jaccard is the metric that we want, and we're sort of connecting these two right now, or in expectation. So what's interesting is normally. You want hash functions to define hash functions so that everything is hashing to different, distinct elements are hashing to different things. You don't want collisions. But here you actually want collisions, not arbitrary collisions, but you want to control the collisions in a certain way to align with a similarity. So similar things you want to collide more. Then the disparate things. ![Slide 40](https://cs336.sg.aidenovak.com/media/slides/l14/040.webp) `33:44` OK, so so the minhash is essentially you actually, let's see, maybe I will OK, so here's the the min hash. It's fairly simple, you take a set, and you hash every element in that set, and you take the minimum element. OK, so, this mean might see, if you're seeing this for the first time, it may seem a little bit strange, why are you taking them in? You can take a max too, doesn't really matter, it's just a way to break ties. So, here's the the picture that you should have in your mind. So, OK, so if you have umum, A, remember A contains 1234, and B contains 1235. So, this is a characteristic matrix representation of these two sets. Right? And so what the random hash function is is doing is that it induces a permutation over the items. ![Slide 41](https://cs336.sg.aidenovak.com/media/slides/l14/041.webp) `34:47` OK, so might be 4 3 1 5 2, or something else. And then you look at which item is first according to this permutation in A, and which item is first in B. `35:00` So, each item has the same probability of being the first. Ok, so if you have, if it the random hash function puts One first, then first in A will be equal to first in B, and if it's two first, then it's the same, if it's three first. it's also the same, but if it the random permutation puts four first Then the first element in A is going to be different from the first element in B, and same with five. `35:39` OK, so, look at this representation. Basically, the random hash function says, elect one of these rows to be you know first, and then when I The mean is just telling me, is the mean the f- the f- the mean equal of A equal the mean of B is basically telling me whether That row has is is the same. OK. so that is the, I guess the proof of why min hash, why this property holds, which is that you have a hash function. In expectation over your random choice of hash functions, that the probability of collision is the similarity metric you want. `36:30` OK, so let's just ch- check this and code. So, I'm going to generate 100 different hash your functions. Each hash function is given by a seed. And I just check whether the min hash is equal to the min hash, and the estimated Jaccard. So, I'm going to just look at the number of of the fraction of batches, and you get 0.6. `36:56` OK, and the key thing with the min hash is that I don't have to do the the and s square thing, I can compute The min hash of A and the min hash of B. and min hash of a different set, and I just look for collisions. `37:16` OK, any questions so so far about Ben Hash? ![Slide 42](https://cs336.sg.aidenovak.com/media/slides/l14/042.webp) `37:29` So, now we can hash our items, but we're not done yet, because a collision doesn't tell us that Jaccard is above some threshold. Which is what we want. We want to find A and B such that Jaccard is greater than 0.99. All we've done is said, OK, well, if we got a collision, the probability of that collision of two things colliding is the Jaccard. ## 8. LSH：用 band 结构锐化阈值 *[37:53 – 49:07] · [▶ 从这里看视频](https://www.youtube.com/watch?v=5sxHosTLPF8&t=2273)* `37:53` But that's not really that useful by itself. It's sarcastic, and I can't really get anything reliable here. ![Slide 43](https://cs336.sg.aidenovak.com/media/slides/l14/043.webp) ![Slide 44](https://cs336.sg.aidenovak.com/media/slides/l14/044.webp) ![Slide 45](https://cs336.sg.aidenovak.com/media/slides/l14/045.webp) `38:01` So, the next thing idea is this thing called locality sensitive hashing, which essentially solves this problem. So, this is a very classic idea in theoretical computer science, and it's quite nice. So, the idea here is that we have A and B colliding with probability equal to Jaccard. OK, so it is true that more similar items will collide more often. But it's very stochastic, like the variance is you know quite large here. So our goal is to have A and B collide if the Jaccard is greater than some threshold. So we in some in some sense have to sharpen these these probabilities somehow. Like the probability can't just be literally equal to the to the card. ![Slide 46](https://cs336.sg.aidenovak.com/media/slides/l14/046.webp) `38:53` So, the solution is to use more hash functions. And these hash functions are going to be independent. So, this part is, you know Gets a bit technical, but we'll walk through it. So, you break the N hash functions into B bands of R hash functions. So, if you have 12 hash functions You have three bands, each band has r, four hash functions. OK, so, if there's a band here, h one to h four, here's a second band, h five to h eight. Third band H9 through H12. ![Slide 47](https://cs336.sg.aidenovak.com/media/slides/l14/047.webp) `39:32` OK, and then so what we're gonna try to do, so so each hash function gives us either collision or not collision. OK, and so every hash function there's a probability of uhcolliding. So the key here is that we want to say compute the probability that A and B collide. We say that A and B collide if for some band, all of its hash functions return the same value. `40:06` OK, so if h five and h six and h seven and h eight all return the kind of the same value. ![Slide 48](https://cs336.sg.aidenovak.com/media/slides/l14/048.webp) `40:18` If H A of A equals H 5 of so, sorry, H 5 of A equals H 5 of B, and H 6 of A equals H 6 of B, and so on. That means this band is triggered, and then I would say they collide, or if this band, if all these say collide, agree, then I say it's collision, or these two. But you don't have to have all the hash functions, you know, return the same value. `40:49` OK, so there's sort of this and/or structure here that is is is doing the the lifting here. And we'll see why this this works. `41:01` So, so now, let's say you have a particular Jaccard of A and and B. What is the probability that A and B collide? According to this definition. So we can calculate this. So let's say that this Jaccard similarity is 0.8. Ok, and we have five you know bands, and each band has 10 hash functions. So then, we can say what is the probability of a fixed band matching? So that's 0.8 to the r, because each band has r hash functions. `41:50` So the probability that they all have to match is 0.8 to the r. OK, so the probability of a band, a a fixed band matching is is generally Quite low, it's exponential in r. ![Slide 49](https://cs336.sg.aidenovak.com/media/slides/l14/049.webp) `42:04` OK, and then now the probability of collision is a probability that some band matches, and that's going to be the probability of one minus the probability of So, this is a probability of a band not matching. and this, if you raise it to the B, there's B bands, this is a probability that All of them don't match, and then one minus that is the probability that some match. `42:32` OK, so the probability collision is point, you know, four. And you expected to be higher because you're given, you know, beach tries to get a match. `42:44` OK. `42:47` So, if you plot this, this is what it looks like. OK, so you have on the x axis, the similarity here. And we were looking at a zero point eight. And and you look at the probability of a a collision here. So we would expect that if the similarity is zero, then it should be zero. If it's one, it's one. But notice that this is kind of interesting s shaped, which is is nice, because this is what we wanted to sharpen. We wanted to basically say if the similarity is Below some threshold, then we want the probability to match to be as low as possible. And if the similarity is above some threshold, we wanted to be as high as possible. `43:34` So we're trying to get this to be like a phase transition. OK, and if you you know plot this this function as a function of similarity, that's what you get. `43:50` OK, so, OK, so. `44:01` So, let's just look at some example. So, let's concretely. ![Slide 50](https://cs336.sg.aidenovak.com/media/slides/l14/050.webp) `44:07` We have similarities 0.7 to 0.98. OK? And we're going to look at how b equals ten or equals ten. So, we look at the collision probability according to our definition, and we see that this gives us a range. Of you know point two five to to one. OK, so if you were to set a threshold, you know this is this is not you know bad. We can like set it here, and you know these with umYou know some probability will you know, there's still false positives, because even if these are below a threshold, let's say 0.9, these might still collide. ![Slide 51](https://cs336.sg.aidenovak.com/media/slides/l14/051.webp) `44:55` But we can filter them out, if we if we want, but the ones above are mostly kept. ![Slide 52](https://cs336.sg.aidenovak.com/media/slides/l14/052.webp) `45:04` So, what happens when you move? So, increasing R. So remember R is a number of hash functions within a bucket. So, when you increase R, the The threshold sharpens and moves the curve to the right. Everything becomes harder to match right? Because you have more hash functions inside of it. Okay, you have the the sharpens that exponent. So, the probability you see you are kind of sharper here, and now everything moves to to the right. And now, it's fairly unlikely that if something has logic card you're going to To match it. ![Slide 53](https://cs336.sg.aidenovak.com/media/slides/l14/053.webp) `45:43` So before it was like point two f five, and now it's you know point zero zero eight. OK, so we're sharpening the probabilities, and then increasing B. Has the effect of moving the c curve to the left. It makes it easier match. B is the number of bands, and some band has to match. If you have more bands, then there's more chances of matching. So, if you look at this, then before and after. So now, we're making before this point nine was only point seven two, and now it's point nine two. And of course, this this also increases, but you know, not by that much. ![Slide 54](https://cs336.sg.aidenovak.com/media/slides/l14/054.webp) `46:23` OK, so, and you can drive these face transition to be as you know as sharp as you want by increasing B and R. But of course, if you could increase B and R too much, then It can be more expensive. ![Slide 55](https://cs336.sg.aidenovak.com/media/slides/l14/055.webp) ![Slide 56](https://cs336.sg.aidenovak.com/media/slides/l14/056.webp) `46:38` OK, so let's look at a more real world's a s a setting. So in this paper, on the duplication, the they had umYou'll be 20 bands, and R is 450. So give you an idea of the magnitude of these these numbers. umAnd in general, the space transition happens at the at the threshold, which is one over b raised to power one over r. So, if you want to find a, you know, if you want to filter based on Jaccard greater than 0.9, then you basically have to set BNR such that this is 0.9. And then by changing b and r, you can make that face transition sharper. ![Slide 57](https://cs336.sg.aidenovak.com/media/slides/l14/057.webp) ![Slide 58](https://cs336.sg.aidenovak.com/media/slides/l14/058.webp) `47:29` OK, so remember the probability of the fixed band matches is one over B. If you have at this threshold, then the probability that A and B collide is you know, approximately, it's a it's a constant, so the probability of collision is point six four. Ok, and this this makes sense, it basically you if you have a face transition, the sort of center of that face transition is like point, you know, 64 and everything below it should go to zero as BNR increases andum. Everything above it should go to one as BNR increases. `48:10` OK, any any questions about LSH? So this this method is called MinHash LSH, because LSH really works for any hash functions for the duplication language model processing, we're using The min hash, which approximates your card, and so, that uhThat basically is the one you should yeah use here. `48:40` Questions about Dedupe. `48:55` So one note about de-dupe is that there's often as we'll see data sets that are coming in, and sometimes the duplication will happen within a data set. ## 9. 数据配比（Data Mixing） *[49:08 – 73:02] · [▶ 从这里看视频](https://www.youtube.com/watch?v=5sxHosTLPF8&t=2948)* ![Slide 59](https://cs336.sg.aidenovak.com/media/slides/l14/059.webp) `49:08` But you actually have to do du duplication across your entire the data set, because often data sets can be redundant. So, you know, sometimes that's not done, but it should be. OK, let's go on to the next topic, which is data mixing. OK, so so far we've transformed data from Really raw HTML or PDFs into text. We filter for high quality. We dedupped, so we have a smaller set of high quality. documents, and umAnd this generally happens for a given, you know, data source. But language models are trained on multiple data sources. So, in Marin, currently, this this website. ![Slide 60](https://cs336.sg.aidenovak.com/media/slides/l14/060.webp) ![Slide 61](https://cs336.sg.aidenovak.com/media/slides/l14/061.webp) `50:00` tracks the different data sources that the next model will be trained on, and you can see that there's bunch of things, there's some neuromorphic, there's Fine, PDS, which we talked about. There's some you know, institutional books. And some code and all these things. OK, so you have all these different, you know, sources. And the question is How do you combine these? So if you look at an older paper, the pile, this is the Set of sources that they had at that time. And they essentially assign a particular weight to each component. So basically you have a distribution of resources. OK, and so where does that way come from? ![Slide 62](https://cs336.sg.aidenovak.com/media/slides/l14/062.webp) `51:02` So, for more, you know, just cut and dry. Let's say you have three sources, you want to find a data mixture, and a data mixture is just a distribution over your sources. ![Slide 63](https://cs336.sg.aidenovak.com/media/slides/l14/063.webp) `51:15` OK, so if you're just thinking about this problem from first principles, well, not from first principles, but like just from what would you, what you might do. vibes, which is, I guess, opposite of first principles, is you just manually set it based on some intuition, which is more often than you might think what people do. Right, so I think this is this is definitely fairly vibe spaced. And even more recent papers, you know you just look at Yeah, maybe you would use some method, and then you just like tweak things. You can also do uniform sampling, which is you just put a uniform distribution over your sources. And you sample every the chunk you you get, you just sample a a source. You can also do a proportional mixing. Which is that you sample proportional to the number of tokens in a source. So, if you have a data set with more, you know, tokens, then you put a higher weight. `52:22` So this is also, you know, generally a rational thing to do. But you might worry that the, if you have a huge low quality data set, that's gonna eat up a lot of your token. So Doesn't seem quite optimal either. ![Slide 64](https://cs336.sg.aidenovak.com/media/slides/l14/064.webp) `52:35` So, intuitively, you should upgrade your higher quality sources. OK, but there's two things that are important to keep in mind. One is that you do want to ensure some diversity, right? So, often sources are incomparable for literature code and papers. You can't really say that you know this paper is higher quality than this code, because they're just incomparable. You know, objects maybe, and if you want your language model to do well, you don't want to just put all your your mass on just papers. ![Slide 65](https://cs336.sg.aidenovak.com/media/slides/l14/065.webp) `53:14` The second thing which I will talk a bit more about is each source is actually finite. So if you put too much weight on a small source Then you need to, you essentially run out of that source, and you're, you need to just repop over it, meaning that you keep keep on training on the same. it literally the same tokens. And this is gonna be a bad for reasons we'll see. OK, so let's let's try to unpack this a little bit more concretely, this last point. So imagine you had just have two sources. A low quality source, you have 10 trillion tokens, and you have a high quality source that has 10 billion tokens. Generally, high quality sources are smaller. OK, so, if you just do a naï a naïve data mixture, let's just do uniform, so half on high, half on low. `54:06` And I'm gonna train for one trillion tokens. So, when I say train for one trillion tokens, that doesn't mean unique tokens, that just means train for that many steps or Divide by batch size essentially, or times batch size. OK, so then you look at, let's look at this, which is the number of Times a particular element in the low quality data set was trained on. So that's the number of epochs, a particular data's point was trained on. So, p low times the train tokens is the number ofuh, Times, I'm want to request tokens from this low quality source, and divide by the total number of low quality sources, soum. This is less than one, so basically that means a 5 percent of the tokens in this low quality data set, I'm even gonna touch and train on once. ![Slide 66](https://cs336.sg.aidenovak.com/media/slides/l14/066.webp) `55:07` And then if you look at high quality, you'll see that the number of epochs is 50, right? Because I only have 10 billion tokens here. And if I'm half half, that means 500 billion tokens. I'm I need to train a 500 billion tokens of high quality data. But I don't have 500 billion tokens of data, so I have to repeat each data point fifty times. `55:37` OK, so this is actually really important, and you know, some big model trains, of the range runs have kind of messed this up, is that you can't just a naively look at The the data sets and define a distribution, and then you sample. Because if you just look at the quality. if you look at the data, you don't you don't, you're sort of missing how many data points are there. OK, does this make sense? Maybe pause for questions. This is a kind of important point. `56:14` OK, so, yeah. `56:21` Products, oh, there there as Today, very good morning. I guess like, does, like why do we need First place, we doesn't it achieve comparable performance at a lower fewer. yeah. so the question is why you need 50 epochs? And I guess the point is that you don't need 50 epochs. Worst case, it's wasting compute. Sorry, best case is wasting compute, and Worst case, you're overfitting. But fifty, the reason it's fifty is that if you just go in and you define this data mixture, then you're going to be end up doing fifty epochs without realizing, unless you're paying close attention. So, this is basically, the lesson is like look at how many epochs you're actually doing, on your data. `57:10` Yeah. How does this? How do the mixtures got express during training? Like, so you switch every step, do you train like steps of ten? Say, oh, I want to ten. Yeah. Five o'clock code, Tennessee. Cause I feel like I was like, there's also an intuition between what decisions are made there. Yeah, good question. `57:26` So the question is like how when you train, how do the mixtures get kind of realized? And so, in general, you you sample, so you have, you wanna fill a batch, right? So you can sample a essentially for each element, which mixture component it comes from. And you fill up a batch with you know sequences from that mixture component. Usually you every sequence comes from one mixture component, you're not sampling like per token. I guess that, so each batch should be fixed. So, I guess the, I make sure assumptions are at the batch level, and then That's all like, oh, there's one step is, one step is, yeah, in general, you want it, to reduce variance. you want to train from multiple, you know, about. Should have some mixture. ![Slide 67](https://cs336.sg.aidenovak.com/media/slides/l14/067.webp) `58:25` OK, so so this idea, I mean this problem has been notice you know quite a long time ago. And so in this paper, they introduce a unimax, and I In there that case, it was they were training multilingual models. And it's like very clear that some languages were very low resource. So, you had to do something about this. So, in this paper, they notice that before, some works woulduh, Essentially, take the proportional mixing and raise it to some power, to kind of flatten it out the distribution. But the idea in this paper is that, let's be a bit more kind of explicit about that. Let's sample the sources uniformly, but with a hard cap on the number of epochs. OK, so this is the key idea here is that you say I'm only going to take 20 epochs over a particular, you know, data source. So, if you found that, then You know, too bad, you go, you you don't get any more tokens, you move on to the next thing. `59:36` So, this is sort of like a a safety net, and in some sense. So in particular, if you look at the number The probability is signed to a source times the number training tokens that should be less record the the cap. `59:51` OK, and there's a simple procedure for actually just de- terming the mixture subject to this constraint. `59:59` OK, so le- let's switch gears a little bit, and talk about how we define the mixture in the first place, right? Because if you have 50 sources, there's 50 numbers you have to fill. And we notice that proportional mixing isn't really enough. Maybe you can try to get estimate some quality metric, and then you can sample proportional to that. But that is also heuristic. So, the the most principled methods and most kind of easiest to understand like what's happening is these regression based mixing. methods, so reg mix, all mix and other papers like this. And the idea here is is kind of is relatively simple. So first, you, let's say you're trying to train a large scale model. `60:50` So, you go to a small scale. Let's say one, you know, I guess, let's say 300 million parameters. And you you try different data mixtures. So, let's say you have three components, you try different mixtures, and then you train these small models. You train a swarm of small models. Each model gives you loss on some target metric. It could be some downstream evals, it could be perplexity, whatever you want. OK, so they use these data points to basically fit a regression where the regression model maps these inputs, which is the the data Mister weights to the loss. `61:34` OK. So now you have basically a very cheap model that tells you if I were to train on this mixture, well, loss what I get. Then you can essentially optimize that you know data mixture. Sorry, optimize that function. For the optimal data mixture. And then that is the data mixture you use to train the large scale model. `62:04` So that's the the kind of the simple procedure, and you notice some kind of similarities with with scaling laws, where you're trying to do some cheap computations to figure out some answers, and then You'll scale up. OK, so there's a few design decisions here. One is, you know, what are the mixtures that you're trying out here? You need a distribution over distributions here. So, often people use a some dirichlet distribution. You also have to define the regression method. People have tried linear models or boosted decision trees. You have to figure out the target here. Often this is based on downstream e values. `62:48` But you have to be very careful not to overfit, right? Because we're doing pre training, and then you know supposedly pre training is supposed to be training this general purpose model, and we're not trying to fit some downstream eVals. If you're not careful, for example, if you have a bunch of code eVals, then well guess what, you're gonna upload all the code data. That's you know, not rocket science, and if you Then go and you say I wanna generate some poetry. You might you know realize that you've over overfit, so you have to be very careful about that. So proportional mixing and uniform mixing don't have this problem because you're not looking, there's no downstream evals. `63:27` And the and the final thing is that, you know how much of a difference between small and large scale, is there? And this is a cost and accuracy trade off. Of course, if you train really really small models, this might not be representative. But if you train large models, to do this, you will. There's no point in doing any of this, you just train, and you're basically doing hyperparameter tuning at the largest scale, which is too expensive. So this Omics paper actually does a nice has this nice table which shows you for a number of different methods that all fall into this kind of umI'm a framework. The si- the size of the proxy models are generally fairly, you know, tens or tens of a millions of parameters. How many, you know, points do you sample mixtures do you sample? M here being the number of parameters in your mixture. `64:24` Oh sorry, the number of domains rather. And you can use linearly, you can use exponential. And then the regression model, you can fit a lo- log linear model whichum, You will tend to work pretty well. And then you can use different ways to solve this optimization your problem. ![Slide 68](https://cs336.sg.aidenovak.com/media/slides/l14/068.webp) `64:50` OK, so there's sort of two leaps of faith here that you just have to be very careful of. Is that the regression model was trained on a bunch of these small proxy runs. And you're optimizing this. So, the hope is that you're the minimizer, your optimal data mixture, that regression model is still accurate. ![Slide 69](https://cs336.sg.aidenovak.com/media/slides/l14/069.webp) `65:14` And this is a little bit kind of, you know, you have to be very careful, because if you, let's say sample a random mixture, you'll probably be fine, because umBecause of you know classic generalization. you sample for a mixture, you sa- and you fit a fu- function, it should be able to predict in distribution. But when you're optimizing, you're e- essentially trying to go to potentially the extremes. Where you might not have as much coverage. So that's one thing to be careful of, and the other thing is that optimal data mixtures You just hope that they transfer first small scale to large scale. And this in general, it seems like at least at the scales that open community works with. Tends to be true or not blatantly false. But clearly there are scale dependent effects. If you just think about the the The previous slide when we're talking about filtering, if you're training for many more tokens, then probably low quality data is is okay. ![Slide 70](https://cs336.sg.aidenovak.com/media/slides/l14/070.webp) ![Slide 71](https://cs336.sg.aidenovak.com/media/slides/l14/071.webp) `66:18` So it's clearly the optimum is not the same, but You just kind of hope for the best here. OK, so hold on. There's one scale dependent effect that we we already talked about, but and we need to address here, which is that umImagine, so we have a lot of low quality data, 10T tokens, 10B tokens of high quality. the data. And if we train a small model on low token counts, you know, remember what what happens is that You might, this is a made up you know, example, but you will might put a lot of mass on your high quality data. Because if a low token counts, you're not epochin, so it's like, oh, wow, Wikipedia is so great, let's just train on Wikipedia. But if you then go and tra- train your large mixture on this If large model on this mixture, then you're gonna end up e-packing a lot on this high quality data, and your overfit. and that's definitely bad. ![Slide 72](https://cs336.sg.aidenovak.com/media/slides/l14/072.webp) `67:24` So, one way to fix this, which is done in the Omex this Omex paper, is you can just cap the number of v box. So that is one solution. There's another solution here which goes by probably multiple names, or one name for it is simulated parking. And the the the principle here is that make your small scale look like your large scale. This is kind of a general theme for the course, we saw it in kind of when taught, you talked about mu p. You prioritize your model so that your your hyperparameters transfer. And so the kind of same principle applies here. And the problem with you're doing this kind of naive scaling is thatum, A low, you can't be not epoch out small scale, and epoch, then epoch in a large scale, because then those are qualitatively different. Up, oh. `68:22` Operations on your data set. So, the way you would handle this is you, you downstream your sources proportionally. ![Slide 73](https://cs336.sg.aidenovak.com/media/slides/l14/073.webp) `68:33` So, so if you have your small runs were 10 billion tokens, and your big run were one trillion tokens, then you haveum, of, you know, one in one hundred kind of downsampling, and and then you basically use the downsample mixture. And so, the idea is that if you have the downsampled, you know, data, then this solution is not gonna look good, because You're not training on all, you're not, you'll get to just train on all Wikipedia. You're gonna train on some minuscule fraction of Wikipedia, and you'll realize, oh, actually, I'm gonna be epochy a lot. And that's gonna have really bad loss, which means that in the optimization, you're gonna look for more balanced approaches. So, you're kind of simulating the data scarcity that you would get at in your high regime at A low scale. ![Slide 74](https://cs336.sg.aidenovak.com/media/slides/l14/074.webp) `69:37` So, to summarize this section, the problem is how do you weigh different sources. Yeah, Wikipedia, you have your common crawl, you have your your code, you have some math data that you You scrape from somewhere, how do you balance them? Regression based mixing is is a nice framework, I think, to think about things. You you estimate your A function form that maps your mixture weights to a loss at small scale, optimize, and then you generalize to large scale. And you just have to be very careful or with this, because of epochin and overfitting kind of issues, and you have to either use cap epochin or simulate epochin. `70:22` So one lesson here is that, if you're trying to optimize anything, you have to be very careful, because you are in danger of optimizing the the wrong OK, any questions about data mixing before we move on? `70:43` When you're a down sample, is there something as like too small that like you can't really generalize as well? `70:53` So, the question is if you're down sampling, is there's a risk maybe you get too small of a data set? Yeah, absolutely. So, I think right, you might get into a case where you just have so few tokens. I think then what will happen is that your optimum will just put very small mass on that. And when you scale up, maybe you get the right set of token. So in principle, like it should work out, but there might be like rounding errors where like you end up instead of training once on the data set, you train like zero times by Accident, but you can you can sort of work around these by just like then you always train once on this data. Yeah. `71:44` OK, so, yeah. So, and data mixing, do you ever, cause like, I guess these sources just happen to align with different topics, but do you ever take like a diverse data set? Try to do like data mixing within. Yeah, yeah. So the question is, do you do, what do you apply data mixing to? The like different, you know, domains or Even within a data set, that's actually I forgot to mention that, so I'm glad you brought it up. So in a Numenta paper, they actually and and sold the omo stuff as well. You basically think about, let's say I give you a just common crawl, right? You can actually break this up, so you can group by domain, so they the AI two folks have this like web organizer, so you can group into topics. `72:30` But you can also quality filter. So then you you have this like two dimensional grid, we have domains, and you have quality, and each of these like cells. Is something that you would data mix. `72:45` So, that's one automatic way to determine the the domains. And then on top of that, you add ex- extra sources that people hand you. `72:57` OK, so I'm gonna quickly go through post training data here. ## 10. 后训练数据与合成数据 *[73:02 – 84:36] · [▶ 从这里看视频](https://www.youtube.com/watch?v=5sxHosTLPF8&t=4382)* ![Slide 75](https://cs336.sg.aidenovak.com/media/slides/l14/075.webp) `73:02` So, up until now, this is basically pre-training or maybe mid-training data. It's generally fairly task agnostic, except for the reg mix stuff where you're you're trying to minimize the loss, but still the data itself is still task relatively task and agnostic and you're trying to developuh, Basic skills. So, when you look at post training, a lot of the data becomes very kind of task, you know, dependent. And I'm not gonna do a comprehensive review. I just want to point out some of interesting post training, you know, data sets that have been recently released in particular for coding. Since that's a of I think great interest these days. So, the general recipe is, you define a set of You know, environments, in the case, think about code, which might be GitHub, you know, repos. You define a set of tasks or or prompts. And then you collect a ma- responses from a strong model or a teacher. `74:09` So, implicitly, you know, at least in open community, almost all the post training data that's most of it is you know synthetically you know generated. You can also replace strong model with a human. But, you know, that is slow and cost a lot of money. But, you know, if you're at the frontier and ofum, If you, I guess a few years ago, you kind of had to go and pay a lot of people a lot of money to give you responses. And nowadays, even at the frontier, you can Do sort of hybrid human AI things. But anyway, the the point is that there's some teacher out there that's giving you responses. OK, so a few Kind of, I'm just gonna talk to a few works here. So, one is this works called Open Thoughts. And this idea here is This came around, it was motivated by, you know, when O one came out, and there was a lot of intention on reasoning, mostly for like math and s- science. `75:15` How do we get really good post training data sets for this? And you know they eventually came out with 1.2 million examples using this teacher model. So the steps are first of allum, There are a bunch of, so, what are the environments and the ta- testing questions? There are a lot of different sources, That they they drew from, their hu- their human sources like Sack exchange, or Numinous Math, and then there's like some synthetic sources as well. These are just a subset. of the coding ones, but there's also more for math and chemistry. there's a big project that involved a lot of different contributions from many people. So you can see that there's you know some of these are like coding exercises, some of these are you know more realistic things that show up onum, I guess that co-golf is not realistic. ![Slide 76](https://cs336.sg.aidenovak.com/media/slides/l14/076.webp) `76:17` but there are some more realistic examples here. So co-review, this is probably more realistic. And they noticed, they did a fairly comprehensive analysis of, you know, given this data set, how do you generate responses? They show that umHaving a few sources was actually good, than rather trying to do all sources. Sampling multiple generations is helpful, like 16. One thing that was interesting is that Having better models aren't necessarily you know better teachers. So for example, Q W Q 32B, which is you know now a very Old and small model is was a better teacher than Deepseek R one, which was at the time probably one of the strongest open models. They also found that basic answer filtering wasn't helpful. And so the entire pipeline looks something, you know like this, where you have a bunch of these sources, you know going in. `77:22` And then you do duplicate, you randomly you know sample, I guess this down sample the questions, and you generate multiple answers, and that gets into the final dataset. So, the 1.2 million Isn't is examples, but divided by 16 gives you the number of actual questions. ![Slide 77](https://cs336.sg.aidenovak.com/media/slides/l14/077.webp) `77:43` OK, so there's an another, so th- that was for you know math and science, and s- and some code. I think more recently, there's been a lot of interest in developing in particular, genetic coding. Also, not just a model that can just generate some code, but actually a model that can do software development. so this swift smith paper, had the idea of you're given a repository. You're using a language model to automatically generate tasks. So, they have an agent that takes a repository and actually makes it usable forum, like installing dependencies and so on. And then you generate task which are generally you know you modify the code in some way, maybe introduce some bugs. And those get verified, and you get some task instances. `78:43` So, these are synthetic tasks, but you get, you know, 50 K of them, which is actually, at the time, which is last year. quite large. So, since then, there's been a bunch of other work which I'll uhdeal mention, so, there's this s- a suite zero a paper from, you know, nvidia, which have this kind of umshow you. okay, i'll also talk about that. so this is a kind of an interesting idea where the observation was that unlike math, sweet task are have just a lot of heavy dependencies. Most GitHub repos, like, don't even run, and you have to install all these dependencies that are out of date, especially if you roll back to like when a PR was It's just kind of a mess, and this is just a nightmare. And so, they were thinking about. how can we actually get a data set that just umOn all the the the repos, and rather than having a repo specific docker image, they notice that the models are actually good enough that they can solve a lot of these tasks. Without execution feedback. ![Slide 78](https://cs336.sg.aidenovak.com/media/slides/l14/078.webp) `80:00` So, you know, if you look at some of these models, if you were allowing edge execution, you get like 80. If you don't allow educate, not education execution, you get almost 70. So which is, you know, not bad for not being able to execute code. So, somehow these models have some internal semantics of of of code. So they were able to generate 300,000 agent trajectories. All these are real GitHub you know PR, so they're realistic unlike the the s- sweep the Smith examples, they use the open hands, a scaffold, and there's some, there's a lot of details like how to prevent agent hacking. So, normally you hand an agent this. instruction, you explore, you test, you implement, and their 30 version was basically saying you cannot run Python code, you can only do like set and grep and All these like basic operations. ![Slide 79](https://cs336.sg.aidenovak.com/media/slides/l14/079.webp) `81:06` OK, so then they distilled from a big coding model, and filtered, because sometimes the coding model will ignore these instructions and still try to execute anyway. And then they're, they showed that with this oh, they also had a t-, 13 k agent trajectories that do require execution, feedback. So they trained models Where they first fine tune on these three zero examples, and then fine tune again on these three hero examples. And they were able to You know, I guess the frontier is still pretty high up, but they were able to make some, you know, progress here. I'll quickly go through three rebound, was essentially another attempt to Grab tons and tons of you know PRs. So, and all these kind of look like get a bunch of GitHub repos. Try to install the repo, like most of them probably, you know, failed and try harder, and then you use a a language model to give you the the responses. So, in this this actually just came out today. So, you can take the Squeeze Zero idea, and you can s- now scale up to 12 million. ![Slide 80](https://cs336.sg.aidenovak.com/media/slides/l14/080.webp) `82:24` Agent trajectories. The nice benefit of 30 is that it's just, you know, almost, it's very lightweight. Here they use the sweep bench, rebench tasks. So, in this one, remember they were trying to get things to execute, they only got 32 thousand of them to execute, and 120 of them didn't execute, so you just, but three zero doesn't care, you can use all of them. This is a very small model, because now the state-of-the-art is quite large. And soum, So, you know, I think as you can see, the data sets are, you know, getting more and more sophisticated. You go from, you know, environment free things like math. to now coding, and now the coding data sets are growing quite a bit. But the general idea is that you you have these these prompts. Which, you. ![Slide 81](https://cs336.sg.aidenovak.com/media/slides/l14/081.webp) `83:20` There's kind of trade offs. You can have fully synthetic, you can have semi synthetic where you have a real environment and synthetic task, where you can have real umAnd the responses generally are co-, coming from just capable models. but they need also need to be good teachers. Code environments are a pain, and you, there's a lot of filtering and other details which we don't have time for. OK, to summarize this lecture, we talked about filtering, so define what good looks like, and then train a lightweight classifier go over your web crawl, and you can get a s- a small subset thatuh. matches what you're looking for. Duplication is im- important to avoid overfitting and saving flops. Mixing, try mixture small scales, extrapolate to large scale. And then we looked at some, you know, post training data. `84:13` I will say that though that a lot of the data work is can be very grungy, it's very domain specific. And requires looking at concrete examples to make these high quality data sets. So, this lecture is not really representative of what data work is like, but hopefully giving you a Idea of the data landscape up there. OK, so that will be it, offer today.