数据来源与数据集
本页 = 官方讲义截图 + 全程文稿（豆包 ASR 转录，已去除语气词）
01 课程引入：接下来两周谈数据
0:04 So today we're gonna talk about data.
02 为什么数据最重要：保密、长尾与人力的长尾问题
0:08 Where we are is that you know how to train a model given data. Last time we saw evaluation, so we know what a good model is, and for the next Two lectures this week, we're going to focus on what data should we train on.
0:25 And I want to argue that data is the most important thing to get right in language models. And one justification for this is if you look at what companies actually disclose. So here is the llama 3 paper. They have full transparency into the architecture. Well, of course, because it's open weight model. They even tell you about their training procedures. But they don't say anything about their data. So, they just say we train from a variety of data sources, and do some stuff. And there's good reasons for this secrecy.
1:02 One is that data is is kind of your comparative secret sauce, comparative dynamics, you don't want like your competitors what you're doing. And the second thing which we'll talk a little bit more about is is copyright liability. You don't want to get sued if you tell people you're training on certain types of data.
1:21 So, data is a long topic in a machine learning, and before foundation models, data work meant you annotate label data for supervised learning. Nowa- days, there's less manual annotation, at least in pre-training. But there's still a lot of curation and cleaning work that needs to be done. And so, fundamentally, this hasn't really changed. Data has always been a bottleneck, and always will be a bottleneck, because it's in some sense of long tail problem. And that scales with human effort. So, if you have a lot of people trying to work on a problem, well, there's only so many people that can work on architectures and systems, but data, well, that's like, you know, especially if you're training a foundation model that's trying to do all these things. You can paralyze that effort easily. So that's why these data teams and these model developers are actually quite quite big.
2:18 So so data comes in at different stages of of of the pipeline. Today we're focusing mostly on pre training. So pre training, you take raw data, these are documents from the web. Then you move into mid training, where you train on more high quality data to enhance certain capabilities, and give long context, things like that. And then finally post training, where you're training on things likeum, Chat transcripts, or if you're doing reinforcement learning, you get some environments that you're training on. This becomes more task specific. In practice, the lines are A bit blurry, and there could be more than three stages, but this is sort of a basic template. The the trend is that we go From training on large amounts of low quality data to smaller amounts of high quality data. So, as a result of this process, you'll see in the literatures, people talk about the base model, this usually means after pre-training and mid-training, instruct models or chat models are after post-training.
3:23 But of course, the lines are becoming blurry enough that what is a base model anymore is really kind of unclear, and more recentlyuh, The largest models are, there's no base model, there's just, Quin 3.5397B, and that's it. There's, you don't see the intermediate ch-, checkpoints.
3:44 So for models that are open source, like Omni from AI two, you can see everything that's happening, and so we'll study some of these models. So for example, in pre-training, there's a bunch of sources which we'll talk about what these mean, but you know, some web pages, academic papers, math pages, proofs. And so on. There's mid training, higher quality web data, instruction data, s-, a bunch of synthetic data often goes here. And then finally post training. There is a bunch of you know chat logs, there're more math and reasoning and coding, and and you do safety at this at this point as well.
4:27 So, you know, what are all these data sets? You know, how do you choose them, and how do you process them? So, that's that's main question.
4:36 OK, so in the spirit of the class, we're gonna talk about data.
03 「训练数据来自整个互联网」并不成立：爬虫、动态内容与技术/法律限制
4:40 I'm gonna go start from scratch. So where does data come from? OK, soum, You know, one might hear in the hallway all language models are trained on the entire internet. So so first of all, this doesn't really quite type check, because for that to be true It would have to be an agent that like an R O agent that goes on the internet and like does stuff. But that's not actually how pretraining works. Slightly more accurately, it's trained on the public worldwide web. This is not quite accurate, for reasons I'll explain. OK, so first of all, the web is actually a bunch of live servers that just exist in the world, and you can connect to. So, take any website, you can go and Download the page.
5:27 Or you can basically send a request and get back a response. So you can't actually train on these live servers, unless you're training a RL agent. So typically what you do is you you have a crawler, or someone builds a crawler, not necessarily you. The crawler needs to discover what web pages are out there, starting with a sets seed set, and it downloads the discovered web pages as it's it's crawling.
5:53 So, but now, still, you can't just run a crawl and download all the web pages on internet. And there's a few reasons for this, One is that a lot of the web content is actually dynamic. So especially these days, a lot of sites or apps. That URL isn't even as fully specification of the content. It's just an app that you interact with. And you need to often click buttons or submit forms to access content. For example, if you're on, you know, Discord or something, it's not like you can just like curl Discord and get a all the content on Discord. OK, so so that's one kind of thing, and a lot of content is actually Known as kind of in the deep web, which is not just like you're, you know, doing web page crawling and following hyperlinks, which is a very kind of traditional model of web crawling. The second thing is.
6:53 that your authentication. So, some pages need a login and account, and generally you have to to to pay. For example, their Facebook, X, LinkedIn, New York Times. There's actually huge amounts of content that are actually locked up behind these wall gardens. So, these are technically on the internet or on the web. But you can't just build a crawler and go crawl your Facebook. I guess if you're Facebook, you can crawl, or you don't need to crawl Facebook, you have the data, and you can train a model, or if you're Your X XAI, you can train on X. but if anyone else, you can't actually access this content.
7:34 So, OK, suppose you don't have the authentication issue, there's still potential restrictions here. So, so there's something called robots dot txt. And this is a file that is usually placed at the root level of a directory such such as mytimes dot com, and it basically tells you what you're allowed to crawl and whatnot. So, this things are a lot of things are disallowed, such as OAI search bot, or perplexity bot, or a bot. I think basicallyuh, ChatGPT user, Claude bot, and all these things. OK, so this is not a tech, this is not a legal restriction, this is just like you're supposed to be a good citizen and look at robot set TXC, and If your name is on that list, then you should not crawl. That's that's basically the the contra- not even a contract, that that is the thing you're supposed to do. S- a lot of websites now use something like CloudFlare to detect and block bot activity. So sometimes you've gone to website, and it's gives you a capture, right?
8:47 That's because it's somehow some algorithm has detected that You might be a bot, and then it's making sure you're human. And if you're actually a bot, then well, you can't. I mean you could try to get around this, but generally this is an obstacle in your way. A website might block certain IP addresses, or or countries, and they might have rate limits. So, there's technical restrictions on what you can actually crawl. And then there's also legal restrictions. So, there's websites have terms of service that Let's say if you go to a website, you have to obey this contract to use the website. And the terms of service often will say if you are a bot, go away. You cannot use this the content of the site for AI training or whatever it might be.
9:37 And and also, if the, if even the terms of service don't say anything, the content of the website might have, you know, a license. uhOr you might not have a license to train, and we'll talk more about this in a bit. So, there's legal restrictions on what you can are allowed to train on. These things are also evolving over time. So there's a nice paper by Sheinlam Prey called cons consent in crisis. And what they did is they examined the restrictions, both the technical restrictions, robots at TXC, and the legal restriction, the term of service. For various URLs in common data sets, we'll talk about later. And the conclusion was that the restrictions have increased over time. So here's a graphic.
10:24 That shows, the top one shows robots at txt. And if you just look at this red, that's the full restrictions. And up until 2023um, Everything was fairly fairly constant, and then up until twenty mid 2023, all of a sudden you see the amount of the fraction of websites that have full restrictions on it has grown to almost like 50 percent here. And for terms, this, there's a. similar trend where I guess in in 2016 No one really put any terms on their pages, and now most pu- pages put some terms, and most of the terms say you can't use this for AI. OK, so even though it was possible to crawl the internet back in, you know, let's say 20 20. The internet actually, at least what you can legallyuh, Crawl is actually much smaller.
11:29 So, of course, these are you know guidelines, and sometimes either intentionally or not intentionally, you could violate these things. So, this is Sometimes you get into these situations where there's this guy who what was this forget the name of this the site. was complaining about anthropic crawling them, hitting their service 1 million times in twenty four hours, which was not good. And then read the docs was also getting hammered. So there's a period where, which I think probably has hopefully been corrected by now, thatum, Crawlers were hammering, you know, websites. And and there the, here this is kind of interesting, is like even before talking about copyright issues. There's I think problems with crawling, because either violating terms of service or robots.txt, or just just blow up down umYou know, generating a lot of server load, which this cost money for the person hosting it. and also degrade service for every everyone else trying to use it.
12:39 So, there are, you know, issues with with crawling. And then of course, there's copyright, which we'll talk about soon. That's a whole another complex can of worms.
12:51 OK, and then finally, there's these things called shadow libraries, there, which are also part of the web. So examples are libgen or Arxiv. And these completely disregard copyright and bypass paywalls. So, someone has collected a ton of of data, usually books and articles that are all Copyright and people have to pay for these, and have made them available for free. And these, there have been a lot take down orders, lawsuits, and these are blocked, butum, Usually these are circumvented because they just make servers in other countries. And the people doing this argue that this is making free what's freely available, what should have been freed, but generally from a legal perspective, this is piracy and copyright infringement, and we'll come back to this point later. So, there are a lot of books and papers on this uhOn these shadow libraries.
13:55 OK, so summary so far, the internet is a huge and messy and you know scary place, and You can't actually get all of it. There are technical restrictions, there's legal restrictions on what data one can access.
04 版权法：许可证、合理使用与诉讼
14:12 So now, next time someone says I trained on the internet, you can tell them all these things you I'll learn.
14:20 OK, so now let's go to the the second pa- part of you know what data can can we use? And this has to do with copyright. So, suppose you were able to obey the terms of service, and you are really a good citizen, and you obey the rate limits. Now, there's still a question of are you allowed to Train on this data. And this is a ongoing question that has not been fully resolved, but there has been a bunch of developments in the last year. So, the legal context for thinking about this question falls into intellectual property law, and this is a a system of, you know, system that is trying to incentivize the creation of intellectual goods. so, i think it's remember to remember the spirit of the law is to not just say You know, technically say no to everything, but to incentivize the creation of intellectual goods. And there's many forms of intellectual property that are covered. There's copyrights, patents, trademark, trademarks and secrets.
15:33 Trade start trademarks and trade secrets. The thing that has is most relevant for language model data is copyright law. So copyright law has a fairly long history going back to the seventeen hundreds in England, and so 1709 was the first time that governments and courts started actually regulating a copyright for this goal of incentivizing, you know, innovation and creation. And in the US, more recently. the copyright act of 1976 is what Think that defines a lot of kind of modern co- copyright. So copyright protection applies to original works of authorship fix, authorship fixed in any tangible medium of expression. Yada, yada, yada. OK, so, a few notes. So, not everything is copyrightable.
16:26 In particular, collections. are not copyrightable. So for example, if you have a a telephone, you know directory, unless there is some creativity and how you arrange things. This is not really copyable. And furthermore, copyright applies to an expression, not the idea. So, you can't copyright an an idea, like you can't copyright theuh, quick sword, algorithm. You can copyright the quick, like implementation of a quick sword algorithm, but not the quick sword algorithm. So, one thing that happened in 1976 is that the barrier to getting a copyright was relaxed a lot, so more things are copyright. So, it used to be that you had to publish something.
17:14 To have something copyright. Now, things just have to be fixed. And what that means is that registration of any sort is not required. For copyright protection, this is in contrast to patents. Patents you have to go pay a lot of money to get a patent, and you would have to file a patent. But the threshold for copyright is very low. For example, you put something on your website, it's copyright. That's it. Now, you, if you want to sue someone, you have to go get and register, but it only cost 65 dollars, so which is much smaller than the lawyer fees that you'll probably pay.
17:54 soum, It's really still a very low low bar. So, one thing to know about copyright is that it lasts 75 years. And then the copyright expires, and whatever that content gets released into the public domain, and this is now you, permissive everyone can use it freely. OK, so a lot of people who have lived in the past, all their great works are in the public domain, which is great for everyone. So, the rationale here is that you're incentivizing innovation. So, when an artist or a creator creates something, you want to protect their, you don't want someone else to just take it. So you protect them for a while, but you know, after 75 years, presumably it's not worth protecting, or they're passed away or something. So, it doesn't make sense to keep things copyright anymore.
18:51 OK, so so basically everything on the internet is copyrighted. OK, so you're saying, wait a minute. So that means is every everything, just if I try not anything that's a copyright violation. Well Not necessarily. So, everything is copyrighted, but you can use copyrighted works, and the way you do it is, A, you either get a license for it. Or B, you appear to appeal to the fair use clause. OK, so a license is from contract law is something that a licensor grants to a licensee, basically says Don't sue me. Like you can you you can use this this work in the ways that the license permits. So, there's a really nice license called you know, Creative Commons, which allows something to kind of act like it was in the public domain.
19:50 enables free distribution of copyright work. So, examples include like Wikipedia, OpenCourseWare, Khan Academy, all these all these things. It was created in 2001, to essentially bridge public domain and existing copyright. Because otherwise you have to wait 75 years. But what if the creator says I don't, I actually want people to use it. They can now have a means, a legal means of saying, I put a Creative Commons license on it, it's gonna act like it's in the public domain. And you know, and everyone can use it. So, so far, things in the public domain, and things that have creative commons, alignances, is are permissive. You can use it.
20:36 But there is many other things that times where you don't have, it's not a creative comments license, it's not in the public domain. You have to get a license. So, if you have money, you can pay someone to give you a license. So, there's a bunch of deals between model developers and Content, I guess platforms that license, allow the mod developers to license the data for training foundation models. OK, so that's the first thing. You can get a license. Either creative comments, or you can pay someone for a license. Now, if you don't want to pay, you can appeal to fair use, which is a more complicated matter. So, this is section 107 of the copyright act.
21:25 And there are b- basically, Fair Use says, I can use this anyway, even if I don't have a license. And there's four factors determine whether for Fair Use supplies. So, the first one is the purpose or character of the use, like what are you trying to do with it? And none of these are, you know, hard rules. They're just Tendencies which have to be waited in in court, if it comes to that. So, for example, if you're trying to do something for education, it is more likely to be fair used than if you're trying to sell and make money off of it. If you're trying to, if you transform something, then that's gonna be more favor than if you just literally host the identical copy. The nature of the actual copyright work is also important. So, things that are more factual are more likely to be fair use than fictional works.
22:17 Right, because if you write a summary of War War, you know, facts about a page about facts about War War Two, is is less, there's less umProtection over you know my, you know, really creative poem. And then you know, the amount of portion of the original work, so using a snippet is a favored over the whole work. So if you just take a little bit, that's not as bad as taking everything. So that's fairly straightforward. And the final thing is is kind of related to original motivation, or was like, why are you, why copyright at all, is to protect and align economic incentives. So, the fact that what the the use has on the market for the original work. So, if you take author's work and you And you provide an alternative, and that decreases amount that the o- original author could monetize their work. That is seen as potentially bad, but if you're transforming in a way, and adding You know, going to a new market or doing something else with it, then that's gonna be treated more favorably.
23:30 OK, so here are some examples of fair use. You can watch movie and write a summary of it. You can im- reimplement an ide- an idea like an algorithm. Not copy the code. And then there was this famous you know, lawsuit between authors guild and Google. So, if you go to Google books You can see snippets of of various books, which are copyright. So, there was a huge you know, uhDeal about whether this was deemed fair use, because you're sort of redistributing you know other copyrighted work, and and eventually Eventually meaning after 11 years. this lawsuit was settled in favor of Google, which may, was why this thing exists. And this has set some precedent for thinking about umWhether you know, language model training as as fair use.
24:32 One thing to note is that copyright is is not about verbatim memorization. So, this might be, I think, for an ML audience, this might be a little bit new, because a lot of papers are focus on verbatim memorization, but that's That's one way you can violate copyright, but that the only one. So for example, plots and characters can't be copyrightable. So you can copy a Harry Potter, the the character, not any particularum, Your book or anything. But there are some exceptions. For example, if you're parodying something, and so you're creating something that is Looks like a derivative, but as long as you're making fun of it, it's actually more likely to be fair use. So, this, yeah. So, a lot of coverage is about semantics, is definitely not about n-gram overlap.
25:23 And also about the economics.
25:27 OK, so what is implication for language models? So first off, copying data, like even if you're not training, like even the mere fact of copying Which is in the word copyright, is potentially a violation already, even if you don't do anything with that. OK? Training a model, you know, I guess this is, This is not really let's see, how do I take that? This is not necessarily a a fact, but is intuitively has a transformative flavor, right? Because it's certainly not It certainly seems different than just like rehosting another work, right? It's doing something of the transformative. And you know, in some sense, the models are really trained on this data as a means to an end, you were trying to extract the general idea, learn about the world.
26:28 And how it works, rather than only just the concrete expression. And the other thing is that regardless of copy copyright Language models can definitely affect you know the market. So so I think this is also remember that's the fourth item for for fair use. So by negatively affecting the market, you are you know more likely to be ruled not various.
26:56 OK, so so fair use is kind of a little bit, you know, slippery. suppose you have fair use, or you have a license. Remember that there're still terms of service that allow prevent you from actually just g- getting a piece of work. For example, YouTube has all these videos which are comeum, licensed, but the terms of service previt prohibits just downloading the videos as a using a bot or like scraping. Right, so which is another layer, so there's multiple layers of restrictions here. OK, so let me talk a little bit about where things are on the legal landscape. So back in 2023, New York Times file suit against OpenAI, saying you train on our news articles. And look, here's evidence we were able to actually prompt a ChatGPT to generate a news article almost verbatim. I don't think this is still this is still pending.
27:58 There's a a case against Anthropic, where the allegation was you pirated millions of books, and you train on these books to to make Claude. Just last year, this was a kind of a landmark ruling, that said actually, this particular instance of training is is fair use. However, anthropic, you're still in trouble because you pirated all these books, and that is a no no, right? So notice that this is Has nothing to do with training, just a mere a fact of cop a pirating, I mean which is this this is not a a new thing, is is is illegal. Andropick actually interestingly had bought and scanned all the books, which is actually fair use. The court said you're allowed to scan, you buy a bunch of books, rip off the binding, scan and digitize it for your own use, that's fair use. But that doesn't absolve you of your sin of pure pirating. So you can't pirate and then buy it and say like actually, never mind what I just did first. So the outcome is that anthropic paid 1.5 billion to settle the author.
29:14 So this is like about 3,000 dollars a book. So, there's also a lawsuit against Meta. The allegation was that you trained on our books. Sorry. And this was, as we'll see later, actually revealed in the the llama paper. And the jud judgment which came right after anthropic was, yes, training is fair use. And then they also torqued some books, so that's still pending, so that's probably Precedent tolds this is also not gonna be good for for meta. OK, so the summary so far is umTraining, you know, so to speak, has been deemed of fair use, or at least has been not deemed not fair use.
30:08 The new rulings have so far been new narrow. It's not to say that any training on any corporate content is is fair use. But in these cases, it's fine. Pirated books is clearly illegal. I guess we already knew that, and but this is still a very active and evolving area.
30:31 OK, any questions about the the sources of data? Think before we get into the actual data sets, I just want to kind of uhum, Set the stage with this kind of more general social context.
30:51 Yeah. The question is what do I think of voice data in 11 labs. I don't know too much about that, so umSo, we can talk offline. Yeah.
31:08 I guess my both of I might like to kind of, oh, initially it's creative comments, but can I update my license later on to sort of block certain usage? And so, like, what is the role of It's like right there. Train on the older version.
31:29 more police officers if the career Yeah, so the question is what happens with time, if you have one license, and then it gets changed, what happen? So, first, I'm not a lawyer. Have a se- se- second, so this umuh, So, the first license, I believe you're still allowed to train on it, but what happens is that the license usually applies to not just, let's say your fixed set of documents.
05 Common Crawl：公共网页抓取存档
31:58 For example Reddit, there's constantly new content being created. And the license being changed means that you wouldn't be able to train on the the later things.
32:12 OK, so let's look at various sources of data. So let's go back to crawling. So most model developers have their own crawler. Because they want to have control over full control over what the data is. Fortunately for the rest of us, if you don't want to uhbuild your own crawler. there's something called Common Crawl, which has been around for quite some time, since 2007. And every month, they run a web crawl, gets about 3~5 billion web pages. Each crawl has some overlap with the previous crawl, but they try to diversify and get new pages. So, there's 300 billion pages so far.
32:59 Which seems, this was from that website, it seems a little bit big, because if you multiply this number by 20, you don't quite get 300 billion, but that's what they say. So, how many URL's are out there? That's really hard to estimate, so the Google search index is at least 100 p- a petabytes, according to them. And you know each common crawl dump has about two twouh, billion, your web pages. So that's about 372 terabytes. And this is not including images, this is just mostly focus on the the text. So crawling is is conceptually straightforward, but all the gory details are in the implementation. You start with a bunch of URLs. And then you you iterate, it's a basic graph traversal.
33:54 So you pop your role from a queue, you download it, and then you look at all the hyperlinks in that page, and then you add them to the queue. And generally this is done in parallel. Over many machines. And there's many decisions here, which paper, whi- which pages to download. To respect, you have to respect our robots at txt, don't overload the server. Sometimes, the websites change. And so, you want a policy that will download you know, frequently change pages, but not spend time downloading pages that don't change. So there's some policy around that, and like I mentioned before, URLs are dynamic. So, both sometimes the same URL leads to different content depending on like some state of the the browser.
34:46 But also different many URL might lead to the same content, so there's a lot of duplication that happens if you're not, you know, careful. In particular mirror sites, Explicitly is about duplication. So, Common Crawl releases their dumps in two formats. One is a work file, so this is the raw HTTP response. Right, so this remember HTTP protocol is that you send a a get, you know, here's a a a URL, and to, and then outcomes a a res HTTP. Kind of response, and it just the workflow is that response. They also do some processing, they convert to a white file, which is necessarily a lossy process. It turns out that this is not necessarily the best way to use the web. The HTML to text, there's many tools for this including Trafalgar and Brazilia parts, and the way you convert actually does matter.
35:48 So, this is from the data compare alon paper, which we'll talk a little about. It is ablation where you look at the web files that common crawl releases versus These other these other tools, and it seems like Trifle Tour and Brazilia Parsa you know better.
36:09 OK, so one thing you can do is just rely on general web crawls. But often, the we- the web is not a uniform place.
06 Wikipedia（以及数据投毒）
36:17 It's not like you there'suh, A bunch of websites, and you just get some random fraction of them, and that's your data set. And there are specific pockets of really interesting, high-quality content. And I'll talk about three of those Wikipedia, GitHub, and Archive. So, Wikipedia, all of you know about, it's been around since 2001, and now there's 67 million articles of Around all these different languages, and it is, it doesn't, Wikipedia is not all content by any means. There's no, you can't have original thought, in it, because everything has to be kind of referenced and cited. So in some sense, you could argue that Wikipedia is it doesn't contain anything, because that's not already on on the on the web. Well, actually, that's not true, because they can also cite books, which are You know, obviously, you can't easily get them, so we could be so good. So, and also there's articles based on notability. So, not everything, not everyone ha, can get a Wikipedia article about them.
37:34 So, anyone on the internet can write this content. This was the radical thing about wikis, And it's sort of still a miracle, I think, to me that this this actually works. It just happens that any vandalisms gets reverted by administrators or bots, I guess these days. There's you know a small number of Wikipedians, like as any sort of peer production system, there's a small set of people who do most of the work. And for example, this guy has like 5 million edits. And one thing about Wikipedia is that every few weeks they do a periodic dump. So basically, they take all Wikipedia package into a niceuh, tar file, and then you can just download that. So this is important because you don't need to crawl Wikipedia. In fact, they don't want you to crawl Wikipedia, they just want you to download this if you're gonna crawl, and this is muchuh, Better than crawling.
38:31 One kind of just a fun aside, or a good to know is that there's this paper that says you can actually Poison Wikipedia. So you might think that this vandalism gets reverted, so if you do s-, attacker tries to do something fishy, you'll just get it rolled back. But what Colleen realizes that these periodic dumps happen at a regular cadence. And so what what you can do is you can go in And you can just edit it right before the dump happens, and then the dump ha- the happens, and then and then the edits get rolled back, but the dump still has the malicious content.And with, if you're. able to inject things into web pages, that means you can cause, there's works that's show that you can cause umThe model to let's say ascribe negative sentiment to any sort of trigger phrase like the iPhone.
07 GitHub 与 Software Heritage
39:34 So, my one takeaway here is that, If you can consider adversaries, even so called high quality contents might contain bad things in them. I think this has been a fix since then.
39:50 OK, so GitHub is a good place for code, and code is important, not just for if you want coding capabilities for your language model, but if you want general umno reasoning. So GitHub, all of you know, again, it's a live service for hosting code repositories, been around for since 2008, about the same. ages common crawl. it has 400 20 million repositories, 28 million are public. each repository containsum, It's not a file, it's, as you know, it's it's a directory with commit history, issues, pull requests and comments. Code has generally a lot of you know duplicates, and Is this because either copying code or or forking code?
40:42 And GitHub has deemed it you're allowed to train on any public repository with a permissive you know license. So MIT or Apache licenses are are fine. So, you, there's, remember there's two types of data when you talk about GitHub. There's like the repository data which you canuh, Just go go and download through the github git protocol. And again, you know you shouldn't scrape github. You should just download the repository. And then the metadata associated with each repository issues and pull requests and so on. These are made actually via the GitHub archive which gives you hourly snapshots of this event stream. Which I think is interest really interesting data.
41:25 It's basically every single comment or star or action on GitHub gets gets recorded. There's this so software heritage foundation that is focused on the repository, not the metadata.
08 arXiv
41:41 And they aggregate GitHubum, GitHub is not the only place where code shows up. There's also smaller entities like GitLab, Bitbucket, and so on, and they aggregate all that those repositories. OK, so, archive, just very briefly, everyone knows archive, this has been around as a site that allows people to share paper since 1991, started with physics, and now Has a lot of other areas in it. Each of the 3 million submissions has the metadata, a PDF, and Optional latex source. Ok, so already you should be thinking, well what does it mean to train on archive? It's not clear because this is well You have to convert the pdf into a tax or you can use a latex source which is a bunch of files. There it's not peer reviewed, but there is some approval process. And authors can choose to maintain the rights, or put in the creative commons, which, so everything is actually very clearly licensed. All the metadata is under permissive license, so you can use that regardless.
42:55 And then you can go, and you look at all the papers that are Creative Commons license, you download those, and you can use them. And I think most archive papers are creative no comments. And then again, this is available. You don't crawl ar- archive, you just go down touh, bulk download it from some website.
43:18 OK, so any questions about source of data? Yeah.
43:29 Are there restrictions on data generated by models? Generate by models. Yeah, I need to get it here. Oh, oh, I see. We'll talk about that, Little later. So the question is, what about model generated like synthetic data? Are you allowed to use it? The short answer is probably yes, yeah.
43:51 Crawl and stuff like that. I guess like the, because you mentioned earlier that there are websites that have a lot of pirated Great. How do you like, how do you ensure that those types of websites don't show up in crawl? Yeah, yeah. So, the question is There's a lot of websites with pirated books. So if you're just doing a web crawl, you can't look at all the websites. So how do you make sure that everything is kosher? And that's part of the difficulty, is you can't. And and so, there's most likely in common crawl, there are copyrighted books and content that you are not supposed to.
44:31 Train on, and you can appear to, you peer, will you appeal to fair use. Remember everything is copyright. Books are not remarkably different from your your website. For example, both are you know copyright it. Probably a book author has more of a you know, if they want to go into court, they'll probably be able to protect that better because it's published than youruh, your website, but but but still. and I'll talk a li- little bit about later. If you really would want to be careful, you can do something else. Show later.
09 BERT 的数据：Wikipedia + BooksCorpus
45:07 OK, so what I'm gonna do now is to now talk about the various data sets that are actually used in building model starting All the way back to 2019. So we've talked about the origin of data and source of data, so you have a feeling for just the general data, you know, landscape. And so, let's start with Bird. OK, so Bird was from 20 18. And they trained back in the day, they just train on Wikipedia and books. So what is this books? So, there's this website called Smashwords. It was created that allow anyone to just put a ebook up for to publish. And they had about half a million books in 2024.
45:53 So in 2015, there was this paper who just scraped Smashwords. Oh, so, sorry, took the books that were free, and and just made a corpus. This was back in innocent days when no one was paying attention to any of this, so this book's corpus was around for Many years in the academic computer, aca-, academic community. Since then, it has been taken down because it violated the terms of service. Ok, just because it was free and you can get it, doesn't mean it's it was legally allowed.
10 WebText / OpenWebText（GPT-2）
46:33 So, one thing about to denote about BERT is that the sequences were documents rather than sentences. All the language modeling research before that Was focused on you know, sentences.
46:50 OK, so in 2019, the GPT two came around, and so what did they do? So, they wanted to get high quality web content. And at that time, people knew about Common Crawl, but somehow the Common Crawl data was too messy. So what they did was came up with this clever idea where You can take You look at pages that are outgoing links from Reddit posts with gradient 3 comma. So, good posts must link to good good Good website, so you grab those those pages, and you get a forty gigabyte of text, a million pages. They never released this data. But there was a open replication of web text, which people use quite bad.
11 CCNet：去重 + 语种识别 + 「像 Wikipedia」的质量过滤
47:46 OK, so so that's a GPD two. So as you're going through this, I guess think about both you know, there's many methods of Filtering and collecting data, and it's interesting to look at these design choices, and OK, so CCNet was developed by at that time Facebook. umSo, the goal was to create large high quality data sets for pre training, and they were in- especially interested in low resource languages. So, they didn't want some very Manual process that only work for for English. So, they did did a bunch of things, they did du duplication, and language identification, to train a classifier to detect the language. And keep only the language that you you're looking for. And for quality filteringuh, The idea that they had was they wanted to keep documents that look like Wikipedia. So Wikipedia w- at the time was deemed goo- good high quality content. And they used a language model that was trained on Wikipedia, and you score probability of a new document under that language model to gauge how Wikipedia like it was.
49:06 So rather than outgoing links to Reddit, they use a language model on Wikipedia.
12 C4：手工规则过滤
49:11 And they they you know show that because you can get much more more data, you can actually outperform training. performing, just training on Wikipedia, and this was a tool that will later appear in some other papers.
49:29 So, Here is another work. This is from Google and 2019. So, this is called C four. So, this paper as actually more famous for the T five model. Which pushes the idea that you should just treat all the NLP tasks in a world as text to text. But actually a big contribution of this paper, which is a very long report, was the C4 data set. And the observation which was ve- very clear at that time, which is that common crawl is mostly not useful for natural language. Right, there was sort of this idea that you have these small data sets, and they were high quality, and then all you had common crawl, which was a mess. And any attempt to train on common crawl just led to to junk result.
50:20 So people were thinking like how do I get of larger but high quality subset of Common Crawl. So, we saw the Reddit idea, we saw the the Wikipedia like language model idea. C four used a different idea, and this is saying let's just define a bunch of rules. Which turn out to be fairly effective, so, Basically, they keep lines that ending punctuation have more than 5 wor- words, remove pages with fewer than three sentences, remove page that can any bad words. Remove things like terms of use, of, you know boilerplate. Interestingly, they remove the curly brace, which, you know, filters out a lot of code. So, they weren't clearly at that time, they weren't thinking about code moles. And they filtered out anything that was not English. Ok, the end result is that you have 156 billions tokens, so, you know, 40 sorry, 800 gigabytes of text.
51:25 So this is much larger than the GPT two data set, which was only 40 gigabytes of of text.
51:31 A bit later, there was some analysis done in C four, which showed, here's the websites that were represented. So, you have a bunch of Wikipedia. Patterns was a very common thing. They also released the webtext like document, they also used this idea of links in Reddit posts. Anduh, From what it post with greater than 3 karma, and filter to those pages. So, they were able to Get using this method 17 gigabytes of text using 12 common crawl dumps. Remember web text was 40 gigabytes.
13 GPT-3：质量分类器过滤
52:17 Which means that common crawl is is you know maybe incomplete, or doesn't have everything, because if it had everything, you should be able to hit 40 gigabytes. And they they show that at that time, you could use this data improve on a bunch of NLP your benchmarks.
52:36 OK, so we've seen using rules, things that look like Wikipedia, red links, and now, let's talk about GPT 3. So GPT 3, the data set was, they did their They use common crawl, they did their own processing. There is this web text, which is essentially the GPT two data set, but expanded. Notice that there's some overlap, common crawl does overlap with this, but this is a some sense a more targeted. Distribution. There is this in the paper, it's described as books one and books two, which is miss internet based books corpora. So this Remains a mystery exactly what it is. And Wikipedia. OK, so So, the result was about 500 you know gigabytes of of text 400 billion tokens.
53:33 And to do the common crawl processing They trained a quality classifier to distinguish things they determined deemed to be high quality from the rest, and they did some fuzzy deduplication becauseum, You know. WebText and Common Crawl had dupes.
14 The Pile：22 个高质量领域（含 Gutenberg / Books3 / StackExchange）
53:54 OK, so this paper follow the idea of using a classifier to dis- to train to do quality classification.
54:05 So after GPT 3 came out, this was I think a big event, and there were a number of efforts to try to do things in the open. So one of the initial efforts was the the Pile, which isum, a grassroots effort, from a lucid AI, where they had a bunch of people In this Discord, just jamming and figuring out high quality sources, and they came up with this list in the end. And today, this is. still pretty interesting, I diverse, it hasuh, You know, the common crawl, as Pubmed. This has this thing called Books3, which, you know, we'll talk about archive github, which we talked about. You know Wikipedia, IRC, another book's corpus, philosophy papers, and so on. And so, you know, they included this Enron emails data set, so Enron went bust in 20 2002 andum, All the emails were released, and this is one of the few email data sets we have. which is a weird distribution, I think, for email, but that's that's where you get. So, Project Goenberg was, so let's talk about books a little bit.
55:27 So, Project Goenberg was startedum, In 1971, and it only includes books that received copyright clearance. So mostly books in the public domain. And there's this packaging of Gumberg into PG 19, which is Gumberg's before 2019.
55:54 And which is I guess most of them, because for them to be in public domain, you have to wait 75 years. So books 3, which is uhKind of this interesting the data set that appeared in the pile. So, this is described actually as Oh, you know, 200K books from a shadow library called BiblioTec. They concludes all books from your favorite authors. And at that time, again, 2020, no one was paying attention. It just sad, you know, people used it to train models. And since then, it has been taken down. So, you cannot use or should not use a books 3 anymore.
56:42 We'll come back to books three again. So, stack exchange, you've probably used quite a bit, although I guess these days Maybe less so because of of a AI. There's a bunch of user contribute questions and answers starting 2008 and umYou know, one thing is is interesting about this data set is that it's a sort of a Q and A format, and this is quite close to kind of a real, you know, application. So, normally you think about pre-training data as, here's Wikipedia articles, just like raw text, right? But some parts of the web are actually look like You'll supervise like what you would want to ask your language model. So, you know, which I think maybe helps the model learn certain types of question answering behaviors. So not everything is like super magically emergent. It's like, well, this type of data does exist on the web.
15 MassiveText（Gopher）：规则过滤与 SafeSearch
57:47 So, there's also uhThis data has metadata which is like the number of votes, which are useful for filtering. And again, this the this data is a released in data dumps, that allows you to just download without crawling it.
58:06 OK, so now in 2021, there was another paper from DeepMind. They train a model called Gopher, which was never released, and actually was sort of subsumed by Chinchilla. but the description of the data, I think is actually really good. You should read this paper because I think it's very thorough in the way that they described the data processing. Well, except for the parts where they don't tell you what's in the data. So, there's there's this thing created this massive web. And they had you know C four, which remember was the data set that we looked at before from Google. They train on books, news, GitHub and Wikipedia, they don't say anything about how they they got that. So, for the massive web, they English, they duped, and their quality filtering was also based on manual rules.
59:03 And part of the reason for this is that They had more control over that. So, I think, a- as you can see, there is sort of this like division between people who wanted to use Rules, and people who wanted to use classifiers.
59:25 So, they got you know a massive them, as a name suggests a massive amount of text.
16 LLaMA / RedPajama / SlimPajama
59:31 But, you know, their model was only trained on like a very small fraction of this of this data set.
59:40 OK, so in 2022, llama one came out, and the data set here, this was actually a a very nice paper in that it detailed their Data processing. So, this is probably one of the last non open fully open mo- models that actually talk about their data processing. So, they have Common Crawl. Process with C C net, which you know about. The classification though was whether the page references was a reference of Wikipedia, not whether it is actually a Wikipedia article. So the idea is that well, maybe Wikipedia articles are like too stylized, and we know that Wikipedia articles references a bunch of other articles which are presumably good, so let's call those the good. oh websites. They had you know C four. They did a bunch of processing on GitHub to keep permissive licenses.
1:00:40 Wikipedia, they trained on, you know, books 3 and Project Gutenberg. So, books 3 really got them in a lot of trouble, because they're just announcing the world that, hey, I trained on this data set, and if you look back Oh, this came from, where did it come from? It came from the pile. Oh, it came from a shadow library. So, you know that's why people are talking about their data anymore. So, they looked at archive, and actually used the the latex to do the p-, the processing, psych exchange, and they got a 1.2 trillion tokens. They didn't actually release this data set, but they had Enough of a description in their processing that this was reproduced by together's red pajama V one data set, which was then used in other, you know, sources.
17 RefinedWeb 与 FineWeb
1:01:33 So, as you can see here, Red Pajama V one initially also contain books three, which has now been kind of you know stripped out.
1:01:44 So, various decisions made by early on about copyright actually influence have a fairly big kind of watershed.
1:01:53 OK, so let's talk about refined web. So, this is a paper thathmm, Try to make a point that web data is all you need, right? So the web in some sense is everything, and then we talked about all these specialized sources like there's GitHub and archive and At stack exchange, and they said, well, what happens if we just stick with the web? So, they did a good job of doing the transformation from HTML to text. They filtered using the Gopher rules, which is basically keep things that look like English. They explicitly at that time still said avoid ML based filtering to avoid biases. I don't want to like find an overly narrow subset of of the web. They did some, you know, deduplication, and they had 5 trillion tokens, they released about 600 billion of them. So, this fine web from Hugging Face was a replication of refined web, but improved it.
1:03:00 So, they took all the common crawl dumps at that time. Did some filtering, again using manual rules, because do not inject biases.
18 Dolma
1:03:09 Dedupe, did P I removal. And got, you know, 15 trillion, no tokens. So you can see that the size of the data, da- these data sets, is growing, you know, quite a bit, And so then there is this Stoma, a data set from AI two. and this is umuh, DataSet that includes their own processing of Common Crawl, the stack which we'll talk about be- before C four, which you know about, and a bunch of other things. And the Reddit data set was from this this project, called push push shift, and at that time you could still get this data, and you can You know train on it. before things got locked down. AI two has our own you know crawl of academic papers, so called Scisematic Solar, so they derive a data set based on that. So let's look at their common crawl processing. So, they use a language identification, they So, this was model based, but quality filtering still avoid model, fil- base filtering.
19 DCLM：基于模型的质量过滤
1:04:26 And then they remove toxicity using a rules and classifier.
1:04:34 And they got three three troy and tokens out of that. So, DCLM, I think was maybe a point where this idea of model based quality filtering started to Really kind of become the the the norm here. So, data company, the initial motivation was to define some sort of pipeline so that People can try different data methods, in a standard way and show results. But I think the main way that I think people use this is just using their data set that they release to train models and do other things. So, they process Common Crawl to produce DCM pool. So, this is completely unfiltered. And if you look at these days, this is massive 240 trillion You know tokens. There's probably more than number of tokens that anyone really trains on, but a lot of this is fairly low quality, so then they have Pipeline where all that gets, you know, filtered if you only keep and you know English. You have some sort of rules to really.
1:05:44 narrow it down. You do de-dupe. And then you do this model based filtering, and you end up with, you know, something that's 1.4 percent. And this is the data set that we'll see actually works. You know, pretty well. So, the model based filtering, and this was, you know, s- s- kind of strangely good. So, to train a classifier, they took umOpen Hermes, which is essentially a instruction data that was generated by GPT four, and Elife five, which is a subreddit withuh, Various questions and answers. So, this is let's see.
1:06:28 Like these are ty-, the type of questions that are in ELI 5.
1:06:35 OK, so anyway, you get the idea. So, it's it's kind of weird, but but somehow this this works. So, the negative examples are anything from refined web, which is remember, it's just like auh, A very loosely filtered version of the web, it's basically the web, and it got 3.8 to train tokens. They train a fast class classifier, think about this is a linear classifier.
20 Nemotron-CC：分类器集成与合成数据改写
1:07:01 And then they show that this this magical classify quality classifier outperforms a bunch of you know other things that they They tried. So, DCAM, th- this became sort of a for a while, at least in the open community, a bit of a gold standard for qualityuh, Filtering.
1:07:25 Let's move on to Nividia. So, Nividia, this is from Nividia. They said that, well, DCLM filters too aggressively. It's mo-, removing like most of the data. And remember they only got uhI mean 3.8 trillion tokens, right? So, we need more tokens. So, what do we do? We are going to umDo something more I guess elaborate.
1:07:58 We're going to prompt our existing model to score fine web documents based on Educational value, so we're gonna prompt the language model and said, is this educational or not? Go to a bunch of labels, train a fast text model. So that's one, a classifier, another classifier is just using the DCLM classifier. So, we're gonna use those classifiers. We're also going to use synthetic data. So, this is probably the one of the umI don't know if it's a, it's probably not the first, but one of the the main data sets that really lead it to synthetic data as a for pre training. So, they had for low quality data as deemed by these classifiers, you use a language model to rephrase it to make it more look like Wikipedia. For high quality data We use a language model to generate various tasks. For example, given a Wikipedia article, I can generate question answers, or I can generate like, please summarize this document.
1:09:04 Extract key information from it and so on. So, they resulted in a data set that was 6 trillion tokens. So, this is quite No, substantially larger than DCLM. And just for kind of reference, this is still kind of Small by some, you know, consideration. So llama three was trained on 15 trillion tokens, qwen three was trained on 36 trillion. Although there is the sort of thisum, It's not clear how big these unique tokens these are, because when you look at token accounts, and language modeling papers, some of it's like repeated, like you do multiple epochs, two epochs, that's twice the number of tokens.So you have to. be careful when you look at those numbers. And they show that this the data set is you know better, their high quality subset of Beats the pre- the previous data sets.
1:10:07 OK, so up until now, we've seen a bunch of different methods for filtering. I think all these sort of look Very kind of similar at some level. You take a web crawl, you either decide I'm gonna use rules to filter, I'm gonna use a model. If I'm gonna use a model, then I have to decide what looks like good data. And then try to train a classifier, and classify all my documents and select.
21 The Stack 与 Stack v2：代码数据
1:10:34 And there's a seems to be a trade off between, you can take a lot of common crawl, you can getum, You know 240 You know trillion tokens if you want, but that's probably gonna be really low quality. Or you can get like one trillion tokens, and There's sort of some you know sweet spot in between. Let me t- talk about two final things. So, code and going back to this question of licensing. So, the stack is a very nice project that was trying to make a really good coding data set. Since it was clear by 2022 that Coding was gonna be really kind of important. So what they did in the initial version is that they clone, you know, 137 you know repos. They kept the ones that were permissively licensed, remove example, near duplicates, and resulted in, you know, one, sorry, three terabytes of code. And then t, in 2024, they had a update.
1:11:42 And here, they took more the the metadata like the issues, comments and PRs. From GitHub, the repositories from the software heritage, which we talked about. They also scrape documentation from various websites by calling them. They did a bunch of processing. So, in GitHub repos, you have binary files, which you probably don't want to train on. They have malware, you get rid of that. A lot of GitHub, especially these PRs are bots, so you have to filter that, deduplicate, do PI reduction, there's a lot of pull requests and stuff, and they basically subsample to kind of keep it representative and the data set manageable. They also this this thing which was, I thought was kind of nice, which is that, there's many programming languages out there. There's a lot of Python, lot of C.
1:12:38 But low resources languages like Nim, which I hadn't even heard of, they're not that common. So what they do is, they compile this code. And then into a low level intermediate language, LVM, which everything like C compile can compile into this. And they've just supposed the low resource language and the intermediate representation. So then, the the the language model can actually learn the mapping between the shared low level representation, which has a lot of data on, and the thing that it does has less data on. And just for fun, you add in all the other, you know, good stuff, that you can. So this is sort of, like, mostly code, and then Plus other things that were used to train their coding models.
1:13:33 And one thing about pull requests is that Their poll request and all the metadata is not by its nature linearize sequence, so there has to be some steps to taken to linearize it. So one important thing you have to decide is how much context to provide. So for example, event might just be I change one line of code, right? And that could just be one line, but presumably to learn That you want to provide some context, maybe a few lines around it, maybe the entire file surrounding that diff. And these are some design decisions that you have to make for umFor linearization, and so they, at the end of the day, the tokens that they train on look something like this, where it's sort of this like XML like structured data where you have like the PR.And then you have a bunch. of diffs.
22 CommonPile：只用许可数据能训出好模型吗
1:14:26 And then for comments, you have basically the events of like a comment was posted, what the review state of thatuh, PR was, and so on, so forth. So, it's not just learning how to generate code, but also the software development process around code.
1:14:45 OK, so finally, I'll talk about the common pile. So recall that almost all the data on the internet is copyrighted. Some of it is permissively licensed, some of it is even in the public domain. And while you can appeal to fair use, and say I'm gonna train on it anyway. This is not quite settled, right? So if you're very very risk averse. Then you say, well, let's, like, if I don't know whether this is OK, that's a no. If you take that attitude, then that's whatuh, CommonPile did. So, can you train a good model with only permissively licensed data?
1:15:29 Or rather, how far you can you get? So, this was a project that went and scoured the the internet for all the different types of Data that they could possibly find that was worth training on, that was permissively licensed, it includes your SackV two and code. umTurns out that a lot of government proceedings are actually permissively, you know, licensed, wikis, someuh, Things on the on the web, some new sites are actually permissively license, academic papers. online, you know forums. Things that are in the public domain, educational resources, and and so on. So in the end, there were 8 terabytes of, you know, data, which is actually pretty good for permissively licensed data. doing this project is actually much harder than maybe it sounds at first glance. It's not just simple as like, oh, you look at the license, you're like, yep, Apache, a good, or or CC, a good. There's some subtleties.
1:16:44 versus license laundering. so, people are kind of sloppy with licenses. so, people might take some copyright work and just, you know, put it on, slap a cc by on it, anyone can write this on the internet. And it's kind of hard to tell whether this is like a real real or not. There's also this a common practice where You know, for example, Doma is is permissively licensed. Remember Doma is a AI two collection, but collection licenses don't extend to the individual. works. So, the collection is like, you know, is, you, I mean first of all, there's a question where collections, I think collections can't be copyright, but you can, I guess you can put a license on. Anything, but the individual works, are not necessarily of, good.
1:17:41 So, you can't just look at data sets on Hugging Face, and you Many datas says a hungry face, you see they have a permissive license. But if you dig deeper, it's actually not permissive license at the individual level. And they also made a decision to s forego training on any synthetic data. Because training language models on syn- on license data is is unclear. So probably it's going f- fine umBecause like technically, these are these are like open way models with MIT license. You can you know do whatever, you can use them however you want, butum, You know, these language models were presumably also trained on on license data, so it's a little bit of kind of data laundering if you are really kind of honest here.
1:18:34 So, so with this data, they compared with a bunch of other models, like the first llama. NPT, and Quine, and on a bunch of benchmarks, they show that this is not bad. I would say it's not nearly as, and it's not as good as a Quine. Models for sure, but it's certainly outperforming the very old kind of ver, this is like, you know, from 203, I guess. So, quite old models.
23 小结：数据不会从天而降
1:19:11 So, I would say the conclusion here is that you can do reasonably, but it's, I think it's still A pretty tough to compete without doing more tokens. But I don't think this is the final word on it, and I think you can probably eek more out of public lic permissive licenses if you try.
1:19:30 OK, so to summarize this lecture here. So, maybe hopefully I now, I've giving you some appreciation that data is a very rich topic. It's not that data just falls from the sky. Or you just go on hugging face, and you download a data data set. Data has to come from somewhere, and there's also a technical, but also social context in which data comes. You know, because at the top, there's the internet is a bunch of live services, right? There's hack to, has, someone has to do the work of producing raw data, whether it be the website that gives you dumps, or someone has to build a crawler. or or something. and then you someone has to make a decision of how to process it into usable form.
1:20:17 filtering, transformation, duplication. All these things have an impact on the quality of your final language model. And notice that filtering is like probably one of the most kind of important things. How do you go from 200 trillion tokens to, you know, less than, like 3 trillion tokens. I think that obviously is a huge reduction, which I think merits a lot of attention.
1:20:45 Data is in some sense a key ingredient that differentiates language models. A lot of language models are roughly the same kind of Transformer architecture. But, you know, data is some depending on how you process it can make a pretty big difference. There's plenty of legal and ethical issues around data, more than I can get into this lecture. And also this this process is very messy. Unlike you know some of the other parts of this class where things are Maybe more based on kind of first principles. You know, data processing is right now, at least a lot just based on kind of vibes you set off. Define this classifier, define this rule, you set some threshold. So, there are many opportunities to improve.
1:21:36 So, as you do your assignment four, maybe think about whether there's a better way to do this, and this could be Maybe is a research direction. OK, so that's it for today. So next time, I'll continue talking about data, I'll do, talk a bit about post training data, and a bit more about you know, filtering.