10 ms·
The training data - all 1.2 trillion tokens - can be downloaded by grabbing each of the 2,084 URLs listed here: https://data.together.xyz/redpajama-data-1T/v1.0
by simonw 4y ago
The training data - all 1.2 trillion tokens - can be downloaded by grabbing each of the 2,084 URLs listed here: https://data.together.xyz/redpajama-data-1T/v1.0.0/urls.txt https://data.together.xyz/redpajama-data-1T/v1.0.0/urls.txt
I ran a HEAD request against them all to sum up the total file size, and it's 2.67TB total.
Here's a Datasette Lite URL that lets you explore the size metadata about those files: https://lite.datasette.io/?json=https://gist.github.com/simonw/73d15c0dd1025d1196829740bacf4464#/data/raw?_facet=top_folder&_facet=top_folders https://lite.datasette.io/?json=https://gist.github.com/simo...
And a SQL query that shows the breakdown across the different sources:
https://lite.datasette.io/?json=https://gist.github.com/simonw/73d15c0dd1025d1196829740bacf4464#/data?sql=select+top_folder%2C+sum%28size_gb%29+from+raw+group+by+top_folder+order+by+sum%28size_gb%29+desc https://lite.datasette.io/?json=https://gist.github.com/simo...
Sizes here are in GB:
common_crawl 1341.6166818914935
c4 806.7667234372348
github 212.1786002581939
wikipedia 111.89125544670969
book 100.43162744678557
arxiv 87.35323827341199
stackexchange 74.54870238155127
Common Crawl is in there a few times - they have the following folders:
common_crawl/2020-05 198 files
common_crawl/2021-04 176 files
common_crawl/2023-06 175 files
common_crawl/2022-05 157 files
common_crawl/2019-30 153 files
And then C4 as well, which is "a colossal, cleaned version of Common Crawl's web crawl corpus. It was based on Common Crawl dataset": https://paperswithcode.com/dataset/c4 https://paperswithcode.com/dataset/c4
- fnands 4y agoNice. Thanks for the summary. So ~4x the size of the Pile, any idea how it stacks up in terms of quality to other big datasets?
- doctoboggan 4y agoI am a little concerned that they have only about 60% of the code tokens (GitHub and stackexchange). Given that so far the only concrete use case I have for LLMs is coding assistance I wouldn't want this open source model to be and less quality in that area. In your opinion do you think this will hamper the model at all? Or is it still more than enough to get good coding assistance?
- simonw 4y agoNo idea! I wonder how hard it would be to fine-tune something built on RedPajama on further code examples to improve performance there.
- sp332 4y agoAs mentioned in the post, the smaller models are trained well past "compute-optimal" amounts of data and I would expect are well into diminishing returns. On the other hand, large models are good one-shot and few-shot learners, and might be able to pick up enough context from your prompt alone to be useable, even if it wasn't specifically trained on your use case.
- Minus0 4y agoIn this context compute optimal isn't quite the same as diminishing returns. If you look at the loss graphs in the Llama paper, you can see that even the curves for the smaller models were still going down at the time they stopped training and weren't anywhere near plateauing yet. LLMs are notoriously data hungry and will take a long time to reach convergence. Compute optimal here means the point at which it makes sense to move from a smaller to a larger model assuming that: (a) you have a fixed compute budget of FLOPs, and (b) you want to train the best model possible. The problem is that this applies only to training and assumes nothing about the cost of inference. If you actually need to deploy these trained models and support them long-term for hundreds, thousands, even millions of people to use, would you rather deploy a 13B model or a 30B model at the same level of quality, even if the 13B model would be more costly to train? There is going to be a point at which these models plateau and further improvement will not be possible without moving to a larger model, but Llama doesn't get there quite yet.
- deleted 4y ago[deleted]
- bkm 4y agoRelevant: https://twitter.com/abacaj/status/1647999551964323844 https://twitter.com/abacaj/status/1647999551964323844
- simonw 4y agoWrote this up as a blog post: https://simonwillison.net/2023/Apr/17/redpajama-data/ https://simonwillison.net/2023/Apr/17/redpajama-data/
- csris 4y agoHi! I'm the VP of Engineering at Together. Thanks for writing up these instructions! FYI, you can also download all the files with one wget command: wget -i https://data.together.xyz/redpajama-data-1T/v1.0.0/urls.txt This is also mentioned on the dataset card for redpajama-data-1T on Huggingface [1]. [1]: https://huggingface.co/datasets/togethercomputer/RedPajama-Data-1T https://huggingface.co/datasets/togethercomputer/RedPajama-D...
- simonw 4y agoI made sure to include that in my blog post - along with a note that you need 2.67TB of disk space first!
- bhaney 4y ago> you need 2.67TB of disk space The data looks like it should compress pretty well. If you use something like btrfs's transparent compression, I wouldn't be surprised if it all fit in less than 0.75TB of disk space while still being usable to any tool that expects uncompressed data. Edit: It looks like some of this data is already compressed, so maybe not.
- csris 4y agoNote that you also need about 5TB of disk for the full decompressed dataset. However, only common crawl are compressed in jsonl.zst, everything else is uncompressed jsonl.
- deleted 4y ago[deleted]
- afro88 4y agoInteresting they're allowed to use stackexchange. I don't know much about the legalities of scraping. Was this an agreement between them, or is it simply ok to scrape and use the data in a model?
- progbits 4y agohttps://stackoverflow.com/help/licensing https://stackoverflow.com/help/licensing Doesn't this imply the produced model has to be CC-BY-SA too?
- wongarsu 4y agoBy that line of reasoning, GitHub copilot would have to be GPL. Until somebody fights about this in court we don't really know. But even in the worst case the CC-BY-SA is one of the easier licenses to fulfill, not much worse than the MIT-licensed code contained in the dataset.
- gattilorenz 4y agoWelcome to this can of worms. CC-BY-SA content needs attribution too, but I don’t see the(se) model(s) in the current state being able to do so. I imagine we’re gonna see the IBM PC bios/Unix/ReactOS “tainted code” arguments again in court, this time is not the human who is more-or-less knowingly responsible for sneaking in copyrighted code.
- gpm 4y agoNo. It means that if they are doing something that is prohibited by copyright law (without a license) then it needs to be CC-BY-SA. The only theory under which training this sort of model is remotely legal is that doing so is not prohibited by copyright law in the first place. If that theory is correct they don't need a license, and they don't need to abide by any terms of licenses that they were granted without asking. If that theory is incorrect, they have to comply with the stackoverflow license, but they also have to not use any of the (massive amounts) of unlicensed training data they are using, and comply with the numerous incompatible licenses other sources of training data are licensed under. In other words it's impossible to do this.
- rcpt 4y agoI'm kind of surprised how small that dataset is