6 ms·

Can someone explain to me like a noob how this ("this" being the data hosting and download access) works? Am I understanding correctly that they are releasing c

by natch 3y ago

Can someone explain to me like a noob how this ("this" being the data hosting and download access) works? Am I understanding correctly that they are releasing code for filtering common crawl data that is out there, and the result of this filtering is the dataset? To further elaborate on this (possibly wrong) understanding: - Each person can then run their own processing, possibly duplicating effort(?) ...but on the good side, giving each person the ability to tweak the pipeline to suit their needs. - There is no torrent of already processed data because __________? - Looking at file lists for this on Hugging Face, some files seem to be stored in Git Large File Storage. Are these already processed files that together constitute the dataset? Or are these Common Crawl files that are selectively listed and pulled for processing? What options are there to preemptively obtain a copy, in case of any possible eventual takedown of the dataset, any assurances about access aside? I am reminded of parts of the pile. Obviously I'm super clueless here... please be gentle and share anything you know or correct anything I've got wrong. I'm not asking about training, if that wasn't obvious. Just about obtaining the dataset.

zhangce 3y ago
What we make available is: -- (A) the dataset after pre-processing the raw CommonCrawl data (e.g., text extraction and language identification) and some minimal filtering; and (B) for each document in (A), we also pre-computed 40+ of "features" (we call the "quality annotations") you can use to further filter it or deduplicate it. For example, one such feature is "how similar this document is to Wikipedia". -- (A) is around 30T tokens, but you might want to use features in (B) to further filter/dedup it down, e.g., to 5T. For example, if in your application documents similar to Wikipedia are the most helpful documents, you can take the top documents with the highest score for the feature "how similar this document is to Wikipedia". Of course, the really interesting case happens when you consider a larger subset of these features (or maybe even automatically learn what the best way of filtering it is). Our goal is to make this as flexible as possible such that you can fit this into your own application. What we have released is both (A) and (B) If you have any questions, please let us know! Thanks for your interests, have fun with the data!
- natch 3y ago
  Thanks. > how similar this document is to Wikipedia So that’s a measure of how similar it is to the background vector of all (language in focus) Wikipedia data?
  - zhangce 3y ago
    There are actually a few ways to do this; and we have four: - `rps_doc_ml_wikiref_score`: a classifier that classifiers random webpage with Wiki references (used in Llama-1) - `ccnet_perplexity`: perplexity of an LM trained on Wikipedia (used in CCNet) - `rps_doc_ml_wikipedia_score`: classifier prediction for the document being a Wikipedia article - `rps_doc_wikipedia_importance`: Used in https://arxiv.org/abs/2302.03169 https://arxiv.org/abs/2302.03169 You can see the full table here: https://together.ai/blog/redpajama-data-v2 https://together.ai/blog/redpajama-data-v2

by natch 3y ago

zhangce 3y ago

What we make available is: -- (A) the dataset after pre-processing the raw CommonCrawl data (e.g., text extraction and language identification) and some minimal filtering; and (B) for each document in (A), we also pre-computed 40+ of "features" (we call the "quality annotations") you can use to further filter it or deduplicate it. For example, one such feature is "how similar this document is to Wikipedia". -- (A) is around 30T tokens, but you might want to use features in (B) to further filter/dedup it down, e.g., to 5T. For example, if in your application documents similar to Wikipedia are the most helpful documents, you can take the top documents with the highest score for the feature "how similar this document is to Wikipedia". Of course, the really interesting case happens when you consider a larger subset of these features (or maybe even automatically learn what the best way of filtering it is). Our goal is to make this as flexible as possible such that you can fit this into your own application. What we have released is both (A) and (B) If you have any questions, please let us know! Thanks for your interests, have fun with the data!

natch 3y ago
Thanks. > how similar this document is to Wikipedia So that’s a measure of how similar it is to the background vector of all (language in focus) Wikipedia data?
- zhangce 3y ago
  There are actually a few ways to do this; and we have four: - `rps_doc_ml_wikiref_score`: a classifier that classifiers random webpage with Wiki references (used in Llama-1) - `ccnet_perplexity`: perplexity of an LM trained on Wikipedia (used in CCNet) - `rps_doc_ml_wikipedia_score`: classifier prediction for the document being a Wikipedia article - `rps_doc_wikipedia_importance`: Used in https://arxiv.org/abs/2302.03169 https://arxiv.org/abs/2302.03169 You can see the full table here: https://together.ai/blog/redpajama-data-v2 https://together.ai/blog/redpajama-data-v2