csris

3 ms·

1. 1.▲
   by csris 2y ago

   Together's Slurm configs model GPUs as GRES resources [1]. Jobs then specify the number of GPUs that they need with flags like `--gres=gpu:8`. [1]: https://slurm.schedmd.com/gres.html
2. 2.▲
   by csris 3y ago

   Note that you also need about 5TB of disk for the full decompressed dataset. However, only common crawl are compressed in jsonl.zst, everything else is uncompressed jsonl.
3. 3.▲
   by csris 3y ago

   Nice catch! We sampled the github dataset to match the total # tokens seen by LLaMA during training: ~64B tokens (they only pass through 0.64 of their total Github dataset according to the paper). We have a lot of Github data and will make
4. 4.▲
   by csris 3y ago

   Hi! I'm the VP of Engineering at Together. Thanks for writing up these instructions! FYI, you can also download all the files with one wget command: wget -i https://data.together.xyz/redpajama-data-1T/v1.0.0&#x2