4 ms·

From a preservation angle, how big is Books3? Is it easy enough for mortals to mirror for that unlikely future where it might be possible to self-train reasonab

From a preservation angle, how big is Books3? Is it easy enough for mortals to mirror for that unlikely future where it might be possible to self-train reasonably good models from scratch if provisioned with data?

bvm 3y ago
800gb
- CamperBob2 3y ago
  Where are you getting that figure?
  - jwitthuhn 3y ago
    That is the size of 'The Pile' which contains books3 among many other things.
banana_giraffe 3y ago
books3.tar.gz itself is ~37gb compressed. Often really the entire "The Pile" dataset (composed of both the mostly compressed archives, along with a ~450gb compressed jsonl compilation of the data) is being discussed. That's around 825gb.
- Tokumei-no-hito 3y ago
  Out of curiosity why is jsonl so popular in the ML space?
  - fbdab103 3y ago
    Not speaking for ML, but I love jsonl as a distribution format. Lots of storage overhead vs a more appropriate bulk container, but it makes it trivial to stream, sample, concatenate, etc. Anything else (eg csv or parquet) is going to require better tooling and/or just a little shell magic to handle headers.
- fbdab103 3y ago
  That is shockingly approachable for a large fraction of English literature.
  - deleted 3y ago
    [deleted]
sirodoht 3y ago
39.52Gb gzipped and 108.79GB uncompressed.
- elahieh 3y ago
  In bytes: 39,516,981,435 books3.tar.gz -- 36.5% compression ratio 108,371,325,720 tar -- uncompressed Recompressed with 7-zip and xz: 25,221,357,605 b3.7z # with flags: -m0=ppmd (23.3%) 27,077,329,052 books3.tar.xz # with flags: -e9 (25.0%) To see what other slower compressors could do, I checked results from a random 1,000 books. MCM would achieve about half the original tar.gz file size, but it's very slow. 104,043,048 q-x11.mcm - 19.0% -- from q.tar 106,181,190 q-m9.mcm 114,564,722 q.tar.bsc-m03-b1000000000 118,636,713 q.tar.bsc-m03 130,621,411 q.ppmd -- 23.9% 137,874,349 q-s29.tar.lzip 138,274,566 qultra.7z 141,815,136 q.lzma 142,626,724 q9.xz -- 26.1% 146,786,015 q.brotli 146,933,603 q.p7zip 146,933,619 q.7z 149,371,106 q9.bz2 198,213,051 q.gz -- gzip -9 -- 36.3% 198,213,193 q.zip 227,074,912 q.lz4 228,463,000 q.lzop 546,201,600 q.tar

bvm 3y ago

800gb

CamperBob2 3y ago
Where are you getting that figure?
- jwitthuhn 3y ago
  That is the size of 'The Pile' which contains books3 among many other things.

banana_giraffe 3y ago

books3.tar.gz itself is ~37gb compressed. Often really the entire "The Pile" dataset (composed of both the mostly compressed archives, along with a ~450gb compressed jsonl compilation of the data) is being discussed. That's around 825gb.

Tokumei-no-hito 3y ago
Out of curiosity why is jsonl so popular in the ML space?
- fbdab103 3y ago
  Not speaking for ML, but I love jsonl as a distribution format. Lots of storage overhead vs a more appropriate bulk container, but it makes it trivial to stream, sample, concatenate, etc. Anything else (eg csv or parquet) is going to require better tooling and/or just a little shell magic to handle headers.
fbdab103 3y ago
That is shockingly approachable for a large fraction of English literature.
- deleted 3y ago
  [deleted]

sirodoht 3y ago

39.52Gb gzipped and 108.79GB uncompressed.

elahieh 3y ago
In bytes: 39,516,981,435 books3.tar.gz -- 36.5% compression ratio 108,371,325,720 tar -- uncompressed Recompressed with 7-zip and xz: 25,221,357,605 b3.7z # with flags: -m0=ppmd (23.3%) 27,077,329,052 books3.tar.xz # with flags: -e9 (25.0%) To see what other slower compressors could do, I checked results from a random 1,000 books. MCM would achieve about half the original tar.gz file size, but it's very slow. 104,043,048 q-x11.mcm - 19.0% -- from q.tar 106,181,190 q-m9.mcm 114,564,722 q.tar.bsc-m03-b1000000000 118,636,713 q.tar.bsc-m03 130,621,411 q.ppmd -- 23.9% 137,874,349 q-s29.tar.lzip 138,274,566 qultra.7z 141,815,136 q.lzma 142,626,724 q9.xz -- 26.1% 146,786,015 q.brotli 146,933,603 q.p7zip 146,933,619 q.7z 149,371,106 q9.bz2 198,213,051 q.gz -- gzip -9 -- 36.3% 198,213,193 q.zip 227,074,912 q.lz4 228,463,000 q.lzop 546,201,600 q.tar