4 ms·

This is an analysis of the "Books3" dataset. The original download link was taken down [1] but it's still available from Academic Torrents [2]. [1] https://tor

• sillysaurusx 3y ago
  There also exists a working direct download link which I can’t post because the Danish Rights Alliance is following me around DMCA’ing anything to that effect. They already struck my Twitter once. But, BitTorrent is probably the only way a dataset like this can survive now. It’s a damn shame since OpenAI are the ones making money, and researchers are simply trying to replicate their scientific efforts. Original books3 announcement thread: https://x.com/theshawwn/status/1320282149329784833?s=61&t=jQbmCk1JqL7depzFWJNuPA https://x.com/theshawwn/status/1320282149329784833?s=61&t=jQ... An article on books3 from Gizmodo, which I quite like: https://gizmodo.com/anti-piracy-group-takes-ai-training-dataset-books3-off-1850743763 https://gizmodo.com/anti-piracy-group-takes-ai-training-data... By the way, you can use aria2c to download just the books3 part of that torrent. Use aria2c —-select-file=44 and pass the torrent url. Takes about 30 minutes. Plus most people probably don’t have 800GB free.
  • marginalia_nu 3y ago
    > Plus most people probably don’t have 800GB free. They should make space. Data sets in general are hugely slept on among hobbyists. Even just what's publicly available, like Wikipedia's dumps; you can do so much fun stuff with them and a few days of compute time!