2 ms·
Gonna say you’re way off there. Once you decompress common crawl and index it for FTS and put it on fast storage you’re in for some serious pain, and that’s bef
by bevekspldnw 3y ago
Gonna say you’re way off there. Once you decompress common crawl and index it for FTS and put it on fast storage you’re in for some serious pain, and that’s before you even put it in your ML pipeline.
Even refined web runs about 2TB once loaded into Postgres with TS vector columns, and that’s a substantially smaller dataset than common crawl.
It’s not just a dumping a to of zip files on your NAS, it’s making the data responsive and usable.
- Dylan16807 3y agoHow important is full text search for training an LLM, compared to a pile of zip files with a gigabyte of text each?
- michaelt 3y agoMaybe not full full text search, but you'll generally want to remove the duplicates and suchlike.
- Dylan16807 3y agoI guess you want some fast extra storage for as long as it takes to run https://github.com/chatnoir-eu/chatnoir-copycat https://github.com/chatnoir-eu/chatnoir-copycat but that's a very temporary thing.
- bevekspldnw 3y agomy guess is mistral is so good for its size because they are doing some real precise pre ingestion selection and sorting. You need FTS to do that work.