2 ms·
How important is full text search for training an LLM, compared to a pile of zip files with a gigabyte of text each?
by Dylan16807 3y ago
How important is full text search for training an LLM, compared to a pile of zip files with a gigabyte of text each?
- michaelt 3y agoMaybe not full full text search, but you'll generally want to remove the duplicates and suchlike.
- Dylan16807 3y agoI guess you want some fast extra storage for as long as it takes to run https://github.com/chatnoir-eu/chatnoir-copycat https://github.com/chatnoir-eu/chatnoir-copycat but that's a very temporary thing.
- bevekspldnw 3y agomy guess is mistral is so good for its size because they are doing some real precise pre ingestion selection and sorting. You need FTS to do that work.