4 ms·
There isn't enough training data though, is there? The "secret sauce" of LLMs is the vast amount of training data available + the compute to process it all.
by MarkusQ 6mo ago
There isn't enough training data though, is there? The "secret sauce" of LLMs is the vast amount of training data available + the compute to process it all.
- algoth1 6mo agoI think you could probably feed a copy of a toki pona grammar book to a big model, and have it produce ‘infinite’ training data
- eden-u4 6mo agoThere are not enough samples in that book to generate new "infinite" data.
- MarkusQ 6mo agoThis is essentially a distillation on the bigger model; you'd wind up surfacing a lot of artifacts from the host model, amplifying them in the same way repeated photocopying introduces errors. https://dailyai.com/2025/05/create-a-replica-of-this-image-dont-change-anything-ai-trend-takes-off/ https://dailyai.com/2025/05/create-a-replica-of-this-image-d...