3 ms·
Pretraining on the test set is all you need. LLM benchmarks are horribly broken. IMHO there is better signal in just looking at parameter counts.
by qeternity 2y ago
Pretraining on the test set is all you need.
LLM benchmarks are horribly broken. IMHO there is better signal in just looking at parameter counts.
- BoorishBears 2y agoLooking at tokens they were trained on is also a really great indicator of world understanding. Llama 3 is a game changer for some usecases because there's finally a model that understands the world deeply as opposed to typical models which can be fine tuned into hyper specific tasks, but generalize poorly, especially in D2C usecases where someone might probe the model's knowledge