6 ms·
This ties into the bias-variance tradeoff (https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff https://en.wikipedia.org/wiki/Bias%E2%80%93variance_trad
by minimaxir 4mo ago
This ties into the bias-variance tradeoff (https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff) common with building non-LLM models. The solutions can only be a) figure out how to get LLMs smaller with similar performance so they don't memorize things/game the benchmarks and b) build benchmarks that are indeed comprehensive for all real-world data, which is infeasible.
- sigbottle 4mo agoI mean, people always say there are tradeoffs, until you reach the next frontier, in which there are tradeoffs at said frontier, and the next, and the next, etc. In one sense, yes, tradeoffs are inescapable as the scope expands to the maximal possible scope. In another sense... it depends on the level of abstraction we're talking about.