7 ms·
How is it not a violation of the bitter lesson? You're trying to correct the model after the fact using human logic, where the bitter lesson would want you to j
by achierius 1y ago
How is it not a violation of the bitter lesson? You're trying to correct the model after the fact using human logic, where the bitter lesson would want you to just train a better model.
Not that I think that goes against your point -- I think it's rather a problem with the bitter lesson.
- Der_Einzige 1y agoThe primary argument for why it's not a violation is that the heuristic is (almost) free. LLM designed samplers are and will probably be better - but in order to start the recursive self-improvement engine a few free heuristics will be needed. The bitter lesson critique is that the human designed heuristics were not free, and harmed the notion of "letting the computer figure it out" by slowing down training. High temp sampling is very important for half-way decent synthetic data generation and thus enabling "letting the computer figure it out" for natural language. Better sampling is the only way to make high temperature generations coherent.