3 ms·
> openAI seem to have forgotten 'The Bitter Lesson'. They have been going at things in an extremely brute force way. Isn't the point of 'The Bitter Lesson' pre
by crubier 2y ago
> openAI seem to have forgotten 'The Bitter Lesson'. They have been going at things in an extremely brute force way.
Isn't the point of 'The Bitter Lesson' precisely that in the end, brute force wins, and hand-crafted optimizations like the ones you mention llama and deepseek use are bound to lose in the end?
- AnotherGoodName 2y agoImho the tldr is that the wins are always from 'scaling search and learning'. Any customisations that aren't related to the above are destined to be overtaken by someone that can improve the scaling of compute. OpenAI do not seem to be doing as much to improve the scaling of the compute in software terms (they are doing a lot in hardware terms admitedly). They have models at the top of the charts for various benchmarks right now but it feels like a temporary win from chasing those benchmarks outside of the focus of scaling compute.