3 ms·

http://www.incompleteideas.net/IncIdeas/BitterLesson.html http://www.incompleteideas.net/IncIdeas/BitterLesson.html > One thing that should be learned from the

http://www.incompleteideas.net/IncIdeas/BitterLesson.html http://www.incompleteideas.net/IncIdeas/BitterLesson.html > One thing that should be learned from the bitter lesson is the great power of general purpose methods, of methods that continue to scale with increased computation even as the available computation becomes very great. The two methods that seem to scale arbitrarily in this way are search and learning.

anon373839 2mo ago
So what? There are physical and economic ceilings on dumb computation scaling.
cyanydeez 2mo ago
americas tech stack always ends up bloated. not everything is worth learning. endlessly knowing about pokemon is not delivering value proposition cancer also grows carelessly.
kaashif 2mo ago
Right, and if you come up with an efficiency gain that makes scaling better, e.g. a 50% reduction in required compute. Or even asymptotic improvements e.g. moving from quadratic to linear. Then you're much much better off. There is nothing about the bitter lesson that says just be dumb and pour money into a hole, you still have to invent the methods to scale well, and being under immense pressure with constraints seems likely to produce that research.
- Enginerrrd 2mo ago
  It reminds me a bit of the tyranny of the rocket equation. You can always scale your fuel to get a little more delta V, with ever diminishing returns. …but for something like a DEEP space/interstellar mission, it almost always pays to wait a few more years for a faster propulsion system because you’ll get there fastest by always delaying your launch and chasing better technology. I’m not sure how well the analogy holds up, or if there’s anything to be learned from it though.
  - RetroTechie 2mo ago
    https://en.wikipedia.org/wiki/Interstellar_travel#Wait_calculation https://en.wikipedia.org/wiki/Interstellar_travel#Wait_calcu... Certainly applies more general imho. Constrained by some resource -> invest resources elsewhere, and/or invest in reducing the constraint(s) encountered.
andai 2mo ago
The implication here is that the only gains left to be had are from scale. That we are already maximally efficient. If that's true, then how has OpenAI repeatedly bragged about reducing the cost of their models by orders of magnitude? (And DeepSeek Flash even more so, of course.) But we have not been maximally efficient, we keep gaining efficiency. If we keep gaining efficiency, why should we assume it is impossible to gain more?

anon373839 2mo ago

So what? There are physical and economic ceilings on dumb computation scaling.

cyanydeez 2mo ago

americas tech stack always ends up bloated. not everything is worth learning. endlessly knowing about pokemon is not delivering value proposition cancer also grows carelessly.

kaashif 2mo ago

Right, and if you come up with an efficiency gain that makes scaling better, e.g. a 50% reduction in required compute. Or even asymptotic improvements e.g. moving from quadratic to linear. Then you're much much better off. There is nothing about the bitter lesson that says just be dumb and pour money into a hole, you still have to invent the methods to scale well, and being under immense pressure with constraints seems likely to produce that research.

Enginerrrd 2mo ago
It reminds me a bit of the tyranny of the rocket equation. You can always scale your fuel to get a little more delta V, with ever diminishing returns. …but for something like a DEEP space/interstellar mission, it almost always pays to wait a few more years for a faster propulsion system because you’ll get there fastest by always delaying your launch and chasing better technology. I’m not sure how well the analogy holds up, or if there’s anything to be learned from it though.
- RetroTechie 2mo ago
  https://en.wikipedia.org/wiki/Interstellar_travel#Wait_calculation https://en.wikipedia.org/wiki/Interstellar_travel#Wait_calcu... Certainly applies more general imho. Constrained by some resource -> invest resources elsewhere, and/or invest in reducing the constraint(s) encountered.

andai 2mo ago

The implication here is that the only gains left to be had are from scale. That we are already maximally efficient. If that's true, then how has OpenAI repeatedly bragged about reducing the cost of their models by orders of magnitude? (And DeepSeek Flash even more so, of course.) But we have not been maximally efficient, we keep gaining efficiency. If we keep gaining efficiency, why should we assume it is impossible to gain more?