5 ms·

For a blog post of 1,200 words the bitter lesson has done more damage to AI research and funding than blowing up a nuclear bomb at neurips would. Every time I

For a blog post of 1,200 words the bitter lesson has done more damage to AI research and funding than blowing up a nuclear bomb at neurips would. Every time I try to write a reasonable blog post about why it's wrong it blows up to tens of thousands of words and no one can be bothered to read it, let alone the supporting citations. In the spirit of low effort anec-data pulled from memory: The raw compute needed to brute force any problem can only be known after the problem is solved. There is no sane upper limit to how much computation, memory and data any given task will take and humans are terrible at estimating how hard tasks actually are. We are after all only 60 years late for the undergraduate summer project that would solve computer vision. Today VLMs are the best brute force approach to solving computer vision we have, and they look like they will take a PB of state to solve and the compute needed to train them will be available some time around 2040. What do we do with the problems that are too hard to solve with the limited compute that we have? Lie down for 80 years and wait for compute to catch up? Or solve a smaller problem using specialized tricks that don't require a $10B super computer to build? The bitter lesson is nothing of the sort, there is plenty of space for thinking hard, and there always will be.

RobinL 2y ago
I appreciate this comment. I'm currently working hard on something seemly straightforward (address matching) and sometimes I feel demotivated because it feels like whatever progress I make, the bitter lesson will get me in the end. Reading your comment made me feel that maybe it's worth the effort after all. I have also taken some comfort in the fact that current LLMs cannot perform this task very well.
- uxcolumbo 2y ago
  Can you provide more about the problems in address matching and what you are trying to solve? Do you mean street address matching? Isn’t that already solved? (excuse the naive question)
  - kridsdale1 2y ago
    I don’t know of any open source solution. (I work in mapping)
  - RobinL 2y ago
    The problem is that people write addresses down in different ways (I'm in the UK) You will typically have a master/canonical list of addresses from an official source. In the UK that's Ordnance Survey's AddressBase. You will then have 'messy addresses' that humans have written down. For simple addresses, they'll often be the same as the master version (e.g. 5 Rainbow Road, Hemel Hempstead, AB1 2BC). But there are many harder addresses that exhibit lots of variations, especially flats and subunits. These may all be the same: Flat A 1 High Street vs 1A High Street vs Basement Flat, 1 High Street There no guarantee there will be a number in the address. In: 'THE OLD FARM COTTAGE PAD FARM BADGERCROFT ROAD PIKING', 'THE' and PAD FARM' may be missing, which doesn't seem like a problem until you find out there's also a PAD FARM COTTAGE on Badgercroft Road. There's no guarantee tokens will be in the same order. FWIW, my work is open source, and it's here: https://github.com/RobinL/uk_address_matcher https://github.com/RobinL/uk_address_matcher
    - uxcolumbo 2y ago
      Thanks for explaining. Sounds messy indeed. Wasn’t there this startup trying to offer an alternative? Three words to describe a location or something. But of course it’s not really easy to change an address system, so using AI to make the existing system less messy might be more pragmatic.
sriku 2y ago
Agree (though that's perhaps not a charitable interpretation of TBL). Prematurely articulated principles can do a lot of damage. Btw I found the blog post to be one of the lowest quality one in terms of information content posted on HN - almost like it was written by chatgpt or something.
fxtentacle 2y ago
In my experience, "the bitter lesson" is just wrong. Yes, a hypothetical AI model that has a suitable architecture so that it can memorize your handcrafted features can improve upon them, if it is perfectly trained. But that's a huge if. There's a reason that large AI models only work with some architectures and you need to randomly initialize tens to hundreds of times and you need exactly the right optimizer with the right hyperparameters and then the right training data in the right order... and the reason is that generic function approximators get stuck in a local minima incredibly easy. People used to write handcrafted features to limit the optimization base area. Now people use specific pretraining data and initialization functions to do the same. It's just a different way to express the same constraints. In short, the "bitter lesson" assumes a change that did not happen yet.
- imtringued 2y ago
  >and the reason is that generic function approximators get stuck in a local minima incredibly easy. The key insight in massively over parameterized models (aka LLMs) has been that all the local minima are very similar. Picking one over the other doesn't actually benefit you that much.
  - fxtentacle 2y ago
    Absolutely not! LLM training quite easily gets stuck. People call it "divergence" or "loss spike". See for example: https://dl.acm.org/doi/10.5555/3524938.3525354 https://dl.acm.org/doi/10.5555/3524938.3525354 "Gradient optimization with attention layers can be notoriously difficult requiring tricks such as learning rate warmup to prevent divergence.“ "we then propose a new weight initialization scheme with theoretical justification, that enables training without warmup" Or "Initialization of Large Language Models via Reparameterization to Mitigate Loss Spikes" https://arxiv.org/html/2410.05052v1 https://arxiv.org/html/2410.05052v1 If all local minima were similar, then these papers would represent a lot of wasted effort. Or said in a different way: If labs like DeepMind are willing to pay for large teams that are trying to optimize initialization, then surely this represents a real existing problem.
    - 2y ago
mmoskal 2y ago
The bitter lesson is about the balance of human ingenuity and compute being thrown at the problem. We've seen a few years of LLM compute being scaled up 10x every year, but this is hitting limits (fabs), and we will see more human effort, as it becomes compartively cheaper. Also the current crop of models are inherently limited. Even for something as simple as following a JSON schema, models alone are not good enough [0] Of course as the Moore law refuses to die, we'll continue seeing 1.5-2x or so every year, but that's far from 10x. [0] https://openai.com/index/introducing-structured-outputs-in-the-api/ https://openai.com/index/introducing-structured-outputs-in-t... - see plot
- noosphr 2y ago
  >Of course as the Moore law refuses to die, we'll continue seeing 1.5-2x or so every year, but that's far from 10x. This is another one of those anec-data throw away sentences that take thousands of words to disprove - with a lot of graphs - that no one reads. More hot takes: Moores law has been effectively dead since the Pentium 4 on CPUs. It's been dead on GPUs since 2020. Right now we're not seeing a 1.5-2x grow of compute per year. We've seen zero growth for 5. The only way that GPUs have gotten faster is by running ever hotter, and building out a trillion dollars worth of data centers. No one cares because the current hotness in AI is transformers which are memory bound in both training and inference. If someone manages to get diffusion models to become the next hotness all of a sudden everyone will realize this is a problem since those are compute bound by a huge margin and current gen GPUs are fire hazards when ran at 100% utilization for weeks on end.
  - fiddlerwoaroof 2y ago
    Yeah, my experience is that the rapid growth of computing capabilities of my childhood more or less ended in 2013: the desktop I built then is fine for most of my uses and the significant performance improvements since then haven’t been CPUs getting noticeably faster but increases in the speed of disks (SSDs) and growth of memory sizes and network speeds.
  - calvinmorrison 2y ago
    the only escape from the Black Pill of AI ruining society is we hit some hard limits and technology stops progressing at such a rapid rate, tapers, and gives humanity a bit of time to adapt culturally.
abetusk 2y ago
In my opinion, you've grossly mischaracterized and misunderstood the bitter lesson. The bitter lesson is not saying that brute force will win out against better algorithms, it's saying that simple, efficient algorithms, in a theoretical sense, will win out against intricate and specialized algorithms. That is, it's saying that worrying about the constant factor in the "big-Oh" is doomed to failure. My reading is that it's not saying that we should abandon polynomial time algorithms in favor of exponential ones, it's saying we should focus on polynomial time algorithms that are conceptually simple and let Moore's law take care of the constant factor differences between intricate, complex and "clever" algorithms. It's saying that Moore's law will obliterate the constant factor differences in algorithms, and potentially the small polynomial exponent differences, not that we shouldn't care about exponential time complexity. The "Bitter Lesson" blog post came out during a time when people were debating whether simple algorithms on large datasets could win out against more complex algorithms with intricate domain knowledge. That is, did AGI, vision, language, translation, speech, etc. need intricate and deep domain knowledge or was it good enough to get coarser and "stupider" algorithms, that were still theoretically efficient/polynomial time efficient algorithms. I can see why you would think this way and I certainly don't claim to have deep insight into Sutton to claim whether my reading is more correct than yours. From my perspective, the good faith reading of Sutton's article is talking about constant factor differences in algorithms, not in differences between polynomial time and exponential time worst case complexities.
- m463 2y ago
  I wonder if an analogy with graphics would be that "simple algorithms that leveraged compute" would not be "brute force everything with 1000 gpus + ray tracing", it might be a pixel shader that simulated water taking the state of the art a bit further.
jncfhnb 2y ago
> The raw compute needed to brute force any problem can only be known after the problem is solved. There is no sane upper limit to how much computation, memory and data any given task will take and humans are terrible at estimating how hard tasks actually are. We are after all only 60 years late for the undergraduate summer project that would solve computer vision. I feel like you’re conflating conceptual difficulty and computational difficulty
bob1029 2y ago
I don't feel like the bitter lesson becomes applicable until you've demonstrated some initial degree of success with your technique on a single machine/GPU/CPU/thread. If you cannot make it work in practical environments, it's going to be an uphill battle the entire way. This is why I've moved toward CPU-only techniques in my experimentation. Being able to execute arbitrary UTMs with high performance provides a significantly richer computational landscape to work with than matrix multiplication. I am perfectly happy with something taking longer as if it's on a linear basis. I.e., adding another CPU provides ~2x search speed. I am NOT happy with ideas like taking a 100x hit on token generation rate because half my parameters are paged out to disk at any given moment due to not having a step-size amount of vram. The rigidity of the GPU solution stack makes exploration of clever techniques largely a slap on the wrist experience. Anything with rapid control flow changes is verboten for a fancy pants cluster. The latency domain of L1 cache in the CPU is impossible to compete with if you need to serialize all of your events. I strongly believe that control flow is where the magic happens. This is where you can cut through 100 million parameters of linear math bullshit and solve the problem with a lookup table and 2 interpreter cycles. You get about half a billion of these cycles to work with per second per thread, so there is a lot of room to play with ideas.
imtringued 2y ago
Meanwhile in robotics everyone is glad that computers are getting faster. There are a lot of things that used to be impossible to do inside a 1000 Hz control loop. >What do we do with the problems that are too hard to solve with the limited compute that we have? Lie down for 80 years and wait for compute to catch up? Or solve a smaller problem using specialized tricks that don't require a $10B super computer to build? Solving a smaller problem using specialized tricks has gone nowhere in robotics. Almost all the advancements in robotics control happened in the last ten years as a result of computers getting faster. We are very close to the cusp of non linear MPC becoming a solved problem for up to 64 degrees of freedom and a horizon of 20 time steps at 1000Hz, but we aren't there yet. It would definitely be possible with an ASIC built for MPC. >The bitter lesson is nothing of the sort, there is plenty of space for thinking hard, and there always will be. The bitter lesson doesn't say that human ingenuity is worthless. It guides it in a useful direction. A lot of human ingenuity was put into compute scaling solutions for transformers.
_cs2017_ 2y ago
"The bitter lesson" didn't merely claim that a sufficiently large amount of compute would obsolete an engineered solution. Its claim was far stronger: the time it takes for the compute growth to catch up with the hand-engineered solution is so short that the investment in the latter won't pay off in the sense of a researcher's personal career investment, or in the sense of a big-tech R&D effort ROI. You may well dispute such a claim, of course. Would be interesting to read your thoughts if you are willing to share them.

RobinL 2y ago

I appreciate this comment. I'm currently working hard on something seemly straightforward (address matching) and sometimes I feel demotivated because it feels like whatever progress I make, the bitter lesson will get me in the end. Reading your comment made me feel that maybe it's worth the effort after all. I have also taken some comfort in the fact that current LLMs cannot perform this task very well.

uxcolumbo 2y ago
Can you provide more about the problems in address matching and what you are trying to solve? Do you mean street address matching? Isn’t that already solved? (excuse the naive question)
- kridsdale1 2y ago
  I don’t know of any open source solution. (I work in mapping)
- RobinL 2y ago
  The problem is that people write addresses down in different ways (I'm in the UK) You will typically have a master/canonical list of addresses from an official source. In the UK that's Ordnance Survey's AddressBase. You will then have 'messy addresses' that humans have written down. For simple addresses, they'll often be the same as the master version (e.g. 5 Rainbow Road, Hemel Hempstead, AB1 2BC). But there are many harder addresses that exhibit lots of variations, especially flats and subunits. These may all be the same: Flat A 1 High Street vs 1A High Street vs Basement Flat, 1 High Street There no guarantee there will be a number in the address. In: 'THE OLD FARM COTTAGE PAD FARM BADGERCROFT ROAD PIKING', 'THE' and PAD FARM' may be missing, which doesn't seem like a problem until you find out there's also a PAD FARM COTTAGE on Badgercroft Road. There's no guarantee tokens will be in the same order. FWIW, my work is open source, and it's here: https://github.com/RobinL/uk_address_matcher https://github.com/RobinL/uk_address_matcher
  - uxcolumbo 2y ago
    Thanks for explaining. Sounds messy indeed. Wasn’t there this startup trying to offer an alternative? Three words to describe a location or something. But of course it’s not really easy to change an address system, so using AI to make the existing system less messy might be more pragmatic.

sriku 2y ago

Agree (though that's perhaps not a charitable interpretation of TBL). Prematurely articulated principles can do a lot of damage. Btw I found the blog post to be one of the lowest quality one in terms of information content posted on HN - almost like it was written by chatgpt or something.

fxtentacle 2y ago

In my experience, "the bitter lesson" is just wrong. Yes, a hypothetical AI model that has a suitable architecture so that it can memorize your handcrafted features can improve upon them, if it is perfectly trained. But that's a huge if. There's a reason that large AI models only work with some architectures and you need to randomly initialize tens to hundreds of times and you need exactly the right optimizer with the right hyperparameters and then the right training data in the right order... and the reason is that generic function approximators get stuck in a local minima incredibly easy. People used to write handcrafted features to limit the optimization base area. Now people use specific pretraining data and initialization functions to do the same. It's just a different way to express the same constraints. In short, the "bitter lesson" assumes a change that did not happen yet.

imtringued 2y ago
>and the reason is that generic function approximators get stuck in a local minima incredibly easy. The key insight in massively over parameterized models (aka LLMs) has been that all the local minima are very similar. Picking one over the other doesn't actually benefit you that much.
- fxtentacle 2y ago
  Absolutely not! LLM training quite easily gets stuck. People call it "divergence" or "loss spike". See for example: https://dl.acm.org/doi/10.5555/3524938.3525354 https://dl.acm.org/doi/10.5555/3524938.3525354 "Gradient optimization with attention layers can be notoriously difficult requiring tricks such as learning rate warmup to prevent divergence.“ "we then propose a new weight initialization scheme with theoretical justification, that enables training without warmup" Or "Initialization of Large Language Models via Reparameterization to Mitigate Loss Spikes" https://arxiv.org/html/2410.05052v1 https://arxiv.org/html/2410.05052v1 If all local minima were similar, then these papers would represent a lot of wasted effort. Or said in a different way: If labs like DeepMind are willing to pay for large teams that are trying to optimize initialization, then surely this represents a real existing problem.
  - 2y ago

mmoskal 2y ago

The bitter lesson is about the balance of human ingenuity and compute being thrown at the problem. We've seen a few years of LLM compute being scaled up 10x every year, but this is hitting limits (fabs), and we will see more human effort, as it becomes compartively cheaper. Also the current crop of models are inherently limited. Even for something as simple as following a JSON schema, models alone are not good enough [0] Of course as the Moore law refuses to die, we'll continue seeing 1.5-2x or so every year, but that's far from 10x. [0] https://openai.com/index/introducing-structured-outputs-in-the-api/ https://openai.com/index/introducing-structured-outputs-in-t... - see plot

noosphr 2y ago
>Of course as the Moore law refuses to die, we'll continue seeing 1.5-2x or so every year, but that's far from 10x. This is another one of those anec-data throw away sentences that take thousands of words to disprove - with a lot of graphs - that no one reads. More hot takes: Moores law has been effectively dead since the Pentium 4 on CPUs. It's been dead on GPUs since 2020. Right now we're not seeing a 1.5-2x grow of compute per year. We've seen zero growth for 5. The only way that GPUs have gotten faster is by running ever hotter, and building out a trillion dollars worth of data centers. No one cares because the current hotness in AI is transformers which are memory bound in both training and inference. If someone manages to get diffusion models to become the next hotness all of a sudden everyone will realize this is a problem since those are compute bound by a huge margin and current gen GPUs are fire hazards when ran at 100% utilization for weeks on end.
- fiddlerwoaroof 2y ago
  Yeah, my experience is that the rapid growth of computing capabilities of my childhood more or less ended in 2013: the desktop I built then is fine for most of my uses and the significant performance improvements since then haven’t been CPUs getting noticeably faster but increases in the speed of disks (SSDs) and growth of memory sizes and network speeds.
- calvinmorrison 2y ago
  the only escape from the Black Pill of AI ruining society is we hit some hard limits and technology stops progressing at such a rapid rate, tapers, and gives humanity a bit of time to adapt culturally.

abetusk 2y ago

In my opinion, you've grossly mischaracterized and misunderstood the bitter lesson. The bitter lesson is not saying that brute force will win out against better algorithms, it's saying that simple, efficient algorithms, in a theoretical sense, will win out against intricate and specialized algorithms. That is, it's saying that worrying about the constant factor in the "big-Oh" is doomed to failure. My reading is that it's not saying that we should abandon polynomial time algorithms in favor of exponential ones, it's saying we should focus on polynomial time algorithms that are conceptually simple and let Moore's law take care of the constant factor differences between intricate, complex and "clever" algorithms. It's saying that Moore's law will obliterate the constant factor differences in algorithms, and potentially the small polynomial exponent differences, not that we shouldn't care about exponential time complexity. The "Bitter Lesson" blog post came out during a time when people were debating whether simple algorithms on large datasets could win out against more complex algorithms with intricate domain knowledge. That is, did AGI, vision, language, translation, speech, etc. need intricate and deep domain knowledge or was it good enough to get coarser and "stupider" algorithms, that were still theoretically efficient/polynomial time efficient algorithms. I can see why you would think this way and I certainly don't claim to have deep insight into Sutton to claim whether my reading is more correct than yours. From my perspective, the good faith reading of Sutton's article is talking about constant factor differences in algorithms, not in differences between polynomial time and exponential time worst case complexities.

m463 2y ago
I wonder if an analogy with graphics would be that "simple algorithms that leveraged compute" would not be "brute force everything with 1000 gpus + ray tracing", it might be a pixel shader that simulated water taking the state of the art a bit further.

jncfhnb 2y ago

> The raw compute needed to brute force any problem can only be known after the problem is solved. There is no sane upper limit to how much computation, memory and data any given task will take and humans are terrible at estimating how hard tasks actually are. We are after all only 60 years late for the undergraduate summer project that would solve computer vision. I feel like you’re conflating conceptual difficulty and computational difficulty

bob1029 2y ago

I don't feel like the bitter lesson becomes applicable until you've demonstrated some initial degree of success with your technique on a single machine/GPU/CPU/thread. If you cannot make it work in practical environments, it's going to be an uphill battle the entire way. This is why I've moved toward CPU-only techniques in my experimentation. Being able to execute arbitrary UTMs with high performance provides a significantly richer computational landscape to work with than matrix multiplication. I am perfectly happy with something taking longer as if it's on a linear basis. I.e., adding another CPU provides ~2x search speed. I am NOT happy with ideas like taking a 100x hit on token generation rate because half my parameters are paged out to disk at any given moment due to not having a step-size amount of vram. The rigidity of the GPU solution stack makes exploration of clever techniques largely a slap on the wrist experience. Anything with rapid control flow changes is verboten for a fancy pants cluster. The latency domain of L1 cache in the CPU is impossible to compete with if you need to serialize all of your events. I strongly believe that control flow is where the magic happens. This is where you can cut through 100 million parameters of linear math bullshit and solve the problem with a lookup table and 2 interpreter cycles. You get about half a billion of these cycles to work with per second per thread, so there is a lot of room to play with ideas.

imtringued 2y ago

Meanwhile in robotics everyone is glad that computers are getting faster. There are a lot of things that used to be impossible to do inside a 1000 Hz control loop. >What do we do with the problems that are too hard to solve with the limited compute that we have? Lie down for 80 years and wait for compute to catch up? Or solve a smaller problem using specialized tricks that don't require a $10B super computer to build? Solving a smaller problem using specialized tricks has gone nowhere in robotics. Almost all the advancements in robotics control happened in the last ten years as a result of computers getting faster. We are very close to the cusp of non linear MPC becoming a solved problem for up to 64 degrees of freedom and a horizon of 20 time steps at 1000Hz, but we aren't there yet. It would definitely be possible with an ASIC built for MPC. >The bitter lesson is nothing of the sort, there is plenty of space for thinking hard, and there always will be. The bitter lesson doesn't say that human ingenuity is worthless. It guides it in a useful direction. A lot of human ingenuity was put into compute scaling solutions for transformers.

_cs2017_ 2y ago

"The bitter lesson" didn't merely claim that a sufficiently large amount of compute would obsolete an engineered solution. Its claim was far stronger: the time it takes for the compute growth to catch up with the hand-engineered solution is so short that the investment in the latter won't pay off in the sense of a researcher's personal career investment, or in the sense of a big-tech R&D effort ROI. You may well dispute such a claim, of course. Would be interesting to read your thoughts if you are willing to share them.