5 ms·

The conclusion of the bitter lesson would be that a large language model trained on chess commentary as well as being trained on millions of chess games would o

The conclusion of the bitter lesson would be that a large language model trained on chess commentary as well as being trained on millions of chess games would outperform stockfish which is only trained on millions of chess games. There’s no evidence at this point that this is true.

iainmerrick 1mo ago
I think you have it backwards. The common mistake is to think “maybe if we use a blend of raw data and hand-crafted heuristics, we’ll get the best of both worlds!” But the bitter lesson says no, beyond a certain point it’s better just to use the data. Thinking that an LLM might be able to improve on purely “big data” machine learning seems to me to be the same incorrect idea. Its “intelligence” is no more useful than human intelligence. The LLM is based on a massive data corpus, sure, but the amount of data specifically about chess in there pales in comparison to just playing billions of games of chess.
- TwelveEyes 1mo ago
  Also, training it on chess books is literally training it on human knowledge, and not the actual game, which is exactly what the bitter lesson says not to do.
- Dylan16807 1mo ago
  > I think you have it backwards. > maybe if we use a blend of raw data and hand-crafted heuristics I don't follow. They're suggesting giving raw chess data to the LLM, no heuristics involved.
  - iainmerrick 1mo ago
    I was replying to this: The conclusion of the bitter lesson would be that a large language model trained on chess commentary as well as being trained on millions of chess games would outperform stockfish which is only trained on millions of chess games. If you can draw any lessons from chess commentary, I think it’s very reasonable to call it “hand-crafted heuristics.”
    - Dylan16807 1mo ago
      Hand-crafted even if you're feeding in the raw commentary? That seems like a weird way to consider it. Wouldn't that make LLMs in general "hand-crafted"? And raw games plus raw commentary is all the data you have. You can make more games but those can be fed to both stockfish and the LLM competitor. So it seems like a valid interpretation of the bitter lesson to me.
      - iainmerrick 1mo ago
        Yeah, "hand-crafted" is a bit of a stretch; I mean their value is in the human insight they contain. The key point I was trying to get at is that the human insights don't contain anything that can't be mined from vast amounts of gameplay. Every human insight can eventually be rediscovered and made rigorous by data (in chess, at least!) In the short term, those insights are useful, but in the longer term, they add nothing at all. Note also that "raw gameplay" here can mean new games -- you can generate as much data as you need, you don't need to rely on real recorded games.
        Dylan16807 1mo ago
        > Every human insight can eventually be rediscovered and made rigorous by data (in chess, at least!) In the short term, those insights are useful, but in the longer term, they add nothing at all. But isn't that the bulk of what we're shoving into LLMs, and it makes them much smarter? If it's useful there but not in a chess AI then that seems like a significant crack in the bitter lesson. > Note also that "raw gameplay" here can mean new games -- you can generate as much data as you need, you don't need to rely on real recorded games. Yeah I mentioned that, generated games are useful. But if we're being fair and letting both AIs use generated game data, does the more general LLM ever actually overtake the specialized stockfish like the bitter lesson suggests? Another way to look at this is that giving the LLM the commentary is a way to avoid complaints of hiding data from the LLM, since it'll have strictly more info than stockfish. But if we cut that from the training data and only give it a basic description of chess and lots of raw game data then it's going to get even worse than it already is at chess. Meanwhile stockfish never had that commentary, just actual hand-crafted heuristics and training on game data, and it's very strong.
antihipocrat 1mo ago
Maybe a future frontier LLM could approach the problem by first building its own stockfish, then applying the subsequent results
- catoc 1mo ago
  Maybe a future LLM after that could approach the problem by first simulating a human brain, then learning from the ‘human’ gameplay. Just kidding of course
- jurgenburgen 1mo ago
  Or maybe an LLM could just tool call stockfish and doesn’t need to have more than a basic understanding of chess. The bitter lesson seems extraordinarily wasteful on the compute side.
nl 1mo ago
> a large language model trained on chess commentary as well as being trained on millions of chess games would outperform stockfish which is only trained on millions of chess games Not really, if anything it's closer to the opposite. The Bitter Lesson essay literally has this as an example: > These researchers wanted methods based on human input to win and were disappointed when they did not.[1] and > Enormous initial efforts went into avoiding search by taking advantage of human knowledge, or of the special features of the game, but all those efforts proved irrelevant, or worse, once search was applied effectively at scale[1] The actual bitter lesson is this: > breakthrough progress eventually arrives by an opposing approach based on scaling computation by search and learning. The eventual success is tinged with bitterness, and often incompletely digested, because it is success over a favored, human-centric approach.[1] Applying to the "LLMs-for-chess" example the bitter lesson approach would be to put many, many more games into the LLM. Does this work? People have trained fairly small LLMs that are competitive Stockfish at the ELO 1500-2000 level, eg: https://github.com/kinggongzilla/chess-bot-3000 https://github.com/kinggongzilla/chess-bot-3000 This seems to be evidence that large LLMs probably don't have as much chess training data as Stockfish does. [1] http://www.incompleteideas.net/IncIdeas/BitterLesson.html http://www.incompleteideas.net/IncIdeas/BitterLesson.html
- ianjbutler 1mo ago
  > These researchers wanted methods based on human input to win and were disappointed when they did not.[1] This was/is basically a strawman though. Like maybe "human input winning" was desirable for chess masters but for computer science wonks? Not the point or the disappoint. It's always neats and scruffies fighting about using some kind of recognizable method (logic) instead of magic (ML). > breakthrough progress eventually arrives by an opposing approach based on scaling computation by search and learning. More to OP's point I think: nowadays when someone wants to beat you over the head with the bitter lesson, they aren't as careful to include learning and search. They want to say learning leads to intuition (magic) whereby we can avoid work (logic/search), and maybe argue or assume from there that neats and scruffies is settled. TBF, something like reasoning in latent space does resemble intuition! But the real lesson is confirmed every time we bother to check, and not very bitter for anyone. Search/learning/logic are ALL always necessary on any sufficiently difficult problems, and hybrids that interleave always outperform everything else. Stockfish being the example in this thread that different camps of absolutists would like to claim, but also all the MCTS examples, evolving examples, and new hybrids all the time. My favorite lately: https://arxiv.org/pdf/2511.08983 https://arxiv.org/pdf/2511.08983
camuel 1mo ago
It's the exact opposite. The bitter lesson is that simply scaling training on more games—including self-play—trumps any hand-crafted human input, whether that's fine-tuning on human commentary or clever engineering tricks. Current models are just high-dimensional interpolation engines. The denser the data sampling, the more accurate the interpolation gets. Given a choice between denser sampling and anything else, denser sampling always wins. That is the bitter lesson. Computer chess is the canonical example of this.
- klipt 1mo ago
  But the harness still matters. In the case of stockfish, the harness is a tree search around the neural network evaluations.
- inigyou 1mo ago
  Denser sampling only seems useful if the problem domain is in some way smooth - interpolatable. If you run it on a fractal problem domain you just learn more special cases. Chess is fractal.
zarzavat 1mo ago
Chess is a brute force search problem. Humans are not good at chess, even a small computer can beat Magnus Carlsen. It would be better to compare models at how well they can write the code for chess engines, otherwise it's just saying that Fable is not a good CPU emulator, which is obvious.

iainmerrick 1mo ago

I think you have it backwards. The common mistake is to think “maybe if we use a blend of raw data and hand-crafted heuristics, we’ll get the best of both worlds!” But the bitter lesson says no, beyond a certain point it’s better just to use the data. Thinking that an LLM might be able to improve on purely “big data” machine learning seems to me to be the same incorrect idea. Its “intelligence” is no more useful than human intelligence. The LLM is based on a massive data corpus, sure, but the amount of data specifically about chess in there pales in comparison to just playing billions of games of chess.

TwelveEyes 1mo ago
Also, training it on chess books is literally training it on human knowledge, and not the actual game, which is exactly what the bitter lesson says not to do.
Dylan16807 1mo ago
> I think you have it backwards. > maybe if we use a blend of raw data and hand-crafted heuristics I don't follow. They're suggesting giving raw chess data to the LLM, no heuristics involved.
- iainmerrick 1mo ago
  I was replying to this: The conclusion of the bitter lesson would be that a large language model trained on chess commentary as well as being trained on millions of chess games would outperform stockfish which is only trained on millions of chess games. If you can draw any lessons from chess commentary, I think it’s very reasonable to call it “hand-crafted heuristics.”
  - Dylan16807 1mo ago
    Hand-crafted even if you're feeding in the raw commentary? That seems like a weird way to consider it. Wouldn't that make LLMs in general "hand-crafted"? And raw games plus raw commentary is all the data you have. You can make more games but those can be fed to both stockfish and the LLM competitor. So it seems like a valid interpretation of the bitter lesson to me.
    - iainmerrick 1mo ago
      Yeah, "hand-crafted" is a bit of a stretch; I mean their value is in the human insight they contain. The key point I was trying to get at is that the human insights don't contain anything that can't be mined from vast amounts of gameplay. Every human insight can eventually be rediscovered and made rigorous by data (in chess, at least!) In the short term, those insights are useful, but in the longer term, they add nothing at all. Note also that "raw gameplay" here can mean new games -- you can generate as much data as you need, you don't need to rely on real recorded games.
      - Dylan16807 1mo ago
        > Every human insight can eventually be rediscovered and made rigorous by data (in chess, at least!) In the short term, those insights are useful, but in the longer term, they add nothing at all. But isn't that the bulk of what we're shoving into LLMs, and it makes them much smarter? If it's useful there but not in a chess AI then that seems like a significant crack in the bitter lesson. > Note also that "raw gameplay" here can mean new games -- you can generate as much data as you need, you don't need to rely on real recorded games. Yeah I mentioned that, generated games are useful. But if we're being fair and letting both AIs use generated game data, does the more general LLM ever actually overtake the specialized stockfish like the bitter lesson suggests? Another way to look at this is that giving the LLM the commentary is a way to avoid complaints of hiding data from the LLM, since it'll have strictly more info than stockfish. But if we cut that from the training data and only give it a basic description of chess and lots of raw game data then it's going to get even worse than it already is at chess. Meanwhile stockfish never had that commentary, just actual hand-crafted heuristics and training on game data, and it's very strong.

antihipocrat 1mo ago

Maybe a future frontier LLM could approach the problem by first building its own stockfish, then applying the subsequent results

catoc 1mo ago
Maybe a future LLM after that could approach the problem by first simulating a human brain, then learning from the ‘human’ gameplay. Just kidding of course
jurgenburgen 1mo ago
Or maybe an LLM could just tool call stockfish and doesn’t need to have more than a basic understanding of chess. The bitter lesson seems extraordinarily wasteful on the compute side.

nl 1mo ago

> a large language model trained on chess commentary as well as being trained on millions of chess games would outperform stockfish which is only trained on millions of chess games Not really, if anything it's closer to the opposite. The Bitter Lesson essay literally has this as an example: > These researchers wanted methods based on human input to win and were disappointed when they did not.[1] and > Enormous initial efforts went into avoiding search by taking advantage of human knowledge, or of the special features of the game, but all those efforts proved irrelevant, or worse, once search was applied effectively at scale[1] The actual bitter lesson is this: > breakthrough progress eventually arrives by an opposing approach based on scaling computation by search and learning. The eventual success is tinged with bitterness, and often incompletely digested, because it is success over a favored, human-centric approach.[1] Applying to the "LLMs-for-chess" example the bitter lesson approach would be to put many, many more games into the LLM. Does this work? People have trained fairly small LLMs that are competitive Stockfish at the ELO 1500-2000 level, eg: https://github.com/kinggongzilla/chess-bot-3000 https://github.com/kinggongzilla/chess-bot-3000 This seems to be evidence that large LLMs probably don't have as much chess training data as Stockfish does. [1] http://www.incompleteideas.net/IncIdeas/BitterLesson.html http://www.incompleteideas.net/IncIdeas/BitterLesson.html

ianjbutler 1mo ago
> These researchers wanted methods based on human input to win and were disappointed when they did not.[1] This was/is basically a strawman though. Like maybe "human input winning" was desirable for chess masters but for computer science wonks? Not the point or the disappoint. It's always neats and scruffies fighting about using some kind of recognizable method (logic) instead of magic (ML). > breakthrough progress eventually arrives by an opposing approach based on scaling computation by search and learning. More to OP's point I think: nowadays when someone wants to beat you over the head with the bitter lesson, they aren't as careful to include learning and search. They want to say learning leads to intuition (magic) whereby we can avoid work (logic/search), and maybe argue or assume from there that neats and scruffies is settled. TBF, something like reasoning in latent space does resemble intuition! But the real lesson is confirmed every time we bother to check, and not very bitter for anyone. Search/learning/logic are ALL always necessary on any sufficiently difficult problems, and hybrids that interleave always outperform everything else. Stockfish being the example in this thread that different camps of absolutists would like to claim, but also all the MCTS examples, evolving examples, and new hybrids all the time. My favorite lately: https://arxiv.org/pdf/2511.08983 https://arxiv.org/pdf/2511.08983

camuel 1mo ago

It's the exact opposite. The bitter lesson is that simply scaling training on more games—including self-play—trumps any hand-crafted human input, whether that's fine-tuning on human commentary or clever engineering tricks. Current models are just high-dimensional interpolation engines. The denser the data sampling, the more accurate the interpolation gets. Given a choice between denser sampling and anything else, denser sampling always wins. That is the bitter lesson. Computer chess is the canonical example of this.

klipt 1mo ago
But the harness still matters. In the case of stockfish, the harness is a tree search around the neural network evaluations.
inigyou 1mo ago
Denser sampling only seems useful if the problem domain is in some way smooth - interpolatable. If you run it on a fractal problem domain you just learn more special cases. Chess is fractal.

zarzavat 1mo ago

Chess is a brute force search problem. Humans are not good at chess, even a small computer can beat Magnus Carlsen. It would be better to compare models at how well they can write the code for chess engines, otherwise it's just saying that Fable is not a good CPU emulator, which is obvious.