3 ms·

Intelligence doesn't matter. To quote "Superintelligence: Paths, Dangers, Strategies": > There is an important sense, however, in which chess-playing AI turned

Intelligence doesn't matter. To quote "Superintelligence: Paths, Dangers, Strategies": > There is an important sense, however, in which chess-playing AI turned out to be a lesser triumph than many imagined it would be. It was once supposed, perhaps not unreasonably, that in order for a computer to play chess at grandmaster level, it would have to be endowed with a high degree of general intelligence. The same thing might happen with LLMs and software engineering: LLMs will not be considered "intelligent" and software engineering will no longer be thought of as something requiring "actual intelligence". Yes, current models can't replace software engineers. But they are getting better at it with every release. And they don't need to be as good as actual software engineers to replace them.

grim_io 1y ago
There is a reason chess was "solved" so fast. The game maps very nicely onto computers in general. A grandmaster chess playing ai is not better at driving a car than my calculator from the 90s.
- muldvarp 1y ago
  Yes, that's my point. AI doesn't need to be general to be useful. LLMs might replace software engineers without ever being "general intelligence".
  - grim_io 1y ago
    Sorry for not making my point clear. I'm arguing that the category of the problem matters a lot. Chess is, compared to self-driving cars and (in my opinion) programming, very limited in its rules, the fixed board size and the lack of "fog of war".
    - romeros1 1y ago
      "It is difficult to get a man to understand something when his salary depends upon his not understanding it" ~ Upton Sinclair Your stance was the widely held stance not just on hacker news but also by the leading proponents of ai when chatgpt was first launched. A lot of people thought the hallucination aspect is something that simply can't be overcome. That LLMs were nothing but glorified stochastic parrots. Well, things have changed quite dramatically lately. AI could plateau. But the pace at which it is improving is pretty scary. Regardless of real "intelligence" or not.. the current reality is that AI can already do quite a lot of traditional software work. This wasn't even remotely true if if you were to go 6 months back.
      - anthem2025 1y ago
        Ironic to post that quote about AI considering the hype is pretty much entirely from people who stand to make obscene wealth from it.
      - lawlessone 1y ago
        >That LLMs were nothing but glorified stochastic parrots. Well yes , now we know they make kids kill themselves. I think we've all fooled ourselves like this beetle https://www.npr.org/sections/krulwich/2013/06/19/193493225/the-love-that-dared-not-speak-its-name-of-a-beetle-for-a-beer-bottle https://www.npr.org/sections/krulwich/2013/06/19/193493225/t... for thousands of years up until 2020 anything that conversed with us could safely be assumed to be another sentient/intelligent being. No we have something that does that, but is neither sentient or intelligent, just a (complex)deterministic mechanism.
      - svara 1y ago
        How will this work exactly? I think I have a pretty good idea of what AI can do for software engineering, because I use it for that nearly every day and I experiment with different models and IDEs. The way that has worked for me is to make prompts very specific, to the point where the prompt itself would not be comprehensible to someone who's not in the field. If you sat a rando with no CS background in front of Cursor, Windsurf or Claude code, what do you suppose would happen? It seems really doubtful to me that overcoming that gap is "just more training", because it would require a qualitatively different sort of product. And even if we came to a point where no technical knowledge of how software actually works was required, you would still need to be precise about the business logic in natural language. Now you're writing computer code in natural language that will read like legalese. At that point you've just invented a new programming language. Now maybe you're thinking, I'll just prompt it with all my email, all my docs, everything I have for context and just ask it to please make my boss happy. But the level of integrative intelligence, combined with specialized world knowledge required for that task is really very far away from what current models can do. The most powerful way that I've found to conceptualize what LLMs do is that they execute routines from huge learnt banks of programs that re-combine stored textual information along common patterns. They're cut and paste engines where the recombination rules are potentially quite complex programs learnt from data. This view fits well with the strengths and weaknesses of LLMs - they are good at combining two well understood solutions into something new, even if vaguely described. But they are quite bad at abstracting textual information into a more fundamental model of program and world state and reasoning at that level. I strongly suspect this is intrinsic to their training, because doing this is simply not required to complete the vast majority of text that could realistically have ended up in training databases. Executing a sophisticated cut&paste scheme is in some ways just too effective; the technical challenge is how do you pose a training problem to force a model to learn beyond that.
    - Seattle3503 1y ago
      Ive heard this described as a kind vs a wicked learning environment.
    - muldvarp 1y ago
      I think I haven't made my point clear enough: Chess was once thought to require general intelligence. Then computing power became cheap enough that using raw compute made computers better than humans. Computers didn't play chess in a very human-like way and there were a few years where you could still beat a computer by playing to its weaknesses. Now you'll never beat a computer at chess ever again. Similarly, many software engineers think that writing software requires general intelligence. Then computing power became cheap enough that training LLMs became possible. Sure, LLMs don't think in a very human-like way: There are some tasks that are trivial for humans and where LLMs struggle but LLMs also outcompete your average software engineer in many other tasks. It's still possible to win against an LLM in an intelligence-off by playing to its weaknesses. It doesn't matter that computers don't have general intelligence when they use raw compute to crush you in chess. And it won't matter that computers don't have general intelligence when they use raw compute to crush you at programming. The proof that software development requires general intelligence is on you. I think the stuff most software engineers do daily doesn't. And I think LLMs will get continously better at it. I certainly don't feel comfortable betting my professional future on software development for the coming decades.
anthem2025 1y ago
[flagged]
manmal 1y ago
LLMs can code, but they can’t engineer IMO. They lack those other parts of the brain that are not the speech center.

grim_io 1y ago

There is a reason chess was "solved" so fast. The game maps very nicely onto computers in general. A grandmaster chess playing ai is not better at driving a car than my calculator from the 90s.

muldvarp 1y ago
Yes, that's my point. AI doesn't need to be general to be useful. LLMs might replace software engineers without ever being "general intelligence".
- grim_io 1y ago
  Sorry for not making my point clear. I'm arguing that the category of the problem matters a lot. Chess is, compared to self-driving cars and (in my opinion) programming, very limited in its rules, the fixed board size and the lack of "fog of war".
  - romeros1 1y ago
    "It is difficult to get a man to understand something when his salary depends upon his not understanding it" ~ Upton Sinclair Your stance was the widely held stance not just on hacker news but also by the leading proponents of ai when chatgpt was first launched. A lot of people thought the hallucination aspect is something that simply can't be overcome. That LLMs were nothing but glorified stochastic parrots. Well, things have changed quite dramatically lately. AI could plateau. But the pace at which it is improving is pretty scary. Regardless of real "intelligence" or not.. the current reality is that AI can already do quite a lot of traditional software work. This wasn't even remotely true if if you were to go 6 months back.
    - anthem2025 1y ago
      Ironic to post that quote about AI considering the hype is pretty much entirely from people who stand to make obscene wealth from it.
    - lawlessone 1y ago
      >That LLMs were nothing but glorified stochastic parrots. Well yes , now we know they make kids kill themselves. I think we've all fooled ourselves like this beetle https://www.npr.org/sections/krulwich/2013/06/19/193493225/the-love-that-dared-not-speak-its-name-of-a-beetle-for-a-beer-bottle https://www.npr.org/sections/krulwich/2013/06/19/193493225/t... for thousands of years up until 2020 anything that conversed with us could safely be assumed to be another sentient/intelligent being. No we have something that does that, but is neither sentient or intelligent, just a (complex)deterministic mechanism.
    - svara 1y ago
      How will this work exactly? I think I have a pretty good idea of what AI can do for software engineering, because I use it for that nearly every day and I experiment with different models and IDEs. The way that has worked for me is to make prompts very specific, to the point where the prompt itself would not be comprehensible to someone who's not in the field. If you sat a rando with no CS background in front of Cursor, Windsurf or Claude code, what do you suppose would happen? It seems really doubtful to me that overcoming that gap is "just more training", because it would require a qualitatively different sort of product. And even if we came to a point where no technical knowledge of how software actually works was required, you would still need to be precise about the business logic in natural language. Now you're writing computer code in natural language that will read like legalese. At that point you've just invented a new programming language. Now maybe you're thinking, I'll just prompt it with all my email, all my docs, everything I have for context and just ask it to please make my boss happy. But the level of integrative intelligence, combined with specialized world knowledge required for that task is really very far away from what current models can do. The most powerful way that I've found to conceptualize what LLMs do is that they execute routines from huge learnt banks of programs that re-combine stored textual information along common patterns. They're cut and paste engines where the recombination rules are potentially quite complex programs learnt from data. This view fits well with the strengths and weaknesses of LLMs - they are good at combining two well understood solutions into something new, even if vaguely described. But they are quite bad at abstracting textual information into a more fundamental model of program and world state and reasoning at that level. I strongly suspect this is intrinsic to their training, because doing this is simply not required to complete the vast majority of text that could realistically have ended up in training databases. Executing a sophisticated cut&paste scheme is in some ways just too effective; the technical challenge is how do you pose a training problem to force a model to learn beyond that.
  - Seattle3503 1y ago
    Ive heard this described as a kind vs a wicked learning environment.
  - muldvarp 1y ago
    I think I haven't made my point clear enough: Chess was once thought to require general intelligence. Then computing power became cheap enough that using raw compute made computers better than humans. Computers didn't play chess in a very human-like way and there were a few years where you could still beat a computer by playing to its weaknesses. Now you'll never beat a computer at chess ever again. Similarly, many software engineers think that writing software requires general intelligence. Then computing power became cheap enough that training LLMs became possible. Sure, LLMs don't think in a very human-like way: There are some tasks that are trivial for humans and where LLMs struggle but LLMs also outcompete your average software engineer in many other tasks. It's still possible to win against an LLM in an intelligence-off by playing to its weaknesses. It doesn't matter that computers don't have general intelligence when they use raw compute to crush you in chess. And it won't matter that computers don't have general intelligence when they use raw compute to crush you at programming. The proof that software development requires general intelligence is on you. I think the stuff most software engineers do daily doesn't. And I think LLMs will get continously better at it. I certainly don't feel comfortable betting my professional future on software development for the coming decades.

anthem2025 1y ago

[flagged]

manmal 1y ago

LLMs can code, but they can’t engineer IMO. They lack those other parts of the brain that are not the speech center.