4 ms·

I worry this is looking at where the ball is now instead of where it's going. The recent disproof of an Erdos conjecture should put to rest the idea that LLMs w

I worry this is looking at where the ball is now instead of where it's going. The recent disproof of an Erdos conjecture should put to rest the idea that LLMs will reach a skill ceiling before they reach superintelligence. I believe we are headed for a world of superintelligent AI where LLMs are much better at logical thinking than humans, the same way that chess engines are much better at chess than humans. In that world there's really nothing humans can offer in terms of logical thinking other than their humanity itself. An 8 year old with Stockfish can beat Magnus Carlsen, and an 8 year old with Codex (and daddy's credit card) will be able to beat me at software engineering.

deleted 5mo ago
[deleted]
sanderjd 5mo ago
Are you confident in putting a timeline on this prediction? One of the reasons I'm increasingly skeptical of this prediction is that I've now lived past a few of the dates I heard people put on the achievement of this level of superintelligence in previous years.
simonw 5mo ago
I don't buy that at all. It doesn't matter how great the LLMs get, the act of creating software using them will still require a great deal of skill. Most people just don't think in terms of software. Try asking a non-developer in your life what their dream software would be for their work, or their hobby. If they don't have what Nilay Patel calls "software brain" I'd be surprised if they came up with something actionable. (For more on software brain see "THE PEOPLE DO NOT YEARN FOR AUTOMATION", which makes the point I"m making here but much, much better: https://www.theverge.com/podcast/917029/software-brain-ai-backlash-databases-automation https://www.theverge.com/podcast/917029/software-brain-ai-ba...) You could give a non-developer the smartest LLM in the world and they wouldn't be able to create GitHub with it, because creating GitHub requires an enormous amount of understanding of what software developers need from a cloud source control tool. Sure, you can argue that the LLM "knows" what GitHub needs already and can guide their human-user to that, but why would a human-user who doesn't understand the domain ask an LLM to do that in the first place?
- ElProlactin 5mo ago
  > Try asking a non-developer in your life what their dream software would be for their work, or their hobby. If they don't have what Nilay Patel calls "software brain" I'd be surprised if they came up with something actionable. I've posted this in numerous comments because I think it bears repeating: there are tech-savvy non-developers who are actually building and shipping stuff with AI. I personally know a few who have been successful in acquiring initial customers. You can say "but their apps won't scale", "their apps aren't secure", etc. and you might be right but these criticisms ignore the fact that most human-built software suffers from issues around scalability, security, etc. What AI in the hands of a relatively tech-savvy person is capable of is building functional, usable applications that are pretty decent compared to what you might get if you paid an experienced contractor tens of thousands of dollars to build. A whole generation of young people has grown up with the internet, smartphones, etc. They might not be trained software engineers or have a "software brain" but in many cases they probably have a better intuitive sense for digital product design than a 30 or 40-something engineer who has been staring at an IDE for the past decade(s).
  - simonw 5mo ago
    > there are tech-savvy non-developers who are actually building and shipping stuff with AI I absolutely believe that. I think those are people with "software brain" who are on their way to becoming real developers. By the point they can write apps that are secure and scale... they'll have learned enough about software development to be employable as software developers. They'll be part of a new breed of developer who never memorized the syntax of a programming language, but they'll still be at the starting point of learning a HUGE volume of other stuff that's necessary to build good software. If we want to stay employed, we need to be notably better at building software than they are.
    - ElProlactin 5mo ago
      > I absolutely believe that. I think those are people with "software brain" who are on their way to becoming real developers. In my opinion, this is a software developer-centric way of thinking that reminds me of the saying, "if all you have is a hammer, everything is a nail." Here's an alternative perspective: For billions of people, technology products are an integral part of daily life. As a result, lots of people have an interest in building technology products, particularly software. Thanks to AI, you no longer need to be a "real developer" to build software. You can learn enough to build things that are commercially viable without seeking to be employed as a developer. > If we want to stay employed, we need to be notably better at building software than they are. While I don't believe that the market for developers will shrink to 0, unfortunately, I think this type of comment reflects the fear, existential angst and denial that has overtaken many people in this industry. The reality is that developers are no different than all the displaced workers who came before them. One day you had a job that seemed secure and capable of providing for a comfortable life and the next you were facing the prospect of diminished wages and unemployment because the world simply needs fewer people with your skills and there's no way around the secular trend. The sad irony is that when software was eating the world and new CompSci grads could take their pick of $150,000+ job offers before ever writing a line of production code, a lot of people in the industry had a smug "tough luck" attitude towards all the workers being displaced by the tech boom. Now it's their turn.
emodendroket 5mo ago
I personally don't see that much sense worrying about this scenario because if it comes true then it doesn't really matter what I do.
necovek 5mo ago
If building software (and even programming, as the basis for it) was just an expression of logical thinking, we would have cornered it long time ago IMO. But then again, logic is really a lot more discrete and well defined and easily expressed with traditional computing than LLMs are (which are probabilistic systems instead and as such require large knowledge bases). We can observe that at a couple hundred billion parameters they behave similarly to a point (in the sense that they can produce similar results), but the challenge is really in understanding the problem's multifaceted structure and competing needs and priorities.
svara 5mo ago
Chess and proofs only work as comparisons to the extent that you can find parts of your job that share their key property: A solution is sought to a problem that can be stated with relatively little information. What prompt would someone have used to get a superhuman coding agent to output the Linux kernel or GTA5? Before you accuse me of moving the goalposts, that's not my point: The examples are there to help think about what humans would still need to do to build complex projects even if the coding itself was perfectly reliable. Both the Linux kernel and GTA5 contain a large amount of incompressible information; humans thought long and hard about how to design them, i.e. about what that thing they were building was even supposed to be.
- wiseowise 5mo ago
  You don't understand, Claude 69 will be able to one-shot GTA6. You NEED to buy into the fearmongering and anxiety.
wiseowise 5mo ago
> In that world there's really nothing humans can offer in terms of logical thinking other than their humanity itself. By your logic anyone who's not in the top 10% of intelligence can't offer anything. The world keeps spinning. > An 8 year old with Stockfish can beat Magnus Carlsen, and an 8 year old with Codex (and daddy's credit card) will be able to beat me at software engineering. That's just nonsense, nobody will work with 8 year old (it's illegal, to start with). Go touch grass.

deleted 5mo ago

[deleted]

sanderjd 5mo ago

Are you confident in putting a timeline on this prediction? One of the reasons I'm increasingly skeptical of this prediction is that I've now lived past a few of the dates I heard people put on the achievement of this level of superintelligence in previous years.

simonw 5mo ago

I don't buy that at all. It doesn't matter how great the LLMs get, the act of creating software using them will still require a great deal of skill. Most people just don't think in terms of software. Try asking a non-developer in your life what their dream software would be for their work, or their hobby. If they don't have what Nilay Patel calls "software brain" I'd be surprised if they came up with something actionable. (For more on software brain see "THE PEOPLE DO NOT YEARN FOR AUTOMATION", which makes the point I"m making here but much, much better: https://www.theverge.com/podcast/917029/software-brain-ai-backlash-databases-automation https://www.theverge.com/podcast/917029/software-brain-ai-ba...) You could give a non-developer the smartest LLM in the world and they wouldn't be able to create GitHub with it, because creating GitHub requires an enormous amount of understanding of what software developers need from a cloud source control tool. Sure, you can argue that the LLM "knows" what GitHub needs already and can guide their human-user to that, but why would a human-user who doesn't understand the domain ask an LLM to do that in the first place?

ElProlactin 5mo ago
> Try asking a non-developer in your life what their dream software would be for their work, or their hobby. If they don't have what Nilay Patel calls "software brain" I'd be surprised if they came up with something actionable. I've posted this in numerous comments because I think it bears repeating: there are tech-savvy non-developers who are actually building and shipping stuff with AI. I personally know a few who have been successful in acquiring initial customers. You can say "but their apps won't scale", "their apps aren't secure", etc. and you might be right but these criticisms ignore the fact that most human-built software suffers from issues around scalability, security, etc. What AI in the hands of a relatively tech-savvy person is capable of is building functional, usable applications that are pretty decent compared to what you might get if you paid an experienced contractor tens of thousands of dollars to build. A whole generation of young people has grown up with the internet, smartphones, etc. They might not be trained software engineers or have a "software brain" but in many cases they probably have a better intuitive sense for digital product design than a 30 or 40-something engineer who has been staring at an IDE for the past decade(s).
- simonw 5mo ago
  > there are tech-savvy non-developers who are actually building and shipping stuff with AI I absolutely believe that. I think those are people with "software brain" who are on their way to becoming real developers. By the point they can write apps that are secure and scale... they'll have learned enough about software development to be employable as software developers. They'll be part of a new breed of developer who never memorized the syntax of a programming language, but they'll still be at the starting point of learning a HUGE volume of other stuff that's necessary to build good software. If we want to stay employed, we need to be notably better at building software than they are.
  - ElProlactin 5mo ago
    > I absolutely believe that. I think those are people with "software brain" who are on their way to becoming real developers. In my opinion, this is a software developer-centric way of thinking that reminds me of the saying, "if all you have is a hammer, everything is a nail." Here's an alternative perspective: For billions of people, technology products are an integral part of daily life. As a result, lots of people have an interest in building technology products, particularly software. Thanks to AI, you no longer need to be a "real developer" to build software. You can learn enough to build things that are commercially viable without seeking to be employed as a developer. > If we want to stay employed, we need to be notably better at building software than they are. While I don't believe that the market for developers will shrink to 0, unfortunately, I think this type of comment reflects the fear, existential angst and denial that has overtaken many people in this industry. The reality is that developers are no different than all the displaced workers who came before them. One day you had a job that seemed secure and capable of providing for a comfortable life and the next you were facing the prospect of diminished wages and unemployment because the world simply needs fewer people with your skills and there's no way around the secular trend. The sad irony is that when software was eating the world and new CompSci grads could take their pick of $150,000+ job offers before ever writing a line of production code, a lot of people in the industry had a smug "tough luck" attitude towards all the workers being displaced by the tech boom. Now it's their turn.

emodendroket 5mo ago

I personally don't see that much sense worrying about this scenario because if it comes true then it doesn't really matter what I do.

necovek 5mo ago

If building software (and even programming, as the basis for it) was just an expression of logical thinking, we would have cornered it long time ago IMO. But then again, logic is really a lot more discrete and well defined and easily expressed with traditional computing than LLMs are (which are probabilistic systems instead and as such require large knowledge bases). We can observe that at a couple hundred billion parameters they behave similarly to a point (in the sense that they can produce similar results), but the challenge is really in understanding the problem's multifaceted structure and competing needs and priorities.

svara 5mo ago

Chess and proofs only work as comparisons to the extent that you can find parts of your job that share their key property: A solution is sought to a problem that can be stated with relatively little information. What prompt would someone have used to get a superhuman coding agent to output the Linux kernel or GTA5? Before you accuse me of moving the goalposts, that's not my point: The examples are there to help think about what humans would still need to do to build complex projects even if the coding itself was perfectly reliable. Both the Linux kernel and GTA5 contain a large amount of incompressible information; humans thought long and hard about how to design them, i.e. about what that thing they were building was even supposed to be.

wiseowise 5mo ago
You don't understand, Claude 69 will be able to one-shot GTA6. You NEED to buy into the fearmongering and anxiety.

wiseowise 5mo ago

> In that world there's really nothing humans can offer in terms of logical thinking other than their humanity itself. By your logic anyone who's not in the top 10% of intelligence can't offer anything. The world keeps spinning. > An 8 year old with Stockfish can beat Magnus Carlsen, and an 8 year old with Codex (and daddy's credit card) will be able to beat me at software engineering. That's just nonsense, nobody will work with 8 year old (it's illegal, to start with). Go touch grass.