4 ms·

Here's what I don't get about the "AI can build all software" scenario. It extrapolates AI capabilities up to a certain, very advanced point, and then, inexplic

by pron 9mo ago

Here's what I don't get about the "AI can build all software" scenario. It extrapolates AI capabilities up to a certain, very advanced point, and then, inexplicably, it stops. If AI is capable enough to "build pretty much anything", why is it not capable enough to also use what it builds (instead of people using it) or, for that matter, to decide what to build? If AI can, say, build air traffic control software as well as humans, why can't it also be the controller as well as humans? If it can build medical diagnosis software and healtchare management software, why can't it offer the diagnosis and prescribe treatment? Is the argument that there's something special about writing software that AI can do as well as people, but not other things? Why is that? I don't know how soon AI will be able to "build pretty much anything", but when it does, Yegge's point that "all software sectors are threatened" seems to be unimaginative. Why not all sectors full stop?

duderific 9mo ago
AI doesn't "want to" control air traffic. It doesn't have any desire or ambition. That's what the humans are for. It is merely a tool like a hammer. The hammer doesn't build the house, it is the human who wields the hammer that builds the house.
- horsawlarway 9mo ago
  This somewhat falls apart the second you realize that current models are already choosing which tools to use all the time. You can argue that's not "desire" but I'm not sure you'd convince me. Frankly - even the other end of your argument is weak. Humans don't particularly want to control air traffic either (otherwise why are we having to pay those air traffic controllers their salaries to be there?). They do it as a function of achieving a broader goal.
- pron 9mo ago
  I didn't say it wants to do anything. It also doesn't want to build software. But why would it be the case that an AI told to build an air traffic control software could successfully do that, but an AI told to make sure planes arrive where they're supposed to safely and on time won't be able to figure out the rest? Now, I'm not saying it's impossible for there to be something that makes the first job significantly easier than the second, but it's strange for me to assume that an AI would definitely be able to do the former soon, yet not the latter. I think it could be reasonable to believe it will be able to do neither or both soon, but I don't understand how we can expect the ability line to just happen to fall between software and pretty much everything else.
analyte123 9mo ago
I’ve heard this called “technical deflation” and it works similarly to how economic deflation can play out, causing you to forego actions in the present because you think they’ll be easier in the future (or in this case, possibly not needed at all). Time will tell if this results in a “software deflationary spiral” or not.
xorcist 9mo ago
Yes, closely related to the question why AI services insists on writing software in ineffective high level languages intended for humans to read, such as Python. Which then needs to be compiled, use large standard libraries etc. Why not output the actual software directly, intended for a computer to run directly?
- deleted 9mo ago
  [deleted]
- kristjansson 9mo ago
  Same reason humans write in higher-level languages instead of machine code? Each additional unit of program text costs energy at write time, so there's a bias toward more compact _representations_ of programs, even if they're less efficient at runtime.
- Aurornis 9mo ago
  Because the written code, variable names, structure, and comments also serve as context for the LLM. This is why LLM written code is often more verbose than human written code. All of those seemingly unnecessary comments everywhere, the excessively descriptive function names, the way everything is broken down into a seemingly excessive number of logical blocks: This is all helpful to the LLM for understanding the code.
  - pron 9mo ago
    Maybe for existing models, but I don't think that's necessarily the case. AI tools that generate or manipulate images or video don't need human words in the format. Also, I'm not sure how much meaningful names help beyond a certain program size. Assembly is quite understandable locally - even for humans, but especially if you're an AI model trained specifically on it - and globally, even codebases in high-level programming languages are too large to grasp just as code.
    - Aurornis 9mo ago
      > AI tools that generate or manipulate images or video don't need human words in the format. AI image manipulation tools use the image as context just like text tools use text for context. You can get an LLM to modify assembly if there’s enough context about what it’s doing, what the input means, what the output means, and so on provided manually. A chunk of assembly by itself is missing a lot of context that is naturally embedded in higher level programming languages, though. Like variable names.
warkdarrior 9mo ago
There are two ways to answer your questions. You are asking how do we choose between (1) generate+run (AI generate software for some task, then we run the software to do that task) and (2) agentic execution (AI simply completes the task). First way to look at this is through the lens of specialization. A software engineer could design and create Emacs, and then a writer could use Emacs to write a top-notch novel. That does not mean that the software engineer could write top-notch novels. Similarly, maybe AI can generate software for any task, but maybe it cannot do that task just as well as the task-specialized software. Second way to look at this is based on costs. Even if AI is as good as specialized software for a given task, the specialized software will likely be more efficient since it uses direct computation (you know, moving and transforming bits around in the CPU and the memory) instead of GPU or TPU-powered multiplications that emulate the direct computation.
- pron 9mo ago
  > Similarly, maybe AI can generate software for any task, but maybe it cannot do that task just as well as the task-specialized software. Yes, maybe, but assuming that is the case in general seems completely arbitrary. Maybe not all jobs are like writing software, but why assume software is especially easy for AI? > Even if AI is as good as specialized software for a given task, the specialized software will likely be more efficient since it uses direct computation Right, but surely an AI that can "build pretty much anything" can also figure out that it should write specialised software for itself to make its job faster or cheaper (after all, to "build pretty much anything", it needs to know about optimisation).
jnpnj 9mo ago
That was my first question. Somehow people thought that a tool able to generate an entire fullstack app from a few sentences is not capable of inferring needs based on context or customer directly.
seanjaii 9mo ago
I have to agree, I cannot quite pinpoint the fixation on LLMs doing away with software engineers/developers. I know teachers who rely on ChatGPT and Claude to do so many parts of their jobs; marking work, providing feedback for a kid's homework, lesson planning and even answering subject matter questions they don't know how to. LLMs work so much better so many professions, right now, than it does for software development. Clearly, even assuming they are wrong far less often and don't hallucinate in the very near future, there is a human element that LLMs certainly can't replace (unless semi-autonomous robots become a thing soon). Why isn't the same thought process applied to writing software? It's obvious if you've worked in this industry for any amount of time just how paramount that human element is to producing useful programs. If current and/or near future AI will replace us, then it's surely replacing almost everyone else, no?
pron 9mo ago
Another way of putting it is that predictions based on an assumption of "exponential growth" of anything (which is presumed to slow down at some point) are almost always wrong. It's saying, sure, we may be at 1 right now, but in a year we'll be at 100, and that's the future we should prepare for! But if you believe that, then if you're off by only a little about the stopping point, we could just as likely have to prepare for a future where we're at 10 or 1000 for a while. Once you talk about exponential growth of some capability, by definition a small error in the timing of when that growth plateaus translates to an error of the plateau capability by an order of magnitude. It's likely to be be much smaller or much greater than your prediction. In other words, you can believe in exponential growth and you can believe you have a good prediction of the future, but you can't believe in both simultaneously. Yegge believes that "the future" will arrive by end of year. If he's off by only a few months in one direction, such a future may not arrive for a while; in the other direction, and "the future" will be quite different.

by pron 9mo ago

duderific 9mo ago

AI doesn't "want to" control air traffic. It doesn't have any desire or ambition. That's what the humans are for. It is merely a tool like a hammer. The hammer doesn't build the house, it is the human who wields the hammer that builds the house.

horsawlarway 9mo ago
This somewhat falls apart the second you realize that current models are already choosing which tools to use all the time. You can argue that's not "desire" but I'm not sure you'd convince me. Frankly - even the other end of your argument is weak. Humans don't particularly want to control air traffic either (otherwise why are we having to pay those air traffic controllers their salaries to be there?). They do it as a function of achieving a broader goal.
pron 9mo ago
I didn't say it wants to do anything. It also doesn't want to build software. But why would it be the case that an AI told to build an air traffic control software could successfully do that, but an AI told to make sure planes arrive where they're supposed to safely and on time won't be able to figure out the rest? Now, I'm not saying it's impossible for there to be something that makes the first job significantly easier than the second, but it's strange for me to assume that an AI would definitely be able to do the former soon, yet not the latter. I think it could be reasonable to believe it will be able to do neither or both soon, but I don't understand how we can expect the ability line to just happen to fall between software and pretty much everything else.

analyte123 9mo ago

I’ve heard this called “technical deflation” and it works similarly to how economic deflation can play out, causing you to forego actions in the present because you think they’ll be easier in the future (or in this case, possibly not needed at all). Time will tell if this results in a “software deflationary spiral” or not.

xorcist 9mo ago

Yes, closely related to the question why AI services insists on writing software in ineffective high level languages intended for humans to read, such as Python. Which then needs to be compiled, use large standard libraries etc. Why not output the actual software directly, intended for a computer to run directly?

deleted 9mo ago
[deleted]
kristjansson 9mo ago
Same reason humans write in higher-level languages instead of machine code? Each additional unit of program text costs energy at write time, so there's a bias toward more compact _representations_ of programs, even if they're less efficient at runtime.
Aurornis 9mo ago
Because the written code, variable names, structure, and comments also serve as context for the LLM. This is why LLM written code is often more verbose than human written code. All of those seemingly unnecessary comments everywhere, the excessively descriptive function names, the way everything is broken down into a seemingly excessive number of logical blocks: This is all helpful to the LLM for understanding the code.
- pron 9mo ago
  Maybe for existing models, but I don't think that's necessarily the case. AI tools that generate or manipulate images or video don't need human words in the format. Also, I'm not sure how much meaningful names help beyond a certain program size. Assembly is quite understandable locally - even for humans, but especially if you're an AI model trained specifically on it - and globally, even codebases in high-level programming languages are too large to grasp just as code.
  - Aurornis 9mo ago
    > AI tools that generate or manipulate images or video don't need human words in the format. AI image manipulation tools use the image as context just like text tools use text for context. You can get an LLM to modify assembly if there’s enough context about what it’s doing, what the input means, what the output means, and so on provided manually. A chunk of assembly by itself is missing a lot of context that is naturally embedded in higher level programming languages, though. Like variable names.

warkdarrior 9mo ago

There are two ways to answer your questions. You are asking how do we choose between (1) generate+run (AI generate software for some task, then we run the software to do that task) and (2) agentic execution (AI simply completes the task). First way to look at this is through the lens of specialization. A software engineer could design and create Emacs, and then a writer could use Emacs to write a top-notch novel. That does not mean that the software engineer could write top-notch novels. Similarly, maybe AI can generate software for any task, but maybe it cannot do that task just as well as the task-specialized software. Second way to look at this is based on costs. Even if AI is as good as specialized software for a given task, the specialized software will likely be more efficient since it uses direct computation (you know, moving and transforming bits around in the CPU and the memory) instead of GPU or TPU-powered multiplications that emulate the direct computation.

pron 9mo ago
> Similarly, maybe AI can generate software for any task, but maybe it cannot do that task just as well as the task-specialized software. Yes, maybe, but assuming that is the case in general seems completely arbitrary. Maybe not all jobs are like writing software, but why assume software is especially easy for AI? > Even if AI is as good as specialized software for a given task, the specialized software will likely be more efficient since it uses direct computation Right, but surely an AI that can "build pretty much anything" can also figure out that it should write specialised software for itself to make its job faster or cheaper (after all, to "build pretty much anything", it needs to know about optimisation).

jnpnj 9mo ago

That was my first question. Somehow people thought that a tool able to generate an entire fullstack app from a few sentences is not capable of inferring needs based on context or customer directly.

seanjaii 9mo ago

I have to agree, I cannot quite pinpoint the fixation on LLMs doing away with software engineers/developers. I know teachers who rely on ChatGPT and Claude to do so many parts of their jobs; marking work, providing feedback for a kid's homework, lesson planning and even answering subject matter questions they don't know how to. LLMs work so much better so many professions, right now, than it does for software development. Clearly, even assuming they are wrong far less often and don't hallucinate in the very near future, there is a human element that LLMs certainly can't replace (unless semi-autonomous robots become a thing soon). Why isn't the same thought process applied to writing software? It's obvious if you've worked in this industry for any amount of time just how paramount that human element is to producing useful programs. If current and/or near future AI will replace us, then it's surely replacing almost everyone else, no?

pron 9mo ago

Another way of putting it is that predictions based on an assumption of "exponential growth" of anything (which is presumed to slow down at some point) are almost always wrong. It's saying, sure, we may be at 1 right now, but in a year we'll be at 100, and that's the future we should prepare for! But if you believe that, then if you're off by only a little about the stopping point, we could just as likely have to prepare for a future where we're at 10 or 1000 for a while. Once you talk about exponential growth of some capability, by definition a small error in the timing of when that growth plateaus translates to an error of the plateau capability by an order of magnitude. It's likely to be be much smaller or much greater than your prediction. In other words, you can believe in exponential growth and you can believe you have a good prediction of the future, but you can't believe in both simultaneously. Yegge believes that "the future" will arrive by end of year. If he's off by only a few months in one direction, such a future may not arrive for a while; in the other direction, and "the future" will be quite different.