10 ms·
I share the sentiment with others here that have become disillusioned with LLMs. Yes, their capabilities are impressive, and their progression is quite striking
by 4death4 3y ago
I share the sentiment with others here that have become disillusioned with LLMs. Yes, their capabilities are impressive, and their progression is quite striking. But what I have realized, about programming specifically, is that I don't think LLMs can do a better job than humans.
To many futurists hopping on the hypetrain, this is probably somewhat of a hot take. But let me explain.
Ultimately, programming is about instructing a computer on what to do. We use programming languages because they're highly efficient representations of what a computer can do. In the case of LLMs, we need to use natural language to specify the program. For simple things, this can work great. But for complex programs, it's not easy to articulate exactly what a program should do. As you drill in to the details, you begin to experience an almost exponential growth in the amount of natural language specification you need. And so I think there is something very flawed in believing that you can replace human programmers with LLMs. In my personal opinion, a single programmer will be able to outproduce a single LLM prompt engineer, purely because programming languages are better at expressing ideas compared to natural language. At a certain point, it becomes faster to write code then to write sufficiently detailed prompts.
As a thought experiment for the reader: consider a complex system you've worked on. And now try to imagine writing down a specification for the system in natural language that an LLM could use to generate the code that powers the system. I honestly don't think I could do it for many of the system I've worked on.
So yes, I think LLMs are cool. And I think they're cool because they give creative power to people who can't program. But for sufficiently complex things, i.e. things people care about, a natural language interface isn't the answer.
- darkclouds 3y ago[dead]
- crabmusket 3y ago> As a thought experiment for the reader: consider a complex system you've worked on. And now try to imagine writing down a specification for the system in natural language that an LLM could use to generate the code that powers the system. But you didn't write it down, so how did the system get made? The answer is that the program exists in your head as much as it exists in bytes in your source files. There might be a mismatch between your head-program and the source-program, or between your head-program and what the customer actually wants. But the existence of the head-program is what Naur was articulating in 1985's "Programming as Theory-Building", and is what LLM proponents are rediscovering.
- 4death4 3y ago> But you didn't write it down, so how did the system get made? We didn't write it down, but we did use natural language to specify it, albeit somewhat indirectly. Some of the specification came from meetings with leadership, PMs, other people in the org, customers, etc. Some of it came from getting drinks with the team and talking about what we wanted to build. Some of it came from conversations with ex-colleagues. Some was inspired by past experience. It was be extremely onerous to put all of that down as input to an LLM.
- crabmusket 3y agoExactly my point! What you were doing was theory-building, creating a mostly-shared mental understanding of the program.
- 4death4 3y agoAh, ok. I see. I originally interpreted your comment as “there was some natural language spec, therefore you could feed it into an LLM” but I see I was mistaken.
- crabmusket 3y agoI highly recommend the Naur paper, I think it is a pretty accurate view of the dynamics of software production.