3 ms·

> I hope we can put to rest the argument that LLMs are only marginally useful in coding - which are often among the top comments on many threads. I suppose thes

by mjr00 2y ago

> I hope we can put to rest the argument that LLMs are only marginally useful in coding - which are often among the top comments on many threads. I suppose these arguments arise from (a) having used only GH copilot which is the worst tool, or (b) not having spent enough time with the tool/llm, or (c) apprehension. I've given up responding to these. Look at the code that was changed[0]. It's a single file. From what I can tell, it's almost purely functional with clearly specified inputs and outputs. There's no need to implement half the code, realize the requirements weren't specified properly, and go back and have a conversation with the PM about it. Which is, you know, what developers actually do. This is the kind of stuff LLMs are great at, but it's not representative of a typical change request by Java Developer #1753 at Fortune 500 Enterprise Company #271. [0] https://github.com/ggerganov/llama.cpp/pull/11453/files https://github.com/ggerganov/llama.cpp/pull/11453/files

simonw 2y ago
"Yeah, but LLMs can't handle millions of lines of crufty old Java" is a guaranteed reply any time this topic comes up. (That's not to say it isn't a valid argument.) Short answer: LLMs are amazingly useful on large codebases, but they are useful in different ways. They aren't going to bang out a new feature perfectly first time, but in the right hands they can dramatically accelerate all sorts of important activities, such as: - Understanding code. If code has no documentation, dumping it into an LLM can help a lot. - Writing individual functions, classes and modules. You have to be good at software architecture and good at prompting to use them in this way - you take on the role of picking out the tasks that can be done independently of the rest of the code. - Writing tests - again, if you have the skill and experience to prompt them in the right way.
- mjr00 2y ago
  Yes, LLMs are very useful, when used properly. But the linked change request is not a good example of how they would be used by a typical software developer. The linked pull request is essentially output from a compiler that's been hardcoded. > Writing individual functions, classes and modules. You have to be good at software architecture and good at prompting to use them in this way - you take on the role of picking out the tasks that can be done independently of the rest of the code. If you have enough skill and understanding to do this, it means you already have enough general software development experience and domain-specific experience and experience with a specific, existing codebase to be in rarefied air. It's like saying, oh yeah a wrench makes plumbing easy. You just need to turn the wrench, and 25 years of plumbing knowledge to know where to turn it. > Writing tests - again, if you have the skill and experience to prompt them in the right way. This is very true and more accessible to most developers, though my big fear is it encourages people to crap out low-value unit tests. Not that they don't love to do that already.
  - simonw 2y ago
    > If you have enough skill and understanding to do this, it means you already have enough general software development experience and domain-specific experience and experience with a specific, existing codebase to be in rarefied air. Yes, exactly. That's why I keep saying that software developers shouldn't be afraid that they'll be out of a job because of LLMs.
- magicalist 2y ago
  > "Yeah, but LLMs can't handle millions of lines of crufty old Java" is a guaranteed reply any time this topic comes up. That's not at all what the GP was saying, though: > There's no need to implement half the code, realize the requirements weren't specified properly, and go back and have a conversation with the PM about it. Which is, you know, what developers actually do.
  - simonw 2y ago
    I was responding to this bit: > This is the kind of stuff LLMs are great at, but it's not representative of a typical change request by Java Developer #1753 at Fortune 500 Enterprise Company #271.

by mjr00 2y ago

simonw 2y ago

"Yeah, but LLMs can't handle millions of lines of crufty old Java" is a guaranteed reply any time this topic comes up. (That's not to say it isn't a valid argument.) Short answer: LLMs are amazingly useful on large codebases, but they are useful in different ways. They aren't going to bang out a new feature perfectly first time, but in the right hands they can dramatically accelerate all sorts of important activities, such as: - Understanding code. If code has no documentation, dumping it into an LLM can help a lot. - Writing individual functions, classes and modules. You have to be good at software architecture and good at prompting to use them in this way - you take on the role of picking out the tasks that can be done independently of the rest of the code. - Writing tests - again, if you have the skill and experience to prompt them in the right way.

mjr00 2y ago
Yes, LLMs are very useful, when used properly. But the linked change request is not a good example of how they would be used by a typical software developer. The linked pull request is essentially output from a compiler that's been hardcoded. > Writing individual functions, classes and modules. You have to be good at software architecture and good at prompting to use them in this way - you take on the role of picking out the tasks that can be done independently of the rest of the code. If you have enough skill and understanding to do this, it means you already have enough general software development experience and domain-specific experience and experience with a specific, existing codebase to be in rarefied air. It's like saying, oh yeah a wrench makes plumbing easy. You just need to turn the wrench, and 25 years of plumbing knowledge to know where to turn it. > Writing tests - again, if you have the skill and experience to prompt them in the right way. This is very true and more accessible to most developers, though my big fear is it encourages people to crap out low-value unit tests. Not that they don't love to do that already.
- simonw 2y ago
  > If you have enough skill and understanding to do this, it means you already have enough general software development experience and domain-specific experience and experience with a specific, existing codebase to be in rarefied air. Yes, exactly. That's why I keep saying that software developers shouldn't be afraid that they'll be out of a job because of LLMs.
magicalist 2y ago
> "Yeah, but LLMs can't handle millions of lines of crufty old Java" is a guaranteed reply any time this topic comes up. That's not at all what the GP was saying, though: > There's no need to implement half the code, realize the requirements weren't specified properly, and go back and have a conversation with the PM about it. Which is, you know, what developers actually do.
- simonw 2y ago
  I was responding to this bit: > This is the kind of stuff LLMs are great at, but it's not representative of a typical change request by Java Developer #1753 at Fortune 500 Enterprise Company #271.