19 ms·

In people's experience with these sorts of tools, have they assisted with maintainance of codebases? This might be directly, or indirectly via more readable, be

by zizee 3y ago

In people's experience with these sorts of tools, have they assisted with maintainance of codebases? This might be directly, or indirectly via more readable, bette organized code. The reason I ask is that these tools seem to excel in helping to write new code. In my experience I think there is an upper limit to the amount of code a single developer can maintain. Eventually you can't keep everything in your head, so maintaining it becomes more effort as you need to stop to familiarize yourself with something. If these tools help to write more code, but do not assist with maintainance, I wonder if we're going to see masses of new code written really quickly, and then everything grinds to a halt, because no one has an intimate understanding of what was written?

throw2321 3y ago
I've been thinking about this for a while now, wrt two points: 1. This will be the end of traditional SWEs and the rise of the age of debuggers, human debuggers who spend their days setting up breakpoints and figuring bugs in a sea of LLM generated code. 2. Hiring will switch from using Leetcode questions to "pull out your debugger and figure out what's wrong with this code".
- w-m 3y ago
  What makes you think the LLM couldn’t run a debugging session from the content of a JIRA ticket and the whole code base + documentation?
  - eichin 3y ago
    Having never seen it, or anything even close to it. (Of course, I'm a little biased by seeing product demos that don't even get "add another item to this list of command line arguments" right; maybe if everyone already believes it works, nobody bothers to actually sell that?)
  - spookie 3y ago
    If the codebase is anything more than a simple Python project... I don't think that'll happen. It just doesn't scale that well. Hell, GPT-4 can't make sense of my own projects.
  - mike_hearn 3y ago
    Who can say what's possible but there are very few "debugging transcripts" for neural nets to train on out there. So it'd have to sort of work out how to operate a debugger via functions, understand the codebase (perhaps large parts of it), intuit what's going wrong, be able to modify the codebase, etc. Lots of work to do there.
  - Workaccount2 3y ago
    Nobody wants to face the potential that their skills are going to greatly decrease in value.
- eichin 3y ago
  The first interview where I was handed some (intentionally) broken C code and gdb was about 15 years ago. I'm not sure that part is a change in developer workflow (this may apply more in systems and embedded though.) I've been paying attention to this too (mostly by following Simon Willison) and I'm still solidly in the "get back to me when this stuff can successfully review a pull request or even interpret a traceback" camp...
- ukuina 3y ago
  Yep, time to start adding source_file.prompt sidecar files next to each generated module so debugging sessions can start at the same initial condition.
ushakov 3y ago
We are doing this for API-Testing now. You should check out our website https://ai.stepci.com https://ai.stepci.com
- somsak2 3y ago
  piece of feedback: it's weird to have a drop-down on "OpenAPI Links" when there are no other options.
  - ushakov 3y ago
    Thanks! We will have more examples coming very soon
gen220 3y ago
> If these tools help to write more code, but do not assist with maintainance, I wonder if we're going to see masses of new code written really quickly, and then everything grinds to a halt, because no one has an intimate understanding of what was written? Yep. Companies using LLMs to "augment" junior developers will get a lot of positive press, but I guess it remains to be seen how much the market consistently rewards this behavior. Consumers will probably see right through it, but the b2b folks might get fleeced for a few years before eventually churning and moving to a higher quality old-fashioned competitor that employs senior talent. But IDK, maybe we'll come up with models that are good at growing and maintaining a coherent codebase. It doesn't seem like an impossible task, given where we are today. But we're pretty far from it still, as you point out.
- bertil 3y ago
  What are the tasks that you envision are key to maintenance? - bug finding and fixing - parsing logs to find optimisation options - refactoring (after several local changes) - given new features, recommending a refactoring? I feel like code assistants are already reasonable help for doing the first two, and the later two are mostly a question of context window. I feel we might end up with code bases split by context sizes, stitched with shared descriptions.
  - gen220 3y ago
    I guess the issue is that programmers work with a really big context window, and need to reason at multiple levels of abstraction depending on the problem. A great idea to solve a problem at one level of abstraction / context might be a terrible "strategic" idea at a higher level of abstraction. This is what separates the "junior" engineers from "senior" engineers, speaking very loosely. IDK, I'm not convinced by all that I've seen, that GPT is capable of that higher-order thinking. I fear it requires a degree of epistemology that GPT fundamentally doesn't possess as a stochastic token-guesser. It never pushes back against a request, or asks if you really intend another question by your first question. It never tries to read through your requirements to grasp the underlying problem that's prompting them. Maybe some combination of static tools, senior caretakers and prompt hackery can get us to a solution that maintains code effectively. But I don't think you can throw out the senior caretakers, their verification involvement is really necessary. And I don't know how conducive this environment would be to developing the next generation of "senior caretakers".
anotherpaulg 3y ago
My open source ai coding tool aider is unique in that it is designed to work with existing code bases. You can jump into an existing git repo and start aaking for changes, new features, etc. https://github.com/paul-gauthier/aider https://github.com/paul-gauthier/aider It helps gpt understand larger code bases by building a "repository map" based on analyzing the abstract syntax tree of all the code in the repo. This is all built using tree-sitter, the same tooling which powers code search and navigation on GitHub and in many popular IDEs. https://aider.chat/docs/repomap.html https://aider.chat/docs/repomap.html
- anotherpaulg 3y ago
  Related to the OpenAI announcement, I've been able to generate some preliminary code editing evaluations of the new GPT models. OpenAI is enforcing very low rate limits on the new GPT-4 model. I will update the results as quickly my rate limit allows. https://news.ycombinator.com/item?id=38172621 https://news.ycombinator.com/item?id=38172621 Also, aider now supports these new models, including `gpt-4-1106-preview` with the massive 128k context window. https://github.com/paul-gauthier/aider/releases/tag/v0.17.0 https://github.com/paul-gauthier/aider/releases/tag/v0.17.0
  - extr 3y ago
    Dude I love how passionate you are about this project. I see you in every GPT thread. Despite all this tech there are so few projects out there trying to make large-repo code editing/generation possible.
    - hanselot 3y ago
      I think that's mostly because most people are skeptical about trusting people with IP. If there was a way to prove that the data was not being funneled into openai's next models, sure, but where is the proof of that? A piece of paper that says you aren't allowed to do something, does not equate to proof of that thing not being done. Personally I believe all code work should be Open Source by default, as that would ensure the lowest quality code gets filtered out and only the best code gets used for production, resulting in the most efficient solutions dominating (aka, less carbon emissions or children being abused or whatever the politicians say today). So as long as IP exists, companies will continue to drive profit to the max at the expense of every possible resource that has not been regulated. Instead of this model, why not banish IP, make everything open all at once and have only the best, most efficient code running, thereby locating missing children faster, or emitting less carbon, or bombing terrorists better or w/e.
jillesvangurp 3y ago
There's a nice code gpt plugin for intellij and vs code. Basically you can select some code and ask it to criticize it, refactor it, optimize it, find bugs in it, document it, explain it, etc. A larger context means that you can potentially fit your entire code base in that. Most people struggle to keep the details in their head of even a small code base. The next level would be deeper integration with tools to ensure that whatever it changes, the tests still have to pass and the code still has to compile. Speaking of tests, writing those is another thing it can do. So, AI assisted salvaging of legacy code bases that would otherwise not be economical to deal with could become a thing. What we can expect over the next years is a lot more AI assisted developer productivity. IMHO it will perform better on statically typed languages as those are simply easier to reason about for tools.

by zizee 3y ago

throw2321 3y ago

I've been thinking about this for a while now, wrt two points: 1. This will be the end of traditional SWEs and the rise of the age of debuggers, human debuggers who spend their days setting up breakpoints and figuring bugs in a sea of LLM generated code. 2. Hiring will switch from using Leetcode questions to "pull out your debugger and figure out what's wrong with this code".

w-m 3y ago
What makes you think the LLM couldn’t run a debugging session from the content of a JIRA ticket and the whole code base + documentation?
- eichin 3y ago
  Having never seen it, or anything even close to it. (Of course, I'm a little biased by seeing product demos that don't even get "add another item to this list of command line arguments" right; maybe if everyone already believes it works, nobody bothers to actually sell that?)
- spookie 3y ago
  If the codebase is anything more than a simple Python project... I don't think that'll happen. It just doesn't scale that well. Hell, GPT-4 can't make sense of my own projects.
- mike_hearn 3y ago
  Who can say what's possible but there are very few "debugging transcripts" for neural nets to train on out there. So it'd have to sort of work out how to operate a debugger via functions, understand the codebase (perhaps large parts of it), intuit what's going wrong, be able to modify the codebase, etc. Lots of work to do there.
- Workaccount2 3y ago
  Nobody wants to face the potential that their skills are going to greatly decrease in value.
eichin 3y ago
The first interview where I was handed some (intentionally) broken C code and gdb was about 15 years ago. I'm not sure that part is a change in developer workflow (this may apply more in systems and embedded though.) I've been paying attention to this too (mostly by following Simon Willison) and I'm still solidly in the "get back to me when this stuff can successfully review a pull request or even interpret a traceback" camp...
ukuina 3y ago
Yep, time to start adding source_file.prompt sidecar files next to each generated module so debugging sessions can start at the same initial condition.

ushakov 3y ago

We are doing this for API-Testing now. You should check out our website https://ai.stepci.com https://ai.stepci.com

somsak2 3y ago
piece of feedback: it's weird to have a drop-down on "OpenAPI Links" when there are no other options.
- ushakov 3y ago
  Thanks! We will have more examples coming very soon

gen220 3y ago

> If these tools help to write more code, but do not assist with maintainance, I wonder if we're going to see masses of new code written really quickly, and then everything grinds to a halt, because no one has an intimate understanding of what was written? Yep. Companies using LLMs to "augment" junior developers will get a lot of positive press, but I guess it remains to be seen how much the market consistently rewards this behavior. Consumers will probably see right through it, but the b2b folks might get fleeced for a few years before eventually churning and moving to a higher quality old-fashioned competitor that employs senior talent. But IDK, maybe we'll come up with models that are good at growing and maintaining a coherent codebase. It doesn't seem like an impossible task, given where we are today. But we're pretty far from it still, as you point out.

bertil 3y ago
What are the tasks that you envision are key to maintenance? - bug finding and fixing - parsing logs to find optimisation options - refactoring (after several local changes) - given new features, recommending a refactoring? I feel like code assistants are already reasonable help for doing the first two, and the later two are mostly a question of context window. I feel we might end up with code bases split by context sizes, stitched with shared descriptions.
- gen220 3y ago
  I guess the issue is that programmers work with a really big context window, and need to reason at multiple levels of abstraction depending on the problem. A great idea to solve a problem at one level of abstraction / context might be a terrible "strategic" idea at a higher level of abstraction. This is what separates the "junior" engineers from "senior" engineers, speaking very loosely. IDK, I'm not convinced by all that I've seen, that GPT is capable of that higher-order thinking. I fear it requires a degree of epistemology that GPT fundamentally doesn't possess as a stochastic token-guesser. It never pushes back against a request, or asks if you really intend another question by your first question. It never tries to read through your requirements to grasp the underlying problem that's prompting them. Maybe some combination of static tools, senior caretakers and prompt hackery can get us to a solution that maintains code effectively. But I don't think you can throw out the senior caretakers, their verification involvement is really necessary. And I don't know how conducive this environment would be to developing the next generation of "senior caretakers".

anotherpaulg 3y ago

My open source ai coding tool aider is unique in that it is designed to work with existing code bases. You can jump into an existing git repo and start aaking for changes, new features, etc. https://github.com/paul-gauthier/aider https://github.com/paul-gauthier/aider It helps gpt understand larger code bases by building a "repository map" based on analyzing the abstract syntax tree of all the code in the repo. This is all built using tree-sitter, the same tooling which powers code search and navigation on GitHub and in many popular IDEs. https://aider.chat/docs/repomap.html https://aider.chat/docs/repomap.html

anotherpaulg 3y ago
Related to the OpenAI announcement, I've been able to generate some preliminary code editing evaluations of the new GPT models. OpenAI is enforcing very low rate limits on the new GPT-4 model. I will update the results as quickly my rate limit allows. https://news.ycombinator.com/item?id=38172621 https://news.ycombinator.com/item?id=38172621 Also, aider now supports these new models, including `gpt-4-1106-preview` with the massive 128k context window. https://github.com/paul-gauthier/aider/releases/tag/v0.17.0 https://github.com/paul-gauthier/aider/releases/tag/v0.17.0
- extr 3y ago
  Dude I love how passionate you are about this project. I see you in every GPT thread. Despite all this tech there are so few projects out there trying to make large-repo code editing/generation possible.
  - hanselot 3y ago
    I think that's mostly because most people are skeptical about trusting people with IP. If there was a way to prove that the data was not being funneled into openai's next models, sure, but where is the proof of that? A piece of paper that says you aren't allowed to do something, does not equate to proof of that thing not being done. Personally I believe all code work should be Open Source by default, as that would ensure the lowest quality code gets filtered out and only the best code gets used for production, resulting in the most efficient solutions dominating (aka, less carbon emissions or children being abused or whatever the politicians say today). So as long as IP exists, companies will continue to drive profit to the max at the expense of every possible resource that has not been regulated. Instead of this model, why not banish IP, make everything open all at once and have only the best, most efficient code running, thereby locating missing children faster, or emitting less carbon, or bombing terrorists better or w/e.

jillesvangurp 3y ago

There's a nice code gpt plugin for intellij and vs code. Basically you can select some code and ask it to criticize it, refactor it, optimize it, find bugs in it, document it, explain it, etc. A larger context means that you can potentially fit your entire code base in that. Most people struggle to keep the details in their head of even a small code base. The next level would be deeper integration with tools to ensure that whatever it changes, the tests still have to pass and the code still has to compile. Speaking of tests, writing those is another thing it can do. So, AI assisted salvaging of legacy code bases that would otherwise not be economical to deal with could become a thing. What we can expect over the next years is a lot more AI assisted developer productivity. IMHO it will perform better on statically typed languages as those are simply easier to reason about for tools.