4 ms·

So it's on par with a junior software engineer?

elil17 4y ago
No, I don't think it's comparable to a junior engineer. It works best for pretty short snippets of code. It also can't decide when to run the program and how to interpret/make changes based on the results.
- mynameisvlad 4y ago
  So… a junior developer? I (mostly) kid but some seniors I know struggle at those things.
  - elil17 4y ago
    I mean I get what you're saying. But it doesn't just struggle with those things - they aren't even the sort of things it can do.
    - retrocryptid 4y ago
      Yes. Of all the comments I've made on HN, that one most deserves to be down-voted. Jr. Devs I know are motivated and reasonably intelligent.
- steve_adams_86 4y ago
  > It also can't decide when to run the program and how to interpret/make changes based on the results. Not quite, but a bit of plumbing can get you closer. Not human using a computer close, but interestingly closer nonetheless. I've been trying to accomplish something akin to this by having a program monitor and alter another program within a virtual machine, using GPT-generated solutions to error traces to correct bugs in the sand boxed program. It watches the program to see when an error occurs, feeds the error to GPT with pertinent code, then tries to splice in the solution. It kind of works. I don't think we're going to see human-levels of success from this in the immediate future, but I was able to write a simple event-based system which alters a program to resolve simple bugs. It even does it on a different git branch, and there is some stubbed out code and prompts for generating tests. In my manual testing, this actually worked too. If the tests passed I was going to have it push the change set and create a PR explaining the changes, tests, etc. I doubt I'll continue now that Copilot is doing this already. My point though is that with the right configuration, the right data and prompts, and a system orchestrating the start/stop/test patterns based on the state of the sandboxed program, you can begin to achieve something akin to an inexperienced person solving bugs. Sometimes it does a terrible job and other times it kind of falls over itself. But we're already leaps and bounds ahead of previous systems, and I just cobbled this together with what's possible via OpenAI's API. The crazy part is that there are so many possible layers. Like say we get our initial solution and we verify that it works. Well, now we can have a system which optimizes the implementation. Like a PR buddy that observes the implementation and determines: should this test be appended to an existing suite of tests? Can the test case simply be added to an existing table-driven test? How can we streamline this patch to avoid an endless stream of additional files and tests to maintain? I think that's actually tractable already. While the success rate won't be 100% today, it'll clearly only improve.
vintermann 4y ago
Sure. One that never gets tired, never gets frustrated, always does his best, and has a pause button.
- geraneum 4y ago
  This applies to managers and many founders too. I imagine it's gonna affect the startup scene more than people expect and the number of current model of startups where you build a custom software solution for a problem may also fall. It applies to all white collar workers. This is the goal of a corporation like Microsoft to develop such a system (to consolidate those added values), and it will happen one day as it happened with physical labor. Although we are not quite there yet. It might be far or not. Who knows?

No, I don't think it's comparable to a junior engineer. It works best for pretty short snippets of code. It also can't decide when to run the program and how to interpret/make changes based on the results.

mynameisvlad 4y ago
So… a junior developer? I (mostly) kid but some seniors I know struggle at those things.
- elil17 4y ago
  I mean I get what you're saying. But it doesn't just struggle with those things - they aren't even the sort of things it can do.
  - retrocryptid 4y ago
    Yes. Of all the comments I've made on HN, that one most deserves to be down-voted. Jr. Devs I know are motivated and reasonably intelligent.
steve_adams_86 4y ago
> It also can't decide when to run the program and how to interpret/make changes based on the results. Not quite, but a bit of plumbing can get you closer. Not human using a computer close, but interestingly closer nonetheless. I've been trying to accomplish something akin to this by having a program monitor and alter another program within a virtual machine, using GPT-generated solutions to error traces to correct bugs in the sand boxed program. It watches the program to see when an error occurs, feeds the error to GPT with pertinent code, then tries to splice in the solution. It kind of works. I don't think we're going to see human-levels of success from this in the immediate future, but I was able to write a simple event-based system which alters a program to resolve simple bugs. It even does it on a different git branch, and there is some stubbed out code and prompts for generating tests. In my manual testing, this actually worked too. If the tests passed I was going to have it push the change set and create a PR explaining the changes, tests, etc. I doubt I'll continue now that Copilot is doing this already. My point though is that with the right configuration, the right data and prompts, and a system orchestrating the start/stop/test patterns based on the state of the sandboxed program, you can begin to achieve something akin to an inexperienced person solving bugs. Sometimes it does a terrible job and other times it kind of falls over itself. But we're already leaps and bounds ahead of previous systems, and I just cobbled this together with what's possible via OpenAI's API. The crazy part is that there are so many possible layers. Like say we get our initial solution and we verify that it works. Well, now we can have a system which optimizes the implementation. Like a PR buddy that observes the implementation and determines: should this test be appended to an existing suite of tests? Can the test case simply be added to an existing table-driven test? How can we streamline this patch to avoid an endless stream of additional files and tests to maintain? I think that's actually tractable already. While the success rate won't be 100% today, it'll clearly only improve.

vintermann 4y ago

Sure. One that never gets tired, never gets frustrated, always does his best, and has a pause button.

geraneum 4y ago
This applies to managers and many founders too. I imagine it's gonna affect the startup scene more than people expect and the number of current model of startups where you build a custom software solution for a problem may also fall. It applies to all white collar workers. This is the goal of a corporation like Microsoft to develop such a system (to consolidate those added values), and it will happen one day as it happened with physical labor. Although we are not quite there yet. It might be far or not. Who knows?