5 ms·

Run the code it writes, if it gives an error, paste it into the chat and 90% of the time the LLM can fix the issue. People are missing the point here - it's no

Run the code it writes, if it gives an error, paste it into the chat and 90% of the time the LLM can fix the issue. People are missing the point here - it's not about writing code in one-shot. An LLM-enabled loop can generate the code and test and refine it until it works

_bramses 3y ago
Completely agree. Any metric unto itself will only show a partial picture. Using LLMs as an iteration partner can solve any problem which is embedded in latent space (which is pretty much all problems that devs get paid to solve). People seemingly desire to zealously defend the LeetCode way of solving problems for some inane reason.
tedunangst 3y ago
Someone posted a transcript of having chatgpt write them a bash script with the comment "see how much easier this was than figuring it out" and it took like a dozen tries of pasting error messages back in. I was infuriated just reading it. I cannot imagine trying to develop this way. But if people like it, whatever blows your hair back. Allegedly the benefit is it will do the boring stuff like write the error handling code for you, but I have no faith it would work if it takes so many tries to get the happy path correct.
- arp242 3y ago
  > I cannot imagine trying to develop this way. But if people like it, whatever blows your hair back. I agree; who am I to comment on how you write the code, as long as it gets written, right? That said, I've already had a "this is really odd code you committed last week, what's up with that?" and the reply was "oh, dunno, that's just what ChatGPT gave me". Meh. Actually writing things yourself does increase your understanding, so it's not the same, not really. It's when they told you to take notes at school: I didn't believe my teachers it'll help retain knowledge on account of being a stubborn little idiot, but they were right! For people who have already done this programming thing for a while I guess it'll work out, but my main worry is the effect it will have on more junior people who will "grow up" on ChatGPT. "Figuring it out" yourself has a lot of value. Just because the code is free of syntax errors doesn't mean it's free of bugs. Shell scripting is a classic case where it's hard to make something work but also buggy. You need to actually understand the code and reason about it. Trail-and-error programming rarely leads to good code. I also fear we'll end up with hard to read overly verbose/repetitive code, "because ChatGPT/copilot will just generate it".
  - teaearlgraycold 3y ago
    Anyone committing code they don’t understand should be put on a PIP.
    - arp242 3y ago
      He was the team lead shrug This is one of several reasons I worked there for a week only.
- wavemode 3y ago
  I also cannot understand how people develop that way. That being said, better workflows do exist. I imagine IDE-integrated LLM tools like Copilot and CodeGPT are much more productivity-enhancing than copy-pasting code between a browser and an editor.
- wokwokwok 3y ago
  I think you’re missing the point here. Language models scale. It doesn’t matter if a single pass doesn’t solve the problem, has syntax errors, etc. A single pass costs a fraction of a cent. You can just automate the process of code -> create variant -> fix from LLM -> apply deterministic tests until the code at least compiles -> pass it to the user; the fact you can’t do that with chatgpt is just because it’s not a coding tool. Heck, there are already companies doing exactly this with vulnerability scanners (ie. the deterministic feedback loop) to suggest security fixes for code. You just repeat a heap of times until you get a solution that passes all the scanners. If it takes 10 tries, or 100 tries, it still is zero effort from a human. Of course, whether the result does the right thing is another matter, but the frustrating ergonomics of copy-paste cycle is because of the ui, not the technology. You will see “surprisingly good” output from professional tools in this space (we already are); but a lot of it is not magic sauce; it’s just the same tools, run multiple times, in a way that saves you doing it manually and just shows the best results, with a pretty ribbon on it.
  - ChatGTP 3y ago
    Sounds kind of silly, just run the thing till it gives the right answer ? How do you know each pass has not introduced a new problem ?
    - wokwokwok 3y ago
      Yup. …and if it works (and it does) then it really goes to show that you (and me, and people in general) have poor intuition about this stuff.
      - ChatGTP 3y ago
        Yeah, please don't try bullshit me dude. It's a silly, potentially dangerous and inefficient way to "code", which is the easiest part of my job. If my job was just coding all day, I'd be happy.
- Etheryte 3y ago
  Man I sure love running buggy bash scripts on my system just to see what happens.
maxbond 3y ago
If you're writing eg a bash script and it contains a difficult to spot bug which causes causes data loss (see [1] caused by a single space), what then? [1] https://github.com/MrMEEE/bumblebee-Old-and-abbandoned/issues/123 https://github.com/MrMEEE/bumblebee-Old-and-abbandoned/issue...
- YeGoblynQueenne 3y ago
  Then you ask ChatGPT to write you an email to apologise to your boss. They'll probably ask ChatGPT to write an email to reply, so that's all good.

_bramses 3y ago

Completely agree. Any metric unto itself will only show a partial picture. Using LLMs as an iteration partner can solve any problem which is embedded in latent space (which is pretty much all problems that devs get paid to solve). People seemingly desire to zealously defend the LeetCode way of solving problems for some inane reason.

tedunangst 3y ago

Someone posted a transcript of having chatgpt write them a bash script with the comment "see how much easier this was than figuring it out" and it took like a dozen tries of pasting error messages back in. I was infuriated just reading it. I cannot imagine trying to develop this way. But if people like it, whatever blows your hair back. Allegedly the benefit is it will do the boring stuff like write the error handling code for you, but I have no faith it would work if it takes so many tries to get the happy path correct.

arp242 3y ago
> I cannot imagine trying to develop this way. But if people like it, whatever blows your hair back. I agree; who am I to comment on how you write the code, as long as it gets written, right? That said, I've already had a "this is really odd code you committed last week, what's up with that?" and the reply was "oh, dunno, that's just what ChatGPT gave me". Meh. Actually writing things yourself does increase your understanding, so it's not the same, not really. It's when they told you to take notes at school: I didn't believe my teachers it'll help retain knowledge on account of being a stubborn little idiot, but they were right! For people who have already done this programming thing for a while I guess it'll work out, but my main worry is the effect it will have on more junior people who will "grow up" on ChatGPT. "Figuring it out" yourself has a lot of value. Just because the code is free of syntax errors doesn't mean it's free of bugs. Shell scripting is a classic case where it's hard to make something work but also buggy. You need to actually understand the code and reason about it. Trail-and-error programming rarely leads to good code. I also fear we'll end up with hard to read overly verbose/repetitive code, "because ChatGPT/copilot will just generate it".
- teaearlgraycold 3y ago
  Anyone committing code they don’t understand should be put on a PIP.
  - arp242 3y ago
    He was the team lead shrug This is one of several reasons I worked there for a week only.
wavemode 3y ago
I also cannot understand how people develop that way. That being said, better workflows do exist. I imagine IDE-integrated LLM tools like Copilot and CodeGPT are much more productivity-enhancing than copy-pasting code between a browser and an editor.
wokwokwok 3y ago
I think you’re missing the point here. Language models scale. It doesn’t matter if a single pass doesn’t solve the problem, has syntax errors, etc. A single pass costs a fraction of a cent. You can just automate the process of code -> create variant -> fix from LLM -> apply deterministic tests until the code at least compiles -> pass it to the user; the fact you can’t do that with chatgpt is just because it’s not a coding tool. Heck, there are already companies doing exactly this with vulnerability scanners (ie. the deterministic feedback loop) to suggest security fixes for code. You just repeat a heap of times until you get a solution that passes all the scanners. If it takes 10 tries, or 100 tries, it still is zero effort from a human. Of course, whether the result does the right thing is another matter, but the frustrating ergonomics of copy-paste cycle is because of the ui, not the technology. You will see “surprisingly good” output from professional tools in this space (we already are); but a lot of it is not magic sauce; it’s just the same tools, run multiple times, in a way that saves you doing it manually and just shows the best results, with a pretty ribbon on it.
- ChatGTP 3y ago
  Sounds kind of silly, just run the thing till it gives the right answer ? How do you know each pass has not introduced a new problem ?
  - wokwokwok 3y ago
    Yup. …and if it works (and it does) then it really goes to show that you (and me, and people in general) have poor intuition about this stuff.
    - ChatGTP 3y ago
      Yeah, please don't try bullshit me dude. It's a silly, potentially dangerous and inefficient way to "code", which is the easiest part of my job. If my job was just coding all day, I'd be happy.
Etheryte 3y ago
Man I sure love running buggy bash scripts on my system just to see what happens.

maxbond 3y ago

If you're writing eg a bash script and it contains a difficult to spot bug which causes causes data loss (see [1] caused by a single space), what then? [1] https://github.com/MrMEEE/bumblebee-Old-and-abbandoned/issues/123 https://github.com/MrMEEE/bumblebee-Old-and-abbandoned/issue...

YeGoblynQueenne 3y ago
Then you ask ChatGPT to write you an email to apologise to your boss. They'll probably ask ChatGPT to write an email to reply, so that's all good.