4 ms·

I'm not talking about generating a few lines instead of importing left-pad. In recent PRs I've had: - Implementing a scheduler from scratch (hundreds of lines)

I'm not talking about generating a few lines instead of importing left-pad. In recent PRs I've had: - Implementing a scheduler from scratch (hundreds of lines), when there are many many libraries for this in Go. - Implementing some complex configuration store that is safe for concurrent access , using generics, reflection, and a whole other host of stuff (additionally hundreds of lines plus more for tests). While I can't say any of the code is bad, it is effectively like importing a library which your team now owns, but worse in that no one really understands it or supports it. Lastly, I could find libraries that are well supported, documented, and active for each of these use-cases fairly quickly.

daxfohl 1y ago
And that may be where the discrepancy comes in. You feel fast because, whoa I created this whole scheduler in ten seconds! But the you also have to spend an hour code reviewing that scheduler, which, still it feels fast to have a good working scheduler in such a short time. But without AI, maybe it feels slow to find and integrate with some existing scheduling library, but in wall clock time it was the same.
- SchemaLoad 1y ago
  The trick is that no one is actually carefully reviewing this stuff. Reviewing code is properly extremely hard. I'd say even harder than writing it from scratch. But there's no minimum amount of work you have to do. If you just do a quick skim over the result, no one will know you didn't carefully review every single detail. Then it gets merged to production full of mistakes.
  - captainkrtek 1y ago
    To add to this: If I as a reviewer don’t know if the author used AI, I can’t even assume a single human (typically the author) has even read any or major parts of the code. I could be the first person reviewing it. Not that it’s a great assumption to make, but it’s also fair to take a PR and register that the author wrote it, understands it, and considers it ready for production. So much work, outside of tech as well, is built on trust at least in part.
    - dm270 1y ago
      I find this disrespectful by the author. I’m sure I’ve had colleagues at work that did this to me: throwing ai generated code at the reviewers with the mindset like "why should I look at it? That's what the reviewer does anyway".
      - SchemaLoad 1y ago
        I always passively call out the submitter on this stuff with comments like "Can you explain to me why you did this? Can you explain what this is expected to return" etc. Usually gets them to sort out their behavior without directly making accusations that could be incorrect. If they really did write or strongly review the code, those questions are easy to answer.
davidcelis 1y ago
Someone vibe coded a PR on my team where there were hundreds of lines doing complex validation of an uploaded CSV file (which we only expected to have two columns) instead of just relying on Ruby's built-in CSV library (i.e. `CSV.parse` would have done everything the AI produced)
- mandeepj 1y ago
  That’s a good example of ‘getting a desired outcome based on prompt’ - use a built-in lib or not.
  - vkou 1y ago
    And when it hallucinates a non-existant library, what are the magic prompts that you give it that makes it stop trying to bullshit you?
    - mandeepj 1y ago
      > what are the magic prompts that you give it that makes it stop trying to bullshit you? Maybe keep your eyes open? :-)
      - vkou 1y ago
        As I thought. And for the record - my eyes are open. I'm aware I'm being bullshitted. I don't trust, I verify. But I also don't have a magical lever that I can pull to make it stop hallucinating. ... and every time I ask if one exists, I get either crickets, or a response that doesn't answer the question.
        mandeepj 1y ago
        Sorry! Didn't mean to BS you. I've not come across a scenario where it hallucinated me with a non-existent library. Can you share what you were trying to do when that happened?

daxfohl 1y ago

And that may be where the discrepancy comes in. You feel fast because, whoa I created this whole scheduler in ten seconds! But the you also have to spend an hour code reviewing that scheduler, which, still it feels fast to have a good working scheduler in such a short time. But without AI, maybe it feels slow to find and integrate with some existing scheduling library, but in wall clock time it was the same.

SchemaLoad 1y ago
The trick is that no one is actually carefully reviewing this stuff. Reviewing code is properly extremely hard. I'd say even harder than writing it from scratch. But there's no minimum amount of work you have to do. If you just do a quick skim over the result, no one will know you didn't carefully review every single detail. Then it gets merged to production full of mistakes.
- captainkrtek 1y ago
  To add to this: If I as a reviewer don’t know if the author used AI, I can’t even assume a single human (typically the author) has even read any or major parts of the code. I could be the first person reviewing it. Not that it’s a great assumption to make, but it’s also fair to take a PR and register that the author wrote it, understands it, and considers it ready for production. So much work, outside of tech as well, is built on trust at least in part.
  - dm270 1y ago
    I find this disrespectful by the author. I’m sure I’ve had colleagues at work that did this to me: throwing ai generated code at the reviewers with the mindset like "why should I look at it? That's what the reviewer does anyway".
    - SchemaLoad 1y ago
      I always passively call out the submitter on this stuff with comments like "Can you explain to me why you did this? Can you explain what this is expected to return" etc. Usually gets them to sort out their behavior without directly making accusations that could be incorrect. If they really did write or strongly review the code, those questions are easy to answer.

davidcelis 1y ago

Someone vibe coded a PR on my team where there were hundreds of lines doing complex validation of an uploaded CSV file (which we only expected to have two columns) instead of just relying on Ruby's built-in CSV library (i.e. `CSV.parse` would have done everything the AI produced)

mandeepj 1y ago
That’s a good example of ‘getting a desired outcome based on prompt’ - use a built-in lib or not.
- vkou 1y ago
  And when it hallucinates a non-existant library, what are the magic prompts that you give it that makes it stop trying to bullshit you?
  - mandeepj 1y ago
    > what are the magic prompts that you give it that makes it stop trying to bullshit you? Maybe keep your eyes open? :-)
    - vkou 1y ago
      As I thought. And for the record - my eyes are open. I'm aware I'm being bullshitted. I don't trust, I verify. But I also don't have a magical lever that I can pull to make it stop hallucinating. ... and every time I ask if one exists, I get either crickets, or a response that doesn't answer the question.
      - mandeepj 1y ago
        Sorry! Didn't mean to BS you. I've not come across a scenario where it hallucinated me with a non-existent library. Can you share what you were trying to do when that happened?