3 ms·

Someone vibe coded a PR on my team where there were hundreds of lines doing complex validation of an uploaded CSV file (which we only expected to have two colum

Someone vibe coded a PR on my team where there were hundreds of lines doing complex validation of an uploaded CSV file (which we only expected to have two columns) instead of just relying on Ruby's built-in CSV library (i.e. `CSV.parse` would have done everything the AI produced)

mandeepj 1y ago
That’s a good example of ‘getting a desired outcome based on prompt’ - use a built-in lib or not.
- vkou 1y ago
  And when it hallucinates a non-existant library, what are the magic prompts that you give it that makes it stop trying to bullshit you?
  - mandeepj 1y ago
    > what are the magic prompts that you give it that makes it stop trying to bullshit you? Maybe keep your eyes open? :-)
    - vkou 1y ago
      As I thought. And for the record - my eyes are open. I'm aware I'm being bullshitted. I don't trust, I verify. But I also don't have a magical lever that I can pull to make it stop hallucinating. ... and every time I ask if one exists, I get either crickets, or a response that doesn't answer the question.
      - mandeepj 1y ago
        Sorry! Didn't mean to BS you. I've not come across a scenario where it hallucinated me with a non-existent library. Can you share what you were trying to do when that happened?
        vkou 1y ago
        I wish I had the transcript. I don't, and I'm afraid that the passage of time has muddied the interaction to the point of uselessness (when it comes to listing specifics).
      - manwe150 1y ago
        Ask it to write tests, then let it run until the tests pass (preferably in a sandbox, far from your git credentials). It is quite good at developing hypotheses and tests for them, if that is what you explicitly ask for. It doesn’t have (much) ego, so it doesn’t care if it is proven wrong and will accept any outcome fairly if it is testable. Although sometimes it comes to the wrong conclusion and doubles down that the fact should be true so it prepares to write and publish a library to make it true
    - rsynnott 1y ago
      Okay, so at this point it is strictly worse than just searching for and reading the very simple docs for the Ruby CSV parser, surely? Because, as part of your verification, you will have to do that _anyway_.
7thpower 1y ago
I wonder how many times the LLM randomly tried to steer back to that library only to get chastised for not following instructions.

mandeepj 1y ago

That’s a good example of ‘getting a desired outcome based on prompt’ - use a built-in lib or not.

vkou 1y ago
And when it hallucinates a non-existant library, what are the magic prompts that you give it that makes it stop trying to bullshit you?
- mandeepj 1y ago
  > what are the magic prompts that you give it that makes it stop trying to bullshit you? Maybe keep your eyes open? :-)
  - vkou 1y ago
    As I thought. And for the record - my eyes are open. I'm aware I'm being bullshitted. I don't trust, I verify. But I also don't have a magical lever that I can pull to make it stop hallucinating. ... and every time I ask if one exists, I get either crickets, or a response that doesn't answer the question.
    - mandeepj 1y ago
      Sorry! Didn't mean to BS you. I've not come across a scenario where it hallucinated me with a non-existent library. Can you share what you were trying to do when that happened?
      - vkou 1y ago
        I wish I had the transcript. I don't, and I'm afraid that the passage of time has muddied the interaction to the point of uselessness (when it comes to listing specifics).
    - manwe150 1y ago
      Ask it to write tests, then let it run until the tests pass (preferably in a sandbox, far from your git credentials). It is quite good at developing hypotheses and tests for them, if that is what you explicitly ask for. It doesn’t have (much) ego, so it doesn’t care if it is proven wrong and will accept any outcome fairly if it is testable. Although sometimes it comes to the wrong conclusion and doubles down that the fact should be true so it prepares to write and publish a library to make it true
  - rsynnott 1y ago
    Okay, so at this point it is strictly worse than just searching for and reading the very simple docs for the Ruby CSV parser, surely? Because, as part of your verification, you will have to do that _anyway_.

7thpower 1y ago

I wonder how many times the LLM randomly tried to steer back to that library only to get chastised for not following instructions.