5 ms·

I spotted the error instantly. With all due respect to your team - this has nothing to do with ChatGPT and everything to do with using a programming model that

I spotted the error instantly. With all due respect to your team - this has nothing to do with ChatGPT and everything to do with using a programming model that your team does not have sufficient expertise in. Even if this error managed to slip by code review, it would have been caught with virtually any monitoring solution, many of which take less than 5 minutes to set up.

arjvik 2y ago
To be fair, if I wasn't looking for this bug I never would have spotted it. That being said, you're entirely right that any monitoring or even the most basic manual testing should have instantly caught this.
- astromaniak 2y ago
  Easy enough, ChatGPT can write verification code along the main line. Just ask.
  - rvnx 2y ago
    + humans could have done this mistake as well
- Akronymus 2y ago
  Those kinda issues are why I ALWAYS make an integration test with calling the same insert multiple times. I did step into that particular trap more than once (passing the result, rather than the function)
- LigmaBaulls 2y ago
  I spotted it right away. If you plan to use a library in your project RTFM
KennyBlanken 2y ago
It's not some innocent mistake. The title is purposefully clickbait / keyword-y, implying that it was chatgpt that made the 'mistake' for SEO and to generate panicked clicks. "We made a programming error in our use of an LLM, didn't do any QA, and it cost us $10k" doesn't generate the C-suite "oh shit what if ChatGPT fucks up, what's our exposure!?" reaction. There's a million middle and upper management posting this article on LinkedIn, guaranteed. It's like the Mr. Beast open-mouth-surprised expression thumbnail nonsense; you feel incredibly compelled to click it. While we're on the subject: LLMs can't make "mistakes." They are not deterministic. They cannot reason, think, or do logic. They are very fancy word salad generators that use a lot of statistical probabilities. By definition they're not capable of "mistakes" because nothing they generate is remotely guaranteed to be correct or accurate. Edit: The mods boosted the post; it got downvoted into oblivion, for obvious reasons, and then skyrocketed instantly in rank, which means they boosted it: https://hnrankings.info/40627558/ https://hnrankings.info/40627558/ Hilarious that a post which is insanely clickbait (which the rules say should result in a title rewrite) got boosted by the mods. I'm sure it's a complete coincidence that the story was apparently authored by someone at a Ycombinator company: https://news.ycombinator.com/item?id=40629998 https://news.ycombinator.com/item?id=40629998
- vsuperpower2020 2y ago
  >By definition they're not capable of "mistakes" because nothing they generate is remotely guaranteed to be correct or accurate. This makes no sense. Only things that are guaranteed to be correct or accurate can make mistakes? Everyone knows what "mistake" means in this context. Nobody cares what your preferred definition of mistake is.
  - southernplaces7 2y ago
    A good analogy would be nature: Is it making a mistake when something created by it fails in some way? Not really, since there's no conscious intent behind it and thus no correct or incorrect reasoning or a guarantee of either. LLMs share that trait.
  - moritzwarhier 2y ago
    A mistake is usually seen as something that happens when someone (or metaphorically also, something) makes an error, but is capable of solving similar problems through understanding. Hard to put in words. But that's roughly what is concerning about the "AI makes mistakes" narratives. It implies they are caused by a (fixable) fault in reasoning or memory. LLM "AI" will always respond that it made a "mistake" when you correct it. It is trained to do so, and humans often behave similarly. It is hard to come up with a good definition for "mistake", yes. That does not change that using this in case of LLM hallucinations is misleading.
- brabel 2y ago
  > By definition they're not capable of "mistakes" because nothing they generate is remotely guaranteed to be correct or accurate. By that logic, nothing is capable of making mistakes :D. > Hilarious that a post which is insanely clickbait (which the rules say should result in a title rewrite) got boosted by the mods. You have a distorted view of what clickbait is and the rules of this site. I suggest you go calm down and try to stop hating on a technology which is just that: a technology! Like any other, it can be misused, but think about why exactly you feel so passionate about this particular technology.
kachapopopow 2y ago
Having no real experience with python I would assume uuid.uuid4() was some schema definition (like in prisma), so honestly the fact that this bug exists is not surprising at all and I would have done the same mistake myself, but yah one kubectl logs would have been able to catch it immediately. ...also from next.js and prisma to python? ...what?
fragmede 2y ago
Interestingly, you know who else spotted the error? ChatGPT-4o. Annoyingly you can't share a chat with an image in it, but pasting in the image of the bad code, and prompting "whats wrong with the code" got ChatGPT to tell me that: * UUID Generation in Primary Key: The default parameter should use the callable uuid.uuid4 directly instead of str(uuid.uuid4()). SQLAlchemy will call the function to generate the value. * Date Default Value: server_default=text("(now())") might not work as expected. Use func.now() for server-side defaults in SQLAlchemy. * Import Statements: Ensure uuid and text from sqlalchemy are imported. * Column Definitions: Consider using DateTime(timezone=True) for datetime columns to handle time zones. It then provided me with corrected code that does id = Column(String, primary_key=True, default=lambda: str(uuid.uuid4()), unique=True, nullable=False) where the addition of lambda: fixes the problem.
- Kesty 2y ago
  ChatGPT-4o might spot it when asking about the code directly, but this was a conversion from js to python, errors where chatgpt/copilot or any other AI will allucinate or make mistakes to be as close as the original code are very common in my experience. The other common issue is if the original code has thinsg chatgpt doesn't like (misspell, slightly wrong formatting) it will fix it automatically, or if he really think you should have added a particular field you didn't add.
deleted 2y ago
[deleted]
aforwardslash 2y ago
This. And it seems the team wasn't able to do basic troubleshooting from either database or application log. This was a simple error - what will happen when transient errors (such as implicit locks on tables, etc) occurs. These guys shouldn't be writing code - at all.

arjvik 2y ago

To be fair, if I wasn't looking for this bug I never would have spotted it. That being said, you're entirely right that any monitoring or even the most basic manual testing should have instantly caught this.

astromaniak 2y ago
Easy enough, ChatGPT can write verification code along the main line. Just ask.
- rvnx 2y ago
  + humans could have done this mistake as well
Akronymus 2y ago
Those kinda issues are why I ALWAYS make an integration test with calling the same insert multiple times. I did step into that particular trap more than once (passing the result, rather than the function)
LigmaBaulls 2y ago
I spotted it right away. If you plan to use a library in your project RTFM

KennyBlanken 2y ago

It's not some innocent mistake. The title is purposefully clickbait / keyword-y, implying that it was chatgpt that made the 'mistake' for SEO and to generate panicked clicks. "We made a programming error in our use of an LLM, didn't do any QA, and it cost us $10k" doesn't generate the C-suite "oh shit what if ChatGPT fucks up, what's our exposure!?" reaction. There's a million middle and upper management posting this article on LinkedIn, guaranteed. It's like the Mr. Beast open-mouth-surprised expression thumbnail nonsense; you feel incredibly compelled to click it. While we're on the subject: LLMs can't make "mistakes." They are not deterministic. They cannot reason, think, or do logic. They are very fancy word salad generators that use a lot of statistical probabilities. By definition they're not capable of "mistakes" because nothing they generate is remotely guaranteed to be correct or accurate. Edit: The mods boosted the post; it got downvoted into oblivion, for obvious reasons, and then skyrocketed instantly in rank, which means they boosted it: https://hnrankings.info/40627558/ https://hnrankings.info/40627558/ Hilarious that a post which is insanely clickbait (which the rules say should result in a title rewrite) got boosted by the mods. I'm sure it's a complete coincidence that the story was apparently authored by someone at a Ycombinator company: https://news.ycombinator.com/item?id=40629998 https://news.ycombinator.com/item?id=40629998

vsuperpower2020 2y ago
>By definition they're not capable of "mistakes" because nothing they generate is remotely guaranteed to be correct or accurate. This makes no sense. Only things that are guaranteed to be correct or accurate can make mistakes? Everyone knows what "mistake" means in this context. Nobody cares what your preferred definition of mistake is.
- southernplaces7 2y ago
  A good analogy would be nature: Is it making a mistake when something created by it fails in some way? Not really, since there's no conscious intent behind it and thus no correct or incorrect reasoning or a guarantee of either. LLMs share that trait.
- moritzwarhier 2y ago
  A mistake is usually seen as something that happens when someone (or metaphorically also, something) makes an error, but is capable of solving similar problems through understanding. Hard to put in words. But that's roughly what is concerning about the "AI makes mistakes" narratives. It implies they are caused by a (fixable) fault in reasoning or memory. LLM "AI" will always respond that it made a "mistake" when you correct it. It is trained to do so, and humans often behave similarly. It is hard to come up with a good definition for "mistake", yes. That does not change that using this in case of LLM hallucinations is misleading.
brabel 2y ago
> By definition they're not capable of "mistakes" because nothing they generate is remotely guaranteed to be correct or accurate. By that logic, nothing is capable of making mistakes :D. > Hilarious that a post which is insanely clickbait (which the rules say should result in a title rewrite) got boosted by the mods. You have a distorted view of what clickbait is and the rules of this site. I suggest you go calm down and try to stop hating on a technology which is just that: a technology! Like any other, it can be misused, but think about why exactly you feel so passionate about this particular technology.

kachapopopow 2y ago

Having no real experience with python I would assume uuid.uuid4() was some schema definition (like in prisma), so honestly the fact that this bug exists is not surprising at all and I would have done the same mistake myself, but yah one kubectl logs would have been able to catch it immediately. ...also from next.js and prisma to python? ...what?

fragmede 2y ago

Interestingly, you know who else spotted the error? ChatGPT-4o. Annoyingly you can't share a chat with an image in it, but pasting in the image of the bad code, and prompting "whats wrong with the code" got ChatGPT to tell me that: * UUID Generation in Primary Key: The default parameter should use the callable uuid.uuid4 directly instead of str(uuid.uuid4()). SQLAlchemy will call the function to generate the value. * Date Default Value: server_default=text("(now())") might not work as expected. Use func.now() for server-side defaults in SQLAlchemy. * Import Statements: Ensure uuid and text from sqlalchemy are imported. * Column Definitions: Consider using DateTime(timezone=True) for datetime columns to handle time zones. It then provided me with corrected code that does id = Column(String, primary_key=True, default=lambda: str(uuid.uuid4()), unique=True, nullable=False) where the addition of lambda: fixes the problem.

Kesty 2y ago
ChatGPT-4o might spot it when asking about the code directly, but this was a conversion from js to python, errors where chatgpt/copilot or any other AI will allucinate or make mistakes to be as close as the original code are very common in my experience. The other common issue is if the original code has thinsg chatgpt doesn't like (misspell, slightly wrong formatting) it will fix it automatically, or if he really think you should have added a particular field you didn't add.

deleted 2y ago

[deleted]

aforwardslash 2y ago

This. And it seems the team wasn't able to do basic troubleshooting from either database or application log. This was a simple error - what will happen when transient errors (such as implicit locks on tables, etc) occurs. These guys shouldn't be writing code - at all.