5 ms·

The blog post is 404ing, here's a Web archive link https://web.archive.org/web/20240610032818/https://asim.bearblog.dev/how-a-single-chatgpt-mistake-cost-us-10

The blog post is 404ing, here's a Web archive link https://web.archive.org/web/20240610032818/https://asim.bearblog.dev/how-a-single-chatgpt-mistake-cost-us-10000/ https://web.archive.org/web/20240610032818/https://asim.bear... The author has added an important edit: > I want to preface this by saying yes the practices here are very bad and embarrassing (and we've since added robust unit/integration tests and alerting/logging), could/should have been avoided, were human errors beyond anything, and very obvious in hindsight. > > This was from a different time under large time constraints at the very earliest stages (first few weeks) of a company. I'm mostly just sharing this as a funny story with unique circumstances surrounding bug reproducibility in prod (due again to our own stupidity) Please read with that in mind

Unfrozen0688 2y ago
[dead]
patates 2y ago
Could they have deleted it because of all the negativity? They did make a silly mistake, but we are humans, and humans, be it individually or collectively, do make silly mistakes.
- KennyBlanken 2y ago
  If you code for a hobby/fun, yeah, sure, it's a silly mistake. If you're earning past six figures, are part of a team of programmers, call yourself an professional / engineer, and have technical management above you like a VP of Engineering, yadda yadda....then it's closer to systematic failure of the company's engineering practices than "mistake." There is a reason we call it software engineering, not software fuckarounding (or, cough, "DevOps Engineeer".) Software engineering practices assume people are going to make mistakes, and implements procedures to reduce the chances of that making it into production, and reduce the impact of those mistakes if they do make it into production.
  - deleted 2y ago
    [deleted]
  - jnsaff2 2y ago
    This is how I felt when a Gitlab employee deleted the production database by doing it in the wrong terminal window.
    - phist_mcgee 2y ago
      Get into the habit of colour-coding your important SSH sessions, red/green/blue can give you a very powerful subconscious check before doing something very silly.
      - bloqs 2y ago
        How do you do this so that it lasts? I find mine resets to default
        phist_mcgee 2y ago
        I use iTerm2 and have different profiles for different directories/remote hosts. iTerm switches the background colour automatically for me. I'm sure other terminal emulators would have similar features. https://bsago.me/tech-notes/change-ssh-background-colour-with-iterm2#:~:text=Profiles%20can%20be%20edited%20by,or%20duplicate%20an%20existing%20one https://bsago.me/tech-notes/change-ssh-background-colour-wit....
        Edd1CC 2y ago
        Can’t you set it in the .zshrc file?
    - HenryBemis 2y ago
      what phist wrote. Color code the backgrounds of your servers, different colors. So anyone who connects to 'take console' in any system is hit by a blinding electric Green/Blue/Red/Yellow and other striking colors. I assume that all systems already have descriptive names App_DEV_Server1, App_PROD_Server5, etc. It also helps if (ofc they would be right??) in separate IP groups/WLANS? If you are running Windows, it's a good idea to use BGINFO.exe by SysInternals (or Winternals as we old people still call it), and display the most relevant info (showing Dev/Prod/UAT/etc.) with big-big-big letters.
- kissgyorgy 2y ago
  If they would have made that mistake by writing code and just misunderstood something or oversaw the problem, fine. But making this mistake by copy-pasting from ChatGPT without proper review is just terrible.
  - sokoloff 2y ago
    I don’t find the source of the error being a careless human writing original code without proper review vs a careless human copy/pasting code without proper review to be significantly different.
    - kissgyorgy 2y ago
      The difference is that everyone knows about hallucinations, so an LLM never can be trusted by default, they still trusted it blindly.
      - sokoloff 2y ago
        Is that really worse than trusting blindly the code of a human, about which "everyone knows about" the bugs that humans write as well.
madiele 2y ago
Seems like he changed the subdomain or something, the article is still up https://0912i390129ionkjan.bearblog.dev/how-a-single-chatgpt-mistake-cost-us-10000/ https://0912i390129ionkjan.bearblog.dev/how-a-single-chatgpt...
- boxed 2y ago
  Also 404 now
2510c39011c5 2y ago
the page on archive.org could not be loaded now... but google cache still serves a copy... https://webcache.googleusercontent.com/search?q=cache%3Ahttps%3A%2F%2Fasim.bearblog.dev%2Fhow-a-single-chatgpt-mistake-cost-us-10000%2F&oq=cache%3Ahttps%3A%2F%2Fasim.bearblog.dev%2Fhow-a-single-chatgpt-mistake-cost-us-10000%2F&gs_lcrp=EgZjaHJvbWUyBggAEEUYOTIGCAEQRRg60gEIMTk5N2owajeoAgCwAgA https://webcache.googleusercontent.com/search?q=cache%3Ahttp...

Unfrozen0688 2y ago

[dead]

patates 2y ago

Could they have deleted it because of all the negativity? They did make a silly mistake, but we are humans, and humans, be it individually or collectively, do make silly mistakes.

KennyBlanken 2y ago
If you code for a hobby/fun, yeah, sure, it's a silly mistake. If you're earning past six figures, are part of a team of programmers, call yourself an professional / engineer, and have technical management above you like a VP of Engineering, yadda yadda....then it's closer to systematic failure of the company's engineering practices than "mistake." There is a reason we call it software engineering, not software fuckarounding (or, cough, "DevOps Engineeer".) Software engineering practices assume people are going to make mistakes, and implements procedures to reduce the chances of that making it into production, and reduce the impact of those mistakes if they do make it into production.
- deleted 2y ago
  [deleted]
- jnsaff2 2y ago
  This is how I felt when a Gitlab employee deleted the production database by doing it in the wrong terminal window.
  - phist_mcgee 2y ago
    Get into the habit of colour-coding your important SSH sessions, red/green/blue can give you a very powerful subconscious check before doing something very silly.
    - bloqs 2y ago
      How do you do this so that it lasts? I find mine resets to default
      - phist_mcgee 2y ago
        I use iTerm2 and have different profiles for different directories/remote hosts. iTerm switches the background colour automatically for me. I'm sure other terminal emulators would have similar features. https://bsago.me/tech-notes/change-ssh-background-colour-with-iterm2#:~:text=Profiles%20can%20be%20edited%20by,or%20duplicate%20an%20existing%20one https://bsago.me/tech-notes/change-ssh-background-colour-wit....
      - Edd1CC 2y ago
        Can’t you set it in the .zshrc file?
  - HenryBemis 2y ago
    what phist wrote. Color code the backgrounds of your servers, different colors. So anyone who connects to 'take console' in any system is hit by a blinding electric Green/Blue/Red/Yellow and other striking colors. I assume that all systems already have descriptive names App_DEV_Server1, App_PROD_Server5, etc. It also helps if (ofc they would be right??) in separate IP groups/WLANS? If you are running Windows, it's a good idea to use BGINFO.exe by SysInternals (or Winternals as we old people still call it), and display the most relevant info (showing Dev/Prod/UAT/etc.) with big-big-big letters.
kissgyorgy 2y ago
If they would have made that mistake by writing code and just misunderstood something or oversaw the problem, fine. But making this mistake by copy-pasting from ChatGPT without proper review is just terrible.
- sokoloff 2y ago
  I don’t find the source of the error being a careless human writing original code without proper review vs a careless human copy/pasting code without proper review to be significantly different.
  - kissgyorgy 2y ago
    The difference is that everyone knows about hallucinations, so an LLM never can be trusted by default, they still trusted it blindly.
    - sokoloff 2y ago
      Is that really worse than trusting blindly the code of a human, about which "everyone knows about" the bugs that humans write as well.

madiele 2y ago

Seems like he changed the subdomain or something, the article is still up https://0912i390129ionkjan.bearblog.dev/how-a-single-chatgpt-mistake-cost-us-10000/ https://0912i390129ionkjan.bearblog.dev/how-a-single-chatgpt...

boxed 2y ago
Also 404 now

2510c39011c5 2y ago

the page on archive.org could not be loaded now... but google cache still serves a copy... https://webcache.googleusercontent.com/search?q=cache%3Ahttps%3A%2F%2Fasim.bearblog.dev%2Fhow-a-single-chatgpt-mistake-cost-us-10000%2F&oq=cache%3Ahttps%3A%2F%2Fasim.bearblog.dev%2Fhow-a-single-chatgpt-mistake-cost-us-10000%2F&gs_lcrp=EgZjaHJvbWUyBggAEEUYOTIGCAEQRRg60gEIMTk5N2owajeoAgCwAgA https://webcache.googleusercontent.com/search?q=cache%3Ahttp...