6 ms·
No, a lack of monitoring cost you $10K. Your app was throwing a database exception and nobody was alerted that this was not only happening, but happening contin
by wavemode 2y ago
No, a lack of monitoring cost you $10K. Your app was throwing a database exception and nobody was alerted that this was not only happening, but happening continuously and in large volumes. Such an alert would have made this a 5-minute investigation rather than 5 days.
If you haven't fixed that alerting deficiency, then you haven't really fixed anything.
- spencerchubb 2y agoI agree this is more of a monitoring mistake, and little to do with chatgpt
- rvnx 2y agoIt could have happened with any programmer writing the code (ChatGPT or not)
- fragmede 2y agoRight? The log message would have said the id isn't unique, and then it would have taken much less time to debug this problem. Programming when everything works is easy, it's handling the problems that makes it hard.
- carbonatom 2y agoAre you guys able to read this post? When I visit the OP post, I see only this text: "Under construction " Looks like the OP removed the post? EDIT: Found archived copy of post: http://web.archive.org/web/20240609213809/https://asim.bearblog.dev/how-a-single-chatgpt-mistake-cost-us-10000/ http://web.archive.org/web/20240609213809/https://asim.bearb...
- sneak 2y agoTBH, if the backend were written in Go, this probably wouldn’t have happened to the extent it did. Somewhere in a log a descriptive error would have shown up. One of the reasons I use Go whenever possible is that it removes a lot of the classic Python footguns. If you are going to rewrite your backend from Javascript, why would you rewrite it in another untyped, error-prone language?
- TheDong 2y agoIn python it's harder to ignore errors than in Go. In go, I've definitely seen: tx, err := db.Tx() defer tx.Commit() // silently ignores the error on committing, which is the important one That would have masked this error so it didn't get logged by the application. In python, if you ignore an exception entirely, like I did that error above, you instead get an exception logged by default. Python's exceptions also include line numbers, where as Go errors by default wouldn't show you _which_ object has a conflict, even if you logged it. In general, python's logs are way better than Go's, and exceptions make it way harder to ignore errors entirely than Go's strategy.
- sneak 2y agoYeah, each have their benefits. I run all my go projects through golangci-lint which is required for merge to master/main, so not checking the error value would not have made it to prod. I suppose there are probably similar checkers for Python that would have caught the passing of a scalar value instead of a function. Perhaps this is an argument for mandatory linting in CI.
- randomdata 2y ago> I've definitely seen What did they say was the thinking behind it? defer tx.Rollback() would make sense, but defer tx.Commit() is nonsensical, regardless of whether or not the error is handled. It seems apparent that the problem there isn't forgetting to check an error, but that someone got their logic all mixed up, confusing rollback with commit.
- TheDong 2y ago"defer tx.Commit() is nonsensical" It's not pure nonsense, it works in the happy path, and it matches the pattern of how people often handle file IO in go. f, err := os.OpenFile(...) defer f.Close() ... which is another place most gophers ignore errors incorrectly. Just like the "defer tx.Commit()" example, it's collocating the idea of setup and cleanup together. Those two patterns are so similar, python handles them in the same way: with db.begin() as conn: # implicit transaction, gets automatically committed with open(...) as f: # implicit file open + close pair, automatically closed You're of course right that go requires more boiler-plate to do the right thing, but the wrong code has no compiler errors and works in the happy path, and fits how people think about the problem in other languages with sane RAII constructs, so of course people will write it.
- asveikau 2y agoI think deploying and then going to sleep is the red flag here. They should have deployed the change at 9am or something and had the workday to monitor issues.
- bruce511 2y agoWhile I agree, it wouldn't have helped. The first n sign-ups per commit worked. So the problem didn't manifest during the day, only when they stopped committing. Now granted, if they'd carried on doing stuff (but not redeployed production) it may have shown up in say mid-aftenoon, or not depending on volume. And of course the general guideline of not deploying anything to production early in the day, or on Friday, is still valid.
- lolinder 2y agoYou can deploy and go to sleep if you have monitoring and alerting and someone getting paged. It shouldn't be a human monitoring for issues anyway, so the only reason to choose 9am over bedtime should be that you don't want to risk a late night page, not that someone will actually be checking up actively on the deployment.
- owl57 2y agoThat is, if you know what to monitor. When you launch an entirely new feature, you often don't know all the ways it will break. And it's not trivial to set up a correct alert for this one. Simple HTTP 5xx threshold wouldn't work because it's high enough to not wake you up whenever cloud provider restarts Postgres, it's too high to catch this. You need either per-endpoint failure rate alerts or something else more clever.
- asveikau 2y agoFor a big launch, you can have your entire team looking at issues in realtime, rather than a small number of people responding to pages in the middle of the night. I think if it's worth it to you there's a big difference.
- huygens6363 2y agoIMO way sooner. Some low hanging fruit tests would catch this before it’d even hit git.
- Edd1CC 2y agoYeah if one instance created one ID, then any integration tests creating more than one user would have failed. There were no testing or logging on a system with live users while doing a refactor between two dynamic languages
- rnts08 2y agoThis gets more and more common, companies and founders does not think about the infrastructure since they believe that their cloud provider of choice is going to do it for them with it's magic. As soon as you expect paying customers in your system you need to have someone with the knowledge and experience to deal with infrastructure. That means logging, monitoring, alerting, security etc. DevOps.. amateurs.
- Kesty 2y agoSure not having tests, is bad. Doing thing with AI without triple checking is dangerous. But not having error logging/alerts on your db ? That's the crazy part. This is a new product, is not legacy code from 20 years ago when they thought it was a neat idea to just throw stuff at the db raw, and check for db errors to do data validation, so alerts are hard because there's so many expected errrors.
- throw156754228 2y agoShould've been picked up at unit, way before monitoring.
- _xnmw 2y agoThe error didn't show up until multiple entities were created. I can forgive someone for not having unit tests for scaling issues.
- huygens6363 2y agoCreating more than one entry is not a “scaling issue”.
- f35caleb 2y agoAgreed. I'm chuckling because I was wrestling with this exact same bug on a FastAPI project last night. I caught it because I have a habit of submitting API endpoints multiple times with the same data to see how the database reacts. Got a key collision when I tried to submit the endpoint the second time and figured out that setup didn't create a new UUID each time. Unit tests are good, yes. Monitoring is also good. But just taking 30 seconds to do some manual testing will catch a LOT of unexpected behavior.
- withinboredom 2y agoI generally follow this process: 1. Get it working: write the code for the desired behavior, not worrying about making it beautiful, testable, whatever. 2. Get it working well: manually testing and finding edge cases, refactoring to get it testable and writing tests to solidify behavior. 3. Get it working fast: optimizing it to be as fast as I need it to be (can sometimes skip this step). No tests should change here, but only new tests.
- lionkor 2y agoMaybe ChatGPT didn't tell them that they needed monitoring?
- makz 2y agoThat was hilarious