3 ms·

I understand how the mistake was made, it seems relatively easy to slip by even when writing code without ChatGPT. But what I don't understand is how this wasn

I understand how the mistake was made, it seems relatively easy to slip by even when writing code without ChatGPT. But what I don't understand is how this wasn't caught after the first failure? Does this company not have any logging? Shouldn't the fact the backend is attempting to reuse UUIDs be immediately obvious from observing the error?

andrewxdiamond 2y ago
They didn’t even know there was an error until the customers came ringing. You always want to know what errors happened before your customers do, logging, alerting, any monitoring at all would have helped them here.
- theamk 2y ago
  They had 5 days... they knew after the first night
- lawgimenez 2y ago
  Experience should tell you to always take a hard look at anything UUIDs.
  - stouset 2y ago
    Also facepalms here: UUIDs as strings and UUIDv4. UUIDs are just 128-bit values. They might be conventionally encoded for humans as hex, but storing them as 36-byte (plus a few more for length) strings is a pointless waste of both space and performance.
    - Kesty 2y ago
      They don't have logs and commit directly to production 10/20 times a day. I don't think 128 bits vs 36 byte performance it's a main concern right now
      - bombela 2y ago
        It is most likely not a performance issues right now, but every pessismisation compound and catch up to you eventually. 36B vs 16B today, tomorrow you need an array of it, and now it isn't cache aligned, and more than twice the overhead. Most likely instead of manipulating a 36B fixed length string, it is handled as a dynamic string, for extra runtime memory allocations, most likely consuming at least 64B per allocation. Etc etc. Do this all over the codebase and now you know why all the moderne software is a sloth on what was a supercomputer 30y ago.
      - stouset 2y ago
        Probably not, but this is something that would have taken all of ten seconds to get right. And the size and performance impact is multiplied at every tier of their application. It’s also not just the size itself. Despite being fixed-size in practice, these are variable-sized strings in application code which now means gajillions of pointless allocations and indirection for everything. There are a ton of knock-on performance consequences here, all on the most heavily-used columns in your data model. Worst of all should they actually succeed, this is going to be absolutely excruciating to fix.
    - sgarland 2y ago
      IFF they’re using MySQL (doubtful), it’s a common mistake due to there not being a native type, and needing to know to use BINARY(16) and casting the string back and forth in code. But in either case – MySQL or Postgres – they’ve still made the classic mistake of using a UUID as a PK, which will tank performance one way or another. They’ll notice right around when it starts to matter.
drexlspivey 2y ago
Five days to figure out this bug is pretty crazy ngl
- fragmede 2y ago
  it wasn't 5 days of only working on this one problem though, it was over 5 calendar days. even if something is gonna take me one day to implement, it's gonna take three days to get enough focus time between all the other meetings and fires to put out in order to actually get a day's worth of coding done
whakim 2y ago
Yeah, I agree with this here. I think it's totally reasonable that something as specific as multiple Stripe subscriptions wouldn't be exercised by normal unit testing; as mentioned in the post, this wouldn't have been an easy error to reproduce via an acceptance test; and I think the focus on ChatGPT is overblown (by both the OP and everyone else) and mistakenly passing a String instead of a Callable to a function that accepts either happens all the time. My gut instinct is that not using an ORM would have prevented this particular issue, but that may just be my bias against ORMs speaking; one could easily imagine a similar bug occurring in a non-database context. My real conclusion is that all the folks crowing that they would have definitely caught this bug are either much better engineers than I am, or (more likely) are just a bit deluded about their own abilities. I am also very confused about the apparent lack of logging or recourse to logging. It's been a while, but if I recall correctly ECS should automatically propagate the resulting Duplicate Key exceptions which were presumably occurring to CloudWatch without a bunch of additional configuration - was that not happening? If it was happening, did no one think to go check what types of Exceptions were happening overnight?
islewis 2y ago
IMO this is the real issue. I guarantee you that they _will_ have another production bug like this sometime in the future (every fast paced project will). You'd hope this next one wont take 5 days to identify.
faizshah 2y ago
This sort of process failure is why there’s a Correction of Errors (postmortem) process at Amazon: https://aws.amazon.com/blogs/mt/why-you-should-develop-a-correction-of-error-coe/ https://aws.amazon.com/blogs/mt/why-you-should-develop-a-cor... Specifically asking why did it take so long to detect and why did it take so long to diagnose is useful in these situations.
- treflop 2y ago
  I’ve seen two types of people approach problems: Type 1 tries to find the error message and figure out what it really, really means by breaking down the error message and system. Type 2 does trial and error on random related things until the problem goes away. I hate to say that I've seen way more type 2s engineers than type 1, but maybe I’m working at the wrong companies.
  - faizshah 2y ago
    The way you want to build and lead an engineering team is so that the individual engineer’s approach to the problem doesn’t matter. The idea is to establish deployment safety and ops mechanisms so Type 2 isn’t even possible and your Sr. Engineers coach the jr. engineers in how to use these mechanisms effectively through training and runbooks. The idea of COE is to figure out what mechanisms (availability tests, metrics/alarms, logs, unit tests) were missing so that you can fill those gaps.
    - treflop 2y ago
      Naturally you want a team of really good people but the fact of the matter is not all companies have that kind of pull.
Kesty 2y ago
Everything can be understandable if this is a small first personal project of someone. Here we are talking 1.65 MILLION CAD $ backed YC company

andrewxdiamond 2y ago

They didn’t even know there was an error until the customers came ringing. You always want to know what errors happened before your customers do, logging, alerting, any monitoring at all would have helped them here.

theamk 2y ago
They had 5 days... they knew after the first night
lawgimenez 2y ago
Experience should tell you to always take a hard look at anything UUIDs.
- stouset 2y ago
  Also facepalms here: UUIDs as strings and UUIDv4. UUIDs are just 128-bit values. They might be conventionally encoded for humans as hex, but storing them as 36-byte (plus a few more for length) strings is a pointless waste of both space and performance.
  - Kesty 2y ago
    They don't have logs and commit directly to production 10/20 times a day. I don't think 128 bits vs 36 byte performance it's a main concern right now
    - bombela 2y ago
      It is most likely not a performance issues right now, but every pessismisation compound and catch up to you eventually. 36B vs 16B today, tomorrow you need an array of it, and now it isn't cache aligned, and more than twice the overhead. Most likely instead of manipulating a 36B fixed length string, it is handled as a dynamic string, for extra runtime memory allocations, most likely consuming at least 64B per allocation. Etc etc. Do this all over the codebase and now you know why all the moderne software is a sloth on what was a supercomputer 30y ago.
    - stouset 2y ago
      Probably not, but this is something that would have taken all of ten seconds to get right. And the size and performance impact is multiplied at every tier of their application. It’s also not just the size itself. Despite being fixed-size in practice, these are variable-sized strings in application code which now means gajillions of pointless allocations and indirection for everything. There are a ton of knock-on performance consequences here, all on the most heavily-used columns in your data model. Worst of all should they actually succeed, this is going to be absolutely excruciating to fix.
  - sgarland 2y ago
    IFF they’re using MySQL (doubtful), it’s a common mistake due to there not being a native type, and needing to know to use BINARY(16) and casting the string back and forth in code. But in either case – MySQL or Postgres – they’ve still made the classic mistake of using a UUID as a PK, which will tank performance one way or another. They’ll notice right around when it starts to matter.

drexlspivey 2y ago

Five days to figure out this bug is pretty crazy ngl

fragmede 2y ago
it wasn't 5 days of only working on this one problem though, it was over 5 calendar days. even if something is gonna take me one day to implement, it's gonna take three days to get enough focus time between all the other meetings and fires to put out in order to actually get a day's worth of coding done

whakim 2y ago

Yeah, I agree with this here. I think it's totally reasonable that something as specific as multiple Stripe subscriptions wouldn't be exercised by normal unit testing; as mentioned in the post, this wouldn't have been an easy error to reproduce via an acceptance test; and I think the focus on ChatGPT is overblown (by both the OP and everyone else) and mistakenly passing a String instead of a Callable to a function that accepts either happens all the time. My gut instinct is that not using an ORM would have prevented this particular issue, but that may just be my bias against ORMs speaking; one could easily imagine a similar bug occurring in a non-database context. My real conclusion is that all the folks crowing that they would have definitely caught this bug are either much better engineers than I am, or (more likely) are just a bit deluded about their own abilities. I am also very confused about the apparent lack of logging or recourse to logging. It's been a while, but if I recall correctly ECS should automatically propagate the resulting Duplicate Key exceptions which were presumably occurring to CloudWatch without a bunch of additional configuration - was that not happening? If it was happening, did no one think to go check what types of Exceptions were happening overnight?

islewis 2y ago

IMO this is the real issue. I guarantee you that they _will_ have another production bug like this sometime in the future (every fast paced project will). You'd hope this next one wont take 5 days to identify.

faizshah 2y ago

This sort of process failure is why there’s a Correction of Errors (postmortem) process at Amazon: https://aws.amazon.com/blogs/mt/why-you-should-develop-a-correction-of-error-coe/ https://aws.amazon.com/blogs/mt/why-you-should-develop-a-cor... Specifically asking why did it take so long to detect and why did it take so long to diagnose is useful in these situations.

treflop 2y ago
I’ve seen two types of people approach problems: Type 1 tries to find the error message and figure out what it really, really means by breaking down the error message and system. Type 2 does trial and error on random related things until the problem goes away. I hate to say that I've seen way more type 2s engineers than type 1, but maybe I’m working at the wrong companies.
- faizshah 2y ago
  The way you want to build and lead an engineering team is so that the individual engineer’s approach to the problem doesn’t matter. The idea is to establish deployment safety and ops mechanisms so Type 2 isn’t even possible and your Sr. Engineers coach the jr. engineers in how to use these mechanisms effectively through training and runbooks. The idea of COE is to figure out what mechanisms (availability tests, metrics/alarms, logs, unit tests) were missing so that you can fill those gaps.
  - treflop 2y ago
    Naturally you want a team of really good people but the fact of the matter is not all companies have that kind of pull.

Kesty 2y ago

Everything can be understandable if this is a small first personal project of someone. Here we are talking 1.65 MILLION CAD $ backed YC company