4 ms·

I'd rather they admit a mistake and learn a lesson from it even if it isn't a good thing to advertise. That said, I agree that you are identifying a more impor

I'd rather they admit a mistake and learn a lesson from it even if it isn't a good thing to advertise. That said, I agree that you are identifying a more important issue here but I also think you are being a bit too subtle about even if I agree with what you are saying. The real lesson that they should have learned from this ordeal is to never push code directly into production --- period. The article never mentions using a testbed or sandbox beforehand, and I kinda feel like they learned a good lesson but it may in fact be the wrong lesson to learn here.

theamk 2y ago
I don't see how testbed/sandbox would have helped, unless they'd also have a dedicated QA person _and_ configured their sandbox so have dramatically fewer instances. Because I can see "create a new subscription" in the manual test plan, but not "create 5x new subscription".
- owl57 2y ago
  > configured their sandbox so have dramatically fewer instances May be a wise thing to do anyway. I would even advocate for the extreme, one instance in sandbox. I've seen quite a variety of bugs that would be easier to detect and debug with one instance — from values accidentally persisted between requests (OP is one example of this very broad class of bugs), to a thundering herd bug catched in staging due to less instances and more worker threads per instance. But can't remember even one "distributed" bug caught in staging. > they'd also have a dedicated QA person This is a new feature, and it's user-facing, business-critical and inherently fragile (external service integration, though it broke for another reason). I hope multiple people manually run such things end to end before deploying to prod, even if none of them are dedicated QA?