4 ms·

Apart from the opening paragraphs, there is zero information what actually went wrong here. I’m super curious how customers ended up seeing other people’s acco

by tnolet 7y ago

Apart from the opening paragraphs, there is zero information what actually went wrong here. I’m super curious how customers ended up seeing other people’s accounts. That seems like a massive major flaw in logic somewhere. Sadly, the article goes into “general history and commentary”-mode way too quick.

wolfgang42 7y ago
> I’m super curious how customers ended up seeing other people’s accounts. I don’t know anything about this particular case, but the usual cause of this is over-zealous caching (usually by an intermediate load balancer or the like) that returns cached pages while ignoring the session data. The result is that one person logs in and gets their account info, and then everyone else who tries to view the same page gets that person’s info instead of their own. For example, this happened to Steam in 2015: https://arstechnica.com/gaming/2015/12/valve-explains-ddos-induced-caching-problem-led-to-xmas-day-steam-data-leaks-and-downtime/ https://arstechnica.com/gaming/2015/12/valve-explains-ddos-i... Of course, there are also other possibilities, but this is the most likely. Alternatives that come to mind (though I’ve never seen any of these actually happen): - A bad RNG creating the same session token for multiple users - Concurrency bug in the web server returning results to the wrong connection - Messed-up migration causing account IDs to be associated with the wrong credentials (maybe IDs from the old and new systems got mixed up)
- bonzini 7y ago
  > Messed-up migration causing account IDs to be associated with the wrong credentials (maybe IDs from the old and new systems got mixed up) I would put money on this one.
heisenbit 7y ago
Read the long report by Slaughter and May (see top HN comment) with 250 pages of details, it is quite insightful and more than enough gory details. New architecture, new platform, implementation takes too long, migration and testing insufficiently planned and when push comes to shove even the too low non functional testing goals not reached but still gone live. With big bang approaches one has to keep in mind that complexity grows non-linear and at a certain size it becomes very hard. This migration was at a size that was getting hard but the approach taken and the way it was executed was clearly (see report) not on the level required. Big bang on smaller scale is the fastest and cheapest. Big bang on larger scale starts having big risks. A more incremental approach may have been better and cost effective with much lower risks. A honest cost comparison may have shown that - provided the big bang approach would have been properly planed with sufficient large scale integration, migration and proofing milestones and buffers for risks inherent in this one-shot approach.
- james_s_tayler 7y ago
  The problem with the big bang rewrite is it violates Gall's law >All complex systems that work evolved from simpler systems that worked
darkerside 7y ago
Agreed. The root cause bring lack of regression testing is laughable. Sure, testing would have and should have caught this, but what exactly was "this"?

by tnolet 7y ago

wolfgang42 7y ago

> I’m super curious how customers ended up seeing other people’s accounts. I don’t know anything about this particular case, but the usual cause of this is over-zealous caching (usually by an intermediate load balancer or the like) that returns cached pages while ignoring the session data. The result is that one person logs in and gets their account info, and then everyone else who tries to view the same page gets that person’s info instead of their own. For example, this happened to Steam in 2015: https://arstechnica.com/gaming/2015/12/valve-explains-ddos-induced-caching-problem-led-to-xmas-day-steam-data-leaks-and-downtime/ https://arstechnica.com/gaming/2015/12/valve-explains-ddos-i... Of course, there are also other possibilities, but this is the most likely. Alternatives that come to mind (though I’ve never seen any of these actually happen): - A bad RNG creating the same session token for multiple users - Concurrency bug in the web server returning results to the wrong connection - Messed-up migration causing account IDs to be associated with the wrong credentials (maybe IDs from the old and new systems got mixed up)

bonzini 7y ago
> Messed-up migration causing account IDs to be associated with the wrong credentials (maybe IDs from the old and new systems got mixed up) I would put money on this one.

heisenbit 7y ago

Read the long report by Slaughter and May (see top HN comment) with 250 pages of details, it is quite insightful and more than enough gory details. New architecture, new platform, implementation takes too long, migration and testing insufficiently planned and when push comes to shove even the too low non functional testing goals not reached but still gone live. With big bang approaches one has to keep in mind that complexity grows non-linear and at a certain size it becomes very hard. This migration was at a size that was getting hard but the approach taken and the way it was executed was clearly (see report) not on the level required. Big bang on smaller scale is the fastest and cheapest. Big bang on larger scale starts having big risks. A more incremental approach may have been better and cost effective with much lower risks. A honest cost comparison may have shown that - provided the big bang approach would have been properly planed with sufficient large scale integration, migration and proofing milestones and buffers for risks inherent in this one-shot approach.

james_s_tayler 7y ago
The problem with the big bang rewrite is it violates Gall's law >All complex systems that work evolved from simpler systems that worked

darkerside 7y ago

Agreed. The root cause bring lack of regression testing is laughable. Sure, testing would have and should have caught this, but what exactly was "this"?