7 ms·
I'm going to read all the conspiracy theories on this post. https://news.ycombinator.com/item?id=20345060 https://news.ycombinator.com/item?id=20345060 (Facebo
by envolt 7y ago
I'm going to read all the conspiracy theories on this post.
https://news.ycombinator.com/item?id=20345060 https://news.ycombinator.com/item?id=20345060
(Facebook, Instagram, and WhatsApp outages)
- deleted 7y ago[deleted]
- 3JPLW 7y agoYou know the old adage that celebrities die in threes? It's actually mathematically supported... or, well, it's supported that they die in 2.718s. Same principle would apply to cloud service outages if all the services and their failures were actually independent. We'd expect them to happen in "clusters" of e: http://ssp.impulsetrain.com/celebrities.html http://ssp.impulsetrain.com/celebrities.html I still love me a good conspiracy theory, but clustering of random (poisson) events is much more likely than you'd expect.
- emerongi 7y agoThat's calculated with the assumption that you can cluster the events within a 1-month period and that the events happen at a fixed rate. We are talking about a period of a few days here and about only a handful of services that tout 99.999(9)% uptime. I'm no mathematician but I don't think it's a great comparison.
- 3JPLW 7y agoNo the cluster size isn't fixed, nor is the event rate — the clusters are chosen purely based upon what an "unusually long" time between events would be. That's precisely how the rate ends up canceling itself out. Given this formulation, the expected cluster size is always e, regardless of how you define celebrity. That's what makes it fun. Now the human psychology part that this doesn't cover is that typically when you have two or three A-listers die, well, then you start seeing all the B- C- and D-listers that also died in the same period that you would have otherwise ignored.
- deleted 7y ago[deleted]
- emerongi 7y agoCan you calculate the chance of the cluster size being >5 when we can cluster events within a one-week period and the rate of events is one every 2 months (debatable, but feels right to me)? I think that would just end this discussion. I don't know how to calculate that, but my intuition says the resulting chance is low.
- appleiigs 7y agoI don't know how to do the math, but my python simulation says zero. import random service_providers = 20 servers_threshold = 5 up_time_threshold = 0.999**7 # prob down in week years = 20 occurrences = 0 # run sim 10,000 times for i in range(0, 10000): servers_down = False # simulate weeks in 20 years for j in range(1, 52*years): how_many_down_this_week = 0 # run up times for service providers for s in range(0, service_providers): up_time = random.random() if up_time > up_time_threshold: how_many_down_this_week += 1 # did they go down? if how_many_down_this_week > servers_threshold: servers_down = True if servers_down == True: occurrences += 1 print(occurrences)
- joshuamorton 7y agoGiven those parameters, you'd expect to see something every 100,000 years or so. (poisson distribution, lambda=1/8, k=5, the result is like 7e-7).
- desdiv 7y agoAWS averages about one major outage per year[0]. Google's own status page[1] list near 100 incidents in the past 365 days, only one of which ended on HN frontpage. [0] https://en.wikipedia.org/wiki/Timeline_of_Amazon_Web_Services#Amazon_Web_Services_Outages https://en.wikipedia.org/wiki/Timeline_of_Amazon_Web_Service... [1] https://status.cloud.google.com/summary https://status.cloud.google.com/summary
- emerongi 7y agoGood correction: the outages do happen at a fixed rate. Regarding GCP, I only see a couple events that have lasted more than 8 hours, not sure how many of those affect more than one subsystem.
- Buge 7y agoBoth of these Google outages made it to the HN frontpage: https://news.ycombinator.com/item?id=20077421 https://news.ycombinator.com/item?id=20077421 https://news.ycombinator.com/item?id=20338263 https://news.ycombinator.com/item?id=20338263 Both of these AWS outages made it to the HN frontpage (and neither are listed in that AWS timeline): https://news.ycombinator.com/item?id=20298719 https://news.ycombinator.com/item?id=20298719 https://news.ycombinator.com/item?id=19749781 https://news.ycombinator.com/item?id=19749781
- api 7y agoFacebook, Google, Apple, Cloudflare, Azure, Amazon, WhatsApp, and we have noticed smaller bits of routing weirdness like servers in Los Angeles not being able to hit GitHub for a few hours a few days ago. This is more like 6-8, not e. It's definitely odd. One non-conspiracy explanation I can imagine is that maybe all these big providers have a bunch of hidden dependencies on each other.
- 3JPLW 7y agoI agree, this is a bigger and really dense cluster, but remember that e is just the average (mean) cluster size. There will be clusters bigger than it. My point is simply that our intuition of randomness biases us to see meaning in clusters when there is none in the first place. In other words, we need to intentionally shift our prior on how unlikely this event is. Yes, it's still unlikely, but it's not as unlikely as you would think.
- inlined 7y agoApple documents their dependencies on other Cloud providers. Facebook has its own DCs but I think they’re sometimes colocated with cloud providers’.
- the-dude 7y agoBut the outages are sequential, not simultaneous
- api 7y agoThey are very dense compared to the usual rate.
- dmix 7y agoPlus one big outage story then makes other minor outages get more headlines... when normally no one cares. Same thing with hurricanes, when a bad one happens in the US they are more likely to document other minor hurricanes in the Caribbean and Latin America. But otherwise plenty of hurricanes hit those areas and it doesn't make major headlines.
- stefco_ 7y ago“When it rains, it pours” is the conventional wisdom distillation of Poisson variable behavior
- philshem 7y ago2.718 is the Euler number https://en.wikipedia.org/wiki/E_(mathematical_constant) https://en.wikipedia.org/wiki/E_(mathematical_constant)
- the-dude 7y agoDude, I wrote that top comment. And I was like iCloud WTF?
- ksec 7y agoOnce is happenstance. Twice is coincidence. The third time it’s enemy action. I don't even know if fourth of Fifth is confirmation.
- cantbecool 7y agoMaybe some submarines are cutting some deep sea cables? In actuality it's probably massive backbone upgrades that are classified, the NSA has to obtain real-time data from somewhere.
- snazz 7y agoWhy would the NSA be doing such low-level (literal wiretapping) work anymore? Useful traffic is almost always encrypted.
- 3JPLW 7y agoClearly this means they've broken public-key cryptography.
- z3t4 7y agoThey log who talk to who
- 2Ccltvcm 7y agoThey have a #ailored #ccess #perations department. Look it up.
- AlexCoventry 7y agoYou sure are playing it safe there.
- earenndil 7y agoFlaws could be discovered in modern communication such that they can decrypt it in 30 years time. How good is 30-year-old crypto today?
- JohnBooty 7y agoKnowing whom is communicating with whom is pretty useful to know, even if you can't decrypt the data.
- helloworldies 7y ago
- the-dude 7y agoActually there are not many actual conspiracies listed there. I will start : - All the companies in the list are getting rid of their Huawei dependencies, replacing routers by domestic ones. - China is flexing its muscles in the trade war : look, we are creating some instability in your e-commerce. Wait until we attack your infra ( electricity, water )
- wybiral 7y agoIf you go down the route of conspiracy theory you have to also believe that the companies who have already issued a PR response are lying and that all of the employees privy to the truth are also staying silent. That would be quite a story.
- the-dude 7y agoThe iPhone and the atomic bomb were kept secret quite well.
- wybiral 7y agoOne company keeping a specific product launch secret is different from three or four independent companies and all of their network engineers all conspiring.
- the-dude 7y agoSure, but the Manhattan project was a huge undertaking with multiple extremely large facilities. What if the NSA identified a critical backdoor and ordered all major companies to decommission and not talk about it? ( NSLs )
- wybiral 7y agoFair enough... But then they all just did a really crappy job of concealing the patch?