23 ms·
Ok, this is too many high-profile, apparently unrelated outages in the last month to be completely a coincidence. Hypotheses: 1) software complexity is escala
by rossdavidh 7y ago
Ok, this is too many high-profile, apparently unrelated outages in the last month to be completely a coincidence. Hypotheses:
1) software complexity is escalating over time, and logically will continue to until something makes it stop. It has now reached the point where even large companies cannot maintain high reliability.
2) internet volume is continually increasing over time, and periodically we hit a point where there are just too many pieces required to make it work (until some change the infrastructure solves that). We had such a point when dialup was no longer enough, and we solved that with fiber. Now we have a chokepoint somewhere else in the system, and it will require a different infrastructure change
3) Russia or China or Iran or somebody is f*(#ing with us, to see what they are able to break if they needed to, if they need to apply leverage to, for example, get sanctions lifted
4) Just a series of unconnected errors at big companies
5) Other possibilities?
- Qwertystop 7y agoRelating to 1: https://www.youtube.com/watch?v=pW-SOdj4Kkk https://www.youtube.com/watch?v=pW-SOdj4Kkk (Jonathan Blow's "Preventing the Collapse of Civilization"... perhaps a melodramatic title, but well-said overall.)
- marenkay 7y agoOr we just managed to construct the biggest circular dependency ever using the whole internet and a combination of all hyped languages and frameworks. That would in turn lead to an insanely fragile system with increasing amounts of failures that seem unexplainable/weird.
- frogpelt 7y ago5) Some of all of the above? Although 3) doesn't have to be the explanation, it is definitely happening all the time.
- idlewords 7y agoWrite a script to fire random events and you will notice they sometimes cluster in ways that look like a pattern.
- rossdavidh 7y agoOh, sure. But Apple, Google, Cloudflare, Stripe, Slack, Microsoft, we're getting to more than five even...
- mattwad 7y agoReddit went down this morning too
- opencl 7y agoAnd now Discord is down!
- gsich 7y agono loss
- root_axis 7y agoThe logic of the GP still applies though. Sites have outages every day so it is inevitable that some large sites will fail around the same time. Also, we know that Cloudflare and Twitter outages were attributed to configuration changes, probably others have benign explanations as well.
- 7y ago
- MrStonedOne 7y agoOperational consistency creates a hidden single point of failure. If everybody is doing the same things and setting things up the same way to ensure reliability then any failures or short comings in that system are shared by all.
- outworlder 7y agoBrains are excellent pattern matchers. Brains also suck at statistics.
- jimmaswell 7y agoThis is the first time I can remember so much happening so close. It's statistically unlikely.
- deleted 7y ago[deleted]
- djtriptych 7y agoI've been suspecting 3) for a few months now, and I'm quite curious how our government would handle it if it _were_ the case. Only a few of these outages have had plausible post-mortems ever made public.
- deleted 7y ago[deleted]
- depr 7y agoso like all software has reached peak complexity this month?
- rossdavidh 7y agoIt seems a bit of a coincidence, yes? Unless they are all copying each other (e.g. all using Kubernetes or what-have-you), in which case it might be less of a coincidence.
- elamje 7y agoThe brain is the greatest pattern matcher in the world. While it is unlikely all of these companies would have major outages in a month, be wary that the subconscious is constantly generating narratives to explain statistical anomalies. Interesting theories nonetheless:)
- dmix 7y agoThe more the conspiracy grows the faster these otherwise minor stories shoot to the top of HN and add to the pattern. It fuels itself.
- alanbernstein 7y ago"Minor" seems inappropriate. Can you remember another time when so many high-profile websites/services have had outages in so short a time span?
- ambrice 7y agoNo. And a year from now I won't remember this time either.
- whatshisface 7y agoSaying that it's an illusory pattern without checking the statistics is no more scientific than saying it's a conspiracy without checking the statistics.
- braythwayt 7y ago> be wary that the subconscious is constantly generating narratives to explain statistical anomalies This comes up all the time in sports. Let's take pool for example. There are various guesstimates floating around, and I do not have access to detailed tournament statistics, but I have heard that in games where sinking a ball on the break is an advantage, for decent players there's maybe a 75% chance that a ball will go down. So once in every four breaks, you won't sink a ball. How often do you fail twice in a row? Once in every sixteen breaks. Failing three times in a row? Once in every 64 breaks. Four times in a row? Once in every 256 breaks. What about five straight breaks without sinking a ball? Once in every 1,024 breaks. That's a lot of breaks. But wait up a moment. Let's ask, "If you miss a break, what're the odds of it becoming a streak of five misses in a row?" The answer is, "One in every 256 streaks of misses will be a streak of five or more misses." 1/256 is not particularly rare, if you play often enough to sink a ball on the break 75% of the time. What is the point of knowing that a streak of five misses in a row is rare but not that rare? Well, if you miss five in a row, do you chalk your cue for break number six as usual? Or do you tell yourself that your break isn't working, and start adjusting your stance, aim, spin, &c? If you start adjusting everything when you get a streak of five misses in a row, you may just make things worse. You have to pay enough attention to your distribution of misses to work out whether a streak of five misses in a row is just the normal 1/256 streaks, or if there really is something amiss. The brain is a great pattern matcher, but it sucks at understanding statistics. --- The flip side of this, of course, is that if you upgrade your brain well enough to understand statistics, you can win a lot of money. If a pro misses five in a row, feel free to wager money that they'll sink a ball on their next break. Your friends may actually give you odds, even though the expectation of winning is 75-25 in your favour.
- matwood 7y agoReddit was also partially down this morning.
- AnIdiotOnTheNet 7y agoIt's #1. The real question is how this isn't blindingly obvious to everyone.
- koonsolo 7y agoBecause we can't all be as smart as you are.
- rossdavidh 7y agoOne possible answer: it's hard to admit that what you've worked really hard at becoming an expert in, might have been a mistake.
- JaRail 7y agoFirst, I think our general uptime metrics are trending upwards. Recovery times tend to be much shorter as well. Big services are bigger, more mission-critical parts can fail. Continuous development culture is designed with failure as part of the process. We don't spend time looking for obscure issues when they'll be easier to find by looking at metrics. This is fine when a staggered deployment can catch an issue with a small number of users. It's bad when that staggered deployment creates a side-effect that isn't fixed by rolling it back. Much harder to fix corrupted metadata, etc. Automated systems can propagate/cascade/snowball mistakes far more quickly than having to manually apply changes. We notice errors more now. Mistakes are instantly news.
- hn_throwaway_99 7y ago> We notice errors more now. Mistakes are instantly news. Heck, just look at Twitter itself from its original "Fail Whale" days where there was so much downtime, to now where even this relatively small amount of downtime is the top story on HN for hours.
- dillonmckay 7y agoSo, when it went down, was there a Fail Whale displayed during this most recent incident?
- Algol 7y agoI think they retired the fail whale some time ago. I looked it up: in 2013, because they didn't want to be associated w/ outages.
- deleted 7y ago[deleted]
- narrator 7y ago6) White House social media conference just started. https://www.10tv.com/article/trump-hosts-white-house-summit-social-media-2019-jul https://www.10tv.com/article/trump-hosts-white-house-summit-...
- moret1979 7y agoOne possibility on 5) Too many KPIs and quarter goals to be reached, too many corners cut.
- bastijn 7y agoObligatory to watch with this comment: "Let's deploy to production" https://youtu.be/5p8wTOr8AbU https://youtu.be/5p8wTOr8AbU
- kirubakaran 7y agoYou know, I've watched a few of those memes over the past, but this one was especially well done, and timed perfectly with his gestures even!
- sampleinajar 7y agoNo idea how I haven't seen this, but it totally made my day.
- jsjohnst 7y agoThe only possible way for me to make it more than 20-30 seconds into that was to mute it. That guy’s laugh is multiple orders of magnitude worse than nails on a chalkboard. Funny story (albeit too real), but man, mute before clicking everyone.
- winrid 7y agoThis hit close to home. Hilarious. Thanks.
- godarni 7y agoLots of people on vacation this time of year. Would be interesting to see if there is a seasonal component to the reliability of these services.
- syn0byte 7y ago"Don't forget to occasionally do that thing I mentioned in passing 2 weeks ago, under my breath, during a klaxon alarm test. Otherwise the errors will DDoS the whole cluster. See you in a week, goodluck!" Nah - that would never happen.
- papito 7y ago#1. I think the art of keeping things simple is being lost. These days people will mush together ten different cloud services and 5,000 dependencies just for a Hello World.
- listic 7y agoI (don't) like how you exclude Russia, China, Iran and somebody from your definition of 'us'.
- rossdavidh 7y agoThe assumption is that Russia, China, Iran are less dependent on Google, Twitter, etc., in part because some of them aren't allowed to operate in those countries, in part because some of them are much less dominant in those markets. 'Us' means 'people who might care that Twitter (or whoever) is down'.
- vkou 7y agoGoogle, Twitter, Reddit, Facebook, etc, all legally operate in Russia.
- bin0 7y agoBut most have regional replacements. WeChat in China, VK (and some Telegram, though it's now blocked?) in Russia. This makes them less reliant on the American originals, which is why governments often encourage home-grown knock-offs.
- Parcle 7y agoHis definition of "us" seems to just be "Americans". Which is fine because he's talking about American companies...
- deleted 7y ago[deleted]
- bromuro 7y agoYes I have been also hit by the same bad feeling. Thanks for pointing it out.
- deleted 7y ago[deleted]
- jonprobably 7y agoIt's summer time and everyone who knows how stuff works is halfway through a drink right now. Probably with their families. Is it a trend year over year for 7/4 +/- a week?
- julienfr112 7y agoWhat about raising temperatures ?
- t0astbread 7y agoOr maybe it's because the internet is tendentially becoming just a few companies' data centers? Afaik Twitter moved to GCP a few months ago. Maybe this is another Google outage?
- azurezyq 7y agoLess likely since it looks fine from GCP status page. Hmm, it seems that Twitter already figured it out, configuration change issues again.
- marenkay 7y agoProbably because we all use Kubernetes and YAML files and 100% of configuration failures are "oh shit, I used two spaces instead of 4, we're fucked".
- Jach 7y agoSomething like this is my bet too, there was a recent post somewhere called something like "why all outages are due to a configuration change". There are monocultures in site reliability ops for big companies, "configuration over code" but with heavy automation too. From my outside view it seems there's a tradeoff when you do that between more frequent smaller issues and less frequent bigger issues. Also reminds me of Google's move away from eventual consistency because with their infrastructure they can make a CP system highly available in practice... except when it isn't, due to a botched configuration change.
- ionforce 7y ago> tendentially Is this a word? You don't mean tangentially? I'm having a crisis right now.
- t0astbread 7y agoIs it not? Sorry if I got it wrong, English isn't my first language. dict.cc (my source of truth for English vocab) says it's a word: https://www.dict.cc/?s=tendentially https://www.dict.cc/?s=tendentially
- mcqueenjordan 7y agohttps://en.wikipedia.org/wiki/Poisson_distribution https://en.wikipedia.org/wiki/Poisson_distribution
- Hamuko 7y ago>It has now reached the point where even large companies cannot maintain high reliability. Waiting for this to be backed up by statistics.
- akhilcacharya 7y agoThe other possibility is intern season (I'm 99.99% joking)
- rossdavidh 7y agoI'm 99.99% laughing (and 0.01% thinking 'uh oh').
- kgraves 7y agoAs much as I don't like interns, I am sure that they wouldn't even touch a system of the scale like Twitter's in my opinion. /s
- sbmassey 7y agoTaking down Twitter could be a huge boon for the economy, though.
- rossdavidh 7y agoProductivity skyrockets!
- bdd 7y ago#4 I work at Facebook. I worked at Twitter. I worked at CloudFlare. The answer is nothing other than #4. #1 has the right premise but the wrong conclusion. Software complexity will continue escalating until it drops by either commoditization or redefining problems. Companies at the scale of FAANG(+T) continually accumulate tech debt in pockets and they eventually become the biggest threats to availability. Not the new shiny things. The sinusoidal pattern of exposure will continue.
- marenkay 7y agoOnly one thing to add: Tech debt is accrued in amounts where every VC fund would get wet pants if tech debt was worth dollars paid out.
- kwizzt 7y agoHow does the fact you worked at those companies relate to #4? Edit: I misread the parent and my question doesn't make a lot of sense. Please ignore it :)
- bdd 7y ago> How does the fact you worked at those companies relate to #4? For Facebook I worked on the incident, previous Wednesday. 9.5 hours of pain... And for my past employers, I still have friends there texting the root causes with facepalm emojis.
- liberte82 7y agoDo tell
- deleted 7y ago[deleted]
- deleted 7y ago[deleted]
- deleted 7y ago[deleted]
- NightlyDev 7y agoSoftware is getting increasingly complex. Why? To ensure better uptime, amongst other things. The funny part is that all the complexity often leads to downtime. A single server would usually have less downtime than Google, Facebook and so on. But Google and Facebook needs this complexity to handle the amount of traffic they're getting. Makes me wonder why people are trying to do stuff like Google when they're not Google. Keeping it simple is the best solution.
- dqpb 7y ago3) Come on man, you can't just go around opening parentheses and then not closing them.
- dv_dt 7y agoAnother possibility: US (or other) authorities are requiring some sort of monitoring software or hardware that where disruption of service is unavoidable during install
- snazz 7y agoKeeping that many mouths shut seems impossible.
- dv_dt 7y agoMost people won't be directly involved in assessing or fixing the fault. "Sorry the network link went down, here is the after analysis report," seems like a reasonable cover. There are many espionage activities which are covered up, only to come out decades later. But really, I don't have any evidence that this possibility is any more or less likely than any other.
- calimac 7y agoThere is a silent civil war going on to free the users from malicious social technocracy
- pennaMan 7y ago>July 11, 2019 7:56PM UTC[Identified] The outage was due to an internal configuration change, which we're now fixing. Some people may be able to access Twitter again and we're working to make sure Twitter is available to everyone as quickly as possible. Seems #4 is the winner
- gniv 7y agoOr #1. I work on critical infrastructure at FAANG and it's frightening how complex things are. The folks who built the systems and knew them inside-out have mostly moved on, and the newbies, like me, don't fully understand how the abstractions leak, what changes cause what side effects etc.
- DaveInTucson 7y ago> Just a series of unconnected errors at big companies Except that "at big companies" is basically selection bias, problems at little companies don't get noticed because they're, well, small companies. And the underlying issue of the "unconnected errors" is that software is rather like the airline industry: things don't really get fixed until there's a sufficiently ugly crash.
- humanfactor 7y ago1/2) These are web apps. Big web apps but web apps none the less. We know what can go wrong theres nothing really new here. How would you quantify "too many pieces to make work". Is 1000 too many , 10000 ???? There are millions pieces of data on your harddrive and they work fine. In general the idea of variance can be solved with redundancy. Maybe there are not enough backups at twitter. 5/4) Incompetent people lead by incompetent people surrounded by yes men and a drug culture. Also having a company that demonizes conservatives which are some of the best engineers (scientist are squares naturally) Human error is bound to happen and software is complex but so are rockets and supply chains. Things can go right and things can go wrong. Usually when they do go wrong there is a human error reason. Does twitter foster a place where human error can occur more frequently that other places? I dont know. I have my bias about the company and any sjw company but thats very anecdotal. Twitter worked yesterday and it doesnt work today. Doesnt really have to mean anything really important except for the fact that there is a blind spot in their process which they need to harden. I guess the first person to ask is the dev op , then the developer. Something wasnt tested enough. That happens in commercial software, deadlines cant wait. 3)Russia / China / Iran ... stop watching CNN. You are parroting talking points. If twitter is crushed America could care less and would probably turn up sanctions, not lift them. Taking down twitter wont cripple anything in America except for certain marketers budgets.
- papito 7y agoScientists are squares but they also have a brain. That's why they are usually not conservatives. Conservatives are not a party, it's a herd of paranoid people who tune into Fox News every night to be told what to be afraid of next, but it's definitely not engineers or scientists.
- sebastianconcpt 7y agoI'd normally go for #4 but hypotheses #3 is starting to be a more plausible explanation for the timely "coincidence".
- rossdavidh 7y agoA friend of mine who is retired military told me there is a saying that "once is bad luck, twice is a coincidence, but three times is enemy action". Doesn't necessarily mean it's true, of course.
- bArray 7y agoFor point #3, there are a few irregularities: 1. Services all going down one after another. 1 goes down - it happens. 2 go down - it happens sometimes. 3 go down - quite a rare sequence of events. But now a large number of silicon valley companies have experienced service outage over the last few weeks. 2. Russian sub that is said to be a "deep sea research vessel" somehow experiences a fire whilst in international waters [1]. It has been suspected that it could have been tapping undersea cables. Let's imagine for a moment a scenario where they were caught in the act, some NATO sub decides to put an end to it and Russia cover it up to save face. 3. Russia announces tests to ensure that it could survive if completely cut off from the internet [2]. A few months later it's like somebody is probing US services in the same way. 4. There is currently a large NATO exercise in a simulated take-over of Russia happening in Countries close to Russia [3]. Of course it's completely possible it's all unconnected, but my tin foil hat brain says there is a game of cloak and daggers going on here. I would say that Russia's incentive for probing the US/NATO is to test it's weakness after undergoing a trade-war with China and raising sanctions against Iran. After all, Russian fighter planes regularly try to fly into UK airspace just to test their rapid response crews [4], this sort of behaviour is typical of them. [1] https://en.wikipedia.org/wiki/Russian_submarine_Losharik https://en.wikipedia.org/wiki/Russian_submarine_Losharik [2] https://techcrunch.com/2019/02/11/russia-internet-turn-off-digital-economy-national-program/ https://techcrunch.com/2019/02/11/russia-internet-turn-off-d... [3] https://sofiaglobe.com/2019/05/13/6000-military-personnel-to-take-part-in-nato-exercise-in-bulgaria-in-may-june-2019/ https://sofiaglobe.com/2019/05/13/6000-military-personnel-to... [4] https://www.theguardian.com/world/2018/jan/15/raf-fighters-intercept-two-russian-bombers-near-uk-airspace https://www.theguardian.com/world/2018/jan/15/raf-fighters-i...
- asark 7y agoThis is a pattern one might see if there were a secret, rolling disclosure of some exceptionally-bad software vulnerability, I'd think. Or same of some kind of serious but limited malware infection across devices of a certain class that sees some use at any major tech company. If you also didn't want to clue anyone else (any other governments) in that you'd found something (in either case), you might fix the problem this way. Though at that point it might be easier to just manufacture some really bad "routing issue" and have everyone fix it at once, under cover of the network problem.
- lovecg 7y agoIt’s #4 but caused by #1. My pet theory is that we’re pretty far into this business cycle, so a lot of new companies had the time to mature, build up complexity, shed people the most knowledgeable with the original architecture, stop caring as much about the competition, and so on. Add Apple to the mix for recent software quality issues.
- taf2 7y ago6) It's summer and lots of engineers are either a. on vacation or b. thinking less clearly
- ljm 7y agoHypergrowth/blitzscaling also introduces entropy. The more you hire, the more plentiful and diverse your bugs will be. It stands out now because the stars aligned. But theses issues have been coming and going for years in patternless form.
- Bluecobra 7y ago4. I think people are too accustomed now to high availability/uptime nowadays. I started using the Internet in the mid 90s. Stuff used to break all the time back in those days. Now I can’t remember the last time I couldn’t reach a website because it has been Slashdotted.
- Angostura 7y ago5) The increasing interconnectedness of things introducing new interdependences so that when one service stumbles so do many others.
- auslander 7y agoSysadmin and DevOps engineer walk into a bar ...
- danellis 7y ago#4. When things are random, they cluster.
- ducktypegoose 7y agoSoftware complexity escalating over time? Please! The new microservices architecture we have been migrating to over the last year or so is so stable and makes tracking down problems a walk in the park. Not to mention the NOSQL database is a dream come true, as long as you don't need to query anything other than the partition key.
- djohnston 7y agoit's end of half. everyone is entering reviews. gotta deliver.. somerthing...
- jayd16 7y agoMy guess is its a slow news time of year coupled with more usage of cloud services which means these types of stories are higher profile.
- johngalt 7y ago5) Operational reliability is both difficult and unsexy. The fancy new feature, increasing traffic, or adding AI to something will generate headlines, accolades, and positive attention. Not having outages is something everyone expects by default. This goes double for work that prevents outages. No one wins awards for what doesn't happen. How many medals are pinned on the guys installing fire sprinklers?
- meristem 7y agoCorollary: Work that prevents outages--or safe work--is SO unsexy it does not get noticed, but work that causes outages is postmortem-ed to death (pun intended).
- rossdavidh 7y agoOk, I have one to add myself: 6) We used to have many small outages at different websites. Now, with so many things that once were separate small sites aggregated on sites like FB, Twitter, Reddit, etc we have a few large sites, so we have aggregated the failures along with that. The failure rate, by this theory, is the same, but we have replaced "many small failures" with "periodic wide-spread failures, big enough to make headlines". Turning many small problems into a few bigger ones. Just another hypothesis.
- daveheq 7y agoI don't believe it's too complex, I believe people are getting lazy. Complexity can be handled by automation, but too often people just want to rush things out for a buck instead of planning out a good product.
- pard68 7y agoNSA firmware updates requiring a reboot.
- foobiekr 7y ago4. And imho all that’s really happening is people are noticing the outages more. This is a good thing. For years too much of the mental model has been “{ cloud, google, Facebook, aws, xxx } never goes down!” That’s been unhealthy. It’s a good thing.
- willart4food 7y agoall of the above!
- root_axis 7y agoThere's also the HN filter bubble which could be presenting a misleading picture of "outage" frequency.
- dboreham 7y agoCould be a Tacoma Narrows Bridge type problem.
- pu8h 7y agoMaybe aliens invasion.
- chrismarlow9 7y agoEverything is made of plastic these days, even software. It's immediately put out as soon as an MVP is ready. Too many managers with zero coding experience. The marketing people have taken the browser. Time to start over.