6 ms·
I really hope that's not true. Ooph. "Twitter data centers were fragile, and Twitter lacked plans and processes to “cold boot.” That meant that if all the cente
by bushbaba 4y ago
I really hope that's not true. Ooph.
"Twitter data centers were fragile, and Twitter lacked plans and processes to “cold boot.” That meant that if all the centers went offline simultaneously, even briefly, Twitter was unsure if they could bring the service back up. Downtime estimates ranged from weeks of round-the-clock work, to permanent irreparable failure"
- chickin 4y agoHow many large tech companies do you think can do that? Each service wants to benefit from the use of other services. Ensuring there aren't dependency loops in this complex graph sounds tricky.
- danielmarkbruce 4y agoGoogle wasn't 100% certain they could cold boot around 2015. Not sure if it's still true. It's crazy difficult to figure out if you can at some point.
- yuhong 4y agoPart of the point of SSD based search engines is they are much easier to cold boot as well.
- ryan_lane 4y agoYou have to have enough people to "be that asshole" that tells people in their design docs that they are creating a dependency loop and need to design things to avoid it. I did that for quite a while at Lyft, and I'm hoping that people continued doing that after I stopped. My guess, though, is that I and others have missed things and that even somewhere that was careful to avoid it, there's still some places that could make cold boots difficult. The best way to avoid this problem is to have a secondary datacenter that is hot, and to never let both fail. Of course, this is what twitter is doing. Designing a DR plan where you can lose multiple primary/secondary datacenters is hard enough that basically no one does it. This is something that really doesn't belong in the whistleblower complaints because it wasn't under his responsibilities, twitter wasn't lying to anyone about having properly implemented DR plans, and twitter's DR plan isn't out of the ordinary across the industry.
- dehrmann 4y agoI suspect a lot of services are in the same state. As things get large, systems become more complex and this is a lot harder to test.
- buildbot 4y agoIf that happens, can the rest of the fucking internet cold boot?
- na85 4y agoI mean all you really need is DNS. If the root servers come back up I don't see why the rest couldn't.
- danielmarkbruce 4y agohalf joking - wouldn't surprise me if many DNS services have a dependency on s3...