4 ms·
This seems to significantly under-report what's going on, see: https://www.theregister.com/2023/04/26/google_cloud_outage/ https://www.theregister.com/2023/04/
by jacquesm 4y ago
This seems to significantly under-report what's going on, see:
https://www.theregister.com/2023/04/26/google_cloud_outage/ https://www.theregister.com/2023/04/26/google_cloud_outage/
There is mention of a fire as well.
- jonatron 4y agoThis doesn't sound as bad as OVH's 2021 fire.
- nik736 4y agoWell, we had pictures very quickly of the OVH fire. Google seems to be not very transparent on what is exactly happening...
- stingraycharles 4y agoThe linked article says that there was a leak in a water cooling system, which in turn ended up in the battery system which caused a fire. But yeah it’s not coming from Google but second hand reports.
- mike_hearn 4y agoIt's not second hand, it's from the colo provider they're using in Paris.
- sschueller 4y agoIf you trench a fire in water in a DC it might be just as bad.
- jacquesm 4y agoI wouldn't draw any conclusions just yet.
- DebtDeflation 4y agoPlot twist: the server racks were made out of sodium.
- ironmagma 4y agoNaCl, the revolutionary Sodium Cloud technology.
- H8crilA 4y agoYou're not far off: the batteries are (probably) made of lithium. Also, why batteries in a datacenter? When you implement a flush() command at the lowest level you're faced with two choices: 1) actually write to disk, then return from the call, 2) write to some cache/RAM and have just enough battery locally to ensure that you can write it to disk even if all power goes out. Then there's the other problem of surviving long enough between a power interruption and diesel generators starting up. But this is a smaller problem, rebooting all instances in a datacenter is less bad than losing some data that was correctly flush()ed by software. Bad flush() behaviour can result in errors that cannot be recovered from without a complicated manual intervention (for example if it causes corrupted and unreadable database files).
- walrus01 4y ago> Also, why batteries in a datacenter? Everything serious in the telecom/ISP infrastructure sector has a big -48VDC battery plant, or preferably separate A and B side -48VDC battery plants, to provide a significant buffer between power going Grid --> AC-to-DC Rectifiers --> Equipment, and when a generator can start up, warm up, and transfer switch does its job. Even if a bunch of servers don't have any UPS or battery backup because they're designed to tolerate individual node (or whole rack, or whole row failures) the core network equipment in a datacenter will still have a huge battery plant. Ideally if you have a chilled water loop for cooling you do not want it anywhere near your big-ass racks of batteries. Or near the racks that contain the rectifiers and DC breakers, distribution bus bars. If you look at the battery racks in a traditional telco CO in the US for instance you will see that all of the cabling and batteries are a minimum of 1 foot off the floor, so that the whole place could theoretically flood and the DC distribution would remain unaffected. Same principle that applies to very traditional setups with wet-cell 2V lead acid batteries also applies to more modern things if building from scratch.
- pcurve 4y agoThe outage has been going on for 40+ hours now... I think this is sort of big.
- madaxe_again 4y agoOh, the irony. A few years ago I implemented a top to bottom ISO27k1 ISMS for a client handling extremely sensitive and mission-critical data for industry. One risk I recommended controls for was that of a fire and/or flood at their primary datacentre for their client-facing offerings - this datacentre. I’ve experienced the misery of a datacentre oops myself, firsthand, twice, and it’s a genuine risk that has to be mitigated. At my insistence, I had them burn hundreds of man-hours ensuring that they could failover to a new environment in a different datacentre with a bare minimum of fuss, as what I arrived to was an all the eggs in one basket situation. It took a fair bit of re-engineering of how deployments worked, how data was replicated, how the environment was configured - but they got there, and the ISMS was put into operation, and was audited cleanly by a reputable auditor, and everyone lived happily ever after. Except… they were acquired by private equity. Who had no truck with all of this costly prancing about with consultants and systems. Risk register? Why do we need this? What value does it add today? ISO27k1? Don’t be silly. We have that certificate. You don’t need it. Dev team, ops team, leadership — almost everyone — ejected and replaced with a few support staff. I see their sites are down.
- jacquesm 4y agoThere's that beautiful German word again... schadenfreude. I have had similar discussions multiple times in the last year and the magic thinking around the cloud is so strong that it is sometimes impossible to get through. The fact that cloud stuff can go down and that in the end it is your data and no amount of cloud credits are going to help you if your data is lost seems to be utterly beyond some people's comprehension.
- illumin8 4y agoThis really has nothing to do with cloud and is more of an "all eggs in one basket" problem. I wish people would stop painting cloud itself as less capable. The fact is, most cloud providers offer multiple regions, which have the capability of giving you more geographic redundancy than most companies that operate in their own datacenters have. Whether you choose to adopt a multi-region or multi-datacenter architecture is really orthogonal to whether you choose cloud or on-prem.