3 ms·

Sounds like this wasn't caused by power surges reaching the equipment but rather an effect of repeated power loss to drive arrays not fully designed to handle i

by miander 11y ago

Sounds like this wasn't caused by power surges reaching the equipment but rather an effect of repeated power loss to drive arrays not fully designed to handle it. The article is pretty unclear. Still sounds like an infrastructure problem though.

magicalist 11y ago
The incident report ismavis posted below (and linked in the article) has far more information: https://status.cloud.google.com/incident/compute/15056#5719570367119360 https://status.cloud.google.com/incident/compute/15056#57195...
- deleted 11y ago
  [deleted]
  - darkr 11y ago
    AFAIK google doesn't use hardware-based arrays in their servers. I think the reference to batteries would more likely be in reference to a DRUPS (Diesel Rotary UPS) which most datacenters run, sometimes with some form of battery in combination with the flywheel. Typically the combination of kinetic energy in the flywheel (and potentially batteries) only hold enough power to last for 30 seconds or so (often as low as 10 seconds), which gives the diesel power generators enough time to come online and take over from there. What I guess is that might have happened is that grid power was lost, they switched over to UPS fine the first time, grid power came back so they swapped back, repeat a few times and on one of those times the batteries didn't have enough charge to keep things going for the generator swap-over.
    - thrownaway2424 11y ago
      Nope. http://www.buildablade.com/images/Google_data_centers-1.jpg http://www.buildablade.com/images/Google_data_centers-1.jpg
    - dekhn 11y ago
      From http://www.cnet.com/news/google-uncloaks-once-secret-server-10209580/ http://www.cnet.com/news/google-uncloaks-once-secret-server-... "Google's big surprise: each server has its own 12-volt battery to supply power if there's a problem with the main source of electricity"
    - walshemj 11y ago
      A so UPS on the cheap a real UPS is always running of the battery Telco style and you are testing your gensets at least weakly and checking the tanks every month or so. At BT they where tested every day for the really sensitive systems.
  - tw04 11y ago
    A proper storage array refuses to boot if the batteries are not sufficient to de-stage cache in a timely fashion, or at the very least keep cache available for the advertised (generally 72 hours) time window if it doesn't have the ability to de-stage the cache to a more permanent medium.

by miander 11y ago

magicalist 11y ago

The incident report ismavis posted below (and linked in the article) has far more information: https://status.cloud.google.com/incident/compute/15056#5719570367119360 https://status.cloud.google.com/incident/compute/15056#57195...

deleted 11y ago
[deleted]
- darkr 11y ago
  AFAIK google doesn't use hardware-based arrays in their servers. I think the reference to batteries would more likely be in reference to a DRUPS (Diesel Rotary UPS) which most datacenters run, sometimes with some form of battery in combination with the flywheel. Typically the combination of kinetic energy in the flywheel (and potentially batteries) only hold enough power to last for 30 seconds or so (often as low as 10 seconds), which gives the diesel power generators enough time to come online and take over from there. What I guess is that might have happened is that grid power was lost, they switched over to UPS fine the first time, grid power came back so they swapped back, repeat a few times and on one of those times the batteries didn't have enough charge to keep things going for the generator swap-over.
  - thrownaway2424 11y ago
    Nope. http://www.buildablade.com/images/Google_data_centers-1.jpg http://www.buildablade.com/images/Google_data_centers-1.jpg
  - dekhn 11y ago
    From http://www.cnet.com/news/google-uncloaks-once-secret-server-10209580/ http://www.cnet.com/news/google-uncloaks-once-secret-server-... "Google's big surprise: each server has its own 12-volt battery to supply power if there's a problem with the main source of electricity"
  - walshemj 11y ago
    A so UPS on the cheap a real UPS is always running of the battery Telco style and you are testing your gensets at least weakly and checking the tanks every month or so. At BT they where tested every day for the really sensitive systems.
- tw04 11y ago
  A proper storage array refuses to boot if the batteries are not sufficient to de-stage cache in a timely fashion, or at the very least keep cache available for the advertised (generally 72 hours) time window if it doesn't have the ability to de-stage the cache to a more permanent medium.