5 ms·

https://www.sfgate.com/news/article/google-electrical-incident-injures-3-17360321.php https://www.sfgate.com/news/article/google-electrical-incide... Three peo

by breput 4y ago

https://www.sfgate.com/news/article/google-electrical-incident-injures-3-17360321.php https://www.sfgate.com/news/article/google-electrical-incide... Three people in critical condition after Google data center 'electrical incident' in the Council Bluffs, Iowa datacenter (us-central region, I believe)

utopcell 4y ago
not at all relevant to this outage
- breput 4y ago
  Do you have anything to add about the root cause?
  - utopcell 4y ago
    Very much doubt I am allowed to. We are racing to recover before EU reaches peak traffic.
dgrin91 4y ago
Seems unrelated. This happened ~10 hours ago.
- prescriptivist 4y ago
  10 hours ago we had a flood of network failures in us-central1 and saw no GCP status changes. We blindly attempted to mitigate in various ways (freezing HPAs because we thought that we were making excessive calls to external infra and getting throttled) and it resolved itself eventually. Maybe we were at fault the entire time but not seeing this issue exposed up on the GCP dashboard is infuriating.
  - zxcvbn4038 4y ago
    At AWS, if the status changes then someone somewhere gets fired, so a lot of time incidents happen without being recorded on the status board. Maybe it’s the same issue with GCP, or maybe concern for their injured peers made everyone forget to update the status. I really hope the later.
    - nomel 4y ago
      > At AWS, if the status changes then someone somewhere gets fired, so a lot of time incidents happen without being recorded on the status board. This can't be true, can it? What's the reason to lie, when the lie would be so incredibly obvious?
      - washbrain 4y ago
        It's not true. The real answer is that execs don't want to pay the costs of slo violations. If the checkmark status green, who could say whether the service was down?
    - Nihmie 4y ago
      It used to be that one of the best things about working at Google was the "blameless postmortem". As long as you are able to learn from an incident and weren't attempting to look at private data, then you could write up a postmortem document and actually use that as part of a promotion packet. Google would loose a key part of its soul if they were to change that.
      - tnova 4y ago
        AWS has a process like that as well, it’s called a COE. GP is either misinformed or making stuff up.
- lupire 4y ago
  Could be related if eng tried to route traffic away from Council Bluffs after the accident (shut down data center for safety inspection/repair?) and failed.
hoofhearted 4y ago
An arc blast isn’t anything to play with! It’s an electrically charged fireball that is 4 times hotter than the sun at the point of the arc. Pretty much instant 3rd degree burns, while simultaneously getting zapped and hit with an explosive shockwave. https://youtu.be/-iClXrd50Z8 https://youtu.be/-iClXrd50Z8 https://youtu.be/PO6see7_ODY https://youtu.be/PO6see7_ODY
- shadowgovt 4y ago
  The physics of those things is scary. If I recall correctly, the mitigating PPE for expecting an arc flash is reflective, because so much of the energy imparted on a victim is straight-up electromagnetic radiation. You get lightbulbed to death.
  - hoofhearted 4y ago
    Yeah an arc flash suit is rated in cal/cm². It gets to a limit where the potential energy from a high voltage piece of equipment is so great that they don’t make a suit for it because the pressure from the shockwave is enough to kill you. Pretty much any utility transformer has this much potential energy, but they design them to make it near impossible to cross phases.

by breput 4y ago

utopcell 4y ago

not at all relevant to this outage

breput 4y ago
Do you have anything to add about the root cause?
- utopcell 4y ago
  Very much doubt I am allowed to. We are racing to recover before EU reaches peak traffic.

dgrin91 4y ago

Seems unrelated. This happened ~10 hours ago.

prescriptivist 4y ago
10 hours ago we had a flood of network failures in us-central1 and saw no GCP status changes. We blindly attempted to mitigate in various ways (freezing HPAs because we thought that we were making excessive calls to external infra and getting throttled) and it resolved itself eventually. Maybe we were at fault the entire time but not seeing this issue exposed up on the GCP dashboard is infuriating.
- zxcvbn4038 4y ago
  At AWS, if the status changes then someone somewhere gets fired, so a lot of time incidents happen without being recorded on the status board. Maybe it’s the same issue with GCP, or maybe concern for their injured peers made everyone forget to update the status. I really hope the later.
  - nomel 4y ago
    > At AWS, if the status changes then someone somewhere gets fired, so a lot of time incidents happen without being recorded on the status board. This can't be true, can it? What's the reason to lie, when the lie would be so incredibly obvious?
    - washbrain 4y ago
      It's not true. The real answer is that execs don't want to pay the costs of slo violations. If the checkmark status green, who could say whether the service was down?
  - Nihmie 4y ago
    It used to be that one of the best things about working at Google was the "blameless postmortem". As long as you are able to learn from an incident and weren't attempting to look at private data, then you could write up a postmortem document and actually use that as part of a promotion packet. Google would loose a key part of its soul if they were to change that.
    - tnova 4y ago
      AWS has a process like that as well, it’s called a COE. GP is either misinformed or making stuff up.
lupire 4y ago
Could be related if eng tried to route traffic away from Council Bluffs after the accident (shut down data center for safety inspection/repair?) and failed.

hoofhearted 4y ago

An arc blast isn’t anything to play with! It’s an electrically charged fireball that is 4 times hotter than the sun at the point of the arc. Pretty much instant 3rd degree burns, while simultaneously getting zapped and hit with an explosive shockwave. https://youtu.be/-iClXrd50Z8 https://youtu.be/-iClXrd50Z8 https://youtu.be/PO6see7_ODY https://youtu.be/PO6see7_ODY

shadowgovt 4y ago
The physics of those things is scary. If I recall correctly, the mitigating PPE for expecting an arc flash is reflective, because so much of the energy imparted on a victim is straight-up electromagnetic radiation. You get lightbulbed to death.
- hoofhearted 4y ago
  Yeah an arc flash suit is rated in cal/cm². It gets to a limit where the potential energy from a high voltage piece of equipment is so great that they don’t make a suit for it because the pressure from the shockwave is enough to kill you. Pretty much any utility transformer has this much potential energy, but they design them to make it near impossible to cross phases.