7 ms·

Could this be related to the storm? I was out shoveling, and came back in to my phone blowing up. Our systems at IronMountain (formerly Fortrust) in Denver al

Could this be related to the storm? I was out shoveling, and came back in to my phone blowing up. Our systems at IronMountain (formerly Fortrust) in Denver all rebooted at once. These are all on redundant power, each systems redundant power supplies connecting to different circuits entering the cabinet, and those two circuits fed from 3 PDUs (two separate, one share). Each of those is supposed to be fed by a separate UPS and generator. Last status update I had says that they are running off generators, but they've been shockingly tight-lipped about it. Don't get me wrong, it was hi-LAR-ious to call into their NOC and have them pretend that I was the only one having problems. "Can you tell me if there is a major data center outage going on?" "We are trying to gather information, we are making a bunch of client phone calls, we will know after we make those calls." "... Why are you making a bunch of client calls if you aren't having an outage?"

taurath 8y ago
“Outage, what outage?” Is a sort of laughable response but all too common with tons of providers.
- bradhe 8y ago
  I'm looking at you, AWS...
  - ceejayoz 8y ago
    A nuclear war could take out an AWS region and we'd get an "increased error rates" informational message.
    - bluetwo 8y ago
      Are you saying that a cold-war-era system like the internet/arpanet meant to survive a nuclear war might be vulnerable to an attack if we take all the code and data and store it in the same place? :-)
      - kerng 8y ago
        Cloud is not much different then Mainframe Computing 2.0. I'm already curious and have been pondering about this for quite a while to understand what/how/when we will see a disruption of this.
        deusofnull 8y ago
        Limewire is gonna make a come back
        yellowapple 8y ago
        Once upon a time Multics' developers predicted that someday computing power would be treated like a public utility that homes and businesses would buy like electricity or water or natural gas. Given the proliferation of minute/hourly billing among service providers, it looks like the Multics folks guessed right. It just happened on top of Unix(-like systems) instead of Multics. I wonder how long it'll be before we start seeing municipal datacenters?
      - bradhe 8y ago
        No he's saying that AWS sucks at updating their status page.
  - WrtCdEvrydy 8y ago
    Remember when S3 failed and the S3 green arrow was being cached on S3. I remember... unfortunately.
- rad_gruchalski 8y ago
  Reminds me when I was contacting Deutsche Telekom last year regarding an outage in Monschau area. "We have no problems". In fact, the whole exchange was down, press got a sniff of it when people could not contact emergency services anymore: https://www.aachener-nachrichten.de/lokales/eifel/netzstoerung-in-den-drei-eifelgemeinden-behoben_aid-24418103 https://www.aachener-nachrichten.de/lokales/eifel/netzstoeru...
- ct520 8y ago
  Man that reminds me of that century link cluster f
rubicon33 8y ago
Classic response for any kind of service provider: Deny, deny, deny, obfuscate, deny, then blame someone else (usually, YOU).
komali2 8y ago
Do they do this to get around and 99.9% uptime agreements?
- Implicated 8y ago
  Probably a combination of that and to curtail the "I just spoke to Brad in Customer Service who confirmed _the whole datacenter if offline_" type posts. But that's my presumption, I don't actually know anything and don't want to imply I do.
- SilverSurfer972 8y ago
  That's the trust issue with current agreements we are solving. If an API is down the bound agreement is enforced instantly with our platform, no lies, no call, no pain. We are actually onboarding companies to try it out! https://stacktical.com https://stacktical.com
  - ezekg 8y ago
    Why does this need a blockchain?
    - SilverSurfer972 8y ago
      TLDR: Because Smart Contracts on the blockchain are the right tool for Secure Digital Agreements. Paperweight contracts are irrelevant in a world of data * A Smart Contract is cheaper to publish that the stack of paper handled by lawyers. * Code is cheap to iterate from whereas traditional SLA are expensive/slow to renegociate. Over time, SLAs drive behaviors that are focused on delivering a minimum level of service at minimum cost to the provider. * A Smart Contract is a code you can trust, understand and expect to behave instantly compared to the traditional SLM. * A Smart Contract is immutable. More about the benefit of Smart Contract over paper/digital agreements: https://www.forbes.com/sites/cognitiveworld/2019/03/10/rise-of-the-smart-contract/ https://www.forbes.com/sites/cognitiveworld/2019/03/10/rise-... Why not a python script? I don't trust the guy that handle that script. Blockchain is very good at bringing trust. I hope I answered your question [edit] formatting
unethical_ban 8y ago
> "We are trying to gather information, we are making a bunch of client phone calls, we will know after we make those calls." I think that is a yes, and he getting ahead by saying "Yes and we have no idea why or ETA so let us do our job". Granted, they should have a status page.
- opportune 8y ago
  Last time I dealt with a cloud provider outage the status page was unresponsive during the outage because the status page had some kind of dependency on the resources that were down...
  - lozaning 8y ago
    I remember that, the status page for AWS stored the red icon on s3 which was the service that was down.
    - opportune 8y ago
      Actually in this case it was Azure related but it's pretty funny that the same thing has happened to the two biggest cloud platforms
- rhizome 8y ago
  I wouldn't be surprised if they think a status page would open up liability for not putting it up soon enough, or for too long, or for some text that turned out to be wrong or unnecessary.
- linsomniac 8y ago
  Sure, I understand the "so let us do our job". I've been on the other side of that. On the other hand, I need information to be able to do my job: Is this only our cabinet having problems and I need to start rolling to the datacenter (in the middle of a giant blizzard)? Is this possibly some sort of problem with our own power infrastructure? Is something on fire (an EPO triggered by fire could cause this)? Did the roof cave in under the weight of the snow we are getting? Is the power stabilized or is there some indication that power might be up and down? In short, I need answers to: Do I need to gracefully take down my site to prevent lost transactions and database corruption? Do I need to switch to our backup site? For context: All of our servers powering off at once and then back on shouldn't be possible. It should require the failure of at least 3 independent pieces of equipment (except at the breaker panel or in our cabinet where it could be only two failures). It is extremely unusual for this to happen, first time it's happened for me and I've been in that facility since 2004. So, yes, I respect that you need to do your job. But I also need to do my job. Plus, I'm pretty sure the guy answering the trouble line, his job WAS talking with the customers. The people working the problem likely didn't include him. This is a huge data center run by a ginormous company. I don't think I was taking him away from twisting a wrench. :-)
jyriand 8y ago
So, a storm in Denver stops me from using Messengner in Estonia? I wonder where the butterfly flapped its wings.
- ljm 8y ago
  Pretty sure it doesn't apply to Facebook, but Amazon's cheapest AWS tiers are around there. Same with Virginia.
  - mcbain 8y ago
    But AWS doesn’t have any regions near Denver.
  - joncrane 8y ago
    Wrong, AWS has no regions near Denver.
- iamgopal 8y ago
  https://en.m.wikipedia.org/wiki/Butterfly_effect https://en.m.wikipedia.org/wiki/Butterfly_effect
rconti 8y ago
"The storm"? It's sunny in the Bay Area for the first time in I don't know how long. I imagine it's nice in other parts of the world as well, other than where this localized "storm" is.
- kornish 8y ago
  Denver is getting slammed right now — power surges everywhere.
- mises 8y ago
  You missed the bomb cyclone that's happening right now in half the country?
  - rconti 8y ago
    Yes. Believe it or not, it's not really major news for those not impacted. We have the local evening news on most nights in the background, haven't heard a thing about it. I also regularly read NYT and WaPo and follow the Internets.
    - mises 8y ago
      It's sad that such outlets don't bother to print about stuff that impacts middle America. They could regain some credibility with a lot of said middle Americans without ever changing their political alignment just by giving them a little more coverage.
- ninju 8y ago
  Colorado weather: Bomb cyclone brings wild winds, big impact as blizzard whips state https://www.denverpost.com/2019/03/13/colorado-weather-bomb-cyclone-blizzard/ https://www.denverpost.com/2019/03/13/colorado-weather-bomb-...
jonstokes 8y ago
My first reaction to "could this be related to the storm," was "oh no, now this QAnon stuff has spread to HN."
erobbins 8y ago
No, FB datacenters are geographically diverse. They do run quarterly 'storms' where a datacenter is shut down to test failover and resiliency. I have no idea if today is one of those days, since I left last year.
- johannes1234321 8y ago
  Theoretically a real shutdown might go in a different way than previous tests or simulations. For instance in a test you might cut the connection completely, while in the real case only some power circuits go down or whatever. For instance GitHub's relatively recent shutdown was due to a fail-over heartbeat not going as expected.
- chipperyman573 8y ago
  Interesting. Out of curiosity, how hard is it to turn the datacenter "back on" in case they discover there's a problem with the failover?
  - londons_explore 8y ago
    Typically that could be done with one undrain command and take about 30 seconds.
    - AceyMan 8y ago
      Holy Batcave, I'm a baby SRE compared to that <bows solemnly>. Mind. Blown¹. /Acey ¹—I don't use FB [true] but I hear it's huge :-P
      - londons_explore 8y ago
        it's only so quick because stuff isn't actually turned off with disks wiped. The machines are still running, with applications loaded, just with no traffic directed towards them.
- bertil 8y ago
jorblumesea 8y ago
A company as large and sophisticated as FB has data centers and cloud services in multiple countries, and in the US, probably colocated data centers. Certainly nothing localized to where you are.
- TallGuyShort 8y ago
  If the outage is at all infrastructure-related, the root cause was something that at some point was local and cascaded. Unless someone git pushed to a repo used by both companies and it's taken all day to get it git revert'd, their redundancy obviously didn't work, did it? There's effectively a category-2 hurricane moving from the Rockies through the mid-west right now.
ct520 8y ago
Awesome we use iron mountain for escrow

taurath 8y ago

“Outage, what outage?” Is a sort of laughable response but all too common with tons of providers.

bradhe 8y ago
I'm looking at you, AWS...
- ceejayoz 8y ago
  A nuclear war could take out an AWS region and we'd get an "increased error rates" informational message.
  - bluetwo 8y ago
    Are you saying that a cold-war-era system like the internet/arpanet meant to survive a nuclear war might be vulnerable to an attack if we take all the code and data and store it in the same place? :-)
    - kerng 8y ago
      Cloud is not much different then Mainframe Computing 2.0. I'm already curious and have been pondering about this for quite a while to understand what/how/when we will see a disruption of this.
      - deusofnull 8y ago
        Limewire is gonna make a come back
      - yellowapple 8y ago
        Once upon a time Multics' developers predicted that someday computing power would be treated like a public utility that homes and businesses would buy like electricity or water or natural gas. Given the proliferation of minute/hourly billing among service providers, it looks like the Multics folks guessed right. It just happened on top of Unix(-like systems) instead of Multics. I wonder how long it'll be before we start seeing municipal datacenters?
    - bradhe 8y ago
      No he's saying that AWS sucks at updating their status page.
- WrtCdEvrydy 8y ago
  Remember when S3 failed and the S3 green arrow was being cached on S3. I remember... unfortunately.
rad_gruchalski 8y ago
Reminds me when I was contacting Deutsche Telekom last year regarding an outage in Monschau area. "We have no problems". In fact, the whole exchange was down, press got a sniff of it when people could not contact emergency services anymore: https://www.aachener-nachrichten.de/lokales/eifel/netzstoerung-in-den-drei-eifelgemeinden-behoben_aid-24418103 https://www.aachener-nachrichten.de/lokales/eifel/netzstoeru...
ct520 8y ago
Man that reminds me of that century link cluster f

rubicon33 8y ago

Classic response for any kind of service provider: Deny, deny, deny, obfuscate, deny, then blame someone else (usually, YOU).

komali2 8y ago

Do they do this to get around and 99.9% uptime agreements?

Implicated 8y ago
Probably a combination of that and to curtail the "I just spoke to Brad in Customer Service who confirmed _the whole datacenter if offline_" type posts. But that's my presumption, I don't actually know anything and don't want to imply I do.
SilverSurfer972 8y ago
That's the trust issue with current agreements we are solving. If an API is down the bound agreement is enforced instantly with our platform, no lies, no call, no pain. We are actually onboarding companies to try it out! https://stacktical.com https://stacktical.com
- ezekg 8y ago
  Why does this need a blockchain?
  - SilverSurfer972 8y ago
    TLDR: Because Smart Contracts on the blockchain are the right tool for Secure Digital Agreements. Paperweight contracts are irrelevant in a world of data * A Smart Contract is cheaper to publish that the stack of paper handled by lawyers. * Code is cheap to iterate from whereas traditional SLA are expensive/slow to renegociate. Over time, SLAs drive behaviors that are focused on delivering a minimum level of service at minimum cost to the provider. * A Smart Contract is a code you can trust, understand and expect to behave instantly compared to the traditional SLM. * A Smart Contract is immutable. More about the benefit of Smart Contract over paper/digital agreements: https://www.forbes.com/sites/cognitiveworld/2019/03/10/rise-of-the-smart-contract/ https://www.forbes.com/sites/cognitiveworld/2019/03/10/rise-... Why not a python script? I don't trust the guy that handle that script. Blockchain is very good at bringing trust. I hope I answered your question [edit] formatting

unethical_ban 8y ago

> "We are trying to gather information, we are making a bunch of client phone calls, we will know after we make those calls." I think that is a yes, and he getting ahead by saying "Yes and we have no idea why or ETA so let us do our job". Granted, they should have a status page.

opportune 8y ago
Last time I dealt with a cloud provider outage the status page was unresponsive during the outage because the status page had some kind of dependency on the resources that were down...
- lozaning 8y ago
  I remember that, the status page for AWS stored the red icon on s3 which was the service that was down.
  - opportune 8y ago
    Actually in this case it was Azure related but it's pretty funny that the same thing has happened to the two biggest cloud platforms
rhizome 8y ago
I wouldn't be surprised if they think a status page would open up liability for not putting it up soon enough, or for too long, or for some text that turned out to be wrong or unnecessary.
linsomniac 8y ago
Sure, I understand the "so let us do our job". I've been on the other side of that. On the other hand, I need information to be able to do my job: Is this only our cabinet having problems and I need to start rolling to the datacenter (in the middle of a giant blizzard)? Is this possibly some sort of problem with our own power infrastructure? Is something on fire (an EPO triggered by fire could cause this)? Did the roof cave in under the weight of the snow we are getting? Is the power stabilized or is there some indication that power might be up and down? In short, I need answers to: Do I need to gracefully take down my site to prevent lost transactions and database corruption? Do I need to switch to our backup site? For context: All of our servers powering off at once and then back on shouldn't be possible. It should require the failure of at least 3 independent pieces of equipment (except at the breaker panel or in our cabinet where it could be only two failures). It is extremely unusual for this to happen, first time it's happened for me and I've been in that facility since 2004. So, yes, I respect that you need to do your job. But I also need to do my job. Plus, I'm pretty sure the guy answering the trouble line, his job WAS talking with the customers. The people working the problem likely didn't include him. This is a huge data center run by a ginormous company. I don't think I was taking him away from twisting a wrench. :-)

jyriand 8y ago

So, a storm in Denver stops me from using Messengner in Estonia? I wonder where the butterfly flapped its wings.

ljm 8y ago
Pretty sure it doesn't apply to Facebook, but Amazon's cheapest AWS tiers are around there. Same with Virginia.
- mcbain 8y ago
  But AWS doesn’t have any regions near Denver.
- joncrane 8y ago
  Wrong, AWS has no regions near Denver.
iamgopal 8y ago
https://en.m.wikipedia.org/wiki/Butterfly_effect https://en.m.wikipedia.org/wiki/Butterfly_effect

rconti 8y ago

"The storm"? It's sunny in the Bay Area for the first time in I don't know how long. I imagine it's nice in other parts of the world as well, other than where this localized "storm" is.

kornish 8y ago
Denver is getting slammed right now — power surges everywhere.
mises 8y ago
You missed the bomb cyclone that's happening right now in half the country?
- rconti 8y ago
  Yes. Believe it or not, it's not really major news for those not impacted. We have the local evening news on most nights in the background, haven't heard a thing about it. I also regularly read NYT and WaPo and follow the Internets.
  - mises 8y ago
    It's sad that such outlets don't bother to print about stuff that impacts middle America. They could regain some credibility with a lot of said middle Americans without ever changing their political alignment just by giving them a little more coverage.
ninju 8y ago
Colorado weather: Bomb cyclone brings wild winds, big impact as blizzard whips state https://www.denverpost.com/2019/03/13/colorado-weather-bomb-cyclone-blizzard/ https://www.denverpost.com/2019/03/13/colorado-weather-bomb-...

jonstokes 8y ago

My first reaction to "could this be related to the storm," was "oh no, now this QAnon stuff has spread to HN."

erobbins 8y ago

No, FB datacenters are geographically diverse. They do run quarterly 'storms' where a datacenter is shut down to test failover and resiliency. I have no idea if today is one of those days, since I left last year.

johannes1234321 8y ago
Theoretically a real shutdown might go in a different way than previous tests or simulations. For instance in a test you might cut the connection completely, while in the real case only some power circuits go down or whatever. For instance GitHub's relatively recent shutdown was due to a fail-over heartbeat not going as expected.
chipperyman573 8y ago
Interesting. Out of curiosity, how hard is it to turn the datacenter "back on" in case they discover there's a problem with the failover?
- londons_explore 8y ago
  Typically that could be done with one undrain command and take about 30 seconds.
  - AceyMan 8y ago
    Holy Batcave, I'm a baby SRE compared to that <bows solemnly>. Mind. Blown¹. /Acey ¹—I don't use FB [true] but I hear it's huge :-P
    - londons_explore 8y ago
      it's only so quick because stuff isn't actually turned off with disks wiped. The machines are still running, with applications loaded, just with no traffic directed towards them.
bertil 8y ago

jorblumesea 8y ago

A company as large and sophisticated as FB has data centers and cloud services in multiple countries, and in the US, probably colocated data centers. Certainly nothing localized to where you are.

TallGuyShort 8y ago
If the outage is at all infrastructure-related, the root cause was something that at some point was local and cascaded. Unless someone git pushed to a repo used by both companies and it's taken all day to get it git revert'd, their redundancy obviously didn't work, did it? There's effectively a category-2 hurricane moving from the Rockies through the mid-west right now.

ct520 8y ago

Awesome we use iron mountain for escrow