3 ms·

We're affected by this issue. And we had our alerts system in Azure as well, so we didn't get alerts about the outage (welp).

pcunite 8y ago
Thanks for sharing. A big outage like this does have the benefit of making things better in the future.
- craftyguy 8y ago
  [citation needed]
  - smudgymcscmudge 8y ago
    Presumably some people will be setting up nagios servers outside of azure now. So that makes things better.
sieabahlpark 8y ago
And thus the lesson of all eggs in one basket has been taught again.
- smudgymcscmudge 8y ago
  Some thought they were spreading eggs across multiple baskets by replicating across regions and trusting Azure was doing the same for their “global” services. As a consumer, the lesson here is that Azure is one, big basket. It would probably be prudent to think of AWS and GCS as single baskets too.
haroldp 8y ago
You made the same mistake as Microsoft, who seemed to have put their Azure status page on Azure: https://www.reddit.com/r/AZURE/comments/9cvgn2/is_there_a_status_page_for_the_status_page/ https://www.reddit.com/r/AZURE/comments/9cvgn2/is_there_a_st... Their status page is back up now, but my stuff's still broken. :\
- lozaning 8y ago
  And Amazon. I recall that during their most recent s3 outage Amazon's status page was green across the board, because somehow all the assets that were supposed to be displayed when things went wrong were themselves hosted on the thing that was down.
  - mrep 8y ago
    Yes, changing the status page required updating a S3 bucket that was located in us-east-1 which is where it went down so they couldn't change it.
    - ti_ranger 8y ago
      And now, S3 has cross-region replication ... And so does DynamoDB, and Aurora ...
ZainRiz 8y ago
That's why T-Mobile's on-call engineers carry around AT&T phones. (source: friend who's an engineer at T-Mobile)
- Theodores 8y ago
  That is a top 'did you know' factoid that I am sure I will tell others. But do AT+T engineers carry T-Mobile phones? If yes then they should put themselves together a deal so that none of the on-call engineers have to worry about running up big bills using their phones. When there are freak weather events they are all in it together.
  - badbug 8y ago
    They would use Verizon if anything
  - i_have_to_speak 8y ago
    "factoid": I don't think it means what you think it means. https://www.merriam-webster.com/dictionary/factoid https://www.merriam-webster.com/dictionary/factoid
    - fphhotchips 8y ago
      The second definition there fits perfectly.
- diskie 8y ago
  I remember that very well (worked for NSN), it wasn't funny. They learned that lesson from massive (unexpected) outage of their mobile network (2009), without possibility to reach technical persons: https://www.lightreading.com/ethernet-ip/server-glitch-crashes-t-mobile-network/d/d-id/667149 https://www.lightreading.com/ethernet-ip/server-glitch-crash...

pcunite 8y ago

Thanks for sharing. A big outage like this does have the benefit of making things better in the future.

craftyguy 8y ago
[citation needed]
- smudgymcscmudge 8y ago
  Presumably some people will be setting up nagios servers outside of azure now. So that makes things better.

sieabahlpark 8y ago

And thus the lesson of all eggs in one basket has been taught again.

smudgymcscmudge 8y ago
Some thought they were spreading eggs across multiple baskets by replicating across regions and trusting Azure was doing the same for their “global” services. As a consumer, the lesson here is that Azure is one, big basket. It would probably be prudent to think of AWS and GCS as single baskets too.

haroldp 8y ago

You made the same mistake as Microsoft, who seemed to have put their Azure status page on Azure: https://www.reddit.com/r/AZURE/comments/9cvgn2/is_there_a_status_page_for_the_status_page/ https://www.reddit.com/r/AZURE/comments/9cvgn2/is_there_a_st... Their status page is back up now, but my stuff's still broken. :\

lozaning 8y ago
And Amazon. I recall that during their most recent s3 outage Amazon's status page was green across the board, because somehow all the assets that were supposed to be displayed when things went wrong were themselves hosted on the thing that was down.
- mrep 8y ago
  Yes, changing the status page required updating a S3 bucket that was located in us-east-1 which is where it went down so they couldn't change it.
  - ti_ranger 8y ago
    And now, S3 has cross-region replication ... And so does DynamoDB, and Aurora ...

ZainRiz 8y ago

That's why T-Mobile's on-call engineers carry around AT&T phones. (source: friend who's an engineer at T-Mobile)

Theodores 8y ago
That is a top 'did you know' factoid that I am sure I will tell others. But do AT+T engineers carry T-Mobile phones? If yes then they should put themselves together a deal so that none of the on-call engineers have to worry about running up big bills using their phones. When there are freak weather events they are all in it together.
- badbug 8y ago
  They would use Verizon if anything
- i_have_to_speak 8y ago
  "factoid": I don't think it means what you think it means. https://www.merriam-webster.com/dictionary/factoid https://www.merriam-webster.com/dictionary/factoid
  - fphhotchips 8y ago
    The second definition there fits perfectly.
diskie 8y ago
I remember that very well (worked for NSN), it wasn't funny. They learned that lesson from massive (unexpected) outage of their mobile network (2009), without possibility to reach technical persons: https://www.lightreading.com/ethernet-ip/server-glitch-crashes-t-mobile-network/d/d-id/667149 https://www.lightreading.com/ethernet-ip/server-glitch-crash...