6 ms·

Looks like this took down the national emergency alert system in Canada. I'm registered as an alert LMD (last mile distributor), and Pelmorex (corporation runni

by smitop 6y ago

Looks like this took down the national emergency alert system in Canada. I'm registered as an alert LMD (last mile distributor), and Pelmorex (corporation running the system) just emailed me to say "Please note that currently there is an unexpected significant outage on Microsoft Azure that is affecting the availability of the NAADS system and other clients globally. The NAAD System feeds are currently not accessible. We are following up on this and we will update you as soon as the issue is resolved.". I sure hope there aren't any emergencies in Canada until this is resolved...

PretzelPirate 6y ago
Why would Canada deploy this to a single provider? A national alert system should have a better DR plan than that.
- gnabgib 6y ago
  Big cloud systems, the ones we all regularly use, get much closer scrutiny than government systems normally (because we all know when Azure, GCP, AWS, Linode, DO, CloudFlare or OVH have a major outage, misconfiguration, or A/C fire). As a "single" provider I'm sure Microsoft is much better at this than the (likely shoestring budget) government group of admins was before they moved. There's more burst capacity (if needed) without paying for redundant equipment most of the year, and ultimately it's probably at a lower cost. Meanwhile Azure is the only cloud with two Canada regions (Canada "Central", Canada East)[0], AWS has one (Canada "Central")[1], GCP has one (Montreal same city as AWS)[2].. there's really only one player if you want to use managed cloud services. [0]: https://azure.microsoft.com/en-us/global-infrastructure/geographies/ https://azure.microsoft.com/en-us/global-infrastructure/geog... [1]: https://aws.amazon.com/about-aws/global-infrastructure/regions_az/ https://aws.amazon.com/about-aws/global-infrastructure/regio... [2]: https://cloud.google.com/about/locations/ https://cloud.google.com/about/locations/
  - jkaplowitz 6y ago
    I think the point was that a national emergency system should redundantly use two cloud providers, not that they made the wrong choice of single cloud provider.
    - gnabgib 6y ago
      The national emergency system doesn't rely on one provider[0] or one cloud, while Pelmorex Corp (The Weather Network) is part of the chain (and a curious one) it isn't the entire system, nor is their choice of provider/hardware a government choice (but it doesn't seem poor). The network isn't down, an aspect may have endured an outage. For significant (but not total) coverage of the country the "Alert Distributors" could be covered with contacting 15 corporations (huzzah anti competitive Canada)... or less if you pick a specific channel (eg Wireless=4), which I imagine is part of alerting protocol (your cable, mobile, radio, TV, and web applications don't all send the same alert) [0]: https://www.publicsafety.gc.ca/cnt/mrgnc-mngmnt/mrgnc-prprdnss/ntnl-pblc-lrtng-sstm-en.aspx https://www.publicsafety.gc.ca/cnt/mrgnc-mngmnt/mrgnc-prprdn...
  - wbl 6y ago
    It's an emergency alerting system. Low bandwidth, broadcast. Perfect for blasting a few kilowatts of RF via digipeters each with their own generators and lead acid battery banks.
fartcannon 6y ago
And here it is, the main problem with outsourcing critical infrastructure. Remote server providers like Microsoft Azure should at most be used as twins/redundant systems to a locally managed system. Governments of the world: pay your IT people more money to prevent brain drain.
- aynsof 6y ago
  Do you think locally managed systems are immune to outages? Or that governments are capable of resourcing their teams sufficiently to do a better job at availability than Microsoft, Google, or Amazon?
  - walrus01 6y ago
    Your comment is a fine example of the standard rhetoric from the 'move everything to the cloud' marketing people, but perhaps organizations should consider that they don't have to go with one cloud provider as a single point of failure. It's the lazy way of abstracting away responsibility and blame to some other party, and in my experience often does not result in better availability than a properly implemented "belt and suspenders" approach.
    - deleted 6y ago
      [deleted]
    - mattzito 6y ago
      There’s a complexity cost, though, right? It’s not free to run two different systems, manage two different billing systems, different tools, pay for cross-provider bandwidth, etc.? I’m sure there are some cases where the cost is worth the complexity, but I don’t think it’s cut and dried as written here. Most cloud providers are very reliable, and my guess is that you are more likely to have a self-inflicted outage due to the complexity of your infrastructure than the cloud provider having an outage.
    - emodendroket 6y ago
      Or it’s borne of experience of doing it oneself being much less reliable than cloud providers even accounting for these rare outages.
smitop 6y ago
Just got another email from them, apparently "the issue appears to have been resolved with Microsoft Azure". Looks like the problem is resolved
electro_blah 6y ago
lol they must have paid someone a shitty salary for someone to come up with a such a crappy system.

by smitop 6y ago

PretzelPirate 6y ago

Why would Canada deploy this to a single provider? A national alert system should have a better DR plan than that.

gnabgib 6y ago
Big cloud systems, the ones we all regularly use, get much closer scrutiny than government systems normally (because we all know when Azure, GCP, AWS, Linode, DO, CloudFlare or OVH have a major outage, misconfiguration, or A/C fire). As a "single" provider I'm sure Microsoft is much better at this than the (likely shoestring budget) government group of admins was before they moved. There's more burst capacity (if needed) without paying for redundant equipment most of the year, and ultimately it's probably at a lower cost. Meanwhile Azure is the only cloud with two Canada regions (Canada "Central", Canada East)[0], AWS has one (Canada "Central")[1], GCP has one (Montreal same city as AWS)[2].. there's really only one player if you want to use managed cloud services. [0]: https://azure.microsoft.com/en-us/global-infrastructure/geographies/ https://azure.microsoft.com/en-us/global-infrastructure/geog... [1]: https://aws.amazon.com/about-aws/global-infrastructure/regions_az/ https://aws.amazon.com/about-aws/global-infrastructure/regio... [2]: https://cloud.google.com/about/locations/ https://cloud.google.com/about/locations/
- jkaplowitz 6y ago
  I think the point was that a national emergency system should redundantly use two cloud providers, not that they made the wrong choice of single cloud provider.
  - gnabgib 6y ago
    The national emergency system doesn't rely on one provider[0] or one cloud, while Pelmorex Corp (The Weather Network) is part of the chain (and a curious one) it isn't the entire system, nor is their choice of provider/hardware a government choice (but it doesn't seem poor). The network isn't down, an aspect may have endured an outage. For significant (but not total) coverage of the country the "Alert Distributors" could be covered with contacting 15 corporations (huzzah anti competitive Canada)... or less if you pick a specific channel (eg Wireless=4), which I imagine is part of alerting protocol (your cable, mobile, radio, TV, and web applications don't all send the same alert) [0]: https://www.publicsafety.gc.ca/cnt/mrgnc-mngmnt/mrgnc-prprdnss/ntnl-pblc-lrtng-sstm-en.aspx https://www.publicsafety.gc.ca/cnt/mrgnc-mngmnt/mrgnc-prprdn...
- wbl 6y ago
  It's an emergency alerting system. Low bandwidth, broadcast. Perfect for blasting a few kilowatts of RF via digipeters each with their own generators and lead acid battery banks.

fartcannon 6y ago

And here it is, the main problem with outsourcing critical infrastructure. Remote server providers like Microsoft Azure should at most be used as twins/redundant systems to a locally managed system. Governments of the world: pay your IT people more money to prevent brain drain.

aynsof 6y ago
Do you think locally managed systems are immune to outages? Or that governments are capable of resourcing their teams sufficiently to do a better job at availability than Microsoft, Google, or Amazon?
- walrus01 6y ago
  Your comment is a fine example of the standard rhetoric from the 'move everything to the cloud' marketing people, but perhaps organizations should consider that they don't have to go with one cloud provider as a single point of failure. It's the lazy way of abstracting away responsibility and blame to some other party, and in my experience often does not result in better availability than a properly implemented "belt and suspenders" approach.
  - deleted 6y ago
    [deleted]
  - mattzito 6y ago
    There’s a complexity cost, though, right? It’s not free to run two different systems, manage two different billing systems, different tools, pay for cross-provider bandwidth, etc.? I’m sure there are some cases where the cost is worth the complexity, but I don’t think it’s cut and dried as written here. Most cloud providers are very reliable, and my guess is that you are more likely to have a self-inflicted outage due to the complexity of your infrastructure than the cloud provider having an outage.
  - emodendroket 6y ago
    Or it’s borne of experience of doing it oneself being much less reliable than cloud providers even accounting for these rare outages.

smitop 6y ago

Just got another email from them, apparently "the issue appears to have been resolved with Microsoft Azure". Looks like the problem is resolved

electro_blah 6y ago

lol they must have paid someone a shitty salary for someone to come up with a such a crappy system.