6 ms·
Quote from FAA: "Our preliminary work has traced the outage to a damaged database file." UK news source The Independent is reporting that Nav Canada's NOTAM sys
by mgsouth 4y ago
Quote from FAA: "Our preliminary work has traced the outage to a damaged database file." UK news source The Independent is reporting that Nav Canada's NOTAM system also suffered an issue.[1] Speculation: corrupting input, either international or North American? E.G. UTF-8, SQL escape, CSV quoting.
edit: Better reporting of Canada's issue from Canada's CBC (and frankly, better reporting about the US, too). [2] "In Canada, pilots were still able to read NOTAMs, but there was an outage that meant new notices couldn't be entered into the system, NAV Canada said on social media." "NAV Canada said it did not believe the outage was related to the one in the U.S., but it said it was investigating."
[1] https://www.independent.co.uk/news/world/americas/canada-flights-system-outage-grounded-b2260500.html https://www.independent.co.uk/news/world/americas/canada-fli...
HN thread https://news.ycombinator.com/item?id=34347520 https://news.ycombinator.com/item?id=34347520
[2] https://www.cbc.ca/news/us-air-travel-chaos-notam-outage-1.6710734 https://www.cbc.ca/news/us-air-travel-chaos-notam-outage-1.6...
- markdown 4y ago> traced the outage to a damaged database file Lil' Bobby Tables strikes again!
- noman-land 4y agoMy first thought as well.
- deleted 4y ago[deleted]
- asveikau 4y ago> "Our preliminary work has traced the outage to a damaged database file." > Speculation: corrupting input, either international or North American? E.G. UTF-8, SQL escape, CSV quoting. I read it as filesystem corruption from a bad disk, coupled with redundancy that doesn't actually work.
- tigerBL00D 4y agoWhat kind of database are they using, I wonder, to end up with such a spectacular failure?
- lopkeny12ko 4y agoWhy would this be an indictment of any specific database technology? If your disk fails and corrupts the filesystem, you're toast, regardless of what database you are using.
- mdavidn 4y agoThe technology to detect and recover from disk failures does exist. RAID and ZFS, for example. I would not expect a disk failure to replicate to the backup.
- tanelpoder 4y agoYep and if you ship WAL transaction logs to standby databases/replicas, corrupt blocks or lost writes in the primary database won't be propagated to the standbys (unlike with OS filesystem or storage-level replication). Edit: Should add "won't be silently propagated"
- ct520 4y agoWorking with critical infrastructure and lack of true in depth oversight it wouldn’t surprise me DR plans were not ever executed or exercised in a meaningful manner.
- ChrisMarshallNY 4y agoThis is quite common. Comprehensive DR testing is really difficult. Many orgs settle for “on paper,” or “in theory” substitutions for real testing. They do it right; no problem. Doing it right, though … there’s the rub …
- ilyt 4y ago
- ericpauley 4y agoThis could be due to some third party service caching the NOTAMs. Even in the US Foreflight had all notams available but just couldn’t fetch news ones.
- brewdad 4y agoI have no idea how often NOTAMs typically get updated but would that explain why flights were able to operate for a few hours before ultimately a ground stop was called for?
- deleted 4y ago[deleted]
- marrone12 4y agoThe CBC is great. I watch the news show every night even as an American.
- lebuffon 4y agoTo paraphrase Sarah Palin: Canada can see the USA from their front window. (A great deal of time is spent looking through that window)
- GnarfGnarf 4y agoFunny! As a Canadian, I am riveted by CNN/MSNBC/PBS/ for news on American political developments.
- eurasiantiger 4y agoSomeone put an emoji in a NOTAM
- ExoticPearTree 4y agoSomething like ... "CAVOK :)"
- bluetwo 4y agoUnicode. AmI right?