4 ms·

good ETLs are usually designed to separate good records from bad records, so even if one or two rows in the stream do not conform to schema - you can put them a

by slt2021 3y ago

good ETLs are usually designed to separate good records from bad records, so even if one or two rows in the stream do not conform to schema - you can put them aside and process the rest. seems like poor engineering

jandrese 3y ago
The problem is that it means you have a plane entering the airspace at some point in the near future and the system doesn't know it is going to be there. The whole point of this is to make sure no two planes are attempting to occupy the same space at the same time. If you don't know where one of the planes will be you can't plan all of the rest to avoid it. The thing that blows my mind is that this was apparently the first time this situation had happened after 15 million records processed. I would have expected it to trigger much more often. It makes me wonder if there wasn't someone who was fixing these as they came up in the 4 hour window, and he just happened to be off that day.
- d1sxeyes 3y ago
  > It makes me wonder if there wasn't someone who was fixing these as they came up in the 4 hour window, and he just happened to be off that day. This is very possible. I know of a guy who does (or at least a few years ago did) 24x7 365 on-call for a piece of mission (although not safety) critical aviation software. Most of his calls were fixing AWBs quickly because otherwise planes would need to take off empty or lose their take-off slot. Although there had been some “bus factor” planning and mitigation around this guy’s role, it involved engaging vendors etc. and would have likely resulted in a lot of disruption in the short term.
  - fbdab103 3y ago
    Please tell me this guy is now wealthy beyond imagination and living a life of leisure?
    - d1sxeyes 3y ago
      I would love to. But it wouldn’t be true.
- zaphar 3y ago
  Bad records aren't supposed to be ignored. They are supposed to be looked at by a human who can determine what to do. Failing the way NATS did means that all future flight plan data including for planes already in the sky are not longer being processed. The safer failure mode was definitely to flag this plan and surface to a human while continuing to process other plans.
- epolanski 3y ago
  One in a 15M chance with 7000 daily flies over the UK handled by nats meant it had a probability to happen at least once in 69 months, it took few months less.
cratermoon 3y ago
I never said it was a good ETL system. Heck, I don't even know if the specs for it even specifies what to do with a bad record - there are at least 300 pages detailing the system. Looking around at other stories, I see repeated mentions of how the circumstances leading to this failure are supposedly extremely rare, "one in 15 million" according to one official[1]. But at 100,000 flights/day (estimated), this kind situation would occur, statistically, twice a year. 1 https://news.sky.com/story/major-flights-disruption-caused-by-extremely-rare-set-of-circumstances-air-traffic-control-chiefs-say-12955708 https://news.sky.com/story/major-flights-disruption-caused-b...
Scarblac 3y ago
This flight plan was correct though, if there was some validation like that then it should have passed. The code that crashed had a bug, it couldn't deal with all valid data.

jandrese 3y ago

The problem is that it means you have a plane entering the airspace at some point in the near future and the system doesn't know it is going to be there. The whole point of this is to make sure no two planes are attempting to occupy the same space at the same time. If you don't know where one of the planes will be you can't plan all of the rest to avoid it. The thing that blows my mind is that this was apparently the first time this situation had happened after 15 million records processed. I would have expected it to trigger much more often. It makes me wonder if there wasn't someone who was fixing these as they came up in the 4 hour window, and he just happened to be off that day.

d1sxeyes 3y ago
> It makes me wonder if there wasn't someone who was fixing these as they came up in the 4 hour window, and he just happened to be off that day. This is very possible. I know of a guy who does (or at least a few years ago did) 24x7 365 on-call for a piece of mission (although not safety) critical aviation software. Most of his calls were fixing AWBs quickly because otherwise planes would need to take off empty or lose their take-off slot. Although there had been some “bus factor” planning and mitigation around this guy’s role, it involved engaging vendors etc. and would have likely resulted in a lot of disruption in the short term.
- fbdab103 3y ago
  Please tell me this guy is now wealthy beyond imagination and living a life of leisure?
  - d1sxeyes 3y ago
    I would love to. But it wouldn’t be true.
zaphar 3y ago
Bad records aren't supposed to be ignored. They are supposed to be looked at by a human who can determine what to do. Failing the way NATS did means that all future flight plan data including for planes already in the sky are not longer being processed. The safer failure mode was definitely to flag this plan and surface to a human while continuing to process other plans.
epolanski 3y ago
One in a 15M chance with 7000 daily flies over the UK handled by nats meant it had a probability to happen at least once in 69 months, it took few months less.

cratermoon 3y ago

I never said it was a good ETL system. Heck, I don't even know if the specs for it even specifies what to do with a bad record - there are at least 300 pages detailing the system. Looking around at other stories, I see repeated mentions of how the circumstances leading to this failure are supposedly extremely rare, "one in 15 million" according to one official[1]. But at 100,000 flights/day (estimated), this kind situation would occur, statistically, twice a year. 1 https://news.sky.com/story/major-flights-disruption-caused-by-extremely-rare-set-of-circumstances-air-traffic-control-chiefs-say-12955708 https://news.sky.com/story/major-flights-disruption-caused-b...

Scarblac 3y ago

This flight plan was correct though, if there was some validation like that then it should have passed. The code that crashed had a bug, it couldn't deal with all valid data.