7 ms·

This is one of the many reasons there should be a universal data standard using a format like JSON. Heavily structured, easy to parse, easy to debug. What you l

This is one of the many reasons there should be a universal data standard using a format like JSON. Heavily structured, easy to parse, easy to debug. What you lose in footprint (i.e., more disk space), you gain in system stability. Imagine a world where everybody uses JSON and if they offer an API, you can just consume the data without a bunch of hoop jumping. Failures like this would vanish overnight.

0xffff2 3y ago
Broadly speaking I think this is done for new systems. What you need to identify here is how and when you transition legacy systems to this new better standard of practice.
- rglover 3y ago
  I'd argue in favor of at least an annual review process. Have a dedicated "feature freeze, emergencies only" period where you evaluate your existing data structures and queue up any necessary work. The only real hang up here is one of bad management. In terms of how, it's really just a question of Schema A to Schema B mapping. Have a small team responsible for collection/organization of all the possible schemas and then another small team responsible for writing the mapping functions to transition existing data. It would require will/force. Ideally, too, jobs of those responsible would be dependent on completion of the task so you couldn't just kick the can. You either do it and do it correctly or you're shopping your resume around.
tristor 3y ago
The problem is systems written in the 1970s in FORTRAN to run on Mainframes don't speak JSON.
- rglover 3y ago
  Great. It should be fixed by replacing the FORTRAN systems with a modern solution. It's not that it can't be done, it's that the engineers don't bother to start the process (which is a side-effect of bad incentive structure at the employment level).
  - tjohns 3y ago
    Many of them have been upgraded. In the US, we've replaced HOST (the old ATC backend system) with ERAM (the modern replacement) as of 2015. However, you have to remember this is a global problem. You need to maintain 100% backwards compatibility with every country on the planet. So even if you upgrade your country's systems to something modern, you still have to support old analog communication links and industry standard data formats.
  - fullspectrumdev 3y ago
    Have you ever been involved in such a migration? It’s invariably a complete clusterfuck.
    - rglover 3y ago
      I haven't, but I'd love to. My approach wouldn't be very "HR friendly," though.
      - count 3y ago
        Ah yes, migration through sheer force of will.
        jakub_g 3y ago
        It's trivial. Only took Amadeus hundreds of developers working for over a decade to migrate off TPF. /s [0] https://amadeus.com/en/insights/blog/celebrating-one-year-fully-open-systems https://amadeus.com/en/insights/blog/celebrating-one-year-fu...
        rglover 3y ago
        In some sense, yes. Notice that most of the responses to what I've said are immediately negative or dismissive of the idea. If that's the starting point (bad mindset), of course nothing gets fixed and you land where we are today. My initial approach would be to weed out anyone with that point of view before any work took place (the "not HR friendly" part being to be purposefully exclusionary). The only way a problem of this scope/scale can be solved is by a team of people with extremely thick skin who are comfortable grabbing a beer and telling jokes after they spent the day telling each other to go f*ck themselves.
fbdab103 3y ago
Airport software predates basically every standard on the planet. I would not be surprised to learn that they have their own bizarro world implementation of ASCII, unix epoch time, etc.
- tjohns 3y ago
  Yes, FPL messages are sent over AFTN, which uses ITA-2 Baudot code instead of ASCII: https://en.wikipedia.org/wiki/Baudot_code https://en.wikipedia.org/wiki/Baudot_code The keyboards used by ATC don't even allow entering symbols: https://www.reddit.com/media?url=https%3A%2F%2Fi.redd.it%2F1ktpo3pi4th21.jpg https://www.reddit.com/media?url=https%3A%2F%2Fi.redd.it%2F1... (There is a modern replacement for AFTN called AMHS, which replaces analog phone lines with X.400 messages over IP... but the system still needs to be backwards compatible for ATC units still using analog links.)
dundarious 3y ago
Parsing the data formats had zero contribution to the problem. They had a problem running an algorithm on the input data, and error reporting when that algorithm failed. Nothing about JSON would improve the situation.
- rglover 3y ago
  Yes, but look at the data. The algorithm was buggy because the input data is a nightmare. If the data didn't look like that, it's very unlikely the bug(s) would have ever existed.
  - dundarious 3y ago
    ADEXP sounds like the universal data standard you want then. The UK just has an existing NATS that cannot understand it without transformation by this problematic algorithm. So the significant part of your suggestion might be to elide the NATS specific processing and upgrade NATS to use ADEXP directly. Using a JSON format changes nothing. Just adds a few more characters to the text representation.
  - schainks 3y ago
    I have seen a bad outages caused by valid JSON whose consumer implemented something incorrectly. I agree with dundarius that "doing this in JSON" would not have changed the likelihood the bug could have manifested.
    - rglover 3y ago
      No change at all? I find that hard to believe. There's also a data design problem here, but the structure of JSON would aid in, not subtract from, that process. The question at hand is: "heavily structured data vs. a blob of text as input into a complex algorithm, which one is preferred?" Unless you're lying, you'd choose the former given the option.
      - dundarious 3y ago
        The issue is using both ADEXP and ICAO4444 waypoints, and doing so in a sloppy way. For the waypoint lists, there is no issue with structurelessness -- the fact that they're lists is pretty obvious, even in the existing formats. Adding some ["",] would not have helped the specific problem, as the relevant structure was already perfectly clear to the implementers. I am not lying when I say the bug would have been equally likely in a JSON format in this specific case.
vb-8448 3y ago
It won't fix anything. JSON is the "standard" today, 15 years ago it was XML and in 15 years we will have protobuf or another new standard.
- rglover 3y ago
  Correct. The other "leg" of a solution to this problem would be to codify migration practices so stagnation at the tech level is a non issue long-term.
  - vb-8448 3y ago
    > codify migration practices I think this won't work: no one really wants to touch a system that works, and people will try to find any excuse to avoid migrating. The reason of this is that everyone prefers systems that work and fails in known way rather new systems that no one knows how can it fail.
    - rglover 3y ago
      Does the system work if it randomly fails and collapses the entire system for days? People generally prefer to be lazy and to not use their brains, show up, and receive a paycheck for the minimum amount of effort. Not to be rude, but that's where this attitude originates. Having a codified process means that attitude can't exist because you're given all of the tools you need to solve the problem.
      - vb-8448 3y ago
        > Having a codified process means that attitude can't exist because you're given all of the tools you need to solve the problem. Yes, but in real life doesn't work. Processes have corner cases. As you said, people are lazy and will do everything to find the corner case to fit in. Just an example from the banking sector. There are processes (and even laws) that force banks to use only certified, supported and regularly patched software: there are still a lot of Windows 2000 servers in their datacenters and will be there for many years.
smarx007 3y ago
There are already standards like XML and RDF Turtle that allow you to clearly communicate vocabulary, such that a property 'iso3779:vin' (shorthand for a made-up URI 'https://ns.iso.org/standard/52200#vin https://ns.iso.org/standard/52200#vin') is interpreted in the same way anywhere in the structures and across API endpoints across companies (unlike JSON, where you need to fight both the existence of multiple labels like 'vin', 'vin_no', 'vinNumber', as well as the fact that the meaning of a property is strongly connected to its place in the JSON tree). The problem is that the added burden is not respected at the small scale and once large scale is reached, the switching costs are too big. And that XML is not cool, naturally. On top of that, RDF Turtle is the only widely used standard graph data format (as opposed to tree-based formats like JSON and XML). This allows you to reduce the hoop jumping when consuming responses from multiple APIs as graph union is a trivial operation, while n-way tree merging is not. Finally, RDF Turtle promotes use of URIs as primary identifiers (the ones exposed to the API consumers) instead of primary keys, bespoke tokens, or UUIDs. Followig this rule makes all identifiers globally unique and dereferenceable (ie, the ID contains the necessary information on how to fetch the resource identified by a given ID). P.S.: The problem at hand was caused by the algorithm that was processing the parsed data, not with the parsing per se. The only improvement a better data format like RDF Turtle would bring is that two different waypoints with the same label would have two different URI identifiers.
- seabass-labrax 3y ago
  Furthermore, there are already XML namespaces for flight plans. These are not, however, used by ATC - only by pilots to load new routes into their aircrafts' navigation computers. I'm not sure whether there is an existing RDF ontology for flight plans; it would probably be of low to medium complexity considering how powerful RDF is and the kind of global-scale users it already has.
nemetroid 3y ago
There are several XML formats for expressing flight plans, most notably ARINC 633 and FIXM.
masklinn 3y ago
The bug here was a processing one, having the data in json would make no difference.
elzbardico 3y ago
The problem was not in the format, but with the way the semantics of the data is understood by the system. It could be fixed-width, XML, json, whatever, and the problem would still be the same.

0xffff2 3y ago

Broadly speaking I think this is done for new systems. What you need to identify here is how and when you transition legacy systems to this new better standard of practice.

rglover 3y ago
I'd argue in favor of at least an annual review process. Have a dedicated "feature freeze, emergencies only" period where you evaluate your existing data structures and queue up any necessary work. The only real hang up here is one of bad management. In terms of how, it's really just a question of Schema A to Schema B mapping. Have a small team responsible for collection/organization of all the possible schemas and then another small team responsible for writing the mapping functions to transition existing data. It would require will/force. Ideally, too, jobs of those responsible would be dependent on completion of the task so you couldn't just kick the can. You either do it and do it correctly or you're shopping your resume around.

tristor 3y ago

The problem is systems written in the 1970s in FORTRAN to run on Mainframes don't speak JSON.

rglover 3y ago
Great. It should be fixed by replacing the FORTRAN systems with a modern solution. It's not that it can't be done, it's that the engineers don't bother to start the process (which is a side-effect of bad incentive structure at the employment level).
- tjohns 3y ago
  Many of them have been upgraded. In the US, we've replaced HOST (the old ATC backend system) with ERAM (the modern replacement) as of 2015. However, you have to remember this is a global problem. You need to maintain 100% backwards compatibility with every country on the planet. So even if you upgrade your country's systems to something modern, you still have to support old analog communication links and industry standard data formats.
- fullspectrumdev 3y ago
  Have you ever been involved in such a migration? It’s invariably a complete clusterfuck.
  - rglover 3y ago
    I haven't, but I'd love to. My approach wouldn't be very "HR friendly," though.
    - count 3y ago
      Ah yes, migration through sheer force of will.
      - jakub_g 3y ago
        It's trivial. Only took Amadeus hundreds of developers working for over a decade to migrate off TPF. /s [0] https://amadeus.com/en/insights/blog/celebrating-one-year-fully-open-systems https://amadeus.com/en/insights/blog/celebrating-one-year-fu...
      - rglover 3y ago
        In some sense, yes. Notice that most of the responses to what I've said are immediately negative or dismissive of the idea. If that's the starting point (bad mindset), of course nothing gets fixed and you land where we are today. My initial approach would be to weed out anyone with that point of view before any work took place (the "not HR friendly" part being to be purposefully exclusionary). The only way a problem of this scope/scale can be solved is by a team of people with extremely thick skin who are comfortable grabbing a beer and telling jokes after they spent the day telling each other to go f*ck themselves.

fbdab103 3y ago

Airport software predates basically every standard on the planet. I would not be surprised to learn that they have their own bizarro world implementation of ASCII, unix epoch time, etc.

tjohns 3y ago
Yes, FPL messages are sent over AFTN, which uses ITA-2 Baudot code instead of ASCII: https://en.wikipedia.org/wiki/Baudot_code https://en.wikipedia.org/wiki/Baudot_code The keyboards used by ATC don't even allow entering symbols: https://www.reddit.com/media?url=https%3A%2F%2Fi.redd.it%2F1ktpo3pi4th21.jpg https://www.reddit.com/media?url=https%3A%2F%2Fi.redd.it%2F1... (There is a modern replacement for AFTN called AMHS, which replaces analog phone lines with X.400 messages over IP... but the system still needs to be backwards compatible for ATC units still using analog links.)

dundarious 3y ago

Parsing the data formats had zero contribution to the problem. They had a problem running an algorithm on the input data, and error reporting when that algorithm failed. Nothing about JSON would improve the situation.

rglover 3y ago
Yes, but look at the data. The algorithm was buggy because the input data is a nightmare. If the data didn't look like that, it's very unlikely the bug(s) would have ever existed.
- dundarious 3y ago
  ADEXP sounds like the universal data standard you want then. The UK just has an existing NATS that cannot understand it without transformation by this problematic algorithm. So the significant part of your suggestion might be to elide the NATS specific processing and upgrade NATS to use ADEXP directly. Using a JSON format changes nothing. Just adds a few more characters to the text representation.
- schainks 3y ago
  I have seen a bad outages caused by valid JSON whose consumer implemented something incorrectly. I agree with dundarius that "doing this in JSON" would not have changed the likelihood the bug could have manifested.
  - rglover 3y ago
    No change at all? I find that hard to believe. There's also a data design problem here, but the structure of JSON would aid in, not subtract from, that process. The question at hand is: "heavily structured data vs. a blob of text as input into a complex algorithm, which one is preferred?" Unless you're lying, you'd choose the former given the option.
    - dundarious 3y ago
      The issue is using both ADEXP and ICAO4444 waypoints, and doing so in a sloppy way. For the waypoint lists, there is no issue with structurelessness -- the fact that they're lists is pretty obvious, even in the existing formats. Adding some ["",] would not have helped the specific problem, as the relevant structure was already perfectly clear to the implementers. I am not lying when I say the bug would have been equally likely in a JSON format in this specific case.

vb-8448 3y ago

It won't fix anything. JSON is the "standard" today, 15 years ago it was XML and in 15 years we will have protobuf or another new standard.

rglover 3y ago
Correct. The other "leg" of a solution to this problem would be to codify migration practices so stagnation at the tech level is a non issue long-term.
- vb-8448 3y ago
  > codify migration practices I think this won't work: no one really wants to touch a system that works, and people will try to find any excuse to avoid migrating. The reason of this is that everyone prefers systems that work and fails in known way rather new systems that no one knows how can it fail.
  - rglover 3y ago
    Does the system work if it randomly fails and collapses the entire system for days? People generally prefer to be lazy and to not use their brains, show up, and receive a paycheck for the minimum amount of effort. Not to be rude, but that's where this attitude originates. Having a codified process means that attitude can't exist because you're given all of the tools you need to solve the problem.
    - vb-8448 3y ago
      > Having a codified process means that attitude can't exist because you're given all of the tools you need to solve the problem. Yes, but in real life doesn't work. Processes have corner cases. As you said, people are lazy and will do everything to find the corner case to fit in. Just an example from the banking sector. There are processes (and even laws) that force banks to use only certified, supported and regularly patched software: there are still a lot of Windows 2000 servers in their datacenters and will be there for many years.

smarx007 3y ago

There are already standards like XML and RDF Turtle that allow you to clearly communicate vocabulary, such that a property 'iso3779:vin' (shorthand for a made-up URI 'https://ns.iso.org/standard/52200#vin https://ns.iso.org/standard/52200#vin') is interpreted in the same way anywhere in the structures and across API endpoints across companies (unlike JSON, where you need to fight both the existence of multiple labels like 'vin', 'vin_no', 'vinNumber', as well as the fact that the meaning of a property is strongly connected to its place in the JSON tree). The problem is that the added burden is not respected at the small scale and once large scale is reached, the switching costs are too big. And that XML is not cool, naturally. On top of that, RDF Turtle is the only widely used standard graph data format (as opposed to tree-based formats like JSON and XML). This allows you to reduce the hoop jumping when consuming responses from multiple APIs as graph union is a trivial operation, while n-way tree merging is not. Finally, RDF Turtle promotes use of URIs as primary identifiers (the ones exposed to the API consumers) instead of primary keys, bespoke tokens, or UUIDs. Followig this rule makes all identifiers globally unique and dereferenceable (ie, the ID contains the necessary information on how to fetch the resource identified by a given ID). P.S.: The problem at hand was caused by the algorithm that was processing the parsed data, not with the parsing per se. The only improvement a better data format like RDF Turtle would bring is that two different waypoints with the same label would have two different URI identifiers.

seabass-labrax 3y ago
Furthermore, there are already XML namespaces for flight plans. These are not, however, used by ATC - only by pilots to load new routes into their aircrafts' navigation computers. I'm not sure whether there is an existing RDF ontology for flight plans; it would probably be of low to medium complexity considering how powerful RDF is and the kind of global-scale users it already has.

nemetroid 3y ago

There are several XML formats for expressing flight plans, most notably ARINC 633 and FIXM.

masklinn 3y ago

The bug here was a processing one, having the data in json would make no difference.

elzbardico 3y ago

The problem was not in the format, but with the way the semantics of the data is understood by the system. It could be fixed-width, XML, json, whatever, and the problem would still be the same.