4 ms·

The NOTAM system is something that a room full of decent engineers could easily build from scratch and make it infinitely better in a short time. It’s essential

by JCM9 4y ago

The NOTAM system is something that a room full of decent engineers could easily build from scratch and make it infinitely better in a short time. It’s essentially just a database of categorized posts with some APIs for sending entries and and returning them when requested. These government IT teams spend way more than what it should cost and end up with bloated ancient tech that barely works. That’s not speaking poor of the engineers (which in my experience can be very good) but of the management and innovation culture of these agencies, which is too often terribly broken. They would say they are “risk averse” but as yesterday highlights their poor approach to this creates a ton of risk.

_fat_santa 4y ago
Well they claim to be "risk averse", the wisdom there seems to be that if you fill out a million forms that qualifies as risk averse because it shows you did due diligence. The problem is they wholesale took that process from the rest of the org and applied it to software which doesn't work. It's like making a crap sandwich and filling out a bunch of forms proving that it's not crap sandwich, rather than just spending the time you would be filling out all those forms on i don't know...not making a crap sandwich.
Someone1234 4y ago
"I haven't seen the requirements, know little to nothing about the system, but I could knock that out in a weekend with a few Red Bulls." This stuff is such cringe, it is the type of response you see from fresh CS students who haven't started working yet and think everything is a piece of demo work where the requirements don't matter and that everything is simple if you just start writing some code. Obviously the NOTAM system isn't up to scratch, that's why we're talking about it. But "hard problems are easy" isn't constructive nor very realistic. I bet someone could put in a three-hour presentation talking about all the complexity that led us to this conclusion.
- JCM9 4y ago
  I hear you. But the flip side is there is often a tendency to massively over complicate things in a way that bakes in valueless complexity and bloat, which is very much the norm in large government systems. I’m practice this attitude is far more common and destructive long term.
  - nonethewiser 4y ago
    I think it's way worse than adding complexity and bloat. There are processes that specifically prevent anyone from understanding or owning the system. The complexity and bloat is a side product of the fact that the work was siloedd, contracted out, and everyone washed their hands off the result. Which also takes exponentially more time and people. Its optimized for inefficiency.
    - gnarbarian 4y ago
      this is extremely accurate. it's like a giant rube Goldberg machine that takes both good code and garbage as input and produces a tremendous amount of garbage that nobody can understand once it comes out the other end.
    - bink 4y ago
      Do you know that each of these systems actually has a defined "owner"? It's part of the FISMA process and every so often these "owners" have to attest that certain security processes are in place and functioning and that no significant changes have been made without a full review... among many other things. That "owner" is also a federal employee, not a contractor. https://en.wikipedia.org/wiki/Federal_Information_Security_Management_Act_of_2002 https://en.wikipedia.org/wiki/Federal_Information_Security_M...
- rattlesnakedave 4y ago
  > This stuff is such cringe, it is the type of response you see from fresh CS students who haven't started working yet and think everything is a piece of demo work where the requirements don't matter and that everything is simple if you just start writing some code. This stuff is such cringe, it is the type of response you see from jaded boomers more focused on box-ticking and punching out as opposed to doing anything new. This is a situation where tossing the whole damn thing out and starting over again would be productive. The systems that lead to the creation of these half-fossilized government projects (that still don't work!) will not change and needs to be tossed.
  - _ea1k 4y ago
    The problem with replacing NOTAMs is that it actually is a fairly simple system that works pretty well from a technical standpoint. The biggest issues in general relate to the data itself and the processes involved.
- nonethewiser 4y ago
  > "I haven't seen the requirements, know little to nothing about the system, but I could knock that out in a weekend with a few Red Bulls." This stuff is such cringe You mean the quote that you just made up?
- CamperBob2 4y ago
  What makes this a hard problem, exactly? It has to work no matter what, but it also doesn't have to do very much.
  - gnarbarian 4y ago
    I've worked directly with this system a number of times. it's basically a pub sub rss feed optimized for low latency and molested by bureaucrats for decades
sumtechguy 4y ago
There is a whole group of people who their job depends on the system existing like it does. One dude I worked with re-did a procurement system with a few spreadsheets. He got tired of waiting for 30 people to 'do it'. As soon as his boss found it the whole project he was on was scrapped and he was demoted shortly thereafter. His 'sin'? He had accidently found a way to put 30 people out of a job. These systems exist like this because our gov wants them that way. Not because they are the best. Yesterday should have been 'run this on these 2 old boxes and make sure they still work' regression test. Sounds like either that step is not there, totally skipped, or does not match reality. Building a real five 9s type system means taking each piece and pondering 'what are the different ways this can fail'. Then mitigating each of those. It is mind numbing tedious work than takes a long time to do. Also most of this is probably run by contract houses. Which means the people who use it do not really 'own it'. Which is by design, for CYA. Which costs more and takes more time because it is all paperwork.
- nonethewiser 4y ago
  And the kicker is that you're paying for it.
phaedrus 4y ago
One common problem is building new things with tech that is already obsolete. Given the choice between new and old, the old is perceived as more tried and tested. Granted it's sometimes difficult to distinguish "new and going to last" from "new and shiny", but when it comes to software choosing to go with "old and tested" can be counter to cyber security since the old software is no longer updated. Of course you also have "old but still developed, and likely to be supported in near-perpetuity". Things like SQLite fall in this category. Unfortunately there's this other problem of management being inexplicably down on anything open-source. I don't know whether they've been exposed to too much FUD from vendors selling proprietary solutions, or just that the idea that security through obscurity is no security at all has failed to reach them.
killerstorm 4y ago
Engineers following current best practices? They'd run a Kubernetes cluster with Kafka. Because those are the best CotS tool for reliability. Battle-tested. The system will be down every week because Kubernetes need patching.

by JCM9 4y ago

_fat_santa 4y ago

Well they claim to be "risk averse", the wisdom there seems to be that if you fill out a million forms that qualifies as risk averse because it shows you did due diligence. The problem is they wholesale took that process from the rest of the org and applied it to software which doesn't work. It's like making a crap sandwich and filling out a bunch of forms proving that it's not crap sandwich, rather than just spending the time you would be filling out all those forms on i don't know...not making a crap sandwich.

Someone1234 4y ago

"I haven't seen the requirements, know little to nothing about the system, but I could knock that out in a weekend with a few Red Bulls." This stuff is such cringe, it is the type of response you see from fresh CS students who haven't started working yet and think everything is a piece of demo work where the requirements don't matter and that everything is simple if you just start writing some code. Obviously the NOTAM system isn't up to scratch, that's why we're talking about it. But "hard problems are easy" isn't constructive nor very realistic. I bet someone could put in a three-hour presentation talking about all the complexity that led us to this conclusion.

JCM9 4y ago
I hear you. But the flip side is there is often a tendency to massively over complicate things in a way that bakes in valueless complexity and bloat, which is very much the norm in large government systems. I’m practice this attitude is far more common and destructive long term.
- nonethewiser 4y ago
  I think it's way worse than adding complexity and bloat. There are processes that specifically prevent anyone from understanding or owning the system. The complexity and bloat is a side product of the fact that the work was siloedd, contracted out, and everyone washed their hands off the result. Which also takes exponentially more time and people. Its optimized for inefficiency.
  - gnarbarian 4y ago
    this is extremely accurate. it's like a giant rube Goldberg machine that takes both good code and garbage as input and produces a tremendous amount of garbage that nobody can understand once it comes out the other end.
  - bink 4y ago
    Do you know that each of these systems actually has a defined "owner"? It's part of the FISMA process and every so often these "owners" have to attest that certain security processes are in place and functioning and that no significant changes have been made without a full review... among many other things. That "owner" is also a federal employee, not a contractor. https://en.wikipedia.org/wiki/Federal_Information_Security_Management_Act_of_2002 https://en.wikipedia.org/wiki/Federal_Information_Security_M...
rattlesnakedave 4y ago
> This stuff is such cringe, it is the type of response you see from fresh CS students who haven't started working yet and think everything is a piece of demo work where the requirements don't matter and that everything is simple if you just start writing some code. This stuff is such cringe, it is the type of response you see from jaded boomers more focused on box-ticking and punching out as opposed to doing anything new. This is a situation where tossing the whole damn thing out and starting over again would be productive. The systems that lead to the creation of these half-fossilized government projects (that still don't work!) will not change and needs to be tossed.
- _ea1k 4y ago
  The problem with replacing NOTAMs is that it actually is a fairly simple system that works pretty well from a technical standpoint. The biggest issues in general relate to the data itself and the processes involved.
nonethewiser 4y ago
> "I haven't seen the requirements, know little to nothing about the system, but I could knock that out in a weekend with a few Red Bulls." This stuff is such cringe You mean the quote that you just made up?
CamperBob2 4y ago
What makes this a hard problem, exactly? It has to work no matter what, but it also doesn't have to do very much.
- gnarbarian 4y ago
  I've worked directly with this system a number of times. it's basically a pub sub rss feed optimized for low latency and molested by bureaucrats for decades

sumtechguy 4y ago

There is a whole group of people who their job depends on the system existing like it does. One dude I worked with re-did a procurement system with a few spreadsheets. He got tired of waiting for 30 people to 'do it'. As soon as his boss found it the whole project he was on was scrapped and he was demoted shortly thereafter. His 'sin'? He had accidently found a way to put 30 people out of a job. These systems exist like this because our gov wants them that way. Not because they are the best. Yesterday should have been 'run this on these 2 old boxes and make sure they still work' regression test. Sounds like either that step is not there, totally skipped, or does not match reality. Building a real five 9s type system means taking each piece and pondering 'what are the different ways this can fail'. Then mitigating each of those. It is mind numbing tedious work than takes a long time to do. Also most of this is probably run by contract houses. Which means the people who use it do not really 'own it'. Which is by design, for CYA. Which costs more and takes more time because it is all paperwork.

nonethewiser 4y ago
And the kicker is that you're paying for it.

phaedrus 4y ago

One common problem is building new things with tech that is already obsolete. Given the choice between new and old, the old is perceived as more tried and tested. Granted it's sometimes difficult to distinguish "new and going to last" from "new and shiny", but when it comes to software choosing to go with "old and tested" can be counter to cyber security since the old software is no longer updated. Of course you also have "old but still developed, and likely to be supported in near-perpetuity". Things like SQLite fall in this category. Unfortunately there's this other problem of management being inexplicably down on anything open-source. I don't know whether they've been exposed to too much FUD from vendors selling proprietary solutions, or just that the idea that security through obscurity is no security at all has failed to reach them.

killerstorm 4y ago

Engineers following current best practices? They'd run a Kubernetes cluster with Kafka. Because those are the best CotS tool for reliability. Battle-tested. The system will be down every week because Kubernetes need patching.