6 ms·

You've got it exactly right. There are a lot of people here who are completely deluded into the "move fast and break things" mindset not realizing that sometime

by alexb_ 4y ago

You've got it exactly right. There are a lot of people here who are completely deluded into the "move fast and break things" mindset not realizing that sometimes you really do not want to move fast, because you REALLY do not want to break things. A corrupted file throwing up panics like this is a good thing, because you don't want corrupted files to pass through like everything is fine.

bastardoperator 4y ago
If the corrupted file is in the backup, it DID pass through like everything was fine. What's clear to me is that the FAA has no post deployment validation, hasn't tested its DR strategy, and that errors can go unseen for long periods of time.
- haswell 4y ago
  > What's clear to me is that the FAA has no post deployment validation, hasn't tested its DR strategy, and that errors can go unseen for long periods of time. It is possible to have all of those mitigations in place and still experience a failure like this. Post deployment validation is only as good as the validations executed. 99% coverage still leaves the door open to failure. A DR strategy is just that - a strategy. A failure of this sort is not an automatic implication that those things do not exist, just that they failed in this particular case. I would find it incredibly surprising that an organization of that complexity could have survived as long as they did without a major incident if none of those things were in place. They’d be either incredibly lucky, or incredibly competent, and if they are the latter, they would not operate without such mitigations in place. It seems far more believable that an organization of the FAA’s age and complexity missed something along the way.
  - bastardoperator 4y ago
    We can make excuses all day long. A simple query of the database/table would have produced an error. Sure, the FAA does some complex stuff, but the tech I see in airplanes looks ancient. I'm willing to bet most of the FAA complexity comes from budget (lack thereof) and old computer systems.
    - thrashh 4y ago
      We found the person who intimately knows how FAA’s system is engineered and who also builds perfect systems
    - haswell 4y ago
      This has nothing to do with excuses - I’m challenging the assertion that “because something bad happened, they must not have any mitigations in place at all”. This seems like a bad case of binary thinking, and my point was that the occurrence of an incident like this is not sufficient to support that claim. It’s just as likely that an ancient process that wasn’t accounted for somewhere in the architecture broke down, and this is how it manifested. Clearly improvements are needed, as is always the case after an outage. That doesn’t justify wild speculation. Anecdote time: I once worked for a large financial institution that makes money when people swipe their credit cards. The system that authorizes purchases is ancient, battle tested, and undergoes minimal change because the cost of an outage could be measured in the millions of $ per minute. Every change was scrutinized, reviewed by multiple groups, discussed with executives, and tested thoroughly. The same system underwent regular DR testing that involved quite a lot of involvement from all related teams. So the day it went down, it was obviously a big deal, and raised all of the natural questions about how such a thing could occur. Turns out it had an unknown transitive dependency on an internal server - a server that had not been rebooted in literally a decade. When that server was rebooted (I think it was a security group insisting it needed patches despite some strong reasons to avoid that when considering the architecture), some of the services never came back up, and everyone quickly learned that a very old change that predated almost everyone there established this unknown dependency. The point of this story is really about the unknowability of sufficiently complex legacy enterprise systems. All of the right processes and procedures won’t necessarily account for that seemingly inconsequential RPC call to an internal system implemented by a grizzled dev shortly before his retirement.
      - wbl 4y ago
        Those were the wrong procedures. If you were regularly rebooting systems left and right you'd learn quickly if things didn't come up.
  - landemva 4y ago
    > incredibly surprising that an organization of that complexity could have survived as long as they did without a major incident I'm not surprised. FAA does not fly each plane. Government organizational complexity helps ensure the government organization survives through next round of Congressional appropriations. Org complexity + opaque oversight + 'safety' + 'homeland security' + taxpayer funded = playing around and more budget. The pilot is responsible for safety. Air travel has rules to avoid collisions (eastbound gets altitude levels different than westbound, pilots shall broadcast on known frequencies) and pilots have distributed intelligence to keep their flight safe. Yes, somehow there needs to be coordination of runway use. Many ways to provide reservations and queuing.
- zhengyi13 4y ago
  I read this incident as they in fact did have a DR strategy, they knew how to execute it, did so successfully, and still failed. The only thing I'm left wondering at this point is whether the corrupt data were a config, or state.
  - krabizzwainch 4y ago
    I don't even know if I'd call this a disaster recovery fail. Depending on what they meant by "corruption", a roughly 6-8 hour turn around time is not awful for a database restore.
justinsaccount 4y ago
People like to think that the alternative to "move fast and break things" is "move slowly and not break things" but it's not, it's "move slowly, break things anyway, then take days to resolve the problem because you never learned how to move fast".
- eckza 4y ago
  That's just not true. Moving faster does not mean moving more effectively. Frantic activity should not be mistaken for progress. I'm OK with the FAA taking "days to resolve the problem" if it means that nobody dies. For what it's worth, my team and I are partial to "move carefully, and tend things".
  - justinsaccount 4y ago
    "Moving faster" doesn't mean "frantic" either. Who said frantic?
    - haswell 4y ago
      Frantic is often a natural outcome of “moving faster” when the environment one is moving in is not conducive to that speed of movement. In my experience, this tendency towards frantic is multiplied the larger and more complex the organization and architecture becomes. The entire point of “move fast” in software circles is to leave behind the constraints of legacy tech and management practices in favor of building something “better”. In a mature org that grew up before these ideas were mainstream, maybe one or two teams can manage to move faster, but invariably they end up depending on other teams, who in turn depend on deeply ingrained and established company culture and procedures. We can talk about why those impediments are a Bad Thing, and I wouldn’t advise a consumer startup to adopt those methodologies in 2022, but there’s still the harsh reality that where they exist, “just move faster” doesn’t help much more than telling a depressed person to “just do cardio every day”. There’s often a lot of inner work that’s gotta happen to make way for the new. The only way I’ve seen this sort of work in a large org is when a brand new “emerging tech” group is spun up and given autonomy to work outside of the legacy norms. This is not perfect either, and seems much better for greenfield projects. When applied to deeply entrenched legacy systems, all of the problems mentioned above come to a head. This also creates a weird in/out group dynamic which tends to further stratify the old tech and widen the gap between the old practices and the new. In the context of this particular conversation though, I think the concept of “move fast” has lost all meaning and has little to offer for an org like the FAA.
berniedurfee 4y ago
Move fast and break things, unless people are involved. Please, don’t break people. People are non-fungible.
indymike 4y ago
Software falls into two buckets: stuff that directly injures and kills people when it fails, and everything else.

by alexb_ 4y ago

bastardoperator 4y ago

If the corrupted file is in the backup, it DID pass through like everything was fine. What's clear to me is that the FAA has no post deployment validation, hasn't tested its DR strategy, and that errors can go unseen for long periods of time.

haswell 4y ago
> What's clear to me is that the FAA has no post deployment validation, hasn't tested its DR strategy, and that errors can go unseen for long periods of time. It is possible to have all of those mitigations in place and still experience a failure like this. Post deployment validation is only as good as the validations executed. 99% coverage still leaves the door open to failure. A DR strategy is just that - a strategy. A failure of this sort is not an automatic implication that those things do not exist, just that they failed in this particular case. I would find it incredibly surprising that an organization of that complexity could have survived as long as they did without a major incident if none of those things were in place. They’d be either incredibly lucky, or incredibly competent, and if they are the latter, they would not operate without such mitigations in place. It seems far more believable that an organization of the FAA’s age and complexity missed something along the way.
- bastardoperator 4y ago
  We can make excuses all day long. A simple query of the database/table would have produced an error. Sure, the FAA does some complex stuff, but the tech I see in airplanes looks ancient. I'm willing to bet most of the FAA complexity comes from budget (lack thereof) and old computer systems.
  - thrashh 4y ago
    We found the person who intimately knows how FAA’s system is engineered and who also builds perfect systems
  - haswell 4y ago
    This has nothing to do with excuses - I’m challenging the assertion that “because something bad happened, they must not have any mitigations in place at all”. This seems like a bad case of binary thinking, and my point was that the occurrence of an incident like this is not sufficient to support that claim. It’s just as likely that an ancient process that wasn’t accounted for somewhere in the architecture broke down, and this is how it manifested. Clearly improvements are needed, as is always the case after an outage. That doesn’t justify wild speculation. Anecdote time: I once worked for a large financial institution that makes money when people swipe their credit cards. The system that authorizes purchases is ancient, battle tested, and undergoes minimal change because the cost of an outage could be measured in the millions of $ per minute. Every change was scrutinized, reviewed by multiple groups, discussed with executives, and tested thoroughly. The same system underwent regular DR testing that involved quite a lot of involvement from all related teams. So the day it went down, it was obviously a big deal, and raised all of the natural questions about how such a thing could occur. Turns out it had an unknown transitive dependency on an internal server - a server that had not been rebooted in literally a decade. When that server was rebooted (I think it was a security group insisting it needed patches despite some strong reasons to avoid that when considering the architecture), some of the services never came back up, and everyone quickly learned that a very old change that predated almost everyone there established this unknown dependency. The point of this story is really about the unknowability of sufficiently complex legacy enterprise systems. All of the right processes and procedures won’t necessarily account for that seemingly inconsequential RPC call to an internal system implemented by a grizzled dev shortly before his retirement.
    - wbl 4y ago
      Those were the wrong procedures. If you were regularly rebooting systems left and right you'd learn quickly if things didn't come up.
- landemva 4y ago
  > incredibly surprising that an organization of that complexity could have survived as long as they did without a major incident I'm not surprised. FAA does not fly each plane. Government organizational complexity helps ensure the government organization survives through next round of Congressional appropriations. Org complexity + opaque oversight + 'safety' + 'homeland security' + taxpayer funded = playing around and more budget. The pilot is responsible for safety. Air travel has rules to avoid collisions (eastbound gets altitude levels different than westbound, pilots shall broadcast on known frequencies) and pilots have distributed intelligence to keep their flight safe. Yes, somehow there needs to be coordination of runway use. Many ways to provide reservations and queuing.
zhengyi13 4y ago
I read this incident as they in fact did have a DR strategy, they knew how to execute it, did so successfully, and still failed. The only thing I'm left wondering at this point is whether the corrupt data were a config, or state.
- krabizzwainch 4y ago
  I don't even know if I'd call this a disaster recovery fail. Depending on what they meant by "corruption", a roughly 6-8 hour turn around time is not awful for a database restore.

justinsaccount 4y ago

People like to think that the alternative to "move fast and break things" is "move slowly and not break things" but it's not, it's "move slowly, break things anyway, then take days to resolve the problem because you never learned how to move fast".

eckza 4y ago
That's just not true. Moving faster does not mean moving more effectively. Frantic activity should not be mistaken for progress. I'm OK with the FAA taking "days to resolve the problem" if it means that nobody dies. For what it's worth, my team and I are partial to "move carefully, and tend things".
- justinsaccount 4y ago
  "Moving faster" doesn't mean "frantic" either. Who said frantic?
  - haswell 4y ago
    Frantic is often a natural outcome of “moving faster” when the environment one is moving in is not conducive to that speed of movement. In my experience, this tendency towards frantic is multiplied the larger and more complex the organization and architecture becomes. The entire point of “move fast” in software circles is to leave behind the constraints of legacy tech and management practices in favor of building something “better”. In a mature org that grew up before these ideas were mainstream, maybe one or two teams can manage to move faster, but invariably they end up depending on other teams, who in turn depend on deeply ingrained and established company culture and procedures. We can talk about why those impediments are a Bad Thing, and I wouldn’t advise a consumer startup to adopt those methodologies in 2022, but there’s still the harsh reality that where they exist, “just move faster” doesn’t help much more than telling a depressed person to “just do cardio every day”. There’s often a lot of inner work that’s gotta happen to make way for the new. The only way I’ve seen this sort of work in a large org is when a brand new “emerging tech” group is spun up and given autonomy to work outside of the legacy norms. This is not perfect either, and seems much better for greenfield projects. When applied to deeply entrenched legacy systems, all of the problems mentioned above come to a head. This also creates a weird in/out group dynamic which tends to further stratify the old tech and widen the gap between the old practices and the new. In the context of this particular conversation though, I think the concept of “move fast” has lost all meaning and has little to offer for an org like the FAA.

berniedurfee 4y ago

Move fast and break things, unless people are involved. Please, don’t break people. People are non-fungible.

indymike 4y ago

Software falls into two buckets: stuff that directly injures and kills people when it fails, and everything else.