4 ms·

Our contract with the Air Force required that we document guarantees from every parts vendor for the servers we built for them that they would keep making those

Our contract with the Air Force required that we document guarantees from every parts vendor for the servers we built for them that they would keep making those parts for at least a decade. They also demanded RHEL exclusively. It's a great example of how extreme stability is more important than any other question in a lot of business decisions.

tourmalinetaco 3y ago
The way my engineering professor put it was so: when even 5 minutes of off-time costs $5,000 of product you want to be sure your systems are stable.
- kijin 3y ago
  $5,000 is pocket change. When it comes to the Air Force, 5 minutes of downtime at the wrong time could cost you a country, or perhaps even a planet.
- BozeWolf 3y ago
  We are at 500k/hour for at least one system i know of at my current client (energy). Not sure what causes these expenses. It can be missing out on trading, or fines. There may be more of these systems :) Lots of redhat there.
  - gdprrrr 3y ago
    I've worked for an automotive company that calculates with 1 Million euros per hour. But that's for a just-in-time assembly line.
- andrewaylett 3y ago
  That's the right sort of ballpark for the systems my team maintains. We don't run RHEL. "Stability" in the RHEL sense is akin to ossification: you don't change anything, and then you can't change anything, and then you've got problems. The costs aren't necessarily as obvious as pursuing stability via dynamism, and in any case can often either be avoided by limiting the lifespan of the system as a whole or at least punted on to a successor. For military applications, the trade off is even more intense: you build things hoping that they'll never be used, but knowing that they need to work when called upon. Stuff that'll be on the front line tomorrow can have a very different set of lifecycle guarantees from stuff that's got a planned life of 25 years but which (from experience) you know will probably still be around in 50.
riku_iki 3y ago
> from every parts vendor for the servers we built for them that they would keep making those parts for at least a decade and do main vendors (amd, intel, nvidia, whoever build motherboards) provide guarantees that they continue production of specific version of product for the decade? > They also demanded RHEL exclusively. It's a great example of how extreme stability is there evidence that rhel is more stable than debian?
- PlutoIsAPlanet 3y ago
  > is there evidence that rhel is more stable than debian? Well, yes Debian versions are supported for 3 years, RHEL for 10+. "Stability" means changes, not crashes (although they can be caused by changes). A piece of software compiled 10 years ago will still run today and the systems can continue to get security updates, without needing to change the rug.
  - riku_iki 3y ago
    > Well, yes Debian versions are supported for 3 years, RHEL for 10+. if upgrade path is easy and stable, it is obviously more sustainable to upgrade versions every N years, compared to situation when you got 10yo EOL infra far behind mainline with no clear way to support it in the future.
    - asoneth 3y ago
      > it is obviously more sustainable to upgrade versions every N years I've heard this kind of thing before, but it's worth considering that utility, government, and infrastructure software may be under different kinds of constraints than what you're used to. Testing field upgrades takes years and the rollout often takes a year or more. There are hundreds of different teams involved and every one of them wants to ensure they don't miss the thing that bricks a submarine, satellite, transmission line, or IRS portal. If you can do this half or a quarter as often you can save taxpayers tens if not hundreds of millions of dollars. And in a surprising number of cases the software is never upgraded for the ~25 year or so lifetime of the hardware because the cost of the upgrade is comparable to the cost of buying new hardware. Sometimes that means some poor development team backports security patches for antique versions of DOS or Unix, but in more cases than you'd think they just airgap the computers and hope for the best.

tourmalinetaco 3y ago

The way my engineering professor put it was so: when even 5 minutes of off-time costs $5,000 of product you want to be sure your systems are stable.

kijin 3y ago
$5,000 is pocket change. When it comes to the Air Force, 5 minutes of downtime at the wrong time could cost you a country, or perhaps even a planet.
BozeWolf 3y ago
We are at 500k/hour for at least one system i know of at my current client (energy). Not sure what causes these expenses. It can be missing out on trading, or fines. There may be more of these systems :) Lots of redhat there.
- gdprrrr 3y ago
  I've worked for an automotive company that calculates with 1 Million euros per hour. But that's for a just-in-time assembly line.
andrewaylett 3y ago
That's the right sort of ballpark for the systems my team maintains. We don't run RHEL. "Stability" in the RHEL sense is akin to ossification: you don't change anything, and then you can't change anything, and then you've got problems. The costs aren't necessarily as obvious as pursuing stability via dynamism, and in any case can often either be avoided by limiting the lifespan of the system as a whole or at least punted on to a successor. For military applications, the trade off is even more intense: you build things hoping that they'll never be used, but knowing that they need to work when called upon. Stuff that'll be on the front line tomorrow can have a very different set of lifecycle guarantees from stuff that's got a planned life of 25 years but which (from experience) you know will probably still be around in 50.

riku_iki 3y ago

> from every parts vendor for the servers we built for them that they would keep making those parts for at least a decade and do main vendors (amd, intel, nvidia, whoever build motherboards) provide guarantees that they continue production of specific version of product for the decade? > They also demanded RHEL exclusively. It's a great example of how extreme stability is there evidence that rhel is more stable than debian?

PlutoIsAPlanet 3y ago
> is there evidence that rhel is more stable than debian? Well, yes Debian versions are supported for 3 years, RHEL for 10+. "Stability" means changes, not crashes (although they can be caused by changes). A piece of software compiled 10 years ago will still run today and the systems can continue to get security updates, without needing to change the rug.
- riku_iki 3y ago
  > Well, yes Debian versions are supported for 3 years, RHEL for 10+. if upgrade path is easy and stable, it is obviously more sustainable to upgrade versions every N years, compared to situation when you got 10yo EOL infra far behind mainline with no clear way to support it in the future.
  - asoneth 3y ago
    > it is obviously more sustainable to upgrade versions every N years I've heard this kind of thing before, but it's worth considering that utility, government, and infrastructure software may be under different kinds of constraints than what you're used to. Testing field upgrades takes years and the rollout often takes a year or more. There are hundreds of different teams involved and every one of them wants to ensure they don't miss the thing that bricks a submarine, satellite, transmission line, or IRS portal. If you can do this half or a quarter as often you can save taxpayers tens if not hundreds of millions of dollars. And in a surprising number of cases the software is never upgraded for the ~25 year or so lifetime of the hardware because the cost of the upgrade is comparable to the cost of buying new hardware. Sometimes that means some poor development team backports security patches for antique versions of DOS or Unix, but in more cases than you'd think they just airgap the computers and hope for the best.