6 ms·

Related article on Space Shuttle Software Quality [0] Excerpt: "But how much work the software does is not what makes it remarkable. What makes it remarkable i

Related article on Space Shuttle Software Quality [0] Excerpt: "But how much work the software does is not what makes it remarkable. What makes it remarkable is how well the software works. This software never crashes. It never needs to be re-booted. This software is bug-free. It is perfect, as perfect as human beings have achieved. Consider these stats : the last three versions of the program — each 420,000 lines long-had just one error each. The last 11 versions of this software had a total of 17 errors. Commercial programs of equivalent complexity would have 5,000 errors." [0] https://archive.is/HX7n4 https://archive.is/HX7n4

deleted 2y ago
[deleted]
emerongi 2y ago
5000 / 17 ≈ 295. Is it a fair assumption to make that a commercial program of equivalent complexity would take 295x fewer man-hours?
- fleischhauf 2y ago
  without any knowledge of the other processes involved in space shuttle software development, I highly doubt that the correlation is that easy
- burkaman 2y ago
  No, I don't think so, 295x is a crazy high factor. The article says 260 people are involved, and let's generously say it took 20 years to write the software (the first mission was 10 years after the program started and it was around for a total of 40 years). Dividing by 295 means a commercial team of the same size could have done it in less than a month. Or with a 10x smaller team, about 8 months. I don't think either of those are plausible.
  - deleted 2y ago
    [deleted]
  - swiftcoder 2y ago
    We're well in mythical-man-month territory as you try and accelerate the timeline, though. Typical software companies can't coordinate a 260 person team quickly enough to even start a project of that size in a month.
- bigiain 2y ago
  As I skimmed that, the text wrapped at the hyphen in man-hours, and my brain autocompleted it to "295x fewer managers" - and it pretty much rings true...
fleischhauf 2y ago
I wonder about spaceX track record
cpf_au 2y ago
One of my all-time favourite articles. Amazing that something from the internet in 1996 is still accessible!
dpedu 2y ago
> Consider these stats : the last three versions of the program — each 420,000 lines long-had just one error each. What exactly do they mean by this? If each of the 3 versions had exactly one bug, isn't this just a weird way of saying the first 2 fixes either didn't work or introduced a new bug?
- daemonologist 2y ago
  Or maybe between one version and the next they only found one bug (there may have been bugs in the first version which weren't fixed until the third or later) - this seems more plausible to me since it's... rather difficult to count bugs until after you know about them. Of course now the greatness of the feat depends on how much testing there was between versions, but given that it was the shuttle there was probably a lot.
  - sqeaky 2y ago
    And then we find a new category of bug, consider how we ran millions of different programs for many billions of CPU hours on all of our x86 CPUs before we learned about Spectre and meltdown.
  - swiftcoder 2y ago
    Not just testing - line-by-line code review of the entire system by a panel of experts. Outside of aerospace/defence/nuclear this style of review is not very common.
    - Symbiote 2y ago
      I don't know about now, but software verification would also be used in consumer electronics. Fixing a bug in 10,000 washing machine control boards is very expensive when it entails sending a technician to every house to replace the circuit board.
      - swiftcoder 2y ago
        Yeah, I guess for devices that can't receive OTA updates that makes sense. Though I fear that segment of the market is rapidly shrinking - televisions have ubiquitous software update capabilities now, and even washing machines are increasingly internet connected. We didn't apply anywhere near that kind of quality control to smartphones or VR headsets. Once users are trained to install OTA updates to fix issues, most of the impetus for extreme quality control outside of the bootloader->OTA update path is gone
rqtwteye 2y ago
It would be interesting to see the NASA approach compared to how SpaceX does things. Considering that they have done manned missions they seem to have very similar requirements.
- dotnet00 2y ago
  It'd depend on what software is under consideration. IIRC the UI in Crew Dragon is using more contemporary stuff, Node.js I think. This is fine because they have redundancy, there's minimal crew control anyway, and there are manual overrides behind a panel below the screens. They have 3 relatively modern CPUs setup to run the same code and error check each other, such that if one has an issue, there's still redundancy when it's rebooting. The software controlling critical systems is probably closer to NASA-esque practices.
  - Onavo 2y ago
    Yeah them running a electron style browser in a box GUI was quite a shocker given how much HN bashed electron when it came out.
    - asynchronous 2y ago
      Can’t hate it if it’s working. They make cool stuff and haven’t had a Challenger incident yet, I wish them the best.
    - swiftcoder 2y ago
      Electron was "bashed" for the resource overhead on consumer PCs. That's not really relevant to an aerospace firm who can spec their hardware to match the exact resource requirements of their tech stack.
deterministic 2y ago
The problem with the Space Shuttle development methology is that it is insanely expensive and slow and still not 100% bug free. It is so expensive and slow that it would be much cheaper, faster, and actually safer to prove the software correct using modern proof assistants. Projects like seL4, CompCert etc. shows how it should be done.

deleted 2y ago

[deleted]

emerongi 2y ago

5000 / 17 ≈ 295. Is it a fair assumption to make that a commercial program of equivalent complexity would take 295x fewer man-hours?

fleischhauf 2y ago
without any knowledge of the other processes involved in space shuttle software development, I highly doubt that the correlation is that easy
burkaman 2y ago
No, I don't think so, 295x is a crazy high factor. The article says 260 people are involved, and let's generously say it took 20 years to write the software (the first mission was 10 years after the program started and it was around for a total of 40 years). Dividing by 295 means a commercial team of the same size could have done it in less than a month. Or with a 10x smaller team, about 8 months. I don't think either of those are plausible.
- deleted 2y ago
  [deleted]
- swiftcoder 2y ago
  We're well in mythical-man-month territory as you try and accelerate the timeline, though. Typical software companies can't coordinate a 260 person team quickly enough to even start a project of that size in a month.
bigiain 2y ago
As I skimmed that, the text wrapped at the hyphen in man-hours, and my brain autocompleted it to "295x fewer managers" - and it pretty much rings true...

fleischhauf 2y ago

I wonder about spaceX track record

cpf_au 2y ago

One of my all-time favourite articles. Amazing that something from the internet in 1996 is still accessible!

dpedu 2y ago

> Consider these stats : the last three versions of the program — each 420,000 lines long-had just one error each. What exactly do they mean by this? If each of the 3 versions had exactly one bug, isn't this just a weird way of saying the first 2 fixes either didn't work or introduced a new bug?

daemonologist 2y ago
Or maybe between one version and the next they only found one bug (there may have been bugs in the first version which weren't fixed until the third or later) - this seems more plausible to me since it's... rather difficult to count bugs until after you know about them. Of course now the greatness of the feat depends on how much testing there was between versions, but given that it was the shuttle there was probably a lot.
- sqeaky 2y ago
  And then we find a new category of bug, consider how we ran millions of different programs for many billions of CPU hours on all of our x86 CPUs before we learned about Spectre and meltdown.
- swiftcoder 2y ago
  Not just testing - line-by-line code review of the entire system by a panel of experts. Outside of aerospace/defence/nuclear this style of review is not very common.
  - Symbiote 2y ago
    I don't know about now, but software verification would also be used in consumer electronics. Fixing a bug in 10,000 washing machine control boards is very expensive when it entails sending a technician to every house to replace the circuit board.
    - swiftcoder 2y ago
      Yeah, I guess for devices that can't receive OTA updates that makes sense. Though I fear that segment of the market is rapidly shrinking - televisions have ubiquitous software update capabilities now, and even washing machines are increasingly internet connected. We didn't apply anywhere near that kind of quality control to smartphones or VR headsets. Once users are trained to install OTA updates to fix issues, most of the impetus for extreme quality control outside of the bootloader->OTA update path is gone

rqtwteye 2y ago

It would be interesting to see the NASA approach compared to how SpaceX does things. Considering that they have done manned missions they seem to have very similar requirements.

dotnet00 2y ago
It'd depend on what software is under consideration. IIRC the UI in Crew Dragon is using more contemporary stuff, Node.js I think. This is fine because they have redundancy, there's minimal crew control anyway, and there are manual overrides behind a panel below the screens. They have 3 relatively modern CPUs setup to run the same code and error check each other, such that if one has an issue, there's still redundancy when it's rebooting. The software controlling critical systems is probably closer to NASA-esque practices.
- Onavo 2y ago
  Yeah them running a electron style browser in a box GUI was quite a shocker given how much HN bashed electron when it came out.
  - asynchronous 2y ago
    Can’t hate it if it’s working. They make cool stuff and haven’t had a Challenger incident yet, I wish them the best.
  - swiftcoder 2y ago
    Electron was "bashed" for the resource overhead on consumer PCs. That's not really relevant to an aerospace firm who can spec their hardware to match the exact resource requirements of their tech stack.

deterministic 2y ago

The problem with the Space Shuttle development methology is that it is insanely expensive and slow and still not 100% bug free. It is so expensive and slow that it would be much cheaper, faster, and actually safer to prove the software correct using modern proof assistants. Projects like seL4, CompCert etc. shows how it should be done.