5 ms·

> this cost several millions to redeploy the (one-line) fix to actually check the weight from the sensor was less than the threshold Well maybe this is the oth

by yafbum 3y ago

> this cost several millions to redeploy the (one-line) fix to actually check the weight from the sensor was less than the threshold Well maybe this is the other, compounding problem. Engineering complex machines with such a high cost of bugfix deployment seems like a big issue. It's funny that as an industry we now know how to safely deploy software updates to hundreds of millions of phones, with security checks, signed firmwares, etc, but doing that on applications with a super high unit price tag seems out of reach... Or maybe, a few millions is like only a thousand hours of flying in jet fuel costs alone, not a big deal...

digging 3y ago
This makes no sense and is difficult to even respond to coherently. > It's funny that as an industry we now know how to safely deploy software updates to hundreds of millions of phones, with security checks, signed firmwares, etc, Either you're completely wrong, because we "as an industry" still push bugs and security flaws, or you're comparing two completely different things. > doing that on applications with a super high unit price tag seems out of reach... is true because of > a few millions is like only a thousand hours of flying in jet fuel costs alone like do you really think they spent millions pushing a line of code? or do you think it's just inherently expensive to fly a jet, and so doing it twice costs more?
- the_sleaze9 3y ago
  I would generally pass this comment by, but it's just so distastefully hostile because you totally missed the point. GP's comment was expressing sardonic disbelief that a modern jet wouldn't be able to receive remote software updates, considering it's so ubiquitous and reliable in other fields, even those with much, much lower costs. Not that developers don't release faults.
  - namaria 3y ago
    People tend to opine on systems engineering as if we had some sort of information superconductor connecting all minds involved. Systems are Hard and complex systems are Harder. Thinking of entire class of failures as 'solved' is kinda like talking about curing cancer. There isn't one thing called cancer, there's hundreds. There's no way to solve complex systems problems for good. Reality, technologies, tooling, people, language, everything changes all the time. And complex systems failure modes that happen today will happen forever.
  - digging 3y ago
    Ahh, then I did misread it entirely. Thanks for stopping by to call me out. It's still probably not a matter of capability... I wouldn't be so cavalier about software updates on my phone if it was holding me thousands of feet above the ground at the time.
    - jacquesm 3y ago
      I already commented on this elsewhere but I came across a company that did OTA updates on a control box in vehicles without checking if the vehicle was in motion or not. And it didn't even really surprise me, it was just one of those things that came up when prepping for that job from a risk assessment. They never even thought of it.
  - kayodelycaon 3y ago
    Remote software updates on military vehicles? Hasn't anyone seen the new Battlestar Galactica? :)
  - Balooga 3y ago
    Imagine remotely bricking a fleet of fighter jets. https://news.ycombinator.com/item?id=35983866 https://news.ycombinator.com/item?id=35983866
    - jacquesm 3y ago
      > Imagine remotely bricking a fleet of fighter jets. > https://news.ycombinator.com/item?id=35983866 https://news.ycombinator.com/item?id=35983866 That's about routers, was that the article you meant?
nvy 3y ago
>It's funny that as an industry we now know how to safely deploy software updates to hundreds of millions of phones, with security checks, signed firmwares, etc, but doing that on applications with a super high unit price tag seems out of reach A bunch of JavaScript dudebros yeeting code out into the ether is not at all comparable to deploying avionics software to a fighter jet. Give your head a shake.
- Two4 3y ago
  I don't think they're referring to dudebros' js, they're referring to systems software and the ability to deliver relatively secure updates over insecure channels. I've even delivered a signed firmware update to a microprocessor in a goddamn washing machine over UART. Why can't we do this for a jet?
  - uxp100 3y ago
    We don’t really know the context of this anecdote, but if you have to completely re-run your test plan on a real plane with real munitions for newly deployed software, which is a pretty good idea, then I could see it costing millions, even if the fix deployed in a minute.
  - nvy 3y ago
    >Why can't we do this for a jet? Well, because the software load of an aircraft is certified as part of the approved type design, for one. If you update the software it requires an engineering approval, because the risks inherent to operating an aircraft and the engineering that goes into mitigating those risks and making them acceptably safe are quite a bit more significant than a washing machine. What's more we're talking about stores clearance (i.e. releasing shit from the aircraft in flight). The attitude behind "Just write that function and flash the firmware" gets people killed.
    - yafbum 3y ago
      I'm not saying just write the function and flash the firmware, but it's not like the super rigid certification process doesn't have its nefarious side effects either. My experience is that the more expensive fixes are, the more humans are willing to turn a blind eye to problems or wish them away.
datadrivenangel 3y ago
a few million dollars works out to a surprisingly small amount of time when you add overhead. Call the bug fix a development team of 20 people taking 3 months end to end from bug discovery to fix deployment. You'll probably have twice that much people time again in project management and communication overhead (1:2 ratio of dev time to communication overhead is actually amazing in defense contexts). Assume total cost per person of 200k per year (after factoring in benefits, overhead, and pork), so 60 people * 3 months * $200k/12 months = 3,000,000 USD. It takes a lot of people to build an aircraft.
DavidVoid 3y ago
> Or maybe, a few millions is like only a thousand hours of flying in jet fuel costs alone, not a big deal... Pretty much tbh. For example, the development of the Saab JAS 39 Gripen (JAS-projektet) is the most expensive industrial project in modern Swedish history at a cost of 120+ billion SEK (11+ billion USD). It was also almost cancelled after a very public crash in central Stockholm at the 1993 Stockholm Water Festival [1]. A crash that should not have happened because the flight should not have been approved in the first place, because they weren't yet confident that they'd completely solved the Pilot-Induced Oscillation (PIO) related issues that wrecked the first prototype 4 years prior (with the same test pilot) [2]. It was basically a miracle that no one was killed or seriously hurt in the Stockholm crash, had the plane hit the nearby bridge or any of the other densely crowded areas then it would've been a very different story. [1] https://youtu.be/mkgShfxTzmo?t=122 https://youtu.be/mkgShfxTzmo?t=122 [2] https://www.youtube.com/watch?v=k6yVU_yYtEc https://www.youtube.com/watch?v=k6yVU_yYtEc

by yafbum 3y ago

digging 3y ago

This makes no sense and is difficult to even respond to coherently. > It's funny that as an industry we now know how to safely deploy software updates to hundreds of millions of phones, with security checks, signed firmwares, etc, Either you're completely wrong, because we "as an industry" still push bugs and security flaws, or you're comparing two completely different things. > doing that on applications with a super high unit price tag seems out of reach... is true because of > a few millions is like only a thousand hours of flying in jet fuel costs alone like do you really think they spent millions pushing a line of code? or do you think it's just inherently expensive to fly a jet, and so doing it twice costs more?

the_sleaze9 3y ago
I would generally pass this comment by, but it's just so distastefully hostile because you totally missed the point. GP's comment was expressing sardonic disbelief that a modern jet wouldn't be able to receive remote software updates, considering it's so ubiquitous and reliable in other fields, even those with much, much lower costs. Not that developers don't release faults.
- namaria 3y ago
  People tend to opine on systems engineering as if we had some sort of information superconductor connecting all minds involved. Systems are Hard and complex systems are Harder. Thinking of entire class of failures as 'solved' is kinda like talking about curing cancer. There isn't one thing called cancer, there's hundreds. There's no way to solve complex systems problems for good. Reality, technologies, tooling, people, language, everything changes all the time. And complex systems failure modes that happen today will happen forever.
- digging 3y ago
  Ahh, then I did misread it entirely. Thanks for stopping by to call me out. It's still probably not a matter of capability... I wouldn't be so cavalier about software updates on my phone if it was holding me thousands of feet above the ground at the time.
  - jacquesm 3y ago
    I already commented on this elsewhere but I came across a company that did OTA updates on a control box in vehicles without checking if the vehicle was in motion or not. And it didn't even really surprise me, it was just one of those things that came up when prepping for that job from a risk assessment. They never even thought of it.
- kayodelycaon 3y ago
  Remote software updates on military vehicles? Hasn't anyone seen the new Battlestar Galactica? :)
- Balooga 3y ago
  Imagine remotely bricking a fleet of fighter jets. https://news.ycombinator.com/item?id=35983866 https://news.ycombinator.com/item?id=35983866
  - jacquesm 3y ago
    > Imagine remotely bricking a fleet of fighter jets. > https://news.ycombinator.com/item?id=35983866 https://news.ycombinator.com/item?id=35983866 That's about routers, was that the article you meant?

nvy 3y ago

>It's funny that as an industry we now know how to safely deploy software updates to hundreds of millions of phones, with security checks, signed firmwares, etc, but doing that on applications with a super high unit price tag seems out of reach A bunch of JavaScript dudebros yeeting code out into the ether is not at all comparable to deploying avionics software to a fighter jet. Give your head a shake.

Two4 3y ago
I don't think they're referring to dudebros' js, they're referring to systems software and the ability to deliver relatively secure updates over insecure channels. I've even delivered a signed firmware update to a microprocessor in a goddamn washing machine over UART. Why can't we do this for a jet?
- uxp100 3y ago
  We don’t really know the context of this anecdote, but if you have to completely re-run your test plan on a real plane with real munitions for newly deployed software, which is a pretty good idea, then I could see it costing millions, even if the fix deployed in a minute.
- nvy 3y ago
  >Why can't we do this for a jet? Well, because the software load of an aircraft is certified as part of the approved type design, for one. If you update the software it requires an engineering approval, because the risks inherent to operating an aircraft and the engineering that goes into mitigating those risks and making them acceptably safe are quite a bit more significant than a washing machine. What's more we're talking about stores clearance (i.e. releasing shit from the aircraft in flight). The attitude behind "Just write that function and flash the firmware" gets people killed.
  - yafbum 3y ago
    I'm not saying just write the function and flash the firmware, but it's not like the super rigid certification process doesn't have its nefarious side effects either. My experience is that the more expensive fixes are, the more humans are willing to turn a blind eye to problems or wish them away.

datadrivenangel 3y ago

a few million dollars works out to a surprisingly small amount of time when you add overhead. Call the bug fix a development team of 20 people taking 3 months end to end from bug discovery to fix deployment. You'll probably have twice that much people time again in project management and communication overhead (1:2 ratio of dev time to communication overhead is actually amazing in defense contexts). Assume total cost per person of 200k per year (after factoring in benefits, overhead, and pork), so 60 people * 3 months * $200k/12 months = 3,000,000 USD. It takes a lot of people to build an aircraft.

DavidVoid 3y ago

> Or maybe, a few millions is like only a thousand hours of flying in jet fuel costs alone, not a big deal... Pretty much tbh. For example, the development of the Saab JAS 39 Gripen (JAS-projektet) is the most expensive industrial project in modern Swedish history at a cost of 120+ billion SEK (11+ billion USD). It was also almost cancelled after a very public crash in central Stockholm at the 1993 Stockholm Water Festival [1]. A crash that should not have happened because the flight should not have been approved in the first place, because they weren't yet confident that they'd completely solved the Pilot-Induced Oscillation (PIO) related issues that wrecked the first prototype 4 years prior (with the same test pilot) [2]. It was basically a miracle that no one was killed or seriously hurt in the Stockholm crash, had the plane hit the nearby bridge or any of the other densely crowded areas then it would've been a very different story. [1] https://youtu.be/mkgShfxTzmo?t=122 https://youtu.be/mkgShfxTzmo?t=122 [2] https://www.youtube.com/watch?v=k6yVU_yYtEc https://www.youtube.com/watch?v=k6yVU_yYtEc