9 ms·

I worked on the control systems for Predators and Reapers back in the mid and late 00s, and the inefficiencies around process were enormous. Safety is extremel

I worked on the control systems for Predators and Reapers back in the mid and late 00s, and the inefficiencies around process were enormous. Safety is extremely important, so you expect some slowness as a result, but it got pretty extreme. I remember one time having to do 6 weeks of testing around a one-line code change because a "helpful" dev fixed a small bug that had no practical impact. Yet because it changed the release build hash, we had to go through a full acceptance test. As you can imagine that incentivized only fixing important bugs, and even those we had to consider whether it was worth it or not. As a result there were a hole pile of bugs that we (and customers) ended up just living with. On a separate note, I'm curious as to whether AI is making an inroads in that space. I would imagine very minimal, if at all, but very curious.

hvb2 3mo ago
Why would those fixes not be batched up? So fix 20% of those and do one round of testing?
- bad_haircut72 3mo ago
  The obvious answer is that the more bugs you batch up, the higher the chances the next build fails - this is why CI became a thing, small iterative changes are safer and lead to greater throughput
  - dataflow 3mo ago
    CI doesn't mean doing all the tests all the time though. The expensive tests still wait until there's a major reason to run them. I had the same question as the parent and I still don't quite see why this can't work.
    - aidenn0 3mo ago
      I didn't work on this project, but I've been involved with similar ones. There is a process for getting a change into version control. Each change needs to have a (virtual) paper trail: motivation, risk analysis, sign-offs &c. If you can't get something into VC quickly, you can't really do CI. The obvious solution would be to have an integration branch that doesn't need the process to get in, do CI testing on that branch and then make the process for merging to the real branch. I've never seen this done personally, but I have been told some places do it, and then you end up with "Change X, which got approved had a dependency on Change Y that didn't get approved and we didn't realize it until now because Change Y was put in the integration branch before Change X"
      - dataflow 3mo ago
        Well, approval is a different beast from passing tests. But also, that's not how I was imagining this. I was imagining maintaining separate branches on top of each release, only combining them (merge or rebase or whatever) when you have a good reason to. That keeps things independent and makes it so you can always cut a release with solely the critical fixes (and test them in isolation, etc.) whenever needed, letting you integrate the noncritical ones opportunistically.
      - HeyLaughingBoy 3mo ago
        That sounds odd. The way it was done in every place I worked, is that a set of changes were approved for a release before they were planned and implemented. We organized the work as expected: each bug/feature on its own branch, with its own set of unit tests, etc., and automated testing applied on each commit. These branches are then merged to the integration branch once they are known good. Before the release process starts, QA would get a copy of the integration branch and test that. The dependency problem doesn't exist, because all the features were already approved to be in the release. The only way there would be a problem is if someone decided late in the game to pull a feature and that feature was a dependency to something else.
        aidenn0 3mo ago
        What I described was for bugfixes, not features. Features were set in stone way before this. I also maintain that it is impossible to know which changes depend on other changes. In one case, applying a bugfix that changed the order of allocations at startup caused vtable corruption somewhere else because it changed how much padding a particular malloc() call was returning, and someone was writing past the end of their allocation. [edit] Also note that what you described is not CI; things are developed on their own branches and not integrated immediately.
        im3w1l 3mo ago
        It's may seem like a nitpick but I think it's a whole different way off looking at it to phrase it as "applying a bugfix that changed the order of allocations at startup exposed memory corruption somewhere else" I do think there is something to be said for both perspectives, especially for code that is extremely critical. With sufficient testing and determinism maybe you can actually make sure that dormant issues stay dormant meaning there is real value in being change-averse. Still it's a very precarious situation having a known memory corruption hoping it's the testing has made sure its benign in practice.
  - Manuel_D 3mo ago
    True, but if each run of CI takes 6 weeks then you're going to vastly hamper development.
tehjoker 3mo ago
I don't really understand how any of this contributes to "defense". Sounds like "offense" to me. Just patrolling the skies over non-white countries and launching missiles at weddings. The reason the Pentagon invests so heavily in this kind of technology is our wars are so indefensible, they can't convince Americans to sacrifice blood in any quantity for other people's natural resources.
- junofan 3mo ago
  They are white [1]. [1]: https://www.documentcloud.org/documents/21211671-1997-revisions-to-omb-statistical-policy-directive-no-15-race-and-ethnic-standards-for-federal-statistics-and-administrative-reporting/#document/p19/a2081999 https://www.documentcloud.org/documents/21211671-1997-revisi...
  - graemep 3mo ago
    They certainly look white to me. Are Spanish people white or Hispanic according to those definitions? 'Original peoples' is an interesting phrase. Neanderthals? Beaker people? A single category for everyone from Pakistanis to Japanese is weird.
    - ivell 3mo ago
      I think they meant Caucasian. https://en.wikipedia.org/wiki/Caucasian_race https://en.wikipedia.org/wiki/Caucasian_race
      - xorcist 3mo ago
        > Introduced in the 1780s by members of the Göttingen school of history, the term denoted one of three purported major races of humans: Caucasoid, Mongoloid, and Negroid Sounds totally legit.
      - graemep 3mo ago
        Caucasian includes South Asians and some East Africans according to that definition.
pjc50 3mo ago
cf the other thread: https://news.ycombinator.com/item?id=48845442 https://news.ycombinator.com/item?id=48845442 ; Ukraine has a hugely inventive and effective drone industry because it has to work. If it doesn't succeed, there is no Ukraine, and everyone involved in making the drones is dead, fled, in a POW camp, or sucked into the internal Russian displacement system away from their family. By comparison, if the US products fail, there's no real negative effect on the mainland United States.
- barrenko 3mo ago
  To add that Ukraine was also USSR's drone research center; not to detract from what you are saying.
- NooneAtAll3 3mo ago
  [flagged]
- rayiner 3mo ago
  Yeah, their tech development in drones is really impressive. An invading army has a way of focusing the mind and bureaucracy.
- letitgo12345 3mo ago
  Well there's long term impact but yea that doesn't create enough political pressure to make process efficient
- 27183 3mo ago
  > if the US products fail, there's no real negative effect on the mainland United States It's even worse than that. Schedules slipping and cost overruns are good things for the manufacturer, because they can charge more on top of their initial contract. Cost-plus ftw.
  - Retric 3mo ago
    Cost plus isn’t nearly as common as it used to be. But you still run into similar issues regardless of the contract structure. Try and build a rail network without anyone in government wanting something changed from the initial design for 20 years.
pixl97 3mo ago
Life critical software that gets visibility by congress tends to be a very bureaucratic process. Your boss doesn't want your commit being the one that causes a worldwide diplomatic issue. I assume that smaller/cheaper drones avoid a lot of this because the stakes aren't near as high and quite a bit of the development occurs in private industry first.
- matwood 3mo ago
  > visibility by congress tends to be a very bureaucratic process See also SpaceX vs. NASA. No way would NASA have been allowed to blow up as many rockets as SpaceX did to finally get to their working solution.
  - deleted 3mo ago
    [deleted]
  - throwaway894345 3mo ago
    Yeah, the anti-regulation people when NASA experiments: "look at all these failures! Cut NASA funding and give public funds to the guy who purchases elections!" The same people when SpaceX blows up a bunch of rockets: "wow, look at the innovation, they move so fast! Cut NASA funding and give public funds to the guy who purchases elections!"
    - ACCount37 3mo ago
      NASA's failures as of late are less "dramatic explosions" and more "delays", "cost overruns" and "lack of ambition so severe it borders on criminal". The last time NASA caught any serious flak was what, the Starliner shitshow? And that was just splash damage from Boeing getting dunked on by everyone at once.
      - throwaway894345 3mo ago
        > lack of ambition so severe it borders on criminal I'm not sure what timeline you're thinking about, but JWST was launched pretty recently and it's pretty ambitious. But more to the point of my earlier post, NASA's "lack of ambition" is probably directly attributable to the "small government" people who penalize ambition in the public sector and praise it in the private, government contractor sector. The incentives to be ambitious in government are perverse when every "failure" is scrutinized and condemned by people who want government to fail so that they can justify taking public money and dumping it into private bank accounts.
general1465 3mo ago
The bureaucratic development process sounds like Autosar in automotive. I am not surprised that newcomers from USA and Chinese auto companies are able to completely dominate in software because Autosar based development has been like giving a birth to a hedgehog. Slow and painful.
cm2012 3mo ago
This is why Ukraine is making equivalent tech now for 1/10th the price. It's great to see.
- yonaguska 3mo ago
  All the drone footage is actually pretty horrific to see.
  - kilroy123 3mo ago
    I agree. Nightmare fuel.
  - meindnoch 3mo ago
    Depends on who's on the receiving end.
  - floatrock 3mo ago
    It's always been pretty horrific. We just didn't have bodycam livestreams invented during trench warfare times.
  - LgWoodenBadger 3mo ago
    Hopefully making the real horrors of war visible to the public/world will make war less frequent. Alas...
    - disgruntledphd2 3mo ago
      People have been saying this since Vietnam (and actually now that I think about a little more, this may actually have happened).
      - randcraw 3mo ago
        In the late 1960s daily TV news coverage revealed the horrors of war to the American public as never before (just as Brady's photos had done in the Civil War). That palpable gruesomeness was instrumental in the rising tide of unpopularity enjoyed by the war (and Johnson) by the elections of 1968.
varispeed 3mo ago
If safety is extremely important, why there were bugs in the first place? Surely these should have been caught before code would get into main, no?
deleted 3mo ago
[deleted]
mmooss 3mo ago
> one time having to do 6 weeks of testing around a one-line code change because a "helpful" dev fixed a small bug that had no practical impact Roll back the change? Also, fix the approval process - no way that should have been approved. Generally speaking that is risk management, an unavoidable engineering tradeoff. In lower stakes situations, for example a critical application or server for a small office, we let low-impact bugs accumulate: Imposing risks, and therefore eventual costs, to avoid minor workarounds and low-impact bugs is poor engineering and risk management. Engineering and all risk management includes tradeoffs. It's easy to criticize the downside of the tradeoff - the same people criticize the reverse decision when the server (or drone) crashes - when someone is not responsible for both sides of it, when they are not accountable for their words when the outcome occurs. That's speaking generally. It's also poor risk management to be overly safe. I don't know about the parents' situation. But drone crashes (risking humans), mission failure, $50 million losses, and associated downtime (including delays) and labor costs, seem like high costs that are worth some pain to avoid.
legitster 3mo ago
This is one of the reasons weapons and technology development overall explodes during wartime. Desperation is the cure for risk aversion. It's also a reason to be skeptical of a military spending a bunch of money developing technology during peacetime. In reality the expensive stuff they went into the war with is always going to be less effective than the cheap stuff they came out with.
- throwaway894345 3mo ago
  Maybe if you're in a war that actually threatens your country. In the US, the Republican Party wages wars (such as the Iran War) pretty exclusively to facilitate borrowing public money and dumping it into the pockets of the rich. What looks like waste to the taxpayer is a feature to Republicans and their paymasters (including Israel, in the case of the Iran War).
Jtsummers 3mo ago
> As you can imagine that incentivized only fixing important bugs, and even those we had to consider whether it was worth it or not. Or you're batching your releases into larger builds because you know it'll take 6 weeks to test regardless. This increases the duration of each development iteration because you have 100 things you want to do and you could do that in, say, 4x13 week efforts, but with the added 6 weeks between iterations (and possibly more after it leaves your shop) that takes a one year effort and turns it into about 1.5. So the program office decides you should do one big release each year, which also ups the risk because a lot of testing that would catch bugs isn't done until the end in that big 6-week test effort. Oops, now your 1 year + 6 week effort just got turned into 1 year + 6 week + (unknown rework time) + 6 weeks. Probably 2 years.
- supriyo-biswas 3mo ago
  My last job was like this, a full round of QAs manually testing, along with writing up a grandiose release document, stakeholder approvals, and whatnot, for something that should just take two days of development work, and an insistence on putting configuration values into the database, supposedly because it's "safer" than deploying a new configuration file.
  - rixed 3mo ago
    If you are in an environment where deploying a new configuration file is easier than adding data in a pre-existing database, you are in a bad place already.
LgWoodenBadger 3mo ago
Surely the dev wasn't able to merge that one-line code fix causing 6 weeks of testing without any other eyes on it and without someone else's PR-like approval...right?
stickfigure 3mo ago
"Software saved the aerospace industry. Every other way of adding cost to an aircraft also adds weight."
HarHarVeryFunny 3mo ago
In contrast I read that Ukraine is approving 4+ new weapons systems PER DAY !! Even when it comes to more expensive things like cruise missiles it seems the planning has to be that some high percentage of them may be shot down (and much higher for slower moving drones), so you really want them cheap and in high volume, with reliability somewhat of a secondary concern.
pc86 3mo ago
This only makes sense if you have to test each fix in complete isolation which seems silly even for government employees and contractor body shops. You can't batch 80 real bug fixes and 20 "silly bug with no practical impact" fixes together?
ACCount37 3mo ago
In Ukraine, the first place that was bombed was the red tape factory. The drone industry was allowed to basically "do whatever as long as it works", consequences be damned. So they use civilian motors, batteries and SoCs, sketchy firmware with zero code inspection, and more. Does it work perfectly? No. It works well enough. I wonder if anyone is going to learn a lesson about overregulation. I'm not sure if "AI for red tape mitigation" is a thing, but "AI for killer drones" sure is. I suspect that "killer drones are insufficiently smart" is easier to fix with AI than "too much red tape". Because the amount of red tape, if unopposed, will expand to consume any capacity of dealing with it, AI or not.
- deleted 3mo ago
  [deleted]
- lokar 3mo ago
  I don’t know the details of that situation, but I have been on the other side of that debate. People say “it’s a one line change” (once they argued it was a 1 bit change!). But lacking a fully controlled and hermetic build system with its own exhaustive test suite you can’t be sure about the relationship between the source and the binary. And that continues to every step to get the binary into production (updating existing devices, etc).
  - ACCount37 3mo ago
    Welcome to Tradeoff Town. Sure, your ultra paranoid checking of everything might catch an extremely rare bug caused by something like interactions between a benign code change and a build system. But is it worth slowing down the development process by that much? Is it worth missing out on an entire generation of technology, like what happened with US and the shift from 00s drone warfare and 20s drone warfare? Usually not.
    - lokar 3mo ago
      It depends. For a JS web front end? Go crazy. For the consensus algorithm in your distributed database? slow down there buddy... You have to ask: how bad could this get? How hard will it be to fix? Could we loose or corrupt user data?
deleted 3mo ago
[deleted]
giardini 3mo ago
a "hole pile" of ...?! I'm surprised that someone who uses such a phrase was working on classified hardware in the "mid and late 00s".
- freedomben 3mo ago
  for sure, no way even an intelligent person could possibly make a text-to-speech typo when writing a comment on an informal internet forum. Definitely a lying liar who lies
giantg2 3mo ago
This just seems like poor planning. If it were planned better, you could group a bunch of smaller bug fixes under the same acceptance test. The full acceptance test is a good practice for safety critical systems. Project management just has to get better at doing smaller unit/int testing on each bug and then grouping those changes under the full acceptance test.
FireBeyond 3mo ago
> Safety is extremely important, so you expect some slowness as a result, but it got pretty extreme. I remember one time having to do 6 weeks of testing around a one-line code change because a "helpful" dev fixed a small bug that had no practical impact. Not to bring Tesla into this, but the contrast here is stunning. From a component manufacturer about the mindset of Tesla: "Hey, we sent you over the new firmware for the component, check it out." (The test suite for this component takes approximately 36 hours to execute.) Three hours later: "This is working so much better, thanks a lot!" "???" "Oh, we just flashed a car we have here and took it out for a drive." "?!?" Oof.
deleted 3mo ago
[deleted]
YeGoblynQueenne 3mo ago
>> On a separate note, I'm curious as to whether AI is making an inroads in that space. I would imagine very minimal, if at all, but very curious. tl;dr, not in practice. Far as I can tell this is still the SOTA in FPOV drone autonomy: https://spectrum.ieee.org/ai-drone-racing https://spectrum.ieee.org/ai-drone-racing Basically, those guys trained a drone to beat human FPOV racers but they had access to the race course layout before the race and trained extensively on it. Can't do that on a battlefield. Also: no shooting at enemy drones during FPOV races. Makes surviving to the end of the race simpler.

hvb2 3mo ago

Why would those fixes not be batched up? So fix 20% of those and do one round of testing?

bad_haircut72 3mo ago
The obvious answer is that the more bugs you batch up, the higher the chances the next build fails - this is why CI became a thing, small iterative changes are safer and lead to greater throughput
- dataflow 3mo ago
  CI doesn't mean doing all the tests all the time though. The expensive tests still wait until there's a major reason to run them. I had the same question as the parent and I still don't quite see why this can't work.
  - aidenn0 3mo ago
    I didn't work on this project, but I've been involved with similar ones. There is a process for getting a change into version control. Each change needs to have a (virtual) paper trail: motivation, risk analysis, sign-offs &c. If you can't get something into VC quickly, you can't really do CI. The obvious solution would be to have an integration branch that doesn't need the process to get in, do CI testing on that branch and then make the process for merging to the real branch. I've never seen this done personally, but I have been told some places do it, and then you end up with "Change X, which got approved had a dependency on Change Y that didn't get approved and we didn't realize it until now because Change Y was put in the integration branch before Change X"
    - dataflow 3mo ago
      Well, approval is a different beast from passing tests. But also, that's not how I was imagining this. I was imagining maintaining separate branches on top of each release, only combining them (merge or rebase or whatever) when you have a good reason to. That keeps things independent and makes it so you can always cut a release with solely the critical fixes (and test them in isolation, etc.) whenever needed, letting you integrate the noncritical ones opportunistically.
    - HeyLaughingBoy 3mo ago
      That sounds odd. The way it was done in every place I worked, is that a set of changes were approved for a release before they were planned and implemented. We organized the work as expected: each bug/feature on its own branch, with its own set of unit tests, etc., and automated testing applied on each commit. These branches are then merged to the integration branch once they are known good. Before the release process starts, QA would get a copy of the integration branch and test that. The dependency problem doesn't exist, because all the features were already approved to be in the release. The only way there would be a problem is if someone decided late in the game to pull a feature and that feature was a dependency to something else.
      - aidenn0 3mo ago
        What I described was for bugfixes, not features. Features were set in stone way before this. I also maintain that it is impossible to know which changes depend on other changes. In one case, applying a bugfix that changed the order of allocations at startup caused vtable corruption somewhere else because it changed how much padding a particular malloc() call was returning, and someone was writing past the end of their allocation. [edit] Also note that what you described is not CI; things are developed on their own branches and not integrated immediately.
        im3w1l 3mo ago
        It's may seem like a nitpick but I think it's a whole different way off looking at it to phrase it as "applying a bugfix that changed the order of allocations at startup exposed memory corruption somewhere else" I do think there is something to be said for both perspectives, especially for code that is extremely critical. With sufficient testing and determinism maybe you can actually make sure that dormant issues stay dormant meaning there is real value in being change-averse. Still it's a very precarious situation having a known memory corruption hoping it's the testing has made sure its benign in practice.
- Manuel_D 3mo ago
  True, but if each run of CI takes 6 weeks then you're going to vastly hamper development.

tehjoker 3mo ago

I don't really understand how any of this contributes to "defense". Sounds like "offense" to me. Just patrolling the skies over non-white countries and launching missiles at weddings. The reason the Pentagon invests so heavily in this kind of technology is our wars are so indefensible, they can't convince Americans to sacrifice blood in any quantity for other people's natural resources.

junofan 3mo ago
They are white [1]. [1]: https://www.documentcloud.org/documents/21211671-1997-revisions-to-omb-statistical-policy-directive-no-15-race-and-ethnic-standards-for-federal-statistics-and-administrative-reporting/#document/p19/a2081999 https://www.documentcloud.org/documents/21211671-1997-revisi...
- graemep 3mo ago
  They certainly look white to me. Are Spanish people white or Hispanic according to those definitions? 'Original peoples' is an interesting phrase. Neanderthals? Beaker people? A single category for everyone from Pakistanis to Japanese is weird.
  - ivell 3mo ago
    I think they meant Caucasian. https://en.wikipedia.org/wiki/Caucasian_race https://en.wikipedia.org/wiki/Caucasian_race
    - xorcist 3mo ago
      > Introduced in the 1780s by members of the Göttingen school of history, the term denoted one of three purported major races of humans: Caucasoid, Mongoloid, and Negroid Sounds totally legit.
    - graemep 3mo ago
      Caucasian includes South Asians and some East Africans according to that definition.

pjc50 3mo ago

cf the other thread: https://news.ycombinator.com/item?id=48845442 https://news.ycombinator.com/item?id=48845442 ; Ukraine has a hugely inventive and effective drone industry because it has to work. If it doesn't succeed, there is no Ukraine, and everyone involved in making the drones is dead, fled, in a POW camp, or sucked into the internal Russian displacement system away from their family. By comparison, if the US products fail, there's no real negative effect on the mainland United States.

barrenko 3mo ago
To add that Ukraine was also USSR's drone research center; not to detract from what you are saying.
NooneAtAll3 3mo ago
[flagged]
rayiner 3mo ago
Yeah, their tech development in drones is really impressive. An invading army has a way of focusing the mind and bureaucracy.
letitgo12345 3mo ago
Well there's long term impact but yea that doesn't create enough political pressure to make process efficient
27183 3mo ago
> if the US products fail, there's no real negative effect on the mainland United States It's even worse than that. Schedules slipping and cost overruns are good things for the manufacturer, because they can charge more on top of their initial contract. Cost-plus ftw.
- Retric 3mo ago
  Cost plus isn’t nearly as common as it used to be. But you still run into similar issues regardless of the contract structure. Try and build a rail network without anyone in government wanting something changed from the initial design for 20 years.

pixl97 3mo ago

Life critical software that gets visibility by congress tends to be a very bureaucratic process. Your boss doesn't want your commit being the one that causes a worldwide diplomatic issue. I assume that smaller/cheaper drones avoid a lot of this because the stakes aren't near as high and quite a bit of the development occurs in private industry first.

matwood 3mo ago
> visibility by congress tends to be a very bureaucratic process See also SpaceX vs. NASA. No way would NASA have been allowed to blow up as many rockets as SpaceX did to finally get to their working solution.
- deleted 3mo ago
  [deleted]
- throwaway894345 3mo ago
  Yeah, the anti-regulation people when NASA experiments: "look at all these failures! Cut NASA funding and give public funds to the guy who purchases elections!" The same people when SpaceX blows up a bunch of rockets: "wow, look at the innovation, they move so fast! Cut NASA funding and give public funds to the guy who purchases elections!"
  - ACCount37 3mo ago
    NASA's failures as of late are less "dramatic explosions" and more "delays", "cost overruns" and "lack of ambition so severe it borders on criminal". The last time NASA caught any serious flak was what, the Starliner shitshow? And that was just splash damage from Boeing getting dunked on by everyone at once.
    - throwaway894345 3mo ago
      > lack of ambition so severe it borders on criminal I'm not sure what timeline you're thinking about, but JWST was launched pretty recently and it's pretty ambitious. But more to the point of my earlier post, NASA's "lack of ambition" is probably directly attributable to the "small government" people who penalize ambition in the public sector and praise it in the private, government contractor sector. The incentives to be ambitious in government are perverse when every "failure" is scrutinized and condemned by people who want government to fail so that they can justify taking public money and dumping it into private bank accounts.

general1465 3mo ago

The bureaucratic development process sounds like Autosar in automotive. I am not surprised that newcomers from USA and Chinese auto companies are able to completely dominate in software because Autosar based development has been like giving a birth to a hedgehog. Slow and painful.

cm2012 3mo ago

This is why Ukraine is making equivalent tech now for 1/10th the price. It's great to see.

yonaguska 3mo ago
All the drone footage is actually pretty horrific to see.
- kilroy123 3mo ago
  I agree. Nightmare fuel.
- meindnoch 3mo ago
  Depends on who's on the receiving end.
- floatrock 3mo ago
  It's always been pretty horrific. We just didn't have bodycam livestreams invented during trench warfare times.
- LgWoodenBadger 3mo ago
  Hopefully making the real horrors of war visible to the public/world will make war less frequent. Alas...
  - disgruntledphd2 3mo ago
    People have been saying this since Vietnam (and actually now that I think about a little more, this may actually have happened).
    - randcraw 3mo ago
      In the late 1960s daily TV news coverage revealed the horrors of war to the American public as never before (just as Brady's photos had done in the Civil War). That palpable gruesomeness was instrumental in the rising tide of unpopularity enjoyed by the war (and Johnson) by the elections of 1968.

varispeed 3mo ago

If safety is extremely important, why there were bugs in the first place? Surely these should have been caught before code would get into main, no?

deleted 3mo ago

[deleted]

mmooss 3mo ago

> one time having to do 6 weeks of testing around a one-line code change because a "helpful" dev fixed a small bug that had no practical impact Roll back the change? Also, fix the approval process - no way that should have been approved. Generally speaking that is risk management, an unavoidable engineering tradeoff. In lower stakes situations, for example a critical application or server for a small office, we let low-impact bugs accumulate: Imposing risks, and therefore eventual costs, to avoid minor workarounds and low-impact bugs is poor engineering and risk management. Engineering and all risk management includes tradeoffs. It's easy to criticize the downside of the tradeoff - the same people criticize the reverse decision when the server (or drone) crashes - when someone is not responsible for both sides of it, when they are not accountable for their words when the outcome occurs. That's speaking generally. It's also poor risk management to be overly safe. I don't know about the parents' situation. But drone crashes (risking humans), mission failure, $50 million losses, and associated downtime (including delays) and labor costs, seem like high costs that are worth some pain to avoid.

legitster 3mo ago

This is one of the reasons weapons and technology development overall explodes during wartime. Desperation is the cure for risk aversion. It's also a reason to be skeptical of a military spending a bunch of money developing technology during peacetime. In reality the expensive stuff they went into the war with is always going to be less effective than the cheap stuff they came out with.

throwaway894345 3mo ago
Maybe if you're in a war that actually threatens your country. In the US, the Republican Party wages wars (such as the Iran War) pretty exclusively to facilitate borrowing public money and dumping it into the pockets of the rich. What looks like waste to the taxpayer is a feature to Republicans and their paymasters (including Israel, in the case of the Iran War).

Jtsummers 3mo ago

> As you can imagine that incentivized only fixing important bugs, and even those we had to consider whether it was worth it or not. Or you're batching your releases into larger builds because you know it'll take 6 weeks to test regardless. This increases the duration of each development iteration because you have 100 things you want to do and you could do that in, say, 4x13 week efforts, but with the added 6 weeks between iterations (and possibly more after it leaves your shop) that takes a one year effort and turns it into about 1.5. So the program office decides you should do one big release each year, which also ups the risk because a lot of testing that would catch bugs isn't done until the end in that big 6-week test effort. Oops, now your 1 year + 6 week effort just got turned into 1 year + 6 week + (unknown rework time) + 6 weeks. Probably 2 years.

supriyo-biswas 3mo ago
My last job was like this, a full round of QAs manually testing, along with writing up a grandiose release document, stakeholder approvals, and whatnot, for something that should just take two days of development work, and an insistence on putting configuration values into the database, supposedly because it's "safer" than deploying a new configuration file.
- rixed 3mo ago
  If you are in an environment where deploying a new configuration file is easier than adding data in a pre-existing database, you are in a bad place already.

LgWoodenBadger 3mo ago

Surely the dev wasn't able to merge that one-line code fix causing 6 weeks of testing without any other eyes on it and without someone else's PR-like approval...right?

stickfigure 3mo ago

"Software saved the aerospace industry. Every other way of adding cost to an aircraft also adds weight."

HarHarVeryFunny 3mo ago

In contrast I read that Ukraine is approving 4+ new weapons systems PER DAY !! Even when it comes to more expensive things like cruise missiles it seems the planning has to be that some high percentage of them may be shot down (and much higher for slower moving drones), so you really want them cheap and in high volume, with reliability somewhat of a secondary concern.

pc86 3mo ago

This only makes sense if you have to test each fix in complete isolation which seems silly even for government employees and contractor body shops. You can't batch 80 real bug fixes and 20 "silly bug with no practical impact" fixes together?

ACCount37 3mo ago

In Ukraine, the first place that was bombed was the red tape factory. The drone industry was allowed to basically "do whatever as long as it works", consequences be damned. So they use civilian motors, batteries and SoCs, sketchy firmware with zero code inspection, and more. Does it work perfectly? No. It works well enough. I wonder if anyone is going to learn a lesson about overregulation. I'm not sure if "AI for red tape mitigation" is a thing, but "AI for killer drones" sure is. I suspect that "killer drones are insufficiently smart" is easier to fix with AI than "too much red tape". Because the amount of red tape, if unopposed, will expand to consume any capacity of dealing with it, AI or not.

deleted 3mo ago
[deleted]
lokar 3mo ago
I don’t know the details of that situation, but I have been on the other side of that debate. People say “it’s a one line change” (once they argued it was a 1 bit change!). But lacking a fully controlled and hermetic build system with its own exhaustive test suite you can’t be sure about the relationship between the source and the binary. And that continues to every step to get the binary into production (updating existing devices, etc).
- ACCount37 3mo ago
  Welcome to Tradeoff Town. Sure, your ultra paranoid checking of everything might catch an extremely rare bug caused by something like interactions between a benign code change and a build system. But is it worth slowing down the development process by that much? Is it worth missing out on an entire generation of technology, like what happened with US and the shift from 00s drone warfare and 20s drone warfare? Usually not.
  - lokar 3mo ago
    It depends. For a JS web front end? Go crazy. For the consensus algorithm in your distributed database? slow down there buddy... You have to ask: how bad could this get? How hard will it be to fix? Could we loose or corrupt user data?

deleted 3mo ago

[deleted]

giardini 3mo ago

a "hole pile" of ...?! I'm surprised that someone who uses such a phrase was working on classified hardware in the "mid and late 00s".

freedomben 3mo ago
for sure, no way even an intelligent person could possibly make a text-to-speech typo when writing a comment on an informal internet forum. Definitely a lying liar who lies

giantg2 3mo ago

This just seems like poor planning. If it were planned better, you could group a bunch of smaller bug fixes under the same acceptance test. The full acceptance test is a good practice for safety critical systems. Project management just has to get better at doing smaller unit/int testing on each bug and then grouping those changes under the full acceptance test.

FireBeyond 3mo ago

> Safety is extremely important, so you expect some slowness as a result, but it got pretty extreme. I remember one time having to do 6 weeks of testing around a one-line code change because a "helpful" dev fixed a small bug that had no practical impact. Not to bring Tesla into this, but the contrast here is stunning. From a component manufacturer about the mindset of Tesla: "Hey, we sent you over the new firmware for the component, check it out." (The test suite for this component takes approximately 36 hours to execute.) Three hours later: "This is working so much better, thanks a lot!" "???" "Oh, we just flashed a car we have here and took it out for a drive." "?!?" Oof.

deleted 3mo ago

[deleted]

YeGoblynQueenne 3mo ago

>> On a separate note, I'm curious as to whether AI is making an inroads in that space. I would imagine very minimal, if at all, but very curious. tl;dr, not in practice. Far as I can tell this is still the SOTA in FPOV drone autonomy: https://spectrum.ieee.org/ai-drone-racing https://spectrum.ieee.org/ai-drone-racing Basically, those guys trained a drone to beat human FPOV racers but they had access to the race course layout before the race and trained extensively on it. Can't do that on a battlefield. Also: no shooting at enemy drones during FPOV races. Makes surviving to the end of the race simpler.