6 ms·

I'm writing a post on software estimation, how it's difficult to estimate things correctly. One of the arguments that I want to make is that estimating poorly i

by tra3 7y ago

I'm writing a post on software estimation, how it's difficult to estimate things correctly. One of the arguments that I want to make is that estimating poorly is not restricted to the software industry (obvious in retrospect). Military and public infrastructure projects are an excellent source of poorly estimated projects. I've been collecting data about F35 specfically, 'cause it's so recent and the cost overruns are just mind boggling. Anyway, here's an article from last year [0] that discusses how project management for the F35 prorgram reclassifies "blockers" and "criticals" into "major" and "medium" (severity levels of the issue). Especially like how the IFF transponder fails to send out a distress call if a pilot ejects, or how the arresting hook (for carrier landings) bends the airframe. [0]: https://www.pogo.org/investigation/2018/08/f-35-program-cutting-corners-to-complete-development/ https://www.pogo.org/investigation/2018/08/f-35-program-cutt...

davidgl 7y ago
Great discussion - https://codeburst.io/software-estimation-in-the-fractal-dimension-914569e2ccb9 https://codeburst.io/software-estimation-in-the-fractal-dime...
laydn 7y ago
Well there is no indication about the severity of these ~800 bugs, but ~800 non-severe bugs in a system this complex is actually pretty good in my opinion.
- Psyladine 7y ago
  >but ~800 non-severe bugs in a system this complex is actually pretty good in my opinion. "The fly-by-wire flight software for the Saab Gripen (a lightweight fighter) went a step further. It disallowed both subroutine calls and backward branches, except for the one at the bottom of the main loop. Control flow went forward only. Sometimes one piece of code had to leave a note for a later piece telling it what to do, but this worked out well for testing: all data was allocated statically, and monitoring those variables gave a clear picture of most everything the software was doing. The software did only the bare essentials, and of course, they were serious about thorough ground testing. No bug has ever been found in the “released for flight” versions of that code. Henry Spencer henry@spsystems.net" Courtesy of John Carmack
  - Roboprog 7y ago
    Interesting. That’s a different world from the business, or development tool, projects I have ever worked on. Never worked on any real time stuff. Looks like they are very big on using a flat finite state machine model. At least at Saab. I’m guessing they didn’t use C++, either, but who knows.
    - scottlocklin 7y ago
      Looks like another argument in favor of Ada: https://www.sigada.org/conf/sa98/papers/frisberg.pdf https://www.sigada.org/conf/sa98/papers/frisberg.pdf
      - kjs3 7y ago
        Yes...this is a big reason why Ada exists.
    - kjs3 7y ago
      Nope...definitely not C++. Check out Jovial, Coral-66, MISRA C & sometimes Fortran. More recently, Ada & SPARK. The F-35? C++.
  - dingaling 7y ago
    Two crashes of Gripens during development led to changes in the flight control software. Now you could argue as to whether those were bugs or requirements deficiencies, but in neither case were the faults caught in preflight testing.
    - pc86 7y ago
      > during development Wasn't the above comment in reply to there being no bugs found in software released for flight?
      - 0xffff2 7y ago
        Either, the Saab "released for flight" claim only applies to fully operational flights, in which case the F-35 is still in development and we haven't seen the "released for flight" version of its software, or test-flight crashes leading to changes in flight software are very strong evidence that the claim is incorrect.
dkarl 7y ago
One of the arguments that I want to make is that estimating poorly is not restricted to the software industry (obvious in retrospect). Thank you. We have a weird obsession in software with ascribing characteristic deficiencies and unique blind spots to ourselves, and treating ourselves as children next to the real "grown-ups" in other fields. (I think we're overcompensating for the "arrogant nerd" stereotype.) In any profession they are aware of their stereotypical tendencies (their déformation professionnelle) and joke about it but don't get obsessed with their own inferiority or paralyzed by stereotype threat. We don't even use it in a way that helps us find the best knowledge from other fields to apply to our own. We start with our own deficiencies, project some idealized version of another field to serve as the opposite, and then try to learn from our imagined idea of how that field works. Early in my career, this took the form of thinking that building architecture and construction did not suffer from the planning and execution difficulties that software did. For example, in Austin the ownership group of the new soccer team is constructing a stadium that is planned to be ready for the start of the 2021 season. When I was a wee software developer in the early 2000s, I was taught that unlike our immature industry rife with childishness and careless arrogance, people who design and build physical structures are careful, methodical, mature, and too aware of the responsibility riding on their shoulders to make mistakes in design and planning. A stadium isn't some silly computer thing; it's part of a multi-billion dollar professional sports league with complicated travel logistics, bound by the obligation of tens of millions in presold season tickets! This responsibility is entrusted to serious people in a mature industry, and everything is planned out meticulously via responsible processes to avoid delays. Seriously. This is how people in software, at least the ones I was exposed to, talked about how to improve the profession. The question was not, what can we learn from other fields that struggle with complicated design and schedule and budget problems, but what is personally wrong with people in software that we have not mastered this process that people in other fields OBVIOUSLY have. But of course when talking to the season ticket sales rep, when we asked about the possibility of the stadium not being ready when the season starts, he said great question, construction delays happen, it has happened to other MLS teams in the past, and the league will rearrange the schedule so that the team plays nothing but road games for the first several months of the year and has their home games later. This has already been figured out, because the "mature" design and construction industry that meets its schedules through the magical power of seriousness only existed in the imagination of circa-2000 software process consultants.
mdorazio 7y ago
Just be careful what generalizations you draw. I've been in enough proposal development meetings to know that the estimates that make it to the client/government entity aren't always the actual estimate. In many cases, one or more persons provided a pretty good estimate and it got shot down immediately because if the proposal included it, it would be rejected. Basically, if there's a good chance the client will eat the cost overruns (as governments usually do for military/space/infrastructure projects), there's a big incentive to deliberately underestimate even when you know the actual number will be significantly higher.
- chasd00 7y ago
  I was about to say, estimates better be aligned to the sales team expectations or they'll be changed so that they do align. It's only after the project is sold do you get a chance to look at the reality of the situation.

by tra3 7y ago

davidgl 7y ago

Great discussion - https://codeburst.io/software-estimation-in-the-fractal-dimension-914569e2ccb9 https://codeburst.io/software-estimation-in-the-fractal-dime...

laydn 7y ago

Well there is no indication about the severity of these ~800 bugs, but ~800 non-severe bugs in a system this complex is actually pretty good in my opinion.

Psyladine 7y ago
>but ~800 non-severe bugs in a system this complex is actually pretty good in my opinion. "The fly-by-wire flight software for the Saab Gripen (a lightweight fighter) went a step further. It disallowed both subroutine calls and backward branches, except for the one at the bottom of the main loop. Control flow went forward only. Sometimes one piece of code had to leave a note for a later piece telling it what to do, but this worked out well for testing: all data was allocated statically, and monitoring those variables gave a clear picture of most everything the software was doing. The software did only the bare essentials, and of course, they were serious about thorough ground testing. No bug has ever been found in the “released for flight” versions of that code. Henry Spencer henry@spsystems.net" Courtesy of John Carmack
- Roboprog 7y ago
  Interesting. That’s a different world from the business, or development tool, projects I have ever worked on. Never worked on any real time stuff. Looks like they are very big on using a flat finite state machine model. At least at Saab. I’m guessing they didn’t use C++, either, but who knows.
  - scottlocklin 7y ago
    Looks like another argument in favor of Ada: https://www.sigada.org/conf/sa98/papers/frisberg.pdf https://www.sigada.org/conf/sa98/papers/frisberg.pdf
    - kjs3 7y ago
      Yes...this is a big reason why Ada exists.
  - kjs3 7y ago
    Nope...definitely not C++. Check out Jovial, Coral-66, MISRA C & sometimes Fortran. More recently, Ada & SPARK. The F-35? C++.
- dingaling 7y ago
  Two crashes of Gripens during development led to changes in the flight control software. Now you could argue as to whether those were bugs or requirements deficiencies, but in neither case were the faults caught in preflight testing.
  - pc86 7y ago
    > during development Wasn't the above comment in reply to there being no bugs found in software released for flight?
    - 0xffff2 7y ago
      Either, the Saab "released for flight" claim only applies to fully operational flights, in which case the F-35 is still in development and we haven't seen the "released for flight" version of its software, or test-flight crashes leading to changes in flight software are very strong evidence that the claim is incorrect.

dkarl 7y ago

One of the arguments that I want to make is that estimating poorly is not restricted to the software industry (obvious in retrospect). Thank you. We have a weird obsession in software with ascribing characteristic deficiencies and unique blind spots to ourselves, and treating ourselves as children next to the real "grown-ups" in other fields. (I think we're overcompensating for the "arrogant nerd" stereotype.) In any profession they are aware of their stereotypical tendencies (their déformation professionnelle) and joke about it but don't get obsessed with their own inferiority or paralyzed by stereotype threat. We don't even use it in a way that helps us find the best knowledge from other fields to apply to our own. We start with our own deficiencies, project some idealized version of another field to serve as the opposite, and then try to learn from our imagined idea of how that field works. Early in my career, this took the form of thinking that building architecture and construction did not suffer from the planning and execution difficulties that software did. For example, in Austin the ownership group of the new soccer team is constructing a stadium that is planned to be ready for the start of the 2021 season. When I was a wee software developer in the early 2000s, I was taught that unlike our immature industry rife with childishness and careless arrogance, people who design and build physical structures are careful, methodical, mature, and too aware of the responsibility riding on their shoulders to make mistakes in design and planning. A stadium isn't some silly computer thing; it's part of a multi-billion dollar professional sports league with complicated travel logistics, bound by the obligation of tens of millions in presold season tickets! This responsibility is entrusted to serious people in a mature industry, and everything is planned out meticulously via responsible processes to avoid delays. Seriously. This is how people in software, at least the ones I was exposed to, talked about how to improve the profession. The question was not, what can we learn from other fields that struggle with complicated design and schedule and budget problems, but what is personally wrong with people in software that we have not mastered this process that people in other fields OBVIOUSLY have. But of course when talking to the season ticket sales rep, when we asked about the possibility of the stadium not being ready when the season starts, he said great question, construction delays happen, it has happened to other MLS teams in the past, and the league will rearrange the schedule so that the team plays nothing but road games for the first several months of the year and has their home games later. This has already been figured out, because the "mature" design and construction industry that meets its schedules through the magical power of seriousness only existed in the imagination of circa-2000 software process consultants.

mdorazio 7y ago

Just be careful what generalizations you draw. I've been in enough proposal development meetings to know that the estimates that make it to the client/government entity aren't always the actual estimate. In many cases, one or more persons provided a pretty good estimate and it got shot down immediately because if the proposal included it, it would be rejected. Basically, if there's a good chance the client will eat the cost overruns (as governments usually do for military/space/infrastructure projects), there's a big incentive to deliberately underestimate even when you know the actual number will be significantly higher.

chasd00 7y ago
I was about to say, estimates better be aligned to the sales team expectations or they'll be changed so that they do align. It's only after the project is sold do you get a chance to look at the reality of the situation.