4 ms·

Very good essay. However, Carmac's view that there is very little quantified info about what works and what doesn't overlooks the work of software engineering p

Very good essay. However, Carmac's view that there is very little quantified info about what works and what doesn't overlooks the work of software engineering practitioners, such as Capers Jones, who has long studied projects quantitatively and published detailed statistics on the effectiveness of different development approaches. His book "The Economics of Software Quality" is an excellent summary of his findings across thousands of projects. Reading it is like the first time you run a profiler and you realize that what you think you knew is true in part, but that other factors you'd not counted are have an outsize effect that you've misestimated.

coreyoconnor 11y ago
I've run into the same opinion from other developers: "there is very little quantified info about what works and what doesn't" In my experience they are looking for the wrong answers. They want an answer of how to design software. How to define the entities, classes, functions etc. Quantitative software development discusses less of design, IMO, and more organizational factors. EG: How a companies organization and processes impact quality. This is not what practitioners expect. Yet, this is where the science of software development is. Thanks for the book reference! An economic perspective on software quality sounds enlightening.
- shadowfox 11y ago
  Now I am curious as to whether there has been any work in quantifying (or at least studying) the impact of processes vs engineering (in the sense of design and development) on quality and long term maintenance of software. My google-fu is clearly failing me.
angersock 11y ago
Thanks for the reading suggestion! That said, it seems that his last practical experience in the field was better than two decades ago ( https://en.wikipedia.org/wiki/Capers_Jones https://en.wikipedia.org/wiki/Capers_Jones ). A criticism of that sort is available to other folks as well--I feel compelled to ask, "Interesting results, but what have you shipped recently?"
- brehaut 11y ago
  "but what have you shipped recently?" Looks like he has shipped 14 books and at least 36 papers on software engineering methodologies.
- hinkley 11y ago
  Don't you think this is a bit like not listening to coaches or sports medicine experts because they haven't been professional players in two decades?
  - angersock 11y ago
    Well, when software engineering is as good for one's personal social life as being an athlete, perhaps I'll buy your argument. :) More seriously, I see what you're saying--that said, there is a huuuuge difference between how we develop software now versus even five years ago, much less ten or twenty. Github. Stack Overflow. Agile methods (ugh). Lean methodologies. Various testing methodologies. Changes in the funding structure of companies. Order of magnitude increase (comfortably) in debugging tools for web and network applications. The entire single-page browser app movement. The entire GPGPU movement. Truly convenient and open collaborative coding platforms. The field, and the people in it, have changed a great deal, and we've learned a lot. I'm increasingly skeptical of professors of professional software engineering who don't, as a day job, professionally engineer software. This applies to other folks--and I mean this nicely--like Uncle Bob or Martin Fowler as well. Folks like Carmack who have been pushing the envelope and getting scarred the hard way for 20 years are probably simply better sources of learning.
    - InclinedPlane 11y ago
      TDD and agile are both about 15 years old, and both built on well-established pre-existing development patterns. Github doesn't change many of the fundamentals of how software improves in quality or how development works. What I've observed in the industry has been a shocking lack of self-awareness about application of best practices, and decidedly half-assed attempts to improve. The state of the industry is not dev shops on the bleeding edge of the best tools, best methodologies, best policies, etc. It's almost universally a tale of failing to even get to "ok" in terms of well-known best (or even better) practices. There are a ton of dev shops that don't do any code reviews at all, and formal code reviews (which have been shown to be one of the best methods for improving code quality) are almost unheard of. Even at places that take testing seriously or do TDD they still typically don't do it very well. That's why folks like Uncle Bob and Martin Fowler continue to have so much traction in the industry, because simply following good advice that was old 20 years ago is still a huge step forward for the average, or even above average, dev shop in the industry today. Saying "we've learned a lot" just tells me that I shouldn't take you seriously, because clearly we haven't. Software dev. is still a shambles. Security is still a nightmare. Performance is still a nightmare. Quality and robustness is still a nightmare. Work/life balance is still a nightmare. Reading books like "The Mythical Man Month" (which is 40 years old!) still holds a ton of lessons that have yet to be taken to heart by the majority of the industry.
InclinedPlane 11y ago
It's shocking how immature software engineering is as an industry. There is a ton of work that hasn't been done and a ton of information that we just don't have, but even so there is a stark lack of adoption of already well-known and well-documented best practices. We very much are not trying our best.
- gozo 11y ago
  I agree with you, but also think we need to understand why this is happening. We know how to build safe factories, but people keep getting hurt in them and once in a while they collapse entirely. If you go to emerging economies or even just compare northern versus southern Europe you'll see that there is no inherent reason why things can't be more sophisticated other than culture and possibly a status quo. It's overwhelmingly just that the sophistication in knowledge, across the board from worker to law maker and in between, doesn't keep up with the demand backed by economic incentives. The widespread solutions in software seem to be to make software engineering more rigorous. Which while I agree with to some extent still largely avoids the problem. Not only do we know that even the best programmers currently can't consistently make programs that don't display catastrophic failure in terms of e.g. compromising data, but what we currently demand from software doesn't fit well into the field of engineering. John Carmack hasn't made a leading game in a decade, because technology isn't enough anymore. It's games like Minecraft, creative with technical problems, and Call of Duty, almost cinematic entertainment, that is pushing the boundaries. I think to a large extent we missed the opportunity to build a good foundation in computing. The Internet in terms of decentralization and security, software in terms of process and robustness, and probably many more examples I don't know about. Partly because the field (and demand) is moving so fast, but also because of the culture in software. I'm always surprised how conservative (not in terms of politic) people in software are and how we collectively seem to agree, despite things like "software eats the world", that it's still a niche thing.
  - InclinedPlane 11y ago
    The reasons are somewhat complicated, though some are straightforward. There is a ton of money in software, and a ton of people who want to be devs. Which means that even if a company makes a lot of mistakes they can still be profitable. And even if they burn out or drive away a ton of talent there is still no end of bodies willing to try to do the work, they just may not have the same level of skills. I think rigor is an important part of the "solution" but in a way that's too easy. It's simplistic to just say "do better", which is sort of what a call for rigor is. Ultimately I think it's not rigor but just seriousness which is called for. Right now the industry is downright delusional and avoiding even talking about some of the fundamentals of software engineering as a practice. So many people want to feel like they're on the bleeding edge, rather than struggling through the mire (which is the reality). We have to get to the stage where people are talking about these things first before we get to the stage where we can do anything about them. Right now the industry is in a rough shape. And the industry doesn't realize this mostly because there's so much money still (and money is a salve for many wounds) and because not long ago the industry was in crisis, so we're still seeing improvement from that state. Merely producing anything of value from a software project is still a big deal, and not a sure thing. Moreover, shipping any major product or service without relying on "crunch mode" or a death march is a rarity in the industry these days. But we can do better. Not just with rigor but with how people use their time and do work. The norm is still too close to everyone running around in firefighting mode all the time, while technical debt blooms and burnout looms. A better mode is to work at a slower pace, get the right work done at a higher quality level, and still maintain good development velocity because you aren't slowed down by your technical debt all the time. There are some, small, pockets of the industry where that's the norm, but it's still a hard sell. At the end of the day the default is still to fall back to the ridiculous notion that software dev is like factory work and you get more bang for your buck by just adding more bodies and by having more butts in seats (or fingers at keyboards) for more hours per week.

coreyoconnor 11y ago

I've run into the same opinion from other developers: "there is very little quantified info about what works and what doesn't" In my experience they are looking for the wrong answers. They want an answer of how to design software. How to define the entities, classes, functions etc. Quantitative software development discusses less of design, IMO, and more organizational factors. EG: How a companies organization and processes impact quality. This is not what practitioners expect. Yet, this is where the science of software development is. Thanks for the book reference! An economic perspective on software quality sounds enlightening.

shadowfox 11y ago
Now I am curious as to whether there has been any work in quantifying (or at least studying) the impact of processes vs engineering (in the sense of design and development) on quality and long term maintenance of software. My google-fu is clearly failing me.

angersock 11y ago

Thanks for the reading suggestion! That said, it seems that his last practical experience in the field was better than two decades ago ( https://en.wikipedia.org/wiki/Capers_Jones https://en.wikipedia.org/wiki/Capers_Jones ). A criticism of that sort is available to other folks as well--I feel compelled to ask, "Interesting results, but what have you shipped recently?"

brehaut 11y ago
"but what have you shipped recently?" Looks like he has shipped 14 books and at least 36 papers on software engineering methodologies.
hinkley 11y ago
Don't you think this is a bit like not listening to coaches or sports medicine experts because they haven't been professional players in two decades?
- angersock 11y ago
  Well, when software engineering is as good for one's personal social life as being an athlete, perhaps I'll buy your argument. :) More seriously, I see what you're saying--that said, there is a huuuuge difference between how we develop software now versus even five years ago, much less ten or twenty. Github. Stack Overflow. Agile methods (ugh). Lean methodologies. Various testing methodologies. Changes in the funding structure of companies. Order of magnitude increase (comfortably) in debugging tools for web and network applications. The entire single-page browser app movement. The entire GPGPU movement. Truly convenient and open collaborative coding platforms. The field, and the people in it, have changed a great deal, and we've learned a lot. I'm increasingly skeptical of professors of professional software engineering who don't, as a day job, professionally engineer software. This applies to other folks--and I mean this nicely--like Uncle Bob or Martin Fowler as well. Folks like Carmack who have been pushing the envelope and getting scarred the hard way for 20 years are probably simply better sources of learning.
  - InclinedPlane 11y ago
    TDD and agile are both about 15 years old, and both built on well-established pre-existing development patterns. Github doesn't change many of the fundamentals of how software improves in quality or how development works. What I've observed in the industry has been a shocking lack of self-awareness about application of best practices, and decidedly half-assed attempts to improve. The state of the industry is not dev shops on the bleeding edge of the best tools, best methodologies, best policies, etc. It's almost universally a tale of failing to even get to "ok" in terms of well-known best (or even better) practices. There are a ton of dev shops that don't do any code reviews at all, and formal code reviews (which have been shown to be one of the best methods for improving code quality) are almost unheard of. Even at places that take testing seriously or do TDD they still typically don't do it very well. That's why folks like Uncle Bob and Martin Fowler continue to have so much traction in the industry, because simply following good advice that was old 20 years ago is still a huge step forward for the average, or even above average, dev shop in the industry today. Saying "we've learned a lot" just tells me that I shouldn't take you seriously, because clearly we haven't. Software dev. is still a shambles. Security is still a nightmare. Performance is still a nightmare. Quality and robustness is still a nightmare. Work/life balance is still a nightmare. Reading books like "The Mythical Man Month" (which is 40 years old!) still holds a ton of lessons that have yet to be taken to heart by the majority of the industry.

InclinedPlane 11y ago

It's shocking how immature software engineering is as an industry. There is a ton of work that hasn't been done and a ton of information that we just don't have, but even so there is a stark lack of adoption of already well-known and well-documented best practices. We very much are not trying our best.

gozo 11y ago
I agree with you, but also think we need to understand why this is happening. We know how to build safe factories, but people keep getting hurt in them and once in a while they collapse entirely. If you go to emerging economies or even just compare northern versus southern Europe you'll see that there is no inherent reason why things can't be more sophisticated other than culture and possibly a status quo. It's overwhelmingly just that the sophistication in knowledge, across the board from worker to law maker and in between, doesn't keep up with the demand backed by economic incentives. The widespread solutions in software seem to be to make software engineering more rigorous. Which while I agree with to some extent still largely avoids the problem. Not only do we know that even the best programmers currently can't consistently make programs that don't display catastrophic failure in terms of e.g. compromising data, but what we currently demand from software doesn't fit well into the field of engineering. John Carmack hasn't made a leading game in a decade, because technology isn't enough anymore. It's games like Minecraft, creative with technical problems, and Call of Duty, almost cinematic entertainment, that is pushing the boundaries. I think to a large extent we missed the opportunity to build a good foundation in computing. The Internet in terms of decentralization and security, software in terms of process and robustness, and probably many more examples I don't know about. Partly because the field (and demand) is moving so fast, but also because of the culture in software. I'm always surprised how conservative (not in terms of politic) people in software are and how we collectively seem to agree, despite things like "software eats the world", that it's still a niche thing.
- InclinedPlane 11y ago
  The reasons are somewhat complicated, though some are straightforward. There is a ton of money in software, and a ton of people who want to be devs. Which means that even if a company makes a lot of mistakes they can still be profitable. And even if they burn out or drive away a ton of talent there is still no end of bodies willing to try to do the work, they just may not have the same level of skills. I think rigor is an important part of the "solution" but in a way that's too easy. It's simplistic to just say "do better", which is sort of what a call for rigor is. Ultimately I think it's not rigor but just seriousness which is called for. Right now the industry is downright delusional and avoiding even talking about some of the fundamentals of software engineering as a practice. So many people want to feel like they're on the bleeding edge, rather than struggling through the mire (which is the reality). We have to get to the stage where people are talking about these things first before we get to the stage where we can do anything about them. Right now the industry is in a rough shape. And the industry doesn't realize this mostly because there's so much money still (and money is a salve for many wounds) and because not long ago the industry was in crisis, so we're still seeing improvement from that state. Merely producing anything of value from a software project is still a big deal, and not a sure thing. Moreover, shipping any major product or service without relying on "crunch mode" or a death march is a rarity in the industry these days. But we can do better. Not just with rigor but with how people use their time and do work. The norm is still too close to everyone running around in firefighting mode all the time, while technical debt blooms and burnout looms. A better mode is to work at a slower pace, get the right work done at a higher quality level, and still maintain good development velocity because you aren't slowed down by your technical debt all the time. There are some, small, pockets of the industry where that's the norm, but it's still a hard sell. At the end of the day the default is still to fall back to the ridiculous notion that software dev is like factory work and you get more bang for your buck by just adding more bodies and by having more butts in seats (or fingers at keyboards) for more hours per week.