4 ms·

I think we agree more than you think ;) In my original article, I quoted some papers and studies. And on twitter, somebody occused me that "All science is crap"

by struppi 11y ago

I think we agree more than you think ;) In my original article, I quoted some papers and studies. And on twitter, somebody occused me that "All science is crap". So I tried to write another article with only my opinion on the subject - The article linked here on HN.

nickpsecurity 11y ago
I'll reread it with that in mind. See if I see another impression. How about that? :)
nickpsecurity 11y ago
Reading it instead of skimming it shows there's indeed plenty of agreement even without the science part. The minimizing f_n reflects well what the science established. Far as defects, problems people report are definitely a start. Fagan Software Inspection Process had the nice idea of standardizing in a list the kind of issues in code that would be treated as defects and fixed by priority. They could be lack of a bounds check, code style issue, whatever team thought was important. So, my definition for defect in such a scheme was "unacceptable code." "Low defect potential." Interesting metric. Often a function of complexity, coupling, amount of state being thrown around, and handling of interface assumptions. Scientists should come up with ways of assessing stuff like that then see what effect it had on defects during changes among many codebases. Of course, the minimizing f_n stuff should help here. "When you find a defect in production a week after deploying the code, it is probably still rather cheap to fix: All the original developers are still there, they still remember what they did, the documentation is still accurate." That's actually a really good point. I don't remember seeing anyone else say it. I'll have to remember that. "Well crafted code is self documenting. When I read the code together with it's tests, I want to be able to understand what is going on. Without reading some external documentation (wikis, ...)" Well-crafted code section is overall good but I'm not sure about this. I've seen code that can work like this. Yet, I've also seen high assurance work where there were abstract specs of what it does w/ pre and post conditions that were easier to understand for the use rather than modification of a module. Could also be fed into analysis tools. Some modern languages include that stuff in the code itself and can export interface documentation. That's not even getting into issues like consistently using a filesystem where all kinds of extra commands are needed for non-obvious reasons that need a whole article to describe. A 1 line operation suddenly takes more like 10. So, I'd say make the code self-documenting where possible but there's potentially good reasons to have the other things. If they're needed or useful. "Good software design is, in my opinion, less subjective than some other terms I described above." I agree. You did alright explaining that. "It's Not Much Slower Anyway The Microsoft study on TDD" You're conflating quality and testing here. They're not the same. There's plenty of low-quality software with lots of testing. Likewise, Cleanroom (extreme example) normally resulted in high quality from app/code structure and verification via code revies before testing. Same with Fagan inspections and OpenBSD's auditing process. All of those had vastly higher quality than TDD projects far as I've seen. And that's without arguing against testing, as you know I'm for it. Maybe renaming anything like that in your articles to specifically focus on benefit or cost of testing. So, this would be "TDD/Testing is not much slower anyway." The arguments you made for it are good and actually apply to all quality techniques. I think you intended that but set it up wrt to testing. So, maybe it's not the title and just something about that section in particular. Sounds nitpicking but my reaction is more that we need to always convey quality as a combo of prevention, code review, and testing. Code review & testing at the least. Gotta keep putting it into their heads until they can't forget. ;) "Let's look back to "Minimizing F_n (For Large n)":" "Executable specifications (a part of the test suite) make sure everybody understands the current functionality that is implemented. And that "documentation" cannot be outdated, because otherwise the tests would fail." All good advice but I counter that this long-running problem is a management and SCM/VCS problem. One can implement a policy and rule that any change to code requires an update of the documentation. Then enforce that. A test suite is also a good idea but not a requirement for dealing with this problem. Not even enough as other verification methods (eg DbyC, static analysis) were needed to catch problems in this area that testing misses. "At some point, your users will recognize that it takes longer and longer until they get software that contains the new features they requested. And they have to pay more and more for it." A really, good point that few mention or argue satisfactorily. Should be one of the top points. Need more ways to illustrate this in numbers or visually so less technical people get it. Maybe just delivery time and cost themselves as a graph going up over time. ;) "putting it all together" Strong, sound conclusions. The science supports it up to a certain quality point where ROI diminishes. You're advocating a realistic one, though. "A former colleague once said TDD would not make sense for them "because our iOS app only consists of a user interface and some server calls, and you cannot really test those. There is no logic in between. And it's really easy to test manually." Well, maybe you can get away with it when you have an app like that." You can counter that software is rarely that simple. There's assumptions built in to various component, failure modes to account for (esp network), and effects of state build-up if it's isn't fully stateful. Hell, tell them a non-bloated web server is "simple" with an "interface and some server calls." Then show them the release notes detailing all the bug fixes. Murphy's Law says the "Simple thing are always hard." Not always, but often enough to justify verification activities. Just popped in my head that it also helps when platform (eg iOS) breaks something with a change and you didn't know about it. Proper tests can isolate that immediately. "Also, your actions now will come back and bite you later." Very important. Most "short-term" jobs end up lasting a long time and even being reused in some way. Besides, getting in habit of doing it right makes doing it right more effortless. The delay between short-term and long-term should get smaller as verification activities become natural. I also like "unlimited downside" concept haha. "And in many organizations, you will have a hard time to argue throwing away "perfectly working code" when the time pressure gets bad..." Always worth remembering, too. Overall, when I'm more thorough in the review, you have a good write-up with only a few things I take issue with. As you predicted. :)
- struppi 11y ago
  Thank you so much for your detailed reply! That's a lot to think about :) Thanks for the hints about defects. I like "unacceptable code", but it's probably still a definition not everyone can live with :) "It's Not Much Slower Anyway The Microsoft study on TDD" "You're conflating quality and testing here. They're not the same." You are right. My arguments here are only for TDD, and I actually did not want to write a TDD article :) I'll think about how to improve that section. Also, CleanRoom is really interesting, but I don't know very much about it and never experienced it. "At some point, your users will recognize that it takes longer and longer..." "Should be one of the top points." I was thinking about the order of the points a lot, and no particular order seemed really right. It now comes at a point where I think I have established all the preconditions: Speed means lead time, cost depends on speed, high external quality means faster, high internal quality means faster. But you're right, the main points I want to bring across come very late in the article. As for the illustrations: Maybe I'll write another article about how users care about internal quality - Because I've worked in so many legacy code projects that I think I have a good view on that :) I like your points about the iOS stuff and the short-term jobs. I'll think about them some more. I'm still not sure if I believe the "unlimited downside" idea 100% (because only very few things are truly unlimited), but it's an interesting concept.
  - nickpsecurity 11y ago
    " Thank you so much for your detailed reply! That's a lot to think about :)" You've very welcome. :) "Thanks for the hints about defects. I like "unacceptable code", but it's probably still a definition not everyone can live with :)" I kind of came up with it off the top of my head. Point is that the code will be rejected on relatively arbitrary criteria. It might be known defects, coding style, what libraries were used... anything. Hard to find a word or phrase covering all that which people will agree with. "Also, CleanRoom is really interesting, but I don't know very much about it and never experienced it." This presentation... http://groups.engin.umd.umich.edu/CIS/course.des/cis376/ppt/lec21.ppt http://groups.engin.umd.umich.edu/CIS/course.des/cis376/ppt/... ...summarizes it for you in about 5 min of reading. The first attempts to make software in a scientific way, engineered software I call it, were done by Margaret Hamilton (founder of our field) at NASA plus academics like Dijkstra and Hoare. They all produced nearly defect-free systems in a systematic, often mathematical way. Harlan Mills took the lessons to industry when developing Cleanroom: a combo of lightweight formal methods, design constraints, spec/code verification, incremental development, and usage-driven testing. He wanted SW products to be as low-defect as HW clean rooms. Result was code at extremely low defect rate that was consistent enough for statistical certification. While popular, Cleanroom teams actually warrantied their software. Altran/Praxis is another company I know that (a) truly engineers software and (b) warranties it at specific defect rates. Defect rates are usually better than the Linux kernel and close to the Space Shuttle's control code. Here's a case study of their Correct by Construction process applied to high-assurance, certificate authority: http://www.anthonyhall.org/c_by_c_secure_system.pdf http://www.anthonyhall.org/c_by_c_secure_system.pdf So, there's you two methods that already maxed out quality (and/or security) on real-world projects. Altran charges an acceptable premium for quality but Cleanroom often cost nothing extra or saved money for same reasons as you article says. Appears the modern programmers still have lessons to learn in software quality from the 1980's. Like I tell people, the old wisdom is often pretty solid even after you fix outdated parts. Our field is just terrible at passing it down. I try to fix that with posts like this so the smart, young people get a boost toward whatever great things they create. Also reduce endless rediscovering of fire and reinventing the wheel. :) "I was thinking about the order of the points a lot, and no particular order seemed really right. It now comes at a point where I think I have established all the preconditions: Speed means lead time, cost depends on speed, high external quality means faster, high internal quality means faster. " I'm going to let you keep working on that formula. You've got the right ingredients. Far as order, there might not be any. Many things in engineering or business are holistic where the parts all feed into each other. The goal is an emergent property of that. So, rather than an order, you'd visualize it like a bubble chart or something that with connections between the components and labels for the effect they have. Just something that's clear that they're all connected, all feed into each other, and ignoring one breaks the process for achieving the goal. Just my dos centimos. "Maybe I'll write another article about how users care about internal quality - Because I've worked in so many legacy code projects that I think I have a good view on that :)" You should. Matter of fact, you should do a lot more work on figuring out how to measure and convey the importance of eliminating technical debt. To laypersons, not geeks. Not enough work here despite its importance. Everyone is instead making arguments for geeks that already get it cuz they're knee deep in the crap or picked the job that didn't have it. Great attempt here: http://tech.ticketmaster.com/2015/06/30/what-ticketmaster-is-doing-about-technical-debt/ http://tech.ticketmaster.com/2015/06/30/what-ticketmaster-is... Tool for metrics here that was interesting. Not tried it, though. http://swreflections.blogspot.com/2012/02/technical-debt-how-much-is-it-really.html http://swreflections.blogspot.com/2012/02/technical-debt-how... " I'm still not sure if I believe the "unlimited downside" idea 100% (because only very few things are truly unlimited), but it's an interesting concept." I just thought it sounded catchy. Not committing to it myself. ;)

by struppi 11y ago

nickpsecurity 11y ago

I'll reread it with that in mind. See if I see another impression. How about that? :)

nickpsecurity 11y ago

Reading it instead of skimming it shows there's indeed plenty of agreement even without the science part. The minimizing f_n reflects well what the science established. Far as defects, problems people report are definitely a start. Fagan Software Inspection Process had the nice idea of standardizing in a list the kind of issues in code that would be treated as defects and fixed by priority. They could be lack of a bounds check, code style issue, whatever team thought was important. So, my definition for defect in such a scheme was "unacceptable code." "Low defect potential." Interesting metric. Often a function of complexity, coupling, amount of state being thrown around, and handling of interface assumptions. Scientists should come up with ways of assessing stuff like that then see what effect it had on defects during changes among many codebases. Of course, the minimizing f_n stuff should help here. "When you find a defect in production a week after deploying the code, it is probably still rather cheap to fix: All the original developers are still there, they still remember what they did, the documentation is still accurate." That's actually a really good point. I don't remember seeing anyone else say it. I'll have to remember that. "Well crafted code is self documenting. When I read the code together with it's tests, I want to be able to understand what is going on. Without reading some external documentation (wikis, ...)" Well-crafted code section is overall good but I'm not sure about this. I've seen code that can work like this. Yet, I've also seen high assurance work where there were abstract specs of what it does w/ pre and post conditions that were easier to understand for the use rather than modification of a module. Could also be fed into analysis tools. Some modern languages include that stuff in the code itself and can export interface documentation. That's not even getting into issues like consistently using a filesystem where all kinds of extra commands are needed for non-obvious reasons that need a whole article to describe. A 1 line operation suddenly takes more like 10. So, I'd say make the code self-documenting where possible but there's potentially good reasons to have the other things. If they're needed or useful. "Good software design is, in my opinion, less subjective than some other terms I described above." I agree. You did alright explaining that. "It's Not Much Slower Anyway The Microsoft study on TDD" You're conflating quality and testing here. They're not the same. There's plenty of low-quality software with lots of testing. Likewise, Cleanroom (extreme example) normally resulted in high quality from app/code structure and verification via code revies before testing. Same with Fagan inspections and OpenBSD's auditing process. All of those had vastly higher quality than TDD projects far as I've seen. And that's without arguing against testing, as you know I'm for it. Maybe renaming anything like that in your articles to specifically focus on benefit or cost of testing. So, this would be "TDD/Testing is not much slower anyway." The arguments you made for it are good and actually apply to all quality techniques. I think you intended that but set it up wrt to testing. So, maybe it's not the title and just something about that section in particular. Sounds nitpicking but my reaction is more that we need to always convey quality as a combo of prevention, code review, and testing. Code review & testing at the least. Gotta keep putting it into their heads until they can't forget. ;) "Let's look back to "Minimizing F_n (For Large n)":" "Executable specifications (a part of the test suite) make sure everybody understands the current functionality that is implemented. And that "documentation" cannot be outdated, because otherwise the tests would fail." All good advice but I counter that this long-running problem is a management and SCM/VCS problem. One can implement a policy and rule that any change to code requires an update of the documentation. Then enforce that. A test suite is also a good idea but not a requirement for dealing with this problem. Not even enough as other verification methods (eg DbyC, static analysis) were needed to catch problems in this area that testing misses. "At some point, your users will recognize that it takes longer and longer until they get software that contains the new features they requested. And they have to pay more and more for it." A really, good point that few mention or argue satisfactorily. Should be one of the top points. Need more ways to illustrate this in numbers or visually so less technical people get it. Maybe just delivery time and cost themselves as a graph going up over time. ;) "putting it all together" Strong, sound conclusions. The science supports it up to a certain quality point where ROI diminishes. You're advocating a realistic one, though. "A former colleague once said TDD would not make sense for them "because our iOS app only consists of a user interface and some server calls, and you cannot really test those. There is no logic in between. And it's really easy to test manually." Well, maybe you can get away with it when you have an app like that." You can counter that software is rarely that simple. There's assumptions built in to various component, failure modes to account for (esp network), and effects of state build-up if it's isn't fully stateful. Hell, tell them a non-bloated web server is "simple" with an "interface and some server calls." Then show them the release notes detailing all the bug fixes. Murphy's Law says the "Simple thing are always hard." Not always, but often enough to justify verification activities. Just popped in my head that it also helps when platform (eg iOS) breaks something with a change and you didn't know about it. Proper tests can isolate that immediately. "Also, your actions now will come back and bite you later." Very important. Most "short-term" jobs end up lasting a long time and even being reused in some way. Besides, getting in habit of doing it right makes doing it right more effortless. The delay between short-term and long-term should get smaller as verification activities become natural. I also like "unlimited downside" concept haha. "And in many organizations, you will have a hard time to argue throwing away "perfectly working code" when the time pressure gets bad..." Always worth remembering, too. Overall, when I'm more thorough in the review, you have a good write-up with only a few things I take issue with. As you predicted. :)

struppi 11y ago
Thank you so much for your detailed reply! That's a lot to think about :) Thanks for the hints about defects. I like "unacceptable code", but it's probably still a definition not everyone can live with :) "It's Not Much Slower Anyway The Microsoft study on TDD" "You're conflating quality and testing here. They're not the same." You are right. My arguments here are only for TDD, and I actually did not want to write a TDD article :) I'll think about how to improve that section. Also, CleanRoom is really interesting, but I don't know very much about it and never experienced it. "At some point, your users will recognize that it takes longer and longer..." "Should be one of the top points." I was thinking about the order of the points a lot, and no particular order seemed really right. It now comes at a point where I think I have established all the preconditions: Speed means lead time, cost depends on speed, high external quality means faster, high internal quality means faster. But you're right, the main points I want to bring across come very late in the article. As for the illustrations: Maybe I'll write another article about how users care about internal quality - Because I've worked in so many legacy code projects that I think I have a good view on that :) I like your points about the iOS stuff and the short-term jobs. I'll think about them some more. I'm still not sure if I believe the "unlimited downside" idea 100% (because only very few things are truly unlimited), but it's an interesting concept.
- nickpsecurity 11y ago
  " Thank you so much for your detailed reply! That's a lot to think about :)" You've very welcome. :) "Thanks for the hints about defects. I like "unacceptable code", but it's probably still a definition not everyone can live with :)" I kind of came up with it off the top of my head. Point is that the code will be rejected on relatively arbitrary criteria. It might be known defects, coding style, what libraries were used... anything. Hard to find a word or phrase covering all that which people will agree with. "Also, CleanRoom is really interesting, but I don't know very much about it and never experienced it." This presentation... http://groups.engin.umd.umich.edu/CIS/course.des/cis376/ppt/lec21.ppt http://groups.engin.umd.umich.edu/CIS/course.des/cis376/ppt/... ...summarizes it for you in about 5 min of reading. The first attempts to make software in a scientific way, engineered software I call it, were done by Margaret Hamilton (founder of our field) at NASA plus academics like Dijkstra and Hoare. They all produced nearly defect-free systems in a systematic, often mathematical way. Harlan Mills took the lessons to industry when developing Cleanroom: a combo of lightweight formal methods, design constraints, spec/code verification, incremental development, and usage-driven testing. He wanted SW products to be as low-defect as HW clean rooms. Result was code at extremely low defect rate that was consistent enough for statistical certification. While popular, Cleanroom teams actually warrantied their software. Altran/Praxis is another company I know that (a) truly engineers software and (b) warranties it at specific defect rates. Defect rates are usually better than the Linux kernel and close to the Space Shuttle's control code. Here's a case study of their Correct by Construction process applied to high-assurance, certificate authority: http://www.anthonyhall.org/c_by_c_secure_system.pdf http://www.anthonyhall.org/c_by_c_secure_system.pdf So, there's you two methods that already maxed out quality (and/or security) on real-world projects. Altran charges an acceptable premium for quality but Cleanroom often cost nothing extra or saved money for same reasons as you article says. Appears the modern programmers still have lessons to learn in software quality from the 1980's. Like I tell people, the old wisdom is often pretty solid even after you fix outdated parts. Our field is just terrible at passing it down. I try to fix that with posts like this so the smart, young people get a boost toward whatever great things they create. Also reduce endless rediscovering of fire and reinventing the wheel. :) "I was thinking about the order of the points a lot, and no particular order seemed really right. It now comes at a point where I think I have established all the preconditions: Speed means lead time, cost depends on speed, high external quality means faster, high internal quality means faster. " I'm going to let you keep working on that formula. You've got the right ingredients. Far as order, there might not be any. Many things in engineering or business are holistic where the parts all feed into each other. The goal is an emergent property of that. So, rather than an order, you'd visualize it like a bubble chart or something that with connections between the components and labels for the effect they have. Just something that's clear that they're all connected, all feed into each other, and ignoring one breaks the process for achieving the goal. Just my dos centimos. "Maybe I'll write another article about how users care about internal quality - Because I've worked in so many legacy code projects that I think I have a good view on that :)" You should. Matter of fact, you should do a lot more work on figuring out how to measure and convey the importance of eliminating technical debt. To laypersons, not geeks. Not enough work here despite its importance. Everyone is instead making arguments for geeks that already get it cuz they're knee deep in the crap or picked the job that didn't have it. Great attempt here: http://tech.ticketmaster.com/2015/06/30/what-ticketmaster-is-doing-about-technical-debt/ http://tech.ticketmaster.com/2015/06/30/what-ticketmaster-is... Tool for metrics here that was interesting. Not tried it, though. http://swreflections.blogspot.com/2012/02/technical-debt-how-much-is-it-really.html http://swreflections.blogspot.com/2012/02/technical-debt-how... " I'm still not sure if I believe the "unlimited downside" idea 100% (because only very few things are truly unlimited), but it's an interesting concept." I just thought it sounded catchy. Not committing to it myself. ;)