6 ms·

Well Crafted Code, Quality, Speed and Budget

20 points by struppi 11y ago

zenogais 11y ago
Curious if the author has read anything by Watts Humphrey? He spent his entire career at SEI arguing this point and attempting to develop the tools and processes to bear it out. This article reads like a repetition of his arguments from A Disicipline for Software Engineering (1995).
- nickpsecurity 11y ago
  I learned in a discussion with "kragen" that "software engineering" sort of took on a life of its own that turned into a nightmare of management-focused stuff harmful to programming. Most programmers, especially mainstream crowd, have an aversion to the term from the permanent association. I started saying CompSci and focusing on prior techniques rather than "engineering" due to that permanent scarring. As in my other comment, I just focus on specific techniques and processes that help programmers do their job with evidence they work. It's my suggestion to you. I'm going to try to dig up and read that paper out of curiosity as I know many useful things came out of software engineering research. Most won't, though, the second they see "SEI" or "software engineering." So, might be best for us to just drop that and focus on improvements to "programming" or "development" with specific techniques.
- struppi 11y ago
  Original author here (sorry, was in bed when it came to the front page ;) ) I did not read that paper (book?), but will add it to my "To Read" pile. Thank you ;)
nickpsecurity 11y ago
I'm mixed about the post, esp the science part. The science of developing robust software is great and pretty consistent going back decades varying mostly in specific tools and tactics. Mainstream programming just doesn't apply it although more adoption in past decade of key techniques. Here's some computer science from the 1960's-1980's used in robust and secure system development (esp Orange Book B3 or CC EAL6) people might want to copy. I'm taking an empirical route where I reference techniques that were applied to many real-world projects with lessons learned in papers or studies that were consistent. All one can do with limited data & these aren't in order of importance. 1. Formal, non-English (eg math/logical) specifications of requirements or abstract design. English is ambiguous and misreadings of it caused countless errors, even back then. CompSci researchers tried formal specs with both English as a start and precise notations (eg Z, VDM, ASM's, statecharts) for clarity in specifics. Result was many inconsistencies caught in highly assured systems and protocol specs before coding even began. 2. High assurance stuff often used mathematical (formal) verification. Whether that worked or made sense was hit and miss. More on it later. Yet, virtually all of them said there was benefit in restrictions on the specs, design, and coding style to fit the provers' limitations. Essentially, they used boring constructs that were easy to analyse and this prevented/caught problems. Don't be too clever with design or code. Wirth and Hansen applied this to language design to bake safety & comprehension in with minimal to low loss in performance. Note: Led to Nick P's Law of Trustworthy Systems: "Tried and true beats novel or new." Always the default. 3. Dijkstra's THE project showed that modular, layered design with careful attention to interfaces (and interface checks) makes for most robust and maintainable software. Later results confirmed this where each module must fit in your head and control graph that's pretty predictable with minimal cycles prevented all kinds of local-becomes-global issues. Many systems flawless (or nearly so) in production were built this way. Dijkstra correctly noted that it was very hard to do this even for smart people and average developer might screw structuring up a lot. Solid prediction... but still worth striving for improvement here. 4. Fagan ran empirical studies at IBM that showed a regular, systematic, code review process caught many problems, even what tests missed. Turned that into formal inspections with the periodicity and prioritizing tuned per organization for right cost-benefit. Was generalized to whole SDLC by others in high robustness areas. Improved every project that used it from then on. Exactly what parameters to use is still open-ended but periodically looking for well-known flaws with reference sheet always works. 5. Testing for every feature, code-path, prior issues outside of code base, and common use-case. All of these have shown repeated benefits. There's a cut-off point for each that's still an open, research problem. However, at a minimum, usage-based testing and regression testing helped many projects achieve either zero or near-zero, user-facing defects in production. That's a very important differentiator as 100 bugs user never experiences is better than 5 that they do regularly. Mills' Cleanroom process combined simple implementation, code review, and usage-testing for insanely-high, statistically-certifiable quality even for amateur teams. 6. By around 60's-70's, it became clear that the language you choose has a significant effect on productivity, defects, maintenance, and integration. Numerous studies were run in industry and military comparing various ones. Certain languages (eg Ada) showed vastly lower defects, equal/better productivity, and great maintenance/integration in every study. Haven't seen many such studies since the 90's and most aren't constructed well to eliminate bias. However, it's grounded in science to claim that certain language choices prevent common negatives and encourage positives. So, it follows to adopt languages that make robust development easier. 7. By the 80's or 90's, it was clear that computers were better at finding certain problems in specs and code than humans. This gave rise to methodologies that put models of system or code into model-checkers and provers to show certain properties always hold (the good) or never show up (the bad). Used successfully with high-assurance safety and security critical systems with results ranging from "somewhat beneficial" to "caught stuff we'd never see or test for." Back then it was unclear how applicable it was. Recent work by Chlipala, Leroy, et al show near perfect results in practice when specs/proofs are right and much wider application than before. Lots of tooling and prior examples means this is a proven way of getting extra quality where high-stakes are worth the cost and where core functionality doesn't change often.. The CompCert C compiler, Eiffel's SCOOP concurrency scheme, and Navy team's EAL7 IPsec VPN are good examples. 8. Static analysis, aka "lightweight formal methods," were devised to deal with specialized skills and labor of above. Getting to the point, tools like Astree Analyzer or SPARK Ada can prove absence of common flaws with little to no false positives without need for mathematicians in the company. Just a half dozen of these tools by themselves found tons of vulnerabilities in real-world software that passed human review and testing. Enough said, eh? 9. Software that succeeded with testing often failed when random stuff came at it, especially malware. This led to various fault-injection methods like fuzz testing to simulate that and find breaking points. The huge number of defects, esp in file formats & protocol engines, found via this method argues for its effectiveness in improving quality. It ties in with stuff above in that well-written code that validates input at interface and preserves invariants throughout execution should simply disregard (or report) such erroneous input. 10. Interface errors themselves posed something like 80+% of problems. This was noted as far back as the 60's in Apollo project when Margaret Hamilton invented software engineering, fault-tolerance, and specification techniques to fight it. Dijkstra and Hoare pushed for pre- and post-conditions plus specific invariants to document the assumptions of code during procedure calls. Modern version is called Design by Contract in Eiffel, Ada, and numerous other languages (even asserts in C). Many deployments and tests showed such interface checks caught many issues, esp assumption violations when new code extended or modified legacy. 11. Concurrency issues caused all kinds of problems. Techniques were devised by Hansen (Concurrent Pascal) and later Meyer et al (SCOOP) to mostly immunize against them at language level with acceptable performance. Languages without that, especially Java, later got brilliant tooling that could reliably find race conditions, deadlocks, or livelocks. Use of any method inevitably found problems in production code that had escaped detection. So, using prior, proven methods to immunize against or detect common errors in concurrency is A Good Thing. Note that shared-nothing, event-driven architectures also emerged but I have less data on them outside that some (NonStop, Erlang) worked extremely well. The above are just a few things that computer science established with supporting evidence from real-world projects so long ago that Windows didn't exist. Anyone applying these lessons got benefits in terms of code quality, security, and maintainability. The rare few applying most or all of them, mainly high assurance community, got results along lines of space shuttle control code with extremely, low defects or zero in production. So, given the past and present results of these methods every time they're put to the test, I'm irritated every time another person talks like there's no good science to quality software. I just listed a bunch of it, it's been tested in production as scientific method requires thousands of time, tweaked probably hundreds, and core approaches remained even if tactics got modified. Now people can feel free to use and improve on the science. CompSci continues to in every area I listed with a chunk of proprietary and FOSS developers using a subset of the techniques. Just need more uptake. Use what's proven. And do note that there's plenty of examples for specific design and implementation decisions for common types of functionality. Many things that were shown to work or not work that could be encoded in libraries, DSL's, templates, whatever. No excuse except for our field's continual failure to learn and hand down the lessons from the past.
- jack9 11y ago
  > The science of developing robust software is great and pretty consistent going back decades varying mostly in specific tools and tactics Nope (since uh-huh/nuh-uh is the starting point you've taken). Most science is BAD SCIENCE and I have doubts about the claims made by either the author or the topics you reference. Almost all formal studies are about specific tools and tactics in small sample sizes. This isn't proven science as much as they are test results. - https://vimeo.com/9270320 https://vimeo.com/9270320 + http://quorumlanguage.com/evidence.php http://quorumlanguage.com/evidence.php should orient you toward what proof is and how we can reference it in a reasonable manner. There will be plenty of bias to argue about how the tests are conducted or what the data demonstrates.
  - nickpsecurity 11y ago
    You reply with a combination of trolling and endorsing a specific link that you believe represents doing it better. The first thing I looked at, syntax study, was already weaker than prior work I've seen on the subject. The language choice was highly biased while not focusing on enough metrics for comparison of strengths and weaknesses. Between field use and its first "evidence," your Quorum is already weaker than most of what I presented in terms of empirical weight. Due to the trolling, I'm hesistant to even watch the video in case his presentation is something likes yours in terms of content. @ non-trolling, HN readers also concerned about science in software Back to science, though, given other readers might be interested in that point. Software development is a combination of human- and machine-driven processes to turn some requirements into working, maintainable code. A subset of this, significant it turns out, can be analyzed for effectiveness with enough different case studies, sample sizes, objective measures, and repeats to make a claim about it with believable evidence. For process stuff, it's not going to be as logically formulated or minutely analyzed as a math equation or something. Not usually, anyway. Instead, it works more like this: 1. A recurring problem exists in the development process or artifact. 2. Measurements are taken of how often that problem occurs in general and for specific scenarios. 3. A hypothesis is formed as to why it exists and a solution to it. 4. The hypothesis is tested by applying the technique to several projects likely to experience the problem with objective records on outcomes and subjective accounts from users for a take on less tangible aspects. 5. Significant reductions in the problem in the objective data provide supporting evidence that the theory is correct. 6. The theory (solution) is tested by others against the problem to find more supporting evidence or counter-examples to the claim. 7. If it's correct, the results continue to speak for themselves in it being a solution with likely modifications as the problem and solution space are further explored. Note: If it's performed by humans, it's wise to either look out for or control for issues like Hawthorne effect. Must be sure that it's the overall solution that's working rather than people's heightened attention to an issue being studied. The above 1-7 have applied to my list of techniques to varying degrees ranging from strong analysis + repeated, field results (eg formal specs, code reviews) to clear theory, models, assessment, and massive field results (eg type-systems, usage-driven testing). The evidence is quite clear and favors them as effective with repeated successes that matched the theory. Many other aspects of programming I left off the list because the evidence or studies were not clear. Or were non-existent. Hence me not including the parent's strawmen ("most science") or red herring (Quorum) on a list of techniques with decades of measured results behind them across many software projects. Science, what little exists in our field, is on my side if the reader is focused on evidence aspects such as the Why, How, and Got Job Done. They collectively make up The Right Thing in robust, system development. YMMV for individual techniques or tools on a given project. Pick best tools for the job, prioritize your development budget, and so on. Common sense. People looking at where to start dealing with recurring issues or overall robustness have an evidence-backed list in my main comment, though. There are also papers and books on each showing where to apply them and most effectively.

20 points by struppi 11y ago

Curious if the author has read anything by Watts Humphrey? He spent his entire career at SEI arguing this point and attempting to develop the tools and processes to bear it out. This article reads like a repetition of his arguments from A Disicipline for Software Engineering (1995).

nickpsecurity 11y ago
I learned in a discussion with "kragen" that "software engineering" sort of took on a life of its own that turned into a nightmare of management-focused stuff harmful to programming. Most programmers, especially mainstream crowd, have an aversion to the term from the permanent association. I started saying CompSci and focusing on prior techniques rather than "engineering" due to that permanent scarring. As in my other comment, I just focus on specific techniques and processes that help programmers do their job with evidence they work. It's my suggestion to you. I'm going to try to dig up and read that paper out of curiosity as I know many useful things came out of software engineering research. Most won't, though, the second they see "SEI" or "software engineering." So, might be best for us to just drop that and focus on improvements to "programming" or "development" with specific techniques.
struppi 11y ago
Original author here (sorry, was in bed when it came to the front page ;) ) I did not read that paper (book?), but will add it to my "To Read" pile. Thank you ;)

nickpsecurity 11y ago

I'm mixed about the post, esp the science part. The science of developing robust software is great and pretty consistent going back decades varying mostly in specific tools and tactics. Mainstream programming just doesn't apply it although more adoption in past decade of key techniques. Here's some computer science from the 1960's-1980's used in robust and secure system development (esp Orange Book B3 or CC EAL6) people might want to copy. I'm taking an empirical route where I reference techniques that were applied to many real-world projects with lessons learned in papers or studies that were consistent. All one can do with limited data & these aren't in order of importance. 1. Formal, non-English (eg math/logical) specifications of requirements or abstract design. English is ambiguous and misreadings of it caused countless errors, even back then. CompSci researchers tried formal specs with both English as a start and precise notations (eg Z, VDM, ASM's, statecharts) for clarity in specifics. Result was many inconsistencies caught in highly assured systems and protocol specs before coding even began. 2. High assurance stuff often used mathematical (formal) verification. Whether that worked or made sense was hit and miss. More on it later. Yet, virtually all of them said there was benefit in restrictions on the specs, design, and coding style to fit the provers' limitations. Essentially, they used boring constructs that were easy to analyse and this prevented/caught problems. Don't be too clever with design or code. Wirth and Hansen applied this to language design to bake safety & comprehension in with minimal to low loss in performance. Note: Led to Nick P's Law of Trustworthy Systems: "Tried and true beats novel or new." Always the default. 3. Dijkstra's THE project showed that modular, layered design with careful attention to interfaces (and interface checks) makes for most robust and maintainable software. Later results confirmed this where each module must fit in your head and control graph that's pretty predictable with minimal cycles prevented all kinds of local-becomes-global issues. Many systems flawless (or nearly so) in production were built this way. Dijkstra correctly noted that it was very hard to do this even for smart people and average developer might screw structuring up a lot. Solid prediction... but still worth striving for improvement here. 4. Fagan ran empirical studies at IBM that showed a regular, systematic, code review process caught many problems, even what tests missed. Turned that into formal inspections with the periodicity and prioritizing tuned per organization for right cost-benefit. Was generalized to whole SDLC by others in high robustness areas. Improved every project that used it from then on. Exactly what parameters to use is still open-ended but periodically looking for well-known flaws with reference sheet always works. 5. Testing for every feature, code-path, prior issues outside of code base, and common use-case. All of these have shown repeated benefits. There's a cut-off point for each that's still an open, research problem. However, at a minimum, usage-based testing and regression testing helped many projects achieve either zero or near-zero, user-facing defects in production. That's a very important differentiator as 100 bugs user never experiences is better than 5 that they do regularly. Mills' Cleanroom process combined simple implementation, code review, and usage-testing for insanely-high, statistically-certifiable quality even for amateur teams. 6. By around 60's-70's, it became clear that the language you choose has a significant effect on productivity, defects, maintenance, and integration. Numerous studies were run in industry and military comparing various ones. Certain languages (eg Ada) showed vastly lower defects, equal/better productivity, and great maintenance/integration in every study. Haven't seen many such studies since the 90's and most aren't constructed well to eliminate bias. However, it's grounded in science to claim that certain language choices prevent common negatives and encourage positives. So, it follows to adopt languages that make robust development easier. 7. By the 80's or 90's, it was clear that computers were better at finding certain problems in specs and code than humans. This gave rise to methodologies that put models of system or code into model-checkers and provers to show certain properties always hold (the good) or never show up (the bad). Used successfully with high-assurance safety and security critical systems with results ranging from "somewhat beneficial" to "caught stuff we'd never see or test for." Back then it was unclear how applicable it was. Recent work by Chlipala, Leroy, et al show near perfect results in practice when specs/proofs are right and much wider application than before. Lots of tooling and prior examples means this is a proven way of getting extra quality where high-stakes are worth the cost and where core functionality doesn't change often.. The CompCert C compiler, Eiffel's SCOOP concurrency scheme, and Navy team's EAL7 IPsec VPN are good examples. 8. Static analysis, aka "lightweight formal methods," were devised to deal with specialized skills and labor of above. Getting to the point, tools like Astree Analyzer or SPARK Ada can prove absence of common flaws with little to no false positives without need for mathematicians in the company. Just a half dozen of these tools by themselves found tons of vulnerabilities in real-world software that passed human review and testing. Enough said, eh? 9. Software that succeeded with testing often failed when random stuff came at it, especially malware. This led to various fault-injection methods like fuzz testing to simulate that and find breaking points. The huge number of defects, esp in file formats & protocol engines, found via this method argues for its effectiveness in improving quality. It ties in with stuff above in that well-written code that validates input at interface and preserves invariants throughout execution should simply disregard (or report) such erroneous input. 10. Interface errors themselves posed something like 80+% of problems. This was noted as far back as the 60's in Apollo project when Margaret Hamilton invented software engineering, fault-tolerance, and specification techniques to fight it. Dijkstra and Hoare pushed for pre- and post-conditions plus specific invariants to document the assumptions of code during procedure calls. Modern version is called Design by Contract in Eiffel, Ada, and numerous other languages (even asserts in C). Many deployments and tests showed such interface checks caught many issues, esp assumption violations when new code extended or modified legacy. 11. Concurrency issues caused all kinds of problems. Techniques were devised by Hansen (Concurrent Pascal) and later Meyer et al (SCOOP) to mostly immunize against them at language level with acceptable performance. Languages without that, especially Java, later got brilliant tooling that could reliably find race conditions, deadlocks, or livelocks. Use of any method inevitably found problems in production code that had escaped detection. So, using prior, proven methods to immunize against or detect common errors in concurrency is A Good Thing. Note that shared-nothing, event-driven architectures also emerged but I have less data on them outside that some (NonStop, Erlang) worked extremely well. The above are just a few things that computer science established with supporting evidence from real-world projects so long ago that Windows didn't exist. Anyone applying these lessons got benefits in terms of code quality, security, and maintainability. The rare few applying most or all of them, mainly high assurance community, got results along lines of space shuttle control code with extremely, low defects or zero in production. So, given the past and present results of these methods every time they're put to the test, I'm irritated every time another person talks like there's no good science to quality software. I just listed a bunch of it, it's been tested in production as scientific method requires thousands of time, tweaked probably hundreds, and core approaches remained even if tactics got modified. Now people can feel free to use and improve on the science. CompSci continues to in every area I listed with a chunk of proprietary and FOSS developers using a subset of the techniques. Just need more uptake. Use what's proven. And do note that there's plenty of examples for specific design and implementation decisions for common types of functionality. Many things that were shown to work or not work that could be encoded in libraries, DSL's, templates, whatever. No excuse except for our field's continual failure to learn and hand down the lessons from the past.

jack9 11y ago
> The science of developing robust software is great and pretty consistent going back decades varying mostly in specific tools and tactics Nope (since uh-huh/nuh-uh is the starting point you've taken). Most science is BAD SCIENCE and I have doubts about the claims made by either the author or the topics you reference. Almost all formal studies are about specific tools and tactics in small sample sizes. This isn't proven science as much as they are test results. - https://vimeo.com/9270320 https://vimeo.com/9270320 + http://quorumlanguage.com/evidence.php http://quorumlanguage.com/evidence.php should orient you toward what proof is and how we can reference it in a reasonable manner. There will be plenty of bias to argue about how the tests are conducted or what the data demonstrates.
- nickpsecurity 11y ago
  You reply with a combination of trolling and endorsing a specific link that you believe represents doing it better. The first thing I looked at, syntax study, was already weaker than prior work I've seen on the subject. The language choice was highly biased while not focusing on enough metrics for comparison of strengths and weaknesses. Between field use and its first "evidence," your Quorum is already weaker than most of what I presented in terms of empirical weight. Due to the trolling, I'm hesistant to even watch the video in case his presentation is something likes yours in terms of content. @ non-trolling, HN readers also concerned about science in software Back to science, though, given other readers might be interested in that point. Software development is a combination of human- and machine-driven processes to turn some requirements into working, maintainable code. A subset of this, significant it turns out, can be analyzed for effectiveness with enough different case studies, sample sizes, objective measures, and repeats to make a claim about it with believable evidence. For process stuff, it's not going to be as logically formulated or minutely analyzed as a math equation or something. Not usually, anyway. Instead, it works more like this: 1. A recurring problem exists in the development process or artifact. 2. Measurements are taken of how often that problem occurs in general and for specific scenarios. 3. A hypothesis is formed as to why it exists and a solution to it. 4. The hypothesis is tested by applying the technique to several projects likely to experience the problem with objective records on outcomes and subjective accounts from users for a take on less tangible aspects. 5. Significant reductions in the problem in the objective data provide supporting evidence that the theory is correct. 6. The theory (solution) is tested by others against the problem to find more supporting evidence or counter-examples to the claim. 7. If it's correct, the results continue to speak for themselves in it being a solution with likely modifications as the problem and solution space are further explored. Note: If it's performed by humans, it's wise to either look out for or control for issues like Hawthorne effect. Must be sure that it's the overall solution that's working rather than people's heightened attention to an issue being studied. The above 1-7 have applied to my list of techniques to varying degrees ranging from strong analysis + repeated, field results (eg formal specs, code reviews) to clear theory, models, assessment, and massive field results (eg type-systems, usage-driven testing). The evidence is quite clear and favors them as effective with repeated successes that matched the theory. Many other aspects of programming I left off the list because the evidence or studies were not clear. Or were non-existent. Hence me not including the parent's strawmen ("most science") or red herring (Quorum) on a list of techniques with decades of measured results behind them across many software projects. Science, what little exists in our field, is on my side if the reader is focused on evidence aspects such as the Why, How, and Got Job Done. They collectively make up The Right Thing in robust, system development. YMMV for individual techniques or tools on a given project. Pick best tools for the job, prioritize your development budget, and so on. Common sense. People looking at where to start dealing with recurring issues or overall robustness have an evidence-backed list in my main comment, though. There are also papers and books on each showing where to apply them and most effectively.