3 ms·

> The science of developing robust software is great and pretty consistent going back decades varying mostly in specific tools and tactics Nope (since uh-huh/n

by jack9 11y ago

> The science of developing robust software is great and pretty consistent going back decades varying mostly in specific tools and tactics Nope (since uh-huh/nuh-uh is the starting point you've taken). Most science is BAD SCIENCE and I have doubts about the claims made by either the author or the topics you reference. Almost all formal studies are about specific tools and tactics in small sample sizes. This isn't proven science as much as they are test results. - https://vimeo.com/9270320 https://vimeo.com/9270320 + http://quorumlanguage.com/evidence.php http://quorumlanguage.com/evidence.php should orient you toward what proof is and how we can reference it in a reasonable manner. There will be plenty of bias to argue about how the tests are conducted or what the data demonstrates.

nickpsecurity 11y ago
You reply with a combination of trolling and endorsing a specific link that you believe represents doing it better. The first thing I looked at, syntax study, was already weaker than prior work I've seen on the subject. The language choice was highly biased while not focusing on enough metrics for comparison of strengths and weaknesses. Between field use and its first "evidence," your Quorum is already weaker than most of what I presented in terms of empirical weight. Due to the trolling, I'm hesistant to even watch the video in case his presentation is something likes yours in terms of content. @ non-trolling, HN readers also concerned about science in software Back to science, though, given other readers might be interested in that point. Software development is a combination of human- and machine-driven processes to turn some requirements into working, maintainable code. A subset of this, significant it turns out, can be analyzed for effectiveness with enough different case studies, sample sizes, objective measures, and repeats to make a claim about it with believable evidence. For process stuff, it's not going to be as logically formulated or minutely analyzed as a math equation or something. Not usually, anyway. Instead, it works more like this: 1. A recurring problem exists in the development process or artifact. 2. Measurements are taken of how often that problem occurs in general and for specific scenarios. 3. A hypothesis is formed as to why it exists and a solution to it. 4. The hypothesis is tested by applying the technique to several projects likely to experience the problem with objective records on outcomes and subjective accounts from users for a take on less tangible aspects. 5. Significant reductions in the problem in the objective data provide supporting evidence that the theory is correct. 6. The theory (solution) is tested by others against the problem to find more supporting evidence or counter-examples to the claim. 7. If it's correct, the results continue to speak for themselves in it being a solution with likely modifications as the problem and solution space are further explored. Note: If it's performed by humans, it's wise to either look out for or control for issues like Hawthorne effect. Must be sure that it's the overall solution that's working rather than people's heightened attention to an issue being studied. The above 1-7 have applied to my list of techniques to varying degrees ranging from strong analysis + repeated, field results (eg formal specs, code reviews) to clear theory, models, assessment, and massive field results (eg type-systems, usage-driven testing). The evidence is quite clear and favors them as effective with repeated successes that matched the theory. Many other aspects of programming I left off the list because the evidence or studies were not clear. Or were non-existent. Hence me not including the parent's strawmen ("most science") or red herring (Quorum) on a list of techniques with decades of measured results behind them across many software projects. Science, what little exists in our field, is on my side if the reader is focused on evidence aspects such as the Why, How, and Got Job Done. They collectively make up The Right Thing in robust, system development. YMMV for individual techniques or tools on a given project. Pick best tools for the job, prioritize your development budget, and so on. Common sense. People looking at where to start dealing with recurring issues or overall robustness have an evidence-backed list in my main comment, though. There are also papers and books on each showing where to apply them and most effectively.
- jack9 11y ago
  > Quorum is already weaker than most of what I presented in terms of empirical weight Saying you have presented empirical weight, is not compelling when you haven't and, frankly, can't. It's telling that the rest of your pointless counter-troll? is devoid of proof when leading with "back to science", which is sad.
  - nickpsecurity 11y ago
    "Saying you have presented empirical weight, is not compelling when you haven't and, frankly, can't." The proof is in the literature I'm referencing rather than me dumping all of it in the comment box. Anyone studying PL, QA, and so on literature should have seen research on effectiveness of basic techniques. Here's a few examples of methods of evaluation I claimed were scientific vs people guessing that their stuff seems better in vague ways. 1. Experiments conducted between similarly skilled groups doing same things with only difference being language or methodology. Those with new technique do significantly better in ways that match the expected properties of technique. Hypothesis based on real-world data, experimental confirmation, fix/reject if necessary, rinse, repeat. 2. Study of problems in software from a specific language identifies specific features as root cause, proposes a fix, the fix is implemented in significant test cases, and the problem goes away entirely or is significantly reduces. This is replicated across groups and codebases. Easiest example here is safe (Ada) or automatic (GC) techniques to manage memory to reduce or eliminate its many, common issues. Supported by theory, sometimes proofs, & countless applications in industry. 3. A certain verification approach espouses specific modeling, analysis, refinement, etc approach to achieve certain goals. The work on formal side translates into finding defects, enhancing performance, etc on the code side. Implies the model or method were correct, esp with diverse replication of results. This happened a lot in Orange Book days for catching security violations but most applicable to hardware with formal, equivalence checking. Theory, some proofs, and confirmation in production. 4. In some cases, properties are mathematically formulated, proven in theorem prover, checked by humans + machines, and tested on real-world examples to further validate them. Two words: type systems. Again, theories, proofs, and real-world validation. Tons of testing, too. Most of the best evidence in the literature works something like that. You're saying none of the above constitute scientific evidence for a claim about a software tool or method? That's a big claim itself that contradicts what most of research community thinks. The people successfully finding bugs with Astree, avoiding out-of-bounds with languages that don't allow it, catching safety violations with protocol checkers, and so on will all be surprised about your claim. They and I thought the tools would work based on evidence presented and field results that confirmed the theories. Turns out we were all drinking spiked Kool-Aid, leading to hallucinations that followed. Or wrong on a technicality. Got a link that briefly explains why all of that is unscientific and what methods count as scientific evidence in software tools? (without 1hr videos) Note: Looking up more on Quorum is on my agenda. Already saved What Works Clearinghouse document. I'm not rejecting it outright in our discussion. Yet, you've indicated proven models, worked case studies, analysis techniques effective on 1+ million plus samples, mathematical proofs... that none of this is science. So, I'm starting with wanting an easy resource on what you think counts as scientific evaluation, argument, and method. Then, I can go from there in future discussions about specific topics and what science says about them.

by jack9 11y ago

nickpsecurity 11y ago

You reply with a combination of trolling and endorsing a specific link that you believe represents doing it better. The first thing I looked at, syntax study, was already weaker than prior work I've seen on the subject. The language choice was highly biased while not focusing on enough metrics for comparison of strengths and weaknesses. Between field use and its first "evidence," your Quorum is already weaker than most of what I presented in terms of empirical weight. Due to the trolling, I'm hesistant to even watch the video in case his presentation is something likes yours in terms of content. @ non-trolling, HN readers also concerned about science in software Back to science, though, given other readers might be interested in that point. Software development is a combination of human- and machine-driven processes to turn some requirements into working, maintainable code. A subset of this, significant it turns out, can be analyzed for effectiveness with enough different case studies, sample sizes, objective measures, and repeats to make a claim about it with believable evidence. For process stuff, it's not going to be as logically formulated or minutely analyzed as a math equation or something. Not usually, anyway. Instead, it works more like this: 1. A recurring problem exists in the development process or artifact. 2. Measurements are taken of how often that problem occurs in general and for specific scenarios. 3. A hypothesis is formed as to why it exists and a solution to it. 4. The hypothesis is tested by applying the technique to several projects likely to experience the problem with objective records on outcomes and subjective accounts from users for a take on less tangible aspects. 5. Significant reductions in the problem in the objective data provide supporting evidence that the theory is correct. 6. The theory (solution) is tested by others against the problem to find more supporting evidence or counter-examples to the claim. 7. If it's correct, the results continue to speak for themselves in it being a solution with likely modifications as the problem and solution space are further explored. Note: If it's performed by humans, it's wise to either look out for or control for issues like Hawthorne effect. Must be sure that it's the overall solution that's working rather than people's heightened attention to an issue being studied. The above 1-7 have applied to my list of techniques to varying degrees ranging from strong analysis + repeated, field results (eg formal specs, code reviews) to clear theory, models, assessment, and massive field results (eg type-systems, usage-driven testing). The evidence is quite clear and favors them as effective with repeated successes that matched the theory. Many other aspects of programming I left off the list because the evidence or studies were not clear. Or were non-existent. Hence me not including the parent's strawmen ("most science") or red herring (Quorum) on a list of techniques with decades of measured results behind them across many software projects. Science, what little exists in our field, is on my side if the reader is focused on evidence aspects such as the Why, How, and Got Job Done. They collectively make up The Right Thing in robust, system development. YMMV for individual techniques or tools on a given project. Pick best tools for the job, prioritize your development budget, and so on. Common sense. People looking at where to start dealing with recurring issues or overall robustness have an evidence-backed list in my main comment, though. There are also papers and books on each showing where to apply them and most effectively.

jack9 11y ago
> Quorum is already weaker than most of what I presented in terms of empirical weight Saying you have presented empirical weight, is not compelling when you haven't and, frankly, can't. It's telling that the rest of your pointless counter-troll? is devoid of proof when leading with "back to science", which is sad.
- nickpsecurity 11y ago
  "Saying you have presented empirical weight, is not compelling when you haven't and, frankly, can't." The proof is in the literature I'm referencing rather than me dumping all of it in the comment box. Anyone studying PL, QA, and so on literature should have seen research on effectiveness of basic techniques. Here's a few examples of methods of evaluation I claimed were scientific vs people guessing that their stuff seems better in vague ways. 1. Experiments conducted between similarly skilled groups doing same things with only difference being language or methodology. Those with new technique do significantly better in ways that match the expected properties of technique. Hypothesis based on real-world data, experimental confirmation, fix/reject if necessary, rinse, repeat. 2. Study of problems in software from a specific language identifies specific features as root cause, proposes a fix, the fix is implemented in significant test cases, and the problem goes away entirely or is significantly reduces. This is replicated across groups and codebases. Easiest example here is safe (Ada) or automatic (GC) techniques to manage memory to reduce or eliminate its many, common issues. Supported by theory, sometimes proofs, & countless applications in industry. 3. A certain verification approach espouses specific modeling, analysis, refinement, etc approach to achieve certain goals. The work on formal side translates into finding defects, enhancing performance, etc on the code side. Implies the model or method were correct, esp with diverse replication of results. This happened a lot in Orange Book days for catching security violations but most applicable to hardware with formal, equivalence checking. Theory, some proofs, and confirmation in production. 4. In some cases, properties are mathematically formulated, proven in theorem prover, checked by humans + machines, and tested on real-world examples to further validate them. Two words: type systems. Again, theories, proofs, and real-world validation. Tons of testing, too. Most of the best evidence in the literature works something like that. You're saying none of the above constitute scientific evidence for a claim about a software tool or method? That's a big claim itself that contradicts what most of research community thinks. The people successfully finding bugs with Astree, avoiding out-of-bounds with languages that don't allow it, catching safety violations with protocol checkers, and so on will all be surprised about your claim. They and I thought the tools would work based on evidence presented and field results that confirmed the theories. Turns out we were all drinking spiked Kool-Aid, leading to hallucinations that followed. Or wrong on a technicality. Got a link that briefly explains why all of that is unscientific and what methods count as scientific evidence in software tools? (without 1hr videos) Note: Looking up more on Quorum is on my agenda. Already saved What Works Clearinghouse document. I'm not rejecting it outright in our discussion. Yet, you've indicated proven models, worked case studies, analysis techniques effective on 1+ million plus samples, mathematical proofs... that none of this is science. So, I'm starting with wanting an easy resource on what you think counts as scientific evaluation, argument, and method. Then, I can go from there in future discussions about specific topics and what science says about them.