4 ms·

>There is not such a thing as a bug "hit rate" because it would require knowledge of unknown bugs, which isn't possible. It is possible to estimate the number

>There is not such a thing as a bug "hit rate" because it would require knowledge of unknown bugs, which isn't possible. It is possible to estimate the number of things you have not seen. The German Tank Problem [0] is a famous example. For software, if you Alice and Bob independently look at a piece of code for bugs, and Bob finds 17, Alice find 21, and 16 overlap, then you have some knowledge of Alice and Bob's rate at finding bugs, and you can estimate the number of bugs expected to be left. As the sample sizes increase you have smaller and smaller confidence intervals, giving quite good knowledge of the count of unknown things. You are assuming once an audit is done, all bugs are found. This is rarely the case, and I doubt has ever been the case on software as complex/large as this piece. He is assuming an audit can find bugs, each with some probability. This is a quite reasonable position, one borne out by practice. His view is true in practice (and theory, since proving software correct is tremendously difficult except). There is ample literature on how many bugs are likely left after an audit, and this audit itself shows that these bugs remained after people (presumably) looked over the code previously. More effort/skill usually turns up more bugs. It's that simple. Do another audit with more skilled eyes, paid more money, given more time, and they will likely find bugs that this one did not. There is a large literature on estimating the number of bugs left after some are found. Here [1] is an example of how this is done. [0] https://en.wikipedia.org/wiki/German_tank_problem https://en.wikipedia.org/wiki/German_tank_problem [1] http://www.johndcook.com/blog/2010/07/13/lincoln-index/ http://www.johndcook.com/blog/2010/07/13/lincoln-index/

dimino 10y ago
His view is not correct, nor are the links you've provided directly relevant to this discussion. The "German Tank Problem" requires knowledge of variables that aren't present in the VeraCrypt security audit. In fact, your second link begs the question, as it requires the prior knowledge of a "bug hit rate", which is defined in the blog post as a fairly arbitrary number, with plenty of caveats, and requires multiple independent audits. In fact, from your linked blog, "Maybe the Lincoln estimate is not at all accurate, but it does tell you to be worried that there may be a lot more bugs to find since the overlap between the two bug lists was so small." I begin to wonder if you actually read your own links...
- ChrisLomont 10y ago
  From the link [1] "There’s a simple statistic called the Lincoln Index that lets you estimate the total number of errors based on the number of errors found" "Suppose two testers independently search for bugs. Let E1 be the number of errors the first tester finds and E2 the number of errors the second tester finds. Let S be the number of errors both testers find. The Lincoln Index estimates the total number of errors as E1 E2/S. " E1 = count from searcher 1 E2 = count from searcher 2 S = Number overlapping If S != 0, then a good estimate of the number of bugs in the code is E1E2/S. One can derive error bounds and confidence intervals if desired Nowhere does this formula need prior knowledge. All is needs is at least two testers to count the bugs each found and the overlaps. This is precisely* the example I gave above. As one analyzes how this works, one finds that this is likely lower than the true bug count. A simple reason is explained in "Beginning Software Engineering" by Rod Stephens, p197, which you can find on google books and read. Alternatively, try searching for "estimate number of software bugs" in Google scholar and you can see a vast literature based exactly on refining methods like this.
  - dimino 10y ago
    > Nowhere does this formula need prior knowledge I never said the Lincoln Index required prior knowledge, I said the German tank problem required prior knowledge that isn't available in the VeraCrypt security audit. Your example may have been copied from the blog post, but you failed to include any of the caveats that were present in the blog post, which are also present in the "vast literature" in Google Scholar. Rod Stephens even says, in the very citation you attempt to make, "Like all the other bug estimation techniques, this one isn’t perfect. It relies on the assumption that the testers have an equal chance to find any particular bug, and that’s probably not true." It does go on to say the Lincon Index, "probably underestimates the true number of bugs." but in that aspect, Rod Stephens disagrees with the body of work available both in your own links, and in the Google Scholar articles. He then cites Wikipedia, so in that he's basically admitting he's not an expert. I am not and have not been saying there are no further bugs in VeraCrypt. I am saying the likelihood of there being "many" more bugs in VeraCrypt is not available, given the data we currently have. We don't have E2, for example, since you're focused so intently on the Lincoln Index.

dimino 10y ago

His view is not correct, nor are the links you've provided directly relevant to this discussion. The "German Tank Problem" requires knowledge of variables that aren't present in the VeraCrypt security audit. In fact, your second link begs the question, as it requires the prior knowledge of a "bug hit rate", which is defined in the blog post as a fairly arbitrary number, with plenty of caveats, and requires multiple independent audits. In fact, from your linked blog, "Maybe the Lincoln estimate is not at all accurate, but it does tell you to be worried that there may be a lot more bugs to find since the overlap between the two bug lists was so small." I begin to wonder if you actually read your own links...

ChrisLomont 10y ago
From the link [1] "There’s a simple statistic called the Lincoln Index that lets you estimate the total number of errors based on the number of errors found" "Suppose two testers independently search for bugs. Let E1 be the number of errors the first tester finds and E2 the number of errors the second tester finds. Let S be the number of errors both testers find. The Lincoln Index estimates the total number of errors as E1 E2/S. " E1 = count from searcher 1 E2 = count from searcher 2 S = Number overlapping If S != 0, then a good estimate of the number of bugs in the code is E1E2/S. One can derive error bounds and confidence intervals if desired Nowhere does this formula need prior knowledge. All is needs is at least two testers to count the bugs each found and the overlaps. This is precisely* the example I gave above. As one analyzes how this works, one finds that this is likely lower than the true bug count. A simple reason is explained in "Beginning Software Engineering" by Rod Stephens, p197, which you can find on google books and read. Alternatively, try searching for "estimate number of software bugs" in Google scholar and you can see a vast literature based exactly on refining methods like this.
- dimino 10y ago
  > Nowhere does this formula need prior knowledge I never said the Lincoln Index required prior knowledge, I said the German tank problem required prior knowledge that isn't available in the VeraCrypt security audit. Your example may have been copied from the blog post, but you failed to include any of the caveats that were present in the blog post, which are also present in the "vast literature" in Google Scholar. Rod Stephens even says, in the very citation you attempt to make, "Like all the other bug estimation techniques, this one isn’t perfect. It relies on the assumption that the testers have an equal chance to find any particular bug, and that’s probably not true." It does go on to say the Lincon Index, "probably underestimates the true number of bugs." but in that aspect, Rod Stephens disagrees with the body of work available both in your own links, and in the Google Scholar articles. He then cites Wikipedia, so in that he's basically admitting he's not an expert. I am not and have not been saying there are no further bugs in VeraCrypt. I am saying the likelihood of there being "many" more bugs in VeraCrypt is not available, given the data we currently have. We don't have E2, for example, since you're focused so intently on the Lincoln Index.