3 ms·

It doesn't matter what I think, it matters what can be supported by evidence, and your emotional narrative aside, the number of bugs found in a piece of softwar

by dimino 10y ago

It doesn't matter what I think, it matters what can be supported by evidence, and your emotional narrative aside, the number of bugs found in a piece of software does not indicate there are more bugs to be found. You did make the argument I presented above, even if you didn't mean to. It is not a type error, it is a conclusion that you're trying to imply but don't want to directly state because it's indefensible, and you know it. There is not such a thing as a bug "hit rate" because it would require knowledge of unknown bugs, which isn't possible. To be abundantly clear, I find your emotional narrative entirely unhelpful to this conversation, but to illustrate how unhelpful, I will play along: UltraCrypt clearly wasn't audited correctly, and will have more bugs in it given the complexity of the problem space and the lack of issues found. SuperCrypt was clearly audited well and will therefore have fewer bugs if Charles continued looking and when Alice looks next week. This, I repeat, is wholly unhelpful, because the same information can be used to create arguments for either point, and is therefore not useful.

greenshackle2 10y ago
Uh, on the contrary, I've found your narrative helpful in showing how one might arrive at the conclusion that finding more bugs is evidence that more won't be found later. I often try to back up abstract arguments with a concrete illustration, it makes things clearer for me. I've clearly failed to make myself understood, I don't think it's useful to continue this conversation.
- dimino 10y ago
  I'm saddened how you'd rather end conversation than change your beliefs.
Kenji 10y ago
I don't understand how you still can't get it. Let us try again. Suppose you're a professor and you had a super-productive student who wrote 100 very interesting papers. You only have time to read 50 of them, selected at random. However, you notice that in 10 of these 50 papers, there are mathematical errors. What do you expect of the remaining papers, how many errors will there be? This is basic statistics and nothing about this argument is emotional. What if you read 50 papers and only one contained mistakes? Would you expect more or less errors now in the remaining papers?
- dimino 10y ago
  Your analogy isn't relevant to this conversation.
  - Kenji 10y ago
    Do you really think they read through the entire source code and understood every facet of it? Code audits are nothing but sampling. Just like in the example I mentioned. I don't mind people disagreeing with me, but you could at least make an effort to tell me why. Unless this bores you, of course.
    - dimino 10y ago
      I don't think you're understanding the conversation that's going on, and I do find it boring to re-state things I've already written.
      - Kenji 10y ago
        I respect that. So, let us not continue and I hope we didn't bore you too much. I wish you a pleasant day.
ChrisLomont 10y ago
>There is not such a thing as a bug "hit rate" because it would require knowledge of unknown bugs, which isn't possible. It is possible to estimate the number of things you have not seen. The German Tank Problem [0] is a famous example. For software, if you Alice and Bob independently look at a piece of code for bugs, and Bob finds 17, Alice find 21, and 16 overlap, then you have some knowledge of Alice and Bob's rate at finding bugs, and you can estimate the number of bugs expected to be left. As the sample sizes increase you have smaller and smaller confidence intervals, giving quite good knowledge of the count of unknown things. You are assuming once an audit is done, all bugs are found. This is rarely the case, and I doubt has ever been the case on software as complex/large as this piece. He is assuming an audit can find bugs, each with some probability. This is a quite reasonable position, one borne out by practice. His view is true in practice (and theory, since proving software correct is tremendously difficult except). There is ample literature on how many bugs are likely left after an audit, and this audit itself shows that these bugs remained after people (presumably) looked over the code previously. More effort/skill usually turns up more bugs. It's that simple. Do another audit with more skilled eyes, paid more money, given more time, and they will likely find bugs that this one did not. There is a large literature on estimating the number of bugs left after some are found. Here [1] is an example of how this is done. [0] https://en.wikipedia.org/wiki/German_tank_problem https://en.wikipedia.org/wiki/German_tank_problem [1] http://www.johndcook.com/blog/2010/07/13/lincoln-index/ http://www.johndcook.com/blog/2010/07/13/lincoln-index/
- dimino 10y ago
  His view is not correct, nor are the links you've provided directly relevant to this discussion. The "German Tank Problem" requires knowledge of variables that aren't present in the VeraCrypt security audit. In fact, your second link begs the question, as it requires the prior knowledge of a "bug hit rate", which is defined in the blog post as a fairly arbitrary number, with plenty of caveats, and requires multiple independent audits. In fact, from your linked blog, "Maybe the Lincoln estimate is not at all accurate, but it does tell you to be worried that there may be a lot more bugs to find since the overlap between the two bug lists was so small." I begin to wonder if you actually read your own links...

by dimino 10y ago

greenshackle2 10y ago

Uh, on the contrary, I've found your narrative helpful in showing how one might arrive at the conclusion that finding more bugs is evidence that more won't be found later. I often try to back up abstract arguments with a concrete illustration, it makes things clearer for me. I've clearly failed to make myself understood, I don't think it's useful to continue this conversation.

dimino 10y ago
I'm saddened how you'd rather end conversation than change your beliefs.

Kenji 10y ago

I don't understand how you still can't get it. Let us try again. Suppose you're a professor and you had a super-productive student who wrote 100 very interesting papers. You only have time to read 50 of them, selected at random. However, you notice that in 10 of these 50 papers, there are mathematical errors. What do you expect of the remaining papers, how many errors will there be? This is basic statistics and nothing about this argument is emotional. What if you read 50 papers and only one contained mistakes? Would you expect more or less errors now in the remaining papers?

dimino 10y ago
Your analogy isn't relevant to this conversation.
- Kenji 10y ago
  Do you really think they read through the entire source code and understood every facet of it? Code audits are nothing but sampling. Just like in the example I mentioned. I don't mind people disagreeing with me, but you could at least make an effort to tell me why. Unless this bores you, of course.
  - dimino 10y ago
    I don't think you're understanding the conversation that's going on, and I do find it boring to re-state things I've already written.
    - Kenji 10y ago
      I respect that. So, let us not continue and I hope we didn't bore you too much. I wish you a pleasant day.

ChrisLomont 10y ago

>There is not such a thing as a bug "hit rate" because it would require knowledge of unknown bugs, which isn't possible. It is possible to estimate the number of things you have not seen. The German Tank Problem [0] is a famous example. For software, if you Alice and Bob independently look at a piece of code for bugs, and Bob finds 17, Alice find 21, and 16 overlap, then you have some knowledge of Alice and Bob's rate at finding bugs, and you can estimate the number of bugs expected to be left. As the sample sizes increase you have smaller and smaller confidence intervals, giving quite good knowledge of the count of unknown things. You are assuming once an audit is done, all bugs are found. This is rarely the case, and I doubt has ever been the case on software as complex/large as this piece. He is assuming an audit can find bugs, each with some probability. This is a quite reasonable position, one borne out by practice. His view is true in practice (and theory, since proving software correct is tremendously difficult except). There is ample literature on how many bugs are likely left after an audit, and this audit itself shows that these bugs remained after people (presumably) looked over the code previously. More effort/skill usually turns up more bugs. It's that simple. Do another audit with more skilled eyes, paid more money, given more time, and they will likely find bugs that this one did not. There is a large literature on estimating the number of bugs left after some are found. Here [1] is an example of how this is done. [0] https://en.wikipedia.org/wiki/German_tank_problem https://en.wikipedia.org/wiki/German_tank_problem [1] http://www.johndcook.com/blog/2010/07/13/lincoln-index/ http://www.johndcook.com/blog/2010/07/13/lincoln-index/

dimino 10y ago
His view is not correct, nor are the links you've provided directly relevant to this discussion. The "German Tank Problem" requires knowledge of variables that aren't present in the VeraCrypt security audit. In fact, your second link begs the question, as it requires the prior knowledge of a "bug hit rate", which is defined in the blog post as a fairly arbitrary number, with plenty of caveats, and requires multiple independent audits. In fact, from your linked blog, "Maybe the Lincoln estimate is not at all accurate, but it does tell you to be worried that there may be a lot more bugs to find since the overlap between the two bug lists was so small." I begin to wonder if you actually read your own links...