5 ms·

Binary search is not random permutation. The hypothesis is "problem lies in this half of the codebase". Hey, if your test passes after commenting out some stu

Binary search is not random permutation. The hypothesis is "problem lies in this half of the codebase". Hey, if your test passes after commenting out some stuff, great.

bcantrill 11y ago
And if that hypothesis is based on data, great. And if there is no other alternative, fine -- but this technique should be viewed as a last resort (and an unusual one), not a first step.
- ectoplasm 11y ago
  Wait a minute. First of all, the data is simply "there is a reproducible bug in the program". The real hypothesis is that a small, localized portion of the code is responsible for the defect. To find it, you can try binary search. If you have a better idea of the location, then by all means. Second, when you have a bug and you write a unit test, you are effectively commenting out the entire codebase except for the function under test. When you have a compiler error, whether it's a syntax/semantics bug in your code or a bug in the compiler, sometimes you need to produce a minimal example, so you have to cut cut cut until the bug is just barely provoked. When you have a pipeline of data transforms, and the end result is suddenly borked, it can work to chop off half the transforms and look at the result. When latex is crashing for some unintelligible reason, just comment out half of your document and see if the problem goes away. Sure it's really dumb if you're just excluding a* .c through m* .c (spaces due to HN formatting rules), but figuring out if the problem is in the first or second half of main is not outrageous. I don't think the guy was presenting it as the first step ("If you have no idea where your bug lives"), but I do agree that it comes across as a little naive, since he should have talked about all of the other techniques available first. So the problem isn't so much the lack of a hypothesis, but the inefficient experimental approach of using a brute force technique indiscriminately. I think the last resort is reached a little sooner for some types of bugs and some experience levels (language, environment, codebase, programming), and yes in many cases it won't even do anything for you. Personally I always liked dtrace. This guy gave a demo of it at my university once, I thought it was great, one of the best talks I've seen.
  - dap 11y ago
    Part of the scientific method is that you generate hypotheses based on the data you have, not that you generate random hypotheses. It's a directed way of thinking. > The real hypothesis is that a small, localized portion of the code is responsible for the defect. To find it, you can try binary search. In your example, you used this as an assumption, not a hypothesis. And it's not quite right: the assumption you made is that the _presence_ of a small portion of code is responsible for the defect. That's very different than saying that the code is more broadly responsible for the defect. In my experience, very few bugs are caused by the mere presence of some code. > When you have a compiler error, whether it's a syntax/semantics bug in your code or a bug in the compiler, sometimes you need to produce a minimal example, so you have to cut cut cut until the bug is just barely provoked. When you have a pipeline of data transforms, and the end result is suddenly borked, it can work to chop off half the transforms and look at the result. When latex is crashing for some unintelligible reason, just comment out half of your document and see if the problem goes away. Those are fine solutions for those very specific, very simple problems. Given the problem space, the assumption that the error is caused directly by the presence of some input is well-founded.
    - ectoplasm 11y ago
      Sorry, I'm kind of confused here. Why is the approach I'm defending considered random and not based on data? Is the generation of random hypotheses in general considered unscientific? What about fuzz testing or pharmaceutical R&D? What is the precise difference between hypotheses and assumptions in the context of the scientific method? What is the difference between the presence of a small portion of code being responsible and the code being more broadly responsible? Why the emphasis on presence?
      - rjurney 11y ago
        Because it seems you're making an assumption about the problem being reproducible and identifiable by running half the code. I've never found this to be the case. Ever. Its a strange idea. Where does this come from? This isn't a hypothesis based on data, it is an assumption. And a bizarre one.

Binary search is not random permutation. The hypothesis is "problem lies in this half of the codebase". Hey, if your test passes after commenting out some stuff, great.

bcantrill 11y ago

And if that hypothesis is based on data, great. And if there is no other alternative, fine -- but this technique should be viewed as a last resort (and an unusual one), not a first step.

ectoplasm 11y ago
Wait a minute. First of all, the data is simply "there is a reproducible bug in the program". The real hypothesis is that a small, localized portion of the code is responsible for the defect. To find it, you can try binary search. If you have a better idea of the location, then by all means. Second, when you have a bug and you write a unit test, you are effectively commenting out the entire codebase except for the function under test. When you have a compiler error, whether it's a syntax/semantics bug in your code or a bug in the compiler, sometimes you need to produce a minimal example, so you have to cut cut cut until the bug is just barely provoked. When you have a pipeline of data transforms, and the end result is suddenly borked, it can work to chop off half the transforms and look at the result. When latex is crashing for some unintelligible reason, just comment out half of your document and see if the problem goes away. Sure it's really dumb if you're just excluding a* .c through m* .c (spaces due to HN formatting rules), but figuring out if the problem is in the first or second half of main is not outrageous. I don't think the guy was presenting it as the first step ("If you have no idea where your bug lives"), but I do agree that it comes across as a little naive, since he should have talked about all of the other techniques available first. So the problem isn't so much the lack of a hypothesis, but the inefficient experimental approach of using a brute force technique indiscriminately. I think the last resort is reached a little sooner for some types of bugs and some experience levels (language, environment, codebase, programming), and yes in many cases it won't even do anything for you. Personally I always liked dtrace. This guy gave a demo of it at my university once, I thought it was great, one of the best talks I've seen.
- dap 11y ago
  Part of the scientific method is that you generate hypotheses based on the data you have, not that you generate random hypotheses. It's a directed way of thinking. > The real hypothesis is that a small, localized portion of the code is responsible for the defect. To find it, you can try binary search. In your example, you used this as an assumption, not a hypothesis. And it's not quite right: the assumption you made is that the _presence_ of a small portion of code is responsible for the defect. That's very different than saying that the code is more broadly responsible for the defect. In my experience, very few bugs are caused by the mere presence of some code. > When you have a compiler error, whether it's a syntax/semantics bug in your code or a bug in the compiler, sometimes you need to produce a minimal example, so you have to cut cut cut until the bug is just barely provoked. When you have a pipeline of data transforms, and the end result is suddenly borked, it can work to chop off half the transforms and look at the result. When latex is crashing for some unintelligible reason, just comment out half of your document and see if the problem goes away. Those are fine solutions for those very specific, very simple problems. Given the problem space, the assumption that the error is caused directly by the presence of some input is well-founded.
  - ectoplasm 11y ago
    Sorry, I'm kind of confused here. Why is the approach I'm defending considered random and not based on data? Is the generation of random hypotheses in general considered unscientific? What about fuzz testing or pharmaceutical R&D? What is the precise difference between hypotheses and assumptions in the context of the scientific method? What is the difference between the presence of a small portion of code being responsible and the code being more broadly responsible? Why the emphasis on presence?
    - rjurney 11y ago
      Because it seems you're making an assumption about the problem being reproducible and identifiable by running half the code. I've never found this to be the case. Ever. Its a strange idea. Where does this come from? This isn't a hypothesis based on data, it is an assumption. And a bizarre one.