3 ms·

Could I? How long would it take? Would I really get the same results or just similar results? If it doesn't work, how can I find out where the bug is (the paper

by lysium 13y ago

Could I? How long would it take? Would I really get the same results or just similar results? If it doesn't work, how can I find out where the bug is (the paper, my interpretation, my implementation)? Compare this to physics: if you drop this ball from that height under these circumstances, it will take that long. Everybody (with the right equipment) can reproduce this result quite easily. Not so in this case, which ironically could be even more reproducible than any physics experiment if it wasn't closed source.

betterunix 13y ago
Compare it to physics indeed: https://en.wikipedia.org/wiki/Large_hadron_collider https://en.wikipedia.org/wiki/Large_hadron_collider
- lysium 13y ago
  You don't try to argue that LHC is closed source, do you?
mikeash 13y ago
Seems to be that rewriting the software is healthier, and a better analog to physics experiments. When you reproduce that physics result, you're using a different ball in a different location and dropping it from a different structure. If you simply took their program and re-ran it, it's like taking the exact same ball they originally used, climbing the exact same structure they used, and dropping it in the exact same spot. When you then get the same result, it doesn't tell you much, because you don't know if the result is coming from the ball, the structure, the location, or just a general universal phenomenon.
- luke-stanley 13y ago
  Rewriting from scratch doesn't help run more experiments, it slows things down.
  - mikeash 13y ago
    The point is not to run more of them, but to run them better. What good does it do to run identical software ten times? These computer-thingies are pretty deterministic. You're basically guaranteed to get the same result each time, but you have no idea if the result is because of the technique being presented, a quirk of the implementation, or just a bug.
    - luke-stanley 13y ago
      Of course, access to source allows running better experiments! Writing the same algorithms from scratch is a clear waste of time! Please stop defending silliness.
- lysium 13y ago
  I can start right away changing any one of these to find out, instead of spending time to try to write a program that may well not exist in the first place. (In this particular case, I admit that posting 1.2k bugs hints to its existence.)
tptacek 13y ago
The important result of the paper isn't the specific list of bugs they found, which after all: (a) depends on the software they fed to the tool and (b) is verifiable without static analysis (just try to trigger the bug). The important result is the validation of the approach, which makes the paper more important than the tool.

by lysium 13y ago

betterunix 13y ago

Compare it to physics indeed: https://en.wikipedia.org/wiki/Large_hadron_collider https://en.wikipedia.org/wiki/Large_hadron_collider

lysium 13y ago
You don't try to argue that LHC is closed source, do you?

mikeash 13y ago

Seems to be that rewriting the software is healthier, and a better analog to physics experiments. When you reproduce that physics result, you're using a different ball in a different location and dropping it from a different structure. If you simply took their program and re-ran it, it's like taking the exact same ball they originally used, climbing the exact same structure they used, and dropping it in the exact same spot. When you then get the same result, it doesn't tell you much, because you don't know if the result is coming from the ball, the structure, the location, or just a general universal phenomenon.

luke-stanley 13y ago
Rewriting from scratch doesn't help run more experiments, it slows things down.
- mikeash 13y ago
  The point is not to run more of them, but to run them better. What good does it do to run identical software ten times? These computer-thingies are pretty deterministic. You're basically guaranteed to get the same result each time, but you have no idea if the result is because of the technique being presented, a quirk of the implementation, or just a bug.
  - luke-stanley 13y ago
    Of course, access to source allows running better experiments! Writing the same algorithms from scratch is a clear waste of time! Please stop defending silliness.
lysium 13y ago
I can start right away changing any one of these to find out, instead of spending time to try to write a program that may well not exist in the first place. (In this particular case, I admit that posting 1.2k bugs hints to its existence.)

tptacek 13y ago

The important result of the paper isn't the specific list of bugs they found, which after all: (a) depends on the software they fed to the tool and (b) is verifiable without static analysis (just try to trigger the bug). The important result is the validation of the approach, which makes the paper more important than the tool.