8 ms·

While I find the reported results impressive, I'm sad to see another publication about a closed-source application. Nobody can reproduce the results or place t

by lysium 13y ago

While I find the reported results impressive, I'm sad to see another publication about a closed-source application. Nobody can reproduce the results or place their work on top of that. Research was not supposed to work that way (at least in my little universe :-), yet it gets published by the IEEE (but what does that mean nowadays?).

therobot24 13y ago
Sometimes research is closed source due to where the funding comes from, I'm working a project that's been funded by two different entities over the years - one of which won't even release the dataset we've been using - so when trying to open source my code I get stuck.
- gwern 13y ago
  > This research was supported by a DARPA grant to CyLab at Carnegie Mellon University (N11AP20005/D11AP00262), a NSF Career grant (CNS0953751), and partial CyLab ARO support from grant DAAD19-02-1-0389 and W911NF-09-1-0273.
  - lysium 13y ago
    I don't know much about these grants, but DAPRA and NSF sound pretty public to me (read: the public pays for something only a handful of people get access to). Correct me if I'm wrong, though.
    - gwern 13y ago
      Yes, that was my point.
xenophonf 13y ago
The paper documents their approach pretty thoroughly, and honestly, if it were me (and I'm a damn-the-torpedos-full-disclosure-open-access-everything kind of guy), I'd think twice about publishing a program that would give the kiddies something to do on a wet, wet, wet, day.
- lysium 13y ago
  Well, the authors stated that they will publish the bugs. Further, publishing a program like this has the potential of finally having software that is not easily exploitable (eg. by requiring every Debian software to pass that program). Now that it is closed source, only a selected set of people will be able to benefit from it.
tptacek 13y ago
The paper documents their approach. You could reproduce their findings by building your own offline/online symbolic execution system.
- lysium 13y ago
  Could I? How long would it take? Would I really get the same results or just similar results? If it doesn't work, how can I find out where the bug is (the paper, my interpretation, my implementation)? Compare this to physics: if you drop this ball from that height under these circumstances, it will take that long. Everybody (with the right equipment) can reproduce this result quite easily. Not so in this case, which ironically could be even more reproducible than any physics experiment if it wasn't closed source.
  - betterunix 13y ago
    Compare it to physics indeed: https://en.wikipedia.org/wiki/Large_hadron_collider https://en.wikipedia.org/wiki/Large_hadron_collider
    - lysium 13y ago
      You don't try to argue that LHC is closed source, do you?
  - mikeash 13y ago
    Seems to be that rewriting the software is healthier, and a better analog to physics experiments. When you reproduce that physics result, you're using a different ball in a different location and dropping it from a different structure. If you simply took their program and re-ran it, it's like taking the exact same ball they originally used, climbing the exact same structure they used, and dropping it in the exact same spot. When you then get the same result, it doesn't tell you much, because you don't know if the result is coming from the ball, the structure, the location, or just a general universal phenomenon.
    - 13y ago
deleted 13y ago
[deleted]

by lysium 13y ago

therobot24 13y ago

Sometimes research is closed source due to where the funding comes from, I'm working a project that's been funded by two different entities over the years - one of which won't even release the dataset we've been using - so when trying to open source my code I get stuck.

gwern 13y ago
> This research was supported by a DARPA grant to CyLab at Carnegie Mellon University (N11AP20005/D11AP00262), a NSF Career grant (CNS0953751), and partial CyLab ARO support from grant DAAD19-02-1-0389 and W911NF-09-1-0273.
- lysium 13y ago
  I don't know much about these grants, but DAPRA and NSF sound pretty public to me (read: the public pays for something only a handful of people get access to). Correct me if I'm wrong, though.
  - gwern 13y ago
    Yes, that was my point.

xenophonf 13y ago

The paper documents their approach pretty thoroughly, and honestly, if it were me (and I'm a damn-the-torpedos-full-disclosure-open-access-everything kind of guy), I'd think twice about publishing a program that would give the kiddies something to do on a wet, wet, wet, day.

lysium 13y ago
Well, the authors stated that they will publish the bugs. Further, publishing a program like this has the potential of finally having software that is not easily exploitable (eg. by requiring every Debian software to pass that program). Now that it is closed source, only a selected set of people will be able to benefit from it.

tptacek 13y ago

The paper documents their approach. You could reproduce their findings by building your own offline/online symbolic execution system.

lysium 13y ago
Could I? How long would it take? Would I really get the same results or just similar results? If it doesn't work, how can I find out where the bug is (the paper, my interpretation, my implementation)? Compare this to physics: if you drop this ball from that height under these circumstances, it will take that long. Everybody (with the right equipment) can reproduce this result quite easily. Not so in this case, which ironically could be even more reproducible than any physics experiment if it wasn't closed source.
- betterunix 13y ago
  Compare it to physics indeed: https://en.wikipedia.org/wiki/Large_hadron_collider https://en.wikipedia.org/wiki/Large_hadron_collider
  - lysium 13y ago
    You don't try to argue that LHC is closed source, do you?
- mikeash 13y ago
  Seems to be that rewriting the software is healthier, and a better analog to physics experiments. When you reproduce that physics result, you're using a different ball in a different location and dropping it from a different structure. If you simply took their program and re-ran it, it's like taking the exact same ball they originally used, climbing the exact same structure they used, and dropping it in the exact same spot. When you then get the same result, it doesn't tell you much, because you don't know if the result is coming from the ball, the structure, the location, or just a general universal phenomenon.
  - 13y ago

deleted 13y ago

[deleted]