4 ms·

Indeed. This post is nothing more than "I did this and look at the results" but offers nothing for us to repeat his tests. Plus yeah it is a decade old, things

by satysin 10y ago

Indeed. This post is nothing more than "I did this and look at the results" but offers nothing for us to repeat his tests. Plus yeah it is a decade old, things have changed a hell of a lot in the past decade.

YeGoblynQueenne 10y ago
>> This post is nothing more than "I did this and look at the results" but offers nothing for us to repeat his tests. That's a valid gripe to have, but it should be said that most scientific papers that report simulation or algorithm results do exactly the same. Reproducing results is totally left to whomever feels like (and has the time to) dispute the claims. Which is why you don't often see much in the way of disputation of performance claims, btw (yeah, ok, also because people take part in competitions where you get the chance to put your code where your mouth is).
- semi-extrinsic 10y ago
  But scientific papers aren't comparing Language X v.2.7.5 vs Language Y v. 16.9.1. They're comparing Algorithm X vs Algorithm Y, and usually also give theoretical results. Unless there is a big dramatic change in computer hardware, results comparing some algorithms using good implementations in a single well-optimized language will likely stand the test of time well.
  - jblow 10y ago
    No, because the comparisons are usually "baseline version of algorithm I want to beat" vs "highly optimized and hand-tweaked version of the algorithm I have a vested interest in."
    - lorenzhs 10y ago
      At least in my (and my colleagues') areas of CS, you usually ask the original authors of a paper you want to compare to. A large percentage of them will be glad to provide you with their code, for it goes a long way towards ensuring that their algorithms aren't grossly misrepresented. If they don't (because the code is a mess, they can't find it or get it to work any more, or for whatever reason), you'll have to re-implement it. We take great care to match their performance and have, on several occasions, beat it (compared to the published results, running on similar hardware). It's a lot more work, but in our group we really try to make fair comparisons. Maybe not everyone is so inclined, but I'm sick of reading these stereotypes over and over again.
      - YeGoblynQueenne 10y ago
        >> Maybe not everyone is so inclined, but I'm sick of reading these stereotypes over and over again. I'm sorry if my comment upset you. I'm really not trying to advance any stereotypes and rather just report what my experience is so far, as a Masters student (rather than a researcher). I've also discussed this with one of my professors, who does research mainly in NLP and he told me the same thing. Personally I find it hard to understand why it's not compulsory for academics to point to a public repository where anyone can find the code that goes along with their papers. I've heard arguments for and against sharing code and particularly data, so I'm not saying that I am necessarily right, but it's really just weird to read "our system beats the best results reported so far" with nothing but a few figures in a table to back that up. And with so much infrastructure and tools around to support sharing code (just think of github) I really don't see what's stopping people.
        lorenzhs 10y ago
        I didn't mean to attack you personally - that sentiment gets echoed a lot on HN and I replied to your comment, meaning to address all of those that I read before but didn't reply to, so please don't take it personally. My area is in algorithmics, which is a lot less of a "hot topic" and there's less competition and more cooperation all-around. That may be an important factor in the willingness to share code. I do agree with you on publishing code and have open-sourced the code to some of my research. Unfortunately I never heard back from anyone, maybe my topics just aren't hot enough ;) I'm a bit unsure whether to publish code for ongoing projects (where some aspects have been published, but others are still in the works for future publications) - in that case there is some incentive to keep the source closed until the project is finished.
- SilasX 10y ago
  That's seems ... unscientific, you know? To make it unnecessarily difficult to replicate your results for purposes of confirmation/refutation?

by satysin 10y ago

YeGoblynQueenne 10y ago

>> This post is nothing more than "I did this and look at the results" but offers nothing for us to repeat his tests. That's a valid gripe to have, but it should be said that most scientific papers that report simulation or algorithm results do exactly the same. Reproducing results is totally left to whomever feels like (and has the time to) dispute the claims. Which is why you don't often see much in the way of disputation of performance claims, btw (yeah, ok, also because people take part in competitions where you get the chance to put your code where your mouth is).

semi-extrinsic 10y ago
But scientific papers aren't comparing Language X v.2.7.5 vs Language Y v. 16.9.1. They're comparing Algorithm X vs Algorithm Y, and usually also give theoretical results. Unless there is a big dramatic change in computer hardware, results comparing some algorithms using good implementations in a single well-optimized language will likely stand the test of time well.
- jblow 10y ago
  No, because the comparisons are usually "baseline version of algorithm I want to beat" vs "highly optimized and hand-tweaked version of the algorithm I have a vested interest in."
  - lorenzhs 10y ago
    At least in my (and my colleagues') areas of CS, you usually ask the original authors of a paper you want to compare to. A large percentage of them will be glad to provide you with their code, for it goes a long way towards ensuring that their algorithms aren't grossly misrepresented. If they don't (because the code is a mess, they can't find it or get it to work any more, or for whatever reason), you'll have to re-implement it. We take great care to match their performance and have, on several occasions, beat it (compared to the published results, running on similar hardware). It's a lot more work, but in our group we really try to make fair comparisons. Maybe not everyone is so inclined, but I'm sick of reading these stereotypes over and over again.
    - YeGoblynQueenne 10y ago
      >> Maybe not everyone is so inclined, but I'm sick of reading these stereotypes over and over again. I'm sorry if my comment upset you. I'm really not trying to advance any stereotypes and rather just report what my experience is so far, as a Masters student (rather than a researcher). I've also discussed this with one of my professors, who does research mainly in NLP and he told me the same thing. Personally I find it hard to understand why it's not compulsory for academics to point to a public repository where anyone can find the code that goes along with their papers. I've heard arguments for and against sharing code and particularly data, so I'm not saying that I am necessarily right, but it's really just weird to read "our system beats the best results reported so far" with nothing but a few figures in a table to back that up. And with so much infrastructure and tools around to support sharing code (just think of github) I really don't see what's stopping people.
      - lorenzhs 10y ago
        I didn't mean to attack you personally - that sentiment gets echoed a lot on HN and I replied to your comment, meaning to address all of those that I read before but didn't reply to, so please don't take it personally. My area is in algorithmics, which is a lot less of a "hot topic" and there's less competition and more cooperation all-around. That may be an important factor in the willingness to share code. I do agree with you on publishing code and have open-sourced the code to some of my research. Unfortunately I never heard back from anyone, maybe my topics just aren't hot enough ;) I'm a bit unsure whether to publish code for ongoing projects (where some aspects have been published, but others are still in the works for future publications) - in that case there is some incentive to keep the source closed until the project is finished.
SilasX 10y ago
That's seems ... unscientific, you know? To make it unnecessarily difficult to replicate your results for purposes of confirmation/refutation?