2 ms·

Valid. But they are presenting the methodology they claim went into the code and allowing you to agree or disagree with the methodology. If you agree, you take

Valid. But they are presenting the methodology they claim went into the code and allowing you to agree or disagree with the methodology. If you agree, you take their answer, if you disagree, you implement a different methodology, which necessitates new code anyways, and publish your contra-finding. We usually converge on answers over years of varying different methods and attempts. No one in their right mind reads a paper and says "well, they found out the sun is actually in M31, I believe that now" They look at the culmination of the literature (fun fact, we still aren't positive exactly where the sun is, or how fast it's moving) and I suppose that is the black box. What I was really referring to though was students who walk up to me and say they gaussian smoothed a sample, and have no idea what I mean when I ask if 3 sigma outliers were used in the fit or trashed. They just used some gaussSmooth algorithm and may or may not know what a gaussian even is.

Blahah 13y ago
I certainly agree that people shouldn't be using techniques they don't understand. But to address the main point - there are two separate things to consider when assessing a paper's methods. First is the methodology, which as you say is always included in the paper. But secondly, and equally importantly, is the implementation. Agreeing with the methodology does not make me confident in the results. Someone wrote (probably) a lot of code to generate the analysis, and the likelihood that it contains bugs is high. They may or may not affect the outcome. Without seeing the code, I'm not going to trust the results. Of course, I don't expect to read the source code of every analysis, but if it's open to scrutiny by the community, and the results are of any importance, it will be validated. The problem is actually worse in many experimental methods, where the results rely completely on the practitioner having done exactly what they say they did and done it correctly. No source code to publish there, but that doesn't excuse not publishing analytical code when it is available.
tripzilch 13y ago
(edit the "code" I'm talking about is complex stuff like numerical simulations etc, if we were talking about the code used for plotting a simple graph or histogram, that's a different matter--although it wouldn't hurt anyone publishing a full paper open source as a make-file with some TeX sources calling a few plotting libraries) > But they are presenting the methodology they claim went into the code and allowing you to agree or disagree with the methodology. If you agree, you take their answer-- No. And this is the whole point. You can agree with the methodology, but as long as you haven't inspected the code, you cannot just accept they actually implemented it right! In fact, chances are, they probably didn't. The best programmers white buggy and incorrect code. And from what I've seen in the field of physics, scientific code is everything but an exception to that. I'd be surprised if it was very different for astrophysics. So by publishing the methodology and the results, but not the code, they put up a nice show. But that's it, not reproducible. So all that's really brought to the table is the methodology and some "say-so" results that nobody can check are accurate.
- anon_barcode 13y ago
  True. Nobody can check if your data + your methodology = the results you claim. But it is quite common for someone to take the same problem, collect their own data and run their own analysis (both steps can be completely different from the original statement-- collecting data of a different type, from a different instrument, etc. Running an analysis of a different paradigm, running the same analysis to a different precision, running the same analysis via a different algorithm, etc.). If you read the arxiv on a daily basis you can see huge academic arguments unfolding over the course of months and years. There seems to be this idea that the conversation goes: "Yo I found this hypervelocity star" "Dope, let's move on" It's actually more like: "Yo I found this hypervelocity star" "Nope, I got spectra and you're wrong" "Well I got ultra high-res spectra and I think he's right" "Actually all of you are forgetting asymmetric drift, this is just a geometry problem, l2angles" "Hey, I sit in my basement and play with MOND, it might help" "My simulations show something completely different" Authors are called out and proven right or wrong on a daily basis, even if we can't watch them code over the shoulder. I actually think that's one of the beauties of it-- most of our methods are invented by trying to prove or disprove something in a new way. The monster codes like GADGET (which ran the millenium sim everyone's seen) are usually made public after ~5 years of proprietariness.
  - tripzilch 13y ago
    > It's actually more like: "Yo I found this hypervelocity star" "Nope, I got spectra and you're wrong" "Well I got ultra high-res spectra and I think he's right" "Actually all of you are forgetting asymmetric drift, this is just a geometry problem, l2angles" "Hey, I sit in my basement and play with MOND, it might help" "My simulations show something completely different" Haha, that actually sounds almost exactly like the Stone Soup parable :) ( http://en.wikipedia.org/wiki/Stone_soup http://en.wikipedia.org/wiki/Stone_soup )

Blahah 13y ago

I certainly agree that people shouldn't be using techniques they don't understand. But to address the main point - there are two separate things to consider when assessing a paper's methods. First is the methodology, which as you say is always included in the paper. But secondly, and equally importantly, is the implementation. Agreeing with the methodology does not make me confident in the results. Someone wrote (probably) a lot of code to generate the analysis, and the likelihood that it contains bugs is high. They may or may not affect the outcome. Without seeing the code, I'm not going to trust the results. Of course, I don't expect to read the source code of every analysis, but if it's open to scrutiny by the community, and the results are of any importance, it will be validated. The problem is actually worse in many experimental methods, where the results rely completely on the practitioner having done exactly what they say they did and done it correctly. No source code to publish there, but that doesn't excuse not publishing analytical code when it is available.

tripzilch 13y ago

(edit the "code" I'm talking about is complex stuff like numerical simulations etc, if we were talking about the code used for plotting a simple graph or histogram, that's a different matter--although it wouldn't hurt anyone publishing a full paper open source as a make-file with some TeX sources calling a few plotting libraries) > But they are presenting the methodology they claim went into the code and allowing you to agree or disagree with the methodology. If you agree, you take their answer-- No. And this is the whole point. You can agree with the methodology, but as long as you haven't inspected the code, you cannot just accept they actually implemented it right! In fact, chances are, they probably didn't. The best programmers white buggy and incorrect code. And from what I've seen in the field of physics, scientific code is everything but an exception to that. I'd be surprised if it was very different for astrophysics. So by publishing the methodology and the results, but not the code, they put up a nice show. But that's it, not reproducible. So all that's really brought to the table is the methodology and some "say-so" results that nobody can check are accurate.

anon_barcode 13y ago
True. Nobody can check if your data + your methodology = the results you claim. But it is quite common for someone to take the same problem, collect their own data and run their own analysis (both steps can be completely different from the original statement-- collecting data of a different type, from a different instrument, etc. Running an analysis of a different paradigm, running the same analysis to a different precision, running the same analysis via a different algorithm, etc.). If you read the arxiv on a daily basis you can see huge academic arguments unfolding over the course of months and years. There seems to be this idea that the conversation goes: "Yo I found this hypervelocity star" "Dope, let's move on" It's actually more like: "Yo I found this hypervelocity star" "Nope, I got spectra and you're wrong" "Well I got ultra high-res spectra and I think he's right" "Actually all of you are forgetting asymmetric drift, this is just a geometry problem, l2angles" "Hey, I sit in my basement and play with MOND, it might help" "My simulations show something completely different" Authors are called out and proven right or wrong on a daily basis, even if we can't watch them code over the shoulder. I actually think that's one of the beauties of it-- most of our methods are invented by trying to prove or disprove something in a new way. The monster codes like GADGET (which ran the millenium sim everyone's seen) are usually made public after ~5 years of proprietariness.
- tripzilch 13y ago
  > It's actually more like: "Yo I found this hypervelocity star" "Nope, I got spectra and you're wrong" "Well I got ultra high-res spectra and I think he's right" "Actually all of you are forgetting asymmetric drift, this is just a geometry problem, l2angles" "Hey, I sit in my basement and play with MOND, it might help" "My simulations show something completely different" Haha, that actually sounds almost exactly like the Stone Soup parable :) ( http://en.wikipedia.org/wiki/Stone_soup http://en.wikipedia.org/wiki/Stone_soup )