5 ms·

IMO the most significant problem isn't any particular statistical approach, but the incredibly archaic nature of scientific publication. I read a paper and thi

IMO the most significant problem isn't any particular statistical approach, but the incredibly archaic nature of scientific publication. I read a paper and think that there's something wonky with the statistics. Can I go to the journal's website and download the full dataset? No. Can I download the source code used to compute those statistics? No. I can e-mail one of the authors and ask nicely for that information, but what are the odds that they'll respond if they know that their work is weak or outright fraudulent? Science is supposed to be transparent, it's supposed to be about sharing information, but there's a big opaque wall concealing anything that won't fit into the archaic size limitations of a paper journal. There's no One Weird Trick to fix the replication crisis, there's no obvious Bayesian replacement for p-values or confidence intervals, but there's an obvious shortcoming in scientific publishing that is a huge impediment to secondary research. A high schools student can't get full credit on a math exam without showing their working - why should scientists be held to a lower standard?

lisper 7y ago
It's even worse than that. The whole peer-review process we have today is stuck in the 19th century. The whole idea that publication and the credibility that is associated with it should be a binary quality -- a paper is either "published in a peer-reviewd publication", and therefore credible, or it isn't and so it isn't -- is absurd on its face. If there are any shades of grey, they are associated with the "prestige" of the publication, which again, if you think about it, is absurd, and flies in the face of the scientific ideal: an idea is either supported by the data or it isn't, and what two randomly selected anonymous referees on the review board of a journal, people who likely have ulterior motives in terms of their own career advancement, have to say about it should be the least of anyone's concern. The right model IMHO is open publication a la arxiv with an associated open review process so that anyone who wants to can weigh in. The wheat can be separated from the chaff by doing a "reviewer-rank" analogous to page rank so that the opinions of the people who know what they're doing count more than the crackpots. But this idea that there's an elite priesthood of anointed Scientists, a.k.a. corporate researchers and tenure-track university faculty, who pass final judgement on all scientific truth really has to end. (And I say this as someone who made their living working successfully within the current system for 15 years. I quit in no small measure because I saw firsthand just how corrupt and dysfunctional it could become, and I was working in a field that is generally considered "hard" science!)
- deleted 7y ago
  [deleted]
- iagovar 7y ago
  You are right, but the parent is on point. When I was a student I tried many times to download datasets, with very little success. Only statistical institutions like INE, Igape and so on happily provided the data used for their studies, which is a given since they publish most of their datasets in their website anyway. This sucks, because maybe you have more questions that could be answered with their datasets, but you can't do shit because this shitty system. I'm not a scientist, and partly because though the uni I was really put of by the bureaucracy and the smell of endogamy, obscure and old system and so on. It's so hard to build new knowledge when everyone is taking care of their little yard and keeping everyone else away... and social sciences need this openness like farmers need their rain.
  - lisper 7y ago
    > You are right, but the parent is on point. Yes, I never meant to imply otherwise.
- sampo 7y ago
  > The whole peer-review process we have today is stuck in the 19th century. There are journals that do light initial screening, then publish the submitted manuscripts online, then publish the whole peer review (reviewer comments and authors' responses) online, and then proceed (or not) to publish the reviewed article. Also anyone on the internet can leave an additional review, or just some questions, during the review process. Also there are journals that have comment sections for the published articles.
  - lisper 7y ago
    Example(s)?
    - sampo 7y ago
      Open peer review process: All European Geosciences Union journals https://www.egu.eu/publications/open-access-journals/ https://www.egu.eu/publications/open-access-journals/ Comment section for published articles: For example all PLOS journals https://journals.plos.org/ https://journals.plos.org/
ALittleLight 7y ago
I agree. Another benefit to having the data easily available is apparent when you find papers that study a question that is very similar to what you're interested in but not quite the same. It would be amazing to get their data and apply your own analysis, not just to check for errors, but to pursue your own questions.

lisper 7y ago

It's even worse than that. The whole peer-review process we have today is stuck in the 19th century. The whole idea that publication and the credibility that is associated with it should be a binary quality -- a paper is either "published in a peer-reviewd publication", and therefore credible, or it isn't and so it isn't -- is absurd on its face. If there are any shades of grey, they are associated with the "prestige" of the publication, which again, if you think about it, is absurd, and flies in the face of the scientific ideal: an idea is either supported by the data or it isn't, and what two randomly selected anonymous referees on the review board of a journal, people who likely have ulterior motives in terms of their own career advancement, have to say about it should be the least of anyone's concern. The right model IMHO is open publication a la arxiv with an associated open review process so that anyone who wants to can weigh in. The wheat can be separated from the chaff by doing a "reviewer-rank" analogous to page rank so that the opinions of the people who know what they're doing count more than the crackpots. But this idea that there's an elite priesthood of anointed Scientists, a.k.a. corporate researchers and tenure-track university faculty, who pass final judgement on all scientific truth really has to end. (And I say this as someone who made their living working successfully within the current system for 15 years. I quit in no small measure because I saw firsthand just how corrupt and dysfunctional it could become, and I was working in a field that is generally considered "hard" science!)

deleted 7y ago
[deleted]
iagovar 7y ago
You are right, but the parent is on point. When I was a student I tried many times to download datasets, with very little success. Only statistical institutions like INE, Igape and so on happily provided the data used for their studies, which is a given since they publish most of their datasets in their website anyway. This sucks, because maybe you have more questions that could be answered with their datasets, but you can't do shit because this shitty system. I'm not a scientist, and partly because though the uni I was really put of by the bureaucracy and the smell of endogamy, obscure and old system and so on. It's so hard to build new knowledge when everyone is taking care of their little yard and keeping everyone else away... and social sciences need this openness like farmers need their rain.
- lisper 7y ago
  > You are right, but the parent is on point. Yes, I never meant to imply otherwise.
sampo 7y ago
> The whole peer-review process we have today is stuck in the 19th century. There are journals that do light initial screening, then publish the submitted manuscripts online, then publish the whole peer review (reviewer comments and authors' responses) online, and then proceed (or not) to publish the reviewed article. Also anyone on the internet can leave an additional review, or just some questions, during the review process. Also there are journals that have comment sections for the published articles.
- lisper 7y ago
  Example(s)?
  - sampo 7y ago
    Open peer review process: All European Geosciences Union journals https://www.egu.eu/publications/open-access-journals/ https://www.egu.eu/publications/open-access-journals/ Comment section for published articles: For example all PLOS journals https://journals.plos.org/ https://journals.plos.org/

ALittleLight 7y ago

I agree. Another benefit to having the data easily available is apparent when you find papers that study a question that is very similar to what you're interested in but not quite the same. It would be amazing to get their data and apply your own analysis, not just to check for errors, but to pursue your own questions.