3 ms·
I can't comment on the field of psychiatry. However, I will comment on my field of physics. The author seems to completely neglect the signal to noise proble
by ylem 9y ago
I can't comment on the field of psychiatry. However, I will comment on my field of physics. The author seems to completely neglect the signal to noise problem. There has been an explosion of journals and scientific publication. It's rather hard to keep up to date on the literature and editors do provide a valuable function in filtering what they think is interesting (which is further filtered by the referees). Also, for the referee process--it's not perfect and will probably not catch complete fraud. But, it is good for checking the basics of whether or not the story being told in a paper is consistent. Also, again related to signal/noise, the editors hopefully choose referees with a minimum level of competence in the field (not always, but usually). Also, in terms of dilution--if there's a blog, there's no guarantee that a critical mass of people will read it to try to give it a critical review. Whereas, even for a journal that is not widely read, the papers in it will have received some peer review where someone in the field tried to take a look through it for obvious errors. I am not convinced by open peer review. Let's consider two scenarios. Suppose there is a leader in the field trying to publish something (or perhaps even a friend or collaborator). If I notice that they are wrong, am I more likely to call them on it if I am anonymous or if they know who I am? One can be idealistic, but researchers are people too. I think that the anonymous review allows people to speak frankly (though it does sometimes let people be mean--that seems to be the cost of anonymity in forums unless there is strong moderation). Let's consider another scenario. Suppose I am a junior researcher and I get a referee comment that I determine is wrong. If it comes from the leader in my field, maybe I won't push back as hard as if it came from someone else junior--again, in an ideal world, it wouldn't matter, but researchers are people too. It would seem that the editor knowing the identity of the referees would serve as a check on bias, not 100 percent, but I think that more is gained than is lost from a closed referee system.
In terms of error correction, there are errata that are published if someone finds an error in a major study and in journals like the physical review are linked back to the original article.
The open data problem is hard. In my particular field, our raw data is available on the web, but the problem is that the meta-data needed to interpret is not. And that's nontrivial. For example, I may perform several operations on my data (for example, background subtraction) before fitting it to a model. We are experimenting with dataflow languages where we embed the series of "filters" that we apply to the data, but if we want someone to be able to reproduce this 20 years from now, then there's a whole ecosystem that would have to be maintained--for example, not just my code, but all of the libraries that it depends on to run. We can describe the basic process in the paper, but for true long term reproducibility, it's a hard problem...There are groups that are working on open-data and reproducible research, so I don't think that there isn't interest, just that it's not as easy a problem as you might think. But, for reduced data, I agree, it would be nice to have that available for papers in a machine readable format...
Open access. This is also a hard problem. Increasingly, funding agencies are requiring that publications be made available in an open format after some embargo period. I think that may be the best we can hope for. Just paying for competent editors requires funding. If we want additional features like data attached to papers (for decades), it will take more funding. That money has to come from somewhere. For typical open access journals, the author pays, but that seems to create difficult incentives--not to mention that it makes it difficult for poorer funded researchers to publish. I'm not sure what the right answer is, because I agree that the public should be able to see the results that they paid for--perhaps an embargo period is the best solution...
I think a blog is a great way for communication of ideas and for education, but I do not think that it is able to replace publication in a refereed journal.