So, I've edited a journal, been on journal editorial boards, and reviewed a lot. I'm skeptical that more transparency in peer review will make a difference at a

So, I've edited a journal, been on journal editorial boards, and reviewed a lot. I'm skeptical that more transparency in peer review will make a difference at all. In fact, some of the proposed solutions might hurt a bit. In the very least, I think there's a tidal wave of change in publishing that's making traditional academic peer review moot. The linked question is interesting for me to think about given one of my relatively recent experiences (this is all in the area of statistics, so kind of directly relevant). I was asked to review an article, and as a counterpoint to one of my concerns, the author cited an in-press paper in a fairly well-respected journal. So I go to look at it, and it makes little sense to me, in that it contradicts a bunch of other things that are known. I look at it closer, and it turns out there's a subtle but important notational error carried through much of the proofs that basically invalidates the whole paper. So I contact the editor of that journal to feel out the response to writing a commentary about the issue, with a fairly detailed explanation of the flaw in the proofs. Instead of being receptive or at least neutral, the editor throws up all sorts of obstacles — not exactly threats, but strong discouragement in the form of a long list of criteria that had to be met, several of which were completely unnecessary. My colleagues (who are also editors of other journals) got even more upset than me, believing that the editor was trying to bury the error and so forth. One even threatened to expose the exchange on twitter or something. The truth is, I don't know that I cared that much about this particular topic to really put the effort into writing a commentary, fighting with the editor to publish it, and skewering the author's ego in the process. My friends and I discussed just putting the commentary on an online archive, but it wasn't clear anyone would make the connection with the paper. Also, for unrelated reasons, I shortly afterward wrote a different paper that was more comprehensive in scope that sort of superceded that flawed proof anyway (that is, if someone read this paper of mine, the results of the first erroneous paper would probably seen as as irrelevant). In this case, the flawed proof/paper was peer reviewed. Making the reviews transparent wouldn't have mattered because in the end the paper was published. Maybe one of the reviewers raised the issues but it was published anyway, so published reviews would tip a reader off? But at that point where are we? If I had done something (and maybe I still will?), I would have just published it in a public academic archive anyway. What, then, is the point of peer reviewed journals? Are we better off just posting papers publicly, and publicly commenting on them? Is stripping anonymity from reviewers a good or bad thing? Won't that discourage rigorous review, for fear of repercussions against reviewers? Is review really all that rigorous anyway? My personal impression is that the volume of academic publishing has increased so much that it's impossible for readers to really keep up, and making it more difficult for scientific consensus to form completely. Publicly available papers in archives is a natural extension of this. What this means is that readers increasingly pick and choose which literature they read and cite, which truth they want to reinforce, and what truth they want to suppress. When you open up peer review to be public, those reviews become just another part of that literature. The peer reviews become blog and twitter posts, which is absolutely fine, but then a reader just selectively picks and chooses which what reviews and blog posts they cite, and so forth and so on. For the record, I'm very much for open publishing, and open discussion of literature. I just think that academics hasn't wrestled with the implications of that, in terms of what it will look like (e.g., amplifying fads, decreased signal to noise ratio, increased feedback loops), and whether it's worthwhile to vigorously maintain anonymous peer review to have that as another form of literature evaluation. There's already a lot of public discussion of papers, and this will only increase regardless of what happens to peer review. Maybe the issue is who does the anonymous review? Maybe just opening up papers to anonymous commentary is the right way to go?