4 ms·

In my experience, journals are still essential to the peer review process and are the primary way in which work is disseminated into the larger community.

by tensor 16y ago

jerf 16y ago
I think it's important to read avar's comment not as "Let's get rid of journals!" and as "Let's replace journals with circles of trust". They're not at all the same thing, and I don't get the feeling you have grappled with the questions avar's comments raise. One might be "What critical feature do journals provide that circle-of-trust based systems couldn't?" I for one can't come up with anything. Editing? Promoting? Verifying? Repeating? Publishing? It's all pretty easy. You can't quite just take reddit or HN off the shelf and deploy it, you want a much stricter trust system, but it's not really that hard, either.
- tensor 16y ago
  How do you get around bias in peer review? Journals use blind reviews so that reviewers feel free to express their real opinions without the risk of damaging their professional relationships. Increasingly, double-blind review is being used as well. This additionally removes the famous author bias. I understand that when double blind reviewing was added to the area of AI research, the distribution of who got things published changed dramatically. I don't see how a circle of trust will help with that. The other part of the problem is getting people who are willing to do thorough reviews. Time is money, and with the massive pressures to publish, it has become somewhat common in CS for professors to hand off reviews to students who may not be qualified. Or alternately, reviews can be too short or not thorough enough. This is really illustrated well in conferences, which ironically are often valued more than journals. The extreme number of submissions coupled with a very limited assessment time and strict page limits leads to some horrible reviews that help no one. Even bad reviews should be useful in the sense that they should point out what is wrong or how to improve the work.
  - jerf 16y ago
    "How do you get around bias in peer review?" It isn't possible. Much better to have multiple intersecting circles of trust than to pretend that there can be one unbiased judgment of a work. And when I say "isn't possible", I mean, "isn't possible". At all. You can't do it. I could all but prove this mathematically with enough time, using terminology and concepts from the machine learning bunch. (Their definition of "bias" is one of the most profound things I learned in my entire 7-year college education.) Double-blind reviews can still be done in the circle-of-trust regimes. Obviously there's a point where the work is unblinded and you have to deal with that, but it hardly strikes me as impossible.
_delirium 16y ago
In computer science, I'd say the first part is true but not really the second, at least in the areas I work in. People definitely use "did it get into a journal?" (actually in CS, more often a good conference, since top conferences have more prestige than journals) as a quality filter, but they don't really get their information by browsing the latest issue of journals. For that, it's much more common to just get PDFs off the internet--- from authors' homepages, Google Scholar, CiteSeer, citeulike, mailing lists, personal suggestions, etc., and then figure out where those PDFs came from later. In some cases, if it's an author I already know of, I don't even bother looking to see where it was published until I actually need to write out a bibtex entry to cite it. Doing things that way, I've cited papers from publication venues I've never even heard of, so they aren't really doing any sort of information-filtering work for me. In those cases, the article may as well have been a tech report for all it mattered to me, since the publication venue was really serving as nothing (from my perspective) except a fancy place to deposit PDFs. Admittedly, that works mainly when I already know something about the author (or someone I know can tell me something about the author). If they're a total unknown, I'd probably be more wary of citing stuff that didn't already get into a venue I respect.
- billswift 16y ago
  Actually, the only real reason I maintain an ACM membership is that I think their online library is a great resource for browsing technical computer science.

In my experience, journals are still essential to the peer review process and are the primary way in which work is disseminated into the larger community.

by tensor 16y ago

In my experience, journals are still essential to the peer review process and are the primary way in which work is disseminated into the larger community.

jerf 16y ago

I think it's important to read avar's comment not as "Let's get rid of journals!" and as "Let's replace journals with circles of trust". They're not at all the same thing, and I don't get the feeling you have grappled with the questions avar's comments raise. One might be "What critical feature do journals provide that circle-of-trust based systems couldn't?" I for one can't come up with anything. Editing? Promoting? Verifying? Repeating? Publishing? It's all pretty easy. You can't quite just take reddit or HN off the shelf and deploy it, you want a much stricter trust system, but it's not really that hard, either.

tensor 16y ago
How do you get around bias in peer review? Journals use blind reviews so that reviewers feel free to express their real opinions without the risk of damaging their professional relationships. Increasingly, double-blind review is being used as well. This additionally removes the famous author bias. I understand that when double blind reviewing was added to the area of AI research, the distribution of who got things published changed dramatically. I don't see how a circle of trust will help with that. The other part of the problem is getting people who are willing to do thorough reviews. Time is money, and with the massive pressures to publish, it has become somewhat common in CS for professors to hand off reviews to students who may not be qualified. Or alternately, reviews can be too short or not thorough enough. This is really illustrated well in conferences, which ironically are often valued more than journals. The extreme number of submissions coupled with a very limited assessment time and strict page limits leads to some horrible reviews that help no one. Even bad reviews should be useful in the sense that they should point out what is wrong or how to improve the work.
- jerf 16y ago
  "How do you get around bias in peer review?" It isn't possible. Much better to have multiple intersecting circles of trust than to pretend that there can be one unbiased judgment of a work. And when I say "isn't possible", I mean, "isn't possible". At all. You can't do it. I could all but prove this mathematically with enough time, using terminology and concepts from the machine learning bunch. (Their definition of "bias" is one of the most profound things I learned in my entire 7-year college education.) Double-blind reviews can still be done in the circle-of-trust regimes. Obviously there's a point where the work is unblinded and you have to deal with that, but it hardly strikes me as impossible.

_delirium 16y ago

In computer science, I'd say the first part is true but not really the second, at least in the areas I work in. People definitely use "did it get into a journal?" (actually in CS, more often a good conference, since top conferences have more prestige than journals) as a quality filter, but they don't really get their information by browsing the latest issue of journals. For that, it's much more common to just get PDFs off the internet--- from authors' homepages, Google Scholar, CiteSeer, citeulike, mailing lists, personal suggestions, etc., and then figure out where those PDFs came from later. In some cases, if it's an author I already know of, I don't even bother looking to see where it was published until I actually need to write out a bibtex entry to cite it. Doing things that way, I've cited papers from publication venues I've never even heard of, so they aren't really doing any sort of information-filtering work for me. In those cases, the article may as well have been a tech report for all it mattered to me, since the publication venue was really serving as nothing (from my perspective) except a fancy place to deposit PDFs. Admittedly, that works mainly when I already know something about the author (or someone I know can tell me something about the author). If they're a total unknown, I'd probably be more wary of citing stuff that didn't already get into a venue I respect.

billswift 16y ago
Actually, the only real reason I maintain an ACM membership is that I think their online library is a great resource for browsing technical computer science.