5 ms·

I’ve been reading these comments because I’m always seeking ideas for how to distribute peer review and provide signals about paper quality. How could we levera

by cnees 9y ago

I’ve been reading these comments because I’m always seeking ideas for how to distribute peer review and provide signals about paper quality. How could we leverage the Internet and a large group of authors and papers to show readers which papers thrown online are of citable quality? Surely there’s something we can do beyond counting page views.

achileas 9y ago
The system as it is requires experts to review - peer review isn't "let any random person review" so I'm not sure leveraging the Internet in the traditional sense is the way to go. There are scientist social networks, but they don't have a lot of traction, and there's something to be said for the refereed process that exists. Both knowledge extraction and signaling paper quality are fascinating, hard problems in modern science. I wish I had answers beyond criticism of the current system, but it might be a problem for people far smarter than I.
- cnees 9y ago
  Supposing one of the scientist social networks implemented the refereed peer review process for some or all papers they host, funding it with ad revenue, APCs, and/or freemium instead of charging universities and readers, what more could they do to improve the system? Better post-publication review/redaction/amnendment for work we don’t realize is wrong until much later? A rating system?
berryg 9y ago
I have not given this much thought or did any research. It is just an idea that popped into my head reading this thread. Why not just replicate the open source software model? Author opens a git repository on a github like repository. Reviewer can comment on the content repository. Article versions are stored and changes are stored for future reference. Even research data can be stored in same repository. Reviewers can build up a reputation by reviewing articles. Articles can even be forked or referenced. Articles with the most references and reviewed by reviewers with a good reputation can thus rise to the "top". Technically everything is in place. Or I am too much of a technical optimist?
- cnees 9y ago
  As a programmer, it’s hard not to wish we had git-for-publishing, git-for-Photoshop, git-for-homework, etc. it’s just such a powerful system! I think the steep learning curve is the reason it hasn’t already caught on beyond where it’s absolutely essential in software engineering. Maybe that means we need a more approachable git. It would also mean getting people onto the platform. Maybe a platform that already has the users could gradually introduce a version control system piece by piece.
  - kd0amg 9y ago
    FWIW, I've already got git-for-homework, git-for-academic-articles, git-for-slide-decks, git-for-lesson-planning, and probably a few other things. I'd be curious to see what passes for version control in fields other than software/CS, but I don't think I've heard of any such thing in use. I don't even hear "git is too complicated" -- people don't seem to talk about (anything I recognize as) version control at all.
- ethelward 9y ago
  That'd be nice if top repos on github were the best ones. But they're quite often the most marketed instead (though these two qualities are not necessarily mutually exclusive).
anjc 9y ago
From what I've seen, the average number of citations for papers published in the previous year is something like 0.3. A significant proportion are never cited. Most people give no shit about most papers. The fact that even the shittest publication will be reviewed by a handful of experts, even though most people will never read the paper, is as good as you're gonna get imo.

by cnees 9y ago

achileas 9y ago

The system as it is requires experts to review - peer review isn't "let any random person review" so I'm not sure leveraging the Internet in the traditional sense is the way to go. There are scientist social networks, but they don't have a lot of traction, and there's something to be said for the refereed process that exists. Both knowledge extraction and signaling paper quality are fascinating, hard problems in modern science. I wish I had answers beyond criticism of the current system, but it might be a problem for people far smarter than I.

cnees 9y ago
Supposing one of the scientist social networks implemented the refereed peer review process for some or all papers they host, funding it with ad revenue, APCs, and/or freemium instead of charging universities and readers, what more could they do to improve the system? Better post-publication review/redaction/amnendment for work we don’t realize is wrong until much later? A rating system?

berryg 9y ago

I have not given this much thought or did any research. It is just an idea that popped into my head reading this thread. Why not just replicate the open source software model? Author opens a git repository on a github like repository. Reviewer can comment on the content repository. Article versions are stored and changes are stored for future reference. Even research data can be stored in same repository. Reviewers can build up a reputation by reviewing articles. Articles can even be forked or referenced. Articles with the most references and reviewed by reviewers with a good reputation can thus rise to the "top". Technically everything is in place. Or I am too much of a technical optimist?

cnees 9y ago
As a programmer, it’s hard not to wish we had git-for-publishing, git-for-Photoshop, git-for-homework, etc. it’s just such a powerful system! I think the steep learning curve is the reason it hasn’t already caught on beyond where it’s absolutely essential in software engineering. Maybe that means we need a more approachable git. It would also mean getting people onto the platform. Maybe a platform that already has the users could gradually introduce a version control system piece by piece.
- kd0amg 9y ago
  FWIW, I've already got git-for-homework, git-for-academic-articles, git-for-slide-decks, git-for-lesson-planning, and probably a few other things. I'd be curious to see what passes for version control in fields other than software/CS, but I don't think I've heard of any such thing in use. I don't even hear "git is too complicated" -- people don't seem to talk about (anything I recognize as) version control at all.
ethelward 9y ago
That'd be nice if top repos on github were the best ones. But they're quite often the most marketed instead (though these two qualities are not necessarily mutually exclusive).

anjc 9y ago

From what I've seen, the average number of citations for papers published in the previous year is something like 0.3. A significant proportion are never cited. Most people give no shit about most papers. The fact that even the shittest publication will be reviewed by a handful of experts, even though most people will never read the paper, is as good as you're gonna get imo.