4 ms·

Well, the idea is to attach responses to downvotes. So you spend several days finding mistakes, you write it up at the review stage, they say "nope" and reject

Well, the idea is to attach responses to downvotes. So you spend several days finding mistakes, you write it up at the review stage, they say "nope" and reject your review. You don't lose any reputation. When they publish, you downvote and post your review as a response (you can just copy and paste - you're right, right now it disappears, but we can make sure reviewers retain access to review comments - that's easy). Sure the office mate can upvote, but other voters will see your critical review, see the mistakes you outline and can affirm you. I guess what it comes down to is this: do you believe the average actor in science/academia is a good actor? IE. Are they honestly trying to be unbiased and advance good science/good work above all else? Or do you believe the average actor is a bad actor (OR that the incentive structure is so strong that even well intentioned actors are corrupted to be mediocre actors)? If the average actor is a good actor, the system works and work will be rated/rewarded correctly. If the average actor is a bad actor, or the incentive structure is too strong, then this system would absolutely fail in the sort of worst case scenarios that have been described here.

eesmith 4y ago
> do you believe the average actor in science/academia is a good actor? I think you've over-simplified your model. I'll build on your analogy. There are many roles. As you've argued, the current system is fundamentally broken, which means most if not all of the roles are bad roles. What happens if a good actor is stuck in a bad role? My simplistic view of modern scientific publication is it started to reduce knowledge hoarding by coupling prestige to publication. ("I'm sorry Mr. Newton, but as Herr Leibniz published first, he gets the credit with creating the calculus of infinitesimals.") The loop is now tightly coupled, leaving us with "publish or perish", and with coarse and unreliable publication metrics used to determine career advancement. As a result, many people deliberately game the system, with methods like "least publishable unit" and "salami slicing". How is the author order determined? Who gets to be on the author list? Are those the method of good actors? Or bad actors? To be clear, my goal is not simply to point out that there is a grey area, but rather that there are competing goals. Your use of "good actor" and "bad actor" oversimplifies the issue by projecting it onto the single axis of "good science." I recently reviewed a paper. It was 20 or so pages long. Nothing about it was all that new - it was about the level of a senior undergrad project. Most of it was a tutorial/walk-through about how to use the software. Had it been the late 1990s, it would have been cutting edge. My review was something like "this should be a 1-2 page software announcement. Nothing seems wrong, but it isn't worth the time for anyone to read all these details in a journal paper, and it isn't worth my time to review it." Is the full 20-page paper good science? Bad science? Is it the result of a bad actor? Or an uninformed actor? Should my review be acted upon or ignored? If published, would my post-publication review make a difference?
- dbingham 4y ago
  I might put that in the middle category - the category that might warrant a quick review with just that criticism and maybe a post-publish response, but no voting. "Bad" here is used to mean dishonest, very poorly done methods, or conclusions way out of step with the actual empiracle results. "Good" is used to mean simply "this was well and honestly done work, with conclusions that may be reasonably drawn from the results". Nothing more. There are massive gradations with in "Good" and a large area between "Good" and "Bad". The intention is for "Good" papers to be upvoted, "Bad" papers downvoted, and everything in between to not be voted on, but to rather receive responses where warranted. The reasoning behind this is that in a lot of ways we've gotten too into the first discovery pieces of this and forgotten that science works in the aggregate, it must be a collaborative enterprise. Any empirical experiment or theoretical work could be biased or flawed in any number of ways, and in ways which we can't necessarily catch in review. That's why replication is so important, but in so many fields it's not happening now due to ideas about what represents a "substantive contribution to the field". Hence, the Replication Crisis. So the goal of using such a simple system and metric, in a lot of ways, is to take us back to brass tacks. This is about building humanity's collective knowledge base - together. All else is secondary. So a "Good" actor in that context is someone who does the thing that contributes to humanity's collective knowledge about the world, including but not limited to: - replicating previous work - getting a null result - getting inconclusive results - helping a peer polish their work so that it's best communicated - helping a peer catch mistakes or spot unexplored avenues for further study In a lot of ways, the prestige seeking is part of the problem, because it works against collaboration. The nice thing about a reputation system where votes are granted for "Good" work in this context, is that it's not zero sum. And indeed, it does a much better job of incentivizing a lot of non-glamorous grind work, because there's a lot of that, and if it's well done, then it merits votes. It probably won't get as many votes in a single go as the ground breaking stuff, but it's a valid path to building reputation. This system definitely does incentivize "Least Publishable Unit", but... I'm not necessarily convinced that's a bad thing. In software engineering, there are a ton of benefits to breaking up a large knowledge base or problem into its smallest digestible chunks. If all of those "Least Publishable Units" are well organized in a single database, that might actually be a benefit rather than a cost. By the way, I'm still chewing on your other unanswered comments. You're raising a lot of good points and giving me a lot to think about and I really, truly appreciate it.
CJefferson 4y ago
I would say academics are just people, and in many cases they want to be employed and successful. At the moment, while the system is very bad in many ways, it used to work (in my opinion, I haven't done proper research into this) because: * People who run "good" journals want their journal to be considered good, so they publish good papers. * People want to get into "good" journals, so they want to write good papers that will be accepted. The existence of these papers is then used as evidence for promotions and grants. * Academics want to read "good" journals, so they get their University to pay for a subscription to get physical copies in the library. Previously this cycle held reasonably well -- it's been broken by the reduced cost of online journals, and journals wanting higher payments. I think a good future system needs to (somehow) recreate this "virtuous cycle" -- making reviews as important as journals could be a good idea, I could imagine following a reviewer I thought was excellent, and a group of reviewers could create their own "virtual journal". I would personally trust "Person X who I know thinks this paper is good" much more than "100 people thought this paper was good, 40 thought it was bad, so +60".
- dbingham 4y ago
  Virtual journals is definitely on the table. The way I'm thinking it would work is that journals can effectively create a team of reviewers on the platform and when you submit a paper, you could request review from their team. Right now I'm thinking you could choose to request open review from everyone in your fields, review from one or more journals, or both at the same time. The journal editors then give their feedback through the review system, but it's still up to the author to choose to publish. If you successfully satisfy the journal, you get their stamp of approval on your paper when you publish. If not, you can still publish at will, but your risk their review team down voting you and you don't get the stamp of approval. > I would personally trust "Person X who I know thinks this paper is good" much more than "100 people thought this paper was good, 40 thought it was bad, so +60". I can definitely understand where that hesitation would come from - especially with experience in places like Reddit, where the crowd is not given guidelines and voting systems decay to the lowest common denominator. My understanding of the research on the topic is that it actually supports the idea that the judgement of the crowd will often be better than the judgement of any individual -- when it's the right crowd, given the right guidelines. That's the model behind StackExchange and it's worked very well there. And that's the theory behind this (proposed) platform. The reputation and field model is geared towards identifying and creating groups of "the right people" for each topic/discipline/field and allowing people to submit their work to those groups. It could create a similar virtual cycle: * People want to support good science and get good review feedback, so they give good review feedback. * People want to support and highlight good science, so they upvote good science, downvote dishonest science, and post responses with critical feedback on mediocre science. * People want to have their work to be upvoted and receive positive responses, so they take review feedback seriously. That's the theory anyway - the jury seems to be very much out on whether or not that can work with academia.

eesmith 4y ago

> do you believe the average actor in science/academia is a good actor? I think you've over-simplified your model. I'll build on your analogy. There are many roles. As you've argued, the current system is fundamentally broken, which means most if not all of the roles are bad roles. What happens if a good actor is stuck in a bad role? My simplistic view of modern scientific publication is it started to reduce knowledge hoarding by coupling prestige to publication. ("I'm sorry Mr. Newton, but as Herr Leibniz published first, he gets the credit with creating the calculus of infinitesimals.") The loop is now tightly coupled, leaving us with "publish or perish", and with coarse and unreliable publication metrics used to determine career advancement. As a result, many people deliberately game the system, with methods like "least publishable unit" and "salami slicing". How is the author order determined? Who gets to be on the author list? Are those the method of good actors? Or bad actors? To be clear, my goal is not simply to point out that there is a grey area, but rather that there are competing goals. Your use of "good actor" and "bad actor" oversimplifies the issue by projecting it onto the single axis of "good science." I recently reviewed a paper. It was 20 or so pages long. Nothing about it was all that new - it was about the level of a senior undergrad project. Most of it was a tutorial/walk-through about how to use the software. Had it been the late 1990s, it would have been cutting edge. My review was something like "this should be a 1-2 page software announcement. Nothing seems wrong, but it isn't worth the time for anyone to read all these details in a journal paper, and it isn't worth my time to review it." Is the full 20-page paper good science? Bad science? Is it the result of a bad actor? Or an uninformed actor? Should my review be acted upon or ignored? If published, would my post-publication review make a difference?

dbingham 4y ago
I might put that in the middle category - the category that might warrant a quick review with just that criticism and maybe a post-publish response, but no voting. "Bad" here is used to mean dishonest, very poorly done methods, or conclusions way out of step with the actual empiracle results. "Good" is used to mean simply "this was well and honestly done work, with conclusions that may be reasonably drawn from the results". Nothing more. There are massive gradations with in "Good" and a large area between "Good" and "Bad". The intention is for "Good" papers to be upvoted, "Bad" papers downvoted, and everything in between to not be voted on, but to rather receive responses where warranted. The reasoning behind this is that in a lot of ways we've gotten too into the first discovery pieces of this and forgotten that science works in the aggregate, it must be a collaborative enterprise. Any empirical experiment or theoretical work could be biased or flawed in any number of ways, and in ways which we can't necessarily catch in review. That's why replication is so important, but in so many fields it's not happening now due to ideas about what represents a "substantive contribution to the field". Hence, the Replication Crisis. So the goal of using such a simple system and metric, in a lot of ways, is to take us back to brass tacks. This is about building humanity's collective knowledge base - together. All else is secondary. So a "Good" actor in that context is someone who does the thing that contributes to humanity's collective knowledge about the world, including but not limited to: - replicating previous work - getting a null result - getting inconclusive results - helping a peer polish their work so that it's best communicated - helping a peer catch mistakes or spot unexplored avenues for further study In a lot of ways, the prestige seeking is part of the problem, because it works against collaboration. The nice thing about a reputation system where votes are granted for "Good" work in this context, is that it's not zero sum. And indeed, it does a much better job of incentivizing a lot of non-glamorous grind work, because there's a lot of that, and if it's well done, then it merits votes. It probably won't get as many votes in a single go as the ground breaking stuff, but it's a valid path to building reputation. This system definitely does incentivize "Least Publishable Unit", but... I'm not necessarily convinced that's a bad thing. In software engineering, there are a ton of benefits to breaking up a large knowledge base or problem into its smallest digestible chunks. If all of those "Least Publishable Units" are well organized in a single database, that might actually be a benefit rather than a cost. By the way, I'm still chewing on your other unanswered comments. You're raising a lot of good points and giving me a lot to think about and I really, truly appreciate it.

CJefferson 4y ago

I would say academics are just people, and in many cases they want to be employed and successful. At the moment, while the system is very bad in many ways, it used to work (in my opinion, I haven't done proper research into this) because: * People who run "good" journals want their journal to be considered good, so they publish good papers. * People want to get into "good" journals, so they want to write good papers that will be accepted. The existence of these papers is then used as evidence for promotions and grants. * Academics want to read "good" journals, so they get their University to pay for a subscription to get physical copies in the library. Previously this cycle held reasonably well -- it's been broken by the reduced cost of online journals, and journals wanting higher payments. I think a good future system needs to (somehow) recreate this "virtuous cycle" -- making reviews as important as journals could be a good idea, I could imagine following a reviewer I thought was excellent, and a group of reviewers could create their own "virtual journal". I would personally trust "Person X who I know thinks this paper is good" much more than "100 people thought this paper was good, 40 thought it was bad, so +60".

dbingham 4y ago
Virtual journals is definitely on the table. The way I'm thinking it would work is that journals can effectively create a team of reviewers on the platform and when you submit a paper, you could request review from their team. Right now I'm thinking you could choose to request open review from everyone in your fields, review from one or more journals, or both at the same time. The journal editors then give their feedback through the review system, but it's still up to the author to choose to publish. If you successfully satisfy the journal, you get their stamp of approval on your paper when you publish. If not, you can still publish at will, but your risk their review team down voting you and you don't get the stamp of approval. > I would personally trust "Person X who I know thinks this paper is good" much more than "100 people thought this paper was good, 40 thought it was bad, so +60". I can definitely understand where that hesitation would come from - especially with experience in places like Reddit, where the crowd is not given guidelines and voting systems decay to the lowest common denominator. My understanding of the research on the topic is that it actually supports the idea that the judgement of the crowd will often be better than the judgement of any individual -- when it's the right crowd, given the right guidelines. That's the model behind StackExchange and it's worked very well there. And that's the theory behind this (proposed) platform. The reputation and field model is geared towards identifying and creating groups of "the right people" for each topic/discipline/field and allowing people to submit their work to those groups. It could create a similar virtual cycle: * People want to support good science and get good review feedback, so they give good review feedback. * People want to support and highlight good science, so they upvote good science, downvote dishonest science, and post responses with critical feedback on mediocre science. * People want to have their work to be upvoted and receive positive responses, so they take review feedback seriously. That's the theory anyway - the jury seems to be very much out on whether or not that can work with academia.