4 ms·

Could manual flagging be abused by a country or agent against another country or agent in order to maliciously discredit papers from that target?

abstractalgo 4y ago
Most definitely. An advantage of having open peer review system is also that this kind of attempts could be detected by tools doing automatic processing of scientific knowledge graphs generated by peer review activity. This, coupled with verification of users, should be able to minimize the risk quite significantly.
- gchamonlive 4y ago
  Seems to me that this wouldn't be enough in case of state actors or large organizations. They would just have to employ real people to carry the fake review bombs.
bittercynic 4y ago
Instead of a flagging system ranking the credibility of papers once for everyone, can we have a system where each user decides which other users they trust to evaluate papers, and ignore any signal from untrusted users? For example, I could L1 trust people to evaluate papers and the trustworthiness of other users, and L2 trust others only to evaluate the trustworthiness of papers. If I notice a spurious signal getting in, I can find out who it came from and prune my list of trusted users. I imagine I'm not the first to think of this. Anyone know of reading material on the topic?
- abstractalgo 4y ago
  The one thing I'm aware of is Brave's Goggles project: https://search.brave.com/help/goggles https://search.brave.com/help/goggles - it basically allows you to set up your own ranking algorithm with a specific syntax (it links to Github repo and, there, to a white paper). I haven't seen more attempts at similar things, which doesn't mean they don't exist though. (There were mentions of a similar feature being added when Elon Musk was about to buy Twitter, but just gossips)
- jpeloquin 4y ago
  Sounds like journal clubs but organized through a specialized tech platform rather than normal communication channels. (Journal clubs are groups of people, usually working in the same field, who pick & present a paper to the rest of the group on an rotating basis.)

abstractalgo 4y ago

Most definitely. An advantage of having open peer review system is also that this kind of attempts could be detected by tools doing automatic processing of scientific knowledge graphs generated by peer review activity. This, coupled with verification of users, should be able to minimize the risk quite significantly.

gchamonlive 4y ago
Seems to me that this wouldn't be enough in case of state actors or large organizations. They would just have to employ real people to carry the fake review bombs.

bittercynic 4y ago

Instead of a flagging system ranking the credibility of papers once for everyone, can we have a system where each user decides which other users they trust to evaluate papers, and ignore any signal from untrusted users? For example, I could L1 trust people to evaluate papers and the trustworthiness of other users, and L2 trust others only to evaluate the trustworthiness of papers. If I notice a spurious signal getting in, I can find out who it came from and prune my list of trusted users. I imagine I'm not the first to think of this. Anyone know of reading material on the topic?

abstractalgo 4y ago
The one thing I'm aware of is Brave's Goggles project: https://search.brave.com/help/goggles https://search.brave.com/help/goggles - it basically allows you to set up your own ranking algorithm with a specific syntax (it links to Github repo and, there, to a white paper). I haven't seen more attempts at similar things, which doesn't mean they don't exist though. (There were mentions of a similar feature being added when Elon Musk was about to buy Twitter, but just gossips)
jpeloquin 4y ago
Sounds like journal clubs but organized through a specialized tech platform rather than normal communication channels. (Journal clubs are groups of people, usually working in the same field, who pick & present a paper to the rest of the group on an rotating basis.)