4 ms·

Having a "reputation system" at all is already problematic in the case of research. Research should not tend to satisfy arbitrary criteria. Nowadays with these

by p4bl0 12y ago

Having a "reputation system" at all is already problematic in the case of research. Research should not tend to satisfy arbitrary criteria. Nowadays with these h-index and impact factor crap, you see for instance journals which require (more or less officially) authors of an accepted paper to cite at least one or two other recent (1 or 2 years old max) papers from the journal. Any reputation system will work only for its own benefit and not for the benefit of research. I'm not at all saying that academics should not be evaluated. Only you can't judge the quality and pertinence of work at the state-of-the-art of so many domains with one (or even a few) number(s).

AnimalMuppet 12y ago
There are more papers than anyone can read. So how do you tell which ones are important? And how do you objectively tell which professors are better than others at doing research? Whatever method you use, someone will try to game it.
- p4bl0 12y ago
  > There are more papers than anyone can read. So how do you tell which ones are important? Let the researcher decide. When an academic applies for a job he/she can attach his/her two or three most relevant papers. > And how do you objectively tell which professors are better than others at doing research? You can't, that's my point, the question makes no sense. > Whatever method you use, someone will try to game it. Then don't. Don't use a method. Evaluate a researcher by actually looking at her/his research. Yes, it takes longer to read a paper than to read a number. But the number makes no sense and who decided that we have to be in a hurry when deciding how to spend mostly-public money on research and researchers? That's actually a good question. What legitimacy have privately held journals, owned by publishers who sell back the results of research to the researchers who gave it to them for free (in the bast case scenario, sometimes you get to pay them to give it to them), to get this much influence on how to spend public money on research? Because that's pretty much what bibliometrics amounts to.
- Retric 12y ago
  There are more webpages than anyone can read. So how do you tell which ones are useful?
- logfromblammo 12y ago
  If they game you, then game them back. Make the publication (and mitigation) of potential attack vectors into the reputation system worth more points than could ever be earned by exploitation of the flaw, and serve exploiters a large penalty whenever they are caught. All you need to do is make the expected value of an attack against the system worth less than just doing some actual research.
  - chaqke 12y ago
    "All you need to do is make the expected value of an attack against the system worth less than just doing some actual research." you'll always have some people willing to take a gamble on the risk to get the reward, especially if it's a low % chance that they get any penalty at all. on top of the conscientious choice, humans are bad at estimating low-likelihood risks.

by p4bl0 12y ago

AnimalMuppet 12y ago

There are more papers than anyone can read. So how do you tell which ones are important? And how do you objectively tell which professors are better than others at doing research? Whatever method you use, someone will try to game it.

p4bl0 12y ago
> There are more papers than anyone can read. So how do you tell which ones are important? Let the researcher decide. When an academic applies for a job he/she can attach his/her two or three most relevant papers. > And how do you objectively tell which professors are better than others at doing research? You can't, that's my point, the question makes no sense. > Whatever method you use, someone will try to game it. Then don't. Don't use a method. Evaluate a researcher by actually looking at her/his research. Yes, it takes longer to read a paper than to read a number. But the number makes no sense and who decided that we have to be in a hurry when deciding how to spend mostly-public money on research and researchers? That's actually a good question. What legitimacy have privately held journals, owned by publishers who sell back the results of research to the researchers who gave it to them for free (in the bast case scenario, sometimes you get to pay them to give it to them), to get this much influence on how to spend public money on research? Because that's pretty much what bibliometrics amounts to.
Retric 12y ago
There are more webpages than anyone can read. So how do you tell which ones are useful?
logfromblammo 12y ago
If they game you, then game them back. Make the publication (and mitigation) of potential attack vectors into the reputation system worth more points than could ever be earned by exploitation of the flaw, and serve exploiters a large penalty whenever they are caught. All you need to do is make the expected value of an attack against the system worth less than just doing some actual research.
- chaqke 12y ago
  "All you need to do is make the expected value of an attack against the system worth less than just doing some actual research." you'll always have some people willing to take a gamble on the risk to get the reward, especially if it's a low % chance that they get any penalty at all. on top of the conscientious choice, humans are bad at estimating low-likelihood risks.