8 ms·

The mathematics of science’s broken reward system

187 points by seycombi 10y ago

AlphaWeaver 10y ago
Interesting, even if a bit meta.
lbhnact 10y ago
I've always wanted a good word for the 'science of studying science'. Never really got past 'Scientology', and that probably won't stick. What's a better term than the vapid 'Metascience', or the clunky 'scientometrics'. How about 'Superscience'? That would be an awesome Doctorate to hang on the wall.
- Retric 10y ago
  Seems to fit under sociology quite well. Economics got it's own branch mostly for historical reasons.
  - jylam 10y ago
    Sociology is already the science of everything social (that is, people, human society).
    - deleted 10y ago
      [deleted]
  - stdbrouw 10y ago
    The sociology of scientific knowledge is a thing. Some crackpots, some very interesting work as well, e.g. "The Baby Factory" about dodgy science in psychology labs http://srd.sagepub.com/content/2/2378023115625071.abstract http://srd.sagepub.com/content/2/2378023115625071.abstract
- Malarkey73 10y ago
  "Epistemology" is probably the best word.
  - lbhnact 10y ago
    Yeah, maybe 'Quantitative Epistemology'. You'd probably be thrown right out of the Philosophy department though.
    - marcosdumay 10y ago
      Isn't the goal exactly getting thrown out of every department?
    - johncolanduoni 10y ago
      Not really, many people who do philosophy of science/math have science/math PhDs as opposed to philosophy or other humanities.
    - fmoralesc 10y ago
      Only the name, the idea not at all. Formal Epistemology is actually a blooming subfield, Bayesian epistemology being a prominent example.
  - pYQAJ6Zm 10y ago
    Noting that epistemology is concerned with the study of (the concept of) knowledge, and may (in principle) go beyond what’s involved in the purely scientific acquisition of such (if some non-scientific method for the acquisition of knowledge is to be accepted). “Philosophy of science,” already mentioned, is probably a better fit. Once you go meta on science, after all, you are (arguably) not doing science anymore, so there is no harm in naming it a kind of philosophy.
- YeGoblynQueenne 10y ago
  There's always epistemology [1]. It's a branch of philosophy, but I don't think you can examine science on a theoretical basis without going there. There's no reason why not to mix theoretical and practical (as in data-driven) tools. [1] https://en.wikipedia.org/wiki/Epistemology https://en.wikipedia.org/wiki/Epistemology
- trentmb 10y ago
  https://en.wikipedia.org/wiki/Philosophy_of_science https://en.wikipedia.org/wiki/Philosophy_of_science
- jsprogrammer 10y ago
  Where does the 't' come from? Shouldn't it be sciology/scienceology/sciencology?
infinity0 10y ago
The challenge will be to fix this whilst keeping the precise, mechanical and quantitative spirit of science alive. The reason why scientists have a natural "disdain" for sociological studies, is because they don't have these qualities. Many of the arguments aren't convincing from a critical angle, and only convincing to people who have a pre-existing bias towards certain conclusions. Yet the scientific culture of today is falling into that trap itself. The article criticises metric-driven incentives, but it is not metrics (the general concept) that are at fault. It is the choice of metric, and the meta-analysis of this that is lacking. These choices are themselves often backed up by non-scientific vague arguments, of the similar sort that scientists often criticise other fields as depending upon. We must certainly not conclude from these studies that quantitative analysis is itself what is at fault. I know the article doesn't explicitly say this, but it hints at it - using suggestive phrasing like "disdained sociological studies" and referring to all "metric incentives" as a single group - and it is a point I have seen made by many non-scientists. That is, using these flaws as a straw man to attack the very qualities of what has made science so successful and useful. To improve the situation, we must reject these straw-man arguments against science, and develop better methods that are more quantitive, over a broader spectrum of what is being analysed, and that are more self-critical.
- YeGoblynQueenne 10y ago
  I don't think the problem of the article is with quantitative methods per se. Rather, I read it as a gripe about the specific metrics employed, frex, the h-index. A problem is that any such metric is only ever going to be a proxy for some assumed real value, and such a proxy is always going to be controversial, particularly if it's used to control access to funding. Which I think at the end of the day is the big issue here. All this soul-searching is just a big attempt to decide what we spend money on. Who gets to be a scientist, and who gets to become a person with a lot of useless knowledge and an incurable sense of failure.
  - infinity0 10y ago
    It could be read either way - it doesn't talk too much about differences between metrics, and groups them as near-equivalent. Anyway the exact intention of the author/article is not so important. My point is to say that one way that you could (and I'm sure some will) interpret it, is not good. Certainly we need to look at our social reward/incentive systems from an adversarial point of view, in science, economics, politics and elsewhere. Game theory is useful for that, and it can help us develop systems that are less-easily susceptible to the flaws that the article mentions.
known 10y ago
https://en.wikipedia.org/wiki/Economic_mobility https://en.wikipedia.org/wiki/Economic_mobility != https://en.wikipedia.org/wiki/Social_mobility https://en.wikipedia.org/wiki/Social_mobility
hedgew 10y ago
Outside of fields with direct penalties for invalid science (think aviation, nuclear reactors, sinking boats, collapsing bridges), we're just relying on the integrity and honor of researchers. On an individual level it's much more effective to cut corners and publish trendy research than it is to make valid science. The situation is only made worse by the fact that universities reward this; because they too are rewarded more for high-profile research than they are for scientific validity. My guess is that most studies in softer sciences are simply erroneous. Many social sciences have already been shown to be unreliable by the replication crisis. My personal experience comes from reviewing hundreds of published studies that evaluated the applied effectiveness of machine learning models. Half had significant statistical errors.
- chestervonwinch 10y ago
  > My personal experience comes from reviewing hundreds of published studies that evaluated the applied effectiveness of machine learning models. In the social sciences? Just curious. > Half had significant statistical errors. Such as? Have you enumerated the typical errors somewhere, so that other can learn to avoid these common pitfalls?
  - Balgair 10y ago
    The stats errors are so common and easy to correct these days that there is a statscheck (ala spellcheck) program (can't find the link, sorry) out of the Belgium that trawls through journals and checks the stats. Mind you, these are peer reviewed and published articles. Still, about half have erroneous stats, sometimes ones that invalidate the paper. Yeah, we all screw up, but it seems like a lot of the screw ups happen much too often. As for a place that has common stats errors and how to fix them? Well, just about every single intro to stats book, the entire R language, just about any library for just about any language, google, etc. The issue is not that the folks are making honest mistakes, that happens, it's that the system is perverse and incentivizes them.
    - c06n 10y ago
      > statscheck You mean this? https://cran.r-project.org/web/packages/statcheck/index.html https://cran.r-project.org/web/packages/statcheck/index.html
hannob 10y ago
The irony is that this is published in Nature, which undoubtedly is a huge contributor to questionable research practices by publishing spectacular findings and ignoring negative results and replications.
- tdaltonc 10y ago
  I think it's more helpful to blame the incentives and the individual actors. Nature isn't malicious or powerful enough to break science all on it's own. It didn't write the rules, it just plays by them. ie Hate the game, not the player.
  - epistasis 10y ago
    In this case, Nature is the game. Nature is far more powerful in science than any individual scientist, or any individual research institution.
nonbel 10y ago
This is presented as if it is a new problem. It stems back to the 1940s with the adoption of NHST (with its arbitrary "statistical significance" metric) first by educational researchers, then by psychologists, followed by spreading like a cancer throughout the research community from there. Things have only been getting worse and worse as the "old guard" in each field retires/dies leaving behind only people trained to think rejecting a strawman hypothesis according to an arbitrary metric, then concluding something about your hypothesis, counts as science.
cuantos 10y ago
You can have science without academia and visa versa. Perhaps it should say that academia has a broken reward system.
Balgair 10y ago
Science has a lot going wrong with it. Here are some I can list: P-values, the h-index (article discusses), the funding crunches and (US specific) the cyclic nature of NIH funding, the massively skewed incentives of publish or perish, the entire idea of trying to assign value to discoveries (article discusses), the 2-body problem of family life and science (at least the Germans expect you to have no life and are clear on that), the work-life non-balance, the relatively very poor pay, the hyper-competition and sabotage (in some fields, labs, or universities), the total ignorance of doctors/politicians that take your work and apply it in appalling ways, the entire racket of scientific publishing, etc. As always, we should point out to newcomers to this discussion of one of the most read scientific paper EVER by John P. Ioannidis "Why Most Published Research Findings Are False" [0]. If you don't have access and don't want to use Sci-Hub or ICanHazPDF, here is a youtube of John discussing his work [1]. You can use the google to really jump into a rabbit hole here. Essentially, flip a coin, heads the paper is right, tails the paper is wrong, it really is that bad these days. Ok, so lets just scrap the whole thing then, right? Here is why not: https://www.youtube.com/watch?v=_Px5sZyxFYc https://www.youtube.com/watch?v=_Px5sZyxFYc https://www.youtube.com/watch?v=XcPuRaSEq1I https://www.youtube.com/watch?v=XcPuRaSEq1I https://www.youtube.com/watch?v=5EL_bLOK8_A https://www.youtube.com/watch?v=5EL_bLOK8_A https://www.youtube.com/watch?v=qXWYSdijCGU https://www.youtube.com/watch?v=qXWYSdijCGU https://www.youtube.com/watch?v=uBh2LxTW0s0 https://www.youtube.com/watch?v=uBh2LxTW0s0 (a good showing of why science is right and helps folks) https://www.youtube.com/watch?v=gs0JQRT6TpY https://www.youtube.com/watch?v=gs0JQRT6TpY So, yeah, science is a mess right now. But if we scientists don't step up and make it not a mess, those scummy money-grubbing scammer dirtbags and their ilk are more than happy to take up the slack. These fucking asshats are going to kill people like you poor uninformed cousins and their kids and take all their money because they think homeopathy is right and the moon landing was faked. So, you young scientist that looks at this pile of garbage that is modern academia, do not despair! Fight the good fight! Yes, you may end up shirtless and ridiculed by the older scientists. But you have to do what you think is right because the rest of this world is depending upon you! Even for that tiny little bullcrap paper you are getting out just to graduate, that matters too. Be in the mud, be in the arena, fight for truth! [0]http://journals.plos.org/plosmedicine/article?id=10.1371/journal.pmed.0020124 http://journals.plos.org/plosmedicine/article?id=10.1371/jou... [1]https://www.youtube.com/watch?v=KOZAV9AvIQE https://www.youtube.com/watch?v=KOZAV9AvIQE
randomsearch 10y ago
I agree that governments should stop using metrics to evaluate science in this way, and instead qualitatively review research output. It's obvious that citation counts are open to manipulation (e.g. through self-citation and encouraging citations from others via various mechanisms) but there's also a very large factor that can't be easily quantified: an enormous amount of good will has been lost in the system. Many scientists who are interested mainly in satisfying their curiosity and contributing to society resent the top-down mismanagement of science in the UK and US. They see a system where those who play political games do well, but the smartest, most dedicated, and passionate researchers are often sidelined or ignored because they do not spend the requisite time playing the political system to artificially boost their reputations. Many times I've seen very talented academics with huge potential leaving academia as they were passed up for promotion, left on a temporary contract indefinitely, or neglected in other ways. Strangely enough, I think that if you don't have pressure to climb the ladder (e.g. a family), then the current system offers great opportunities to do outstanding science. If you choose to stay low on the ladder and spend the vast majority of your time actually focusing on research problems, you can get far more actual research done than those chasing promotion or esteem. And because so many people are focused on their citation count, or some political game, the competition isn't as strong as it should be - you can make real advances if you quietly focus and leave the politicians to fight each other.
jsprogrammer 10y ago
Couldn't this model be used to detect and alert on bad actors?
BurningFrog 10y ago
Where does this broken system come from? Did anybody construct it, or has it just emerged out of the surrounding world? Is it the same across the planet?
- danieltillett 10y ago
  Like most problems it comes from too many people chasing too few dollars. The ultimate cause was the construction of a system that trains too many people for the money avalible to employ them. The current system was deliberately constructed by politicians after WWII when we had more money (and demand for science) than scientists. They set out to create more scientists by designing a system where each scientist trained 10 to 100s of additional scientists (PhDs). This worked great until around the mid 1970s when the supply of scientists finally caught up with the money. Since then the problem has been getting worse.
foolrush 10y ago
Science hasn't existed forever, and Foucault already smashed to bits of any sort of "evolution with direction" account of science in The Order of Things. The most disturbing thing appears to be that many folks aren't aware of Foucault's work.

187 points by seycombi 10y ago

Interesting, even if a bit meta.

I've always wanted a good word for the 'science of studying science'. Never really got past 'Scientology', and that probably won't stick. What's a better term than the vapid 'Metascience', or the clunky 'scientometrics'. How about 'Superscience'? That would be an awesome Doctorate to hang on the wall.

Retric 10y ago
Seems to fit under sociology quite well. Economics got it's own branch mostly for historical reasons.
- jylam 10y ago
  Sociology is already the science of everything social (that is, people, human society).
  - deleted 10y ago
    [deleted]
- stdbrouw 10y ago
  The sociology of scientific knowledge is a thing. Some crackpots, some very interesting work as well, e.g. "The Baby Factory" about dodgy science in psychology labs http://srd.sagepub.com/content/2/2378023115625071.abstract http://srd.sagepub.com/content/2/2378023115625071.abstract
Malarkey73 10y ago
"Epistemology" is probably the best word.
- lbhnact 10y ago
  Yeah, maybe 'Quantitative Epistemology'. You'd probably be thrown right out of the Philosophy department though.
  - marcosdumay 10y ago
    Isn't the goal exactly getting thrown out of every department?
  - johncolanduoni 10y ago
    Not really, many people who do philosophy of science/math have science/math PhDs as opposed to philosophy or other humanities.
  - fmoralesc 10y ago
    Only the name, the idea not at all. Formal Epistemology is actually a blooming subfield, Bayesian epistemology being a prominent example.
- pYQAJ6Zm 10y ago
  Noting that epistemology is concerned with the study of (the concept of) knowledge, and may (in principle) go beyond what’s involved in the purely scientific acquisition of such (if some non-scientific method for the acquisition of knowledge is to be accepted). “Philosophy of science,” already mentioned, is probably a better fit. Once you go meta on science, after all, you are (arguably) not doing science anymore, so there is no harm in naming it a kind of philosophy.
YeGoblynQueenne 10y ago
There's always epistemology [1]. It's a branch of philosophy, but I don't think you can examine science on a theoretical basis without going there. There's no reason why not to mix theoretical and practical (as in data-driven) tools. [1] https://en.wikipedia.org/wiki/Epistemology https://en.wikipedia.org/wiki/Epistemology
trentmb 10y ago
https://en.wikipedia.org/wiki/Philosophy_of_science https://en.wikipedia.org/wiki/Philosophy_of_science
jsprogrammer 10y ago
Where does the 't' come from? Shouldn't it be sciology/scienceology/sciencology?

infinity0 10y ago

The challenge will be to fix this whilst keeping the precise, mechanical and quantitative spirit of science alive. The reason why scientists have a natural "disdain" for sociological studies, is because they don't have these qualities. Many of the arguments aren't convincing from a critical angle, and only convincing to people who have a pre-existing bias towards certain conclusions. Yet the scientific culture of today is falling into that trap itself. The article criticises metric-driven incentives, but it is not metrics (the general concept) that are at fault. It is the choice of metric, and the meta-analysis of this that is lacking. These choices are themselves often backed up by non-scientific vague arguments, of the similar sort that scientists often criticise other fields as depending upon. We must certainly not conclude from these studies that quantitative analysis is itself what is at fault. I know the article doesn't explicitly say this, but it hints at it - using suggestive phrasing like "disdained sociological studies" and referring to all "metric incentives" as a single group - and it is a point I have seen made by many non-scientists. That is, using these flaws as a straw man to attack the very qualities of what has made science so successful and useful. To improve the situation, we must reject these straw-man arguments against science, and develop better methods that are more quantitive, over a broader spectrum of what is being analysed, and that are more self-critical.

YeGoblynQueenne 10y ago
I don't think the problem of the article is with quantitative methods per se. Rather, I read it as a gripe about the specific metrics employed, frex, the h-index. A problem is that any such metric is only ever going to be a proxy for some assumed real value, and such a proxy is always going to be controversial, particularly if it's used to control access to funding. Which I think at the end of the day is the big issue here. All this soul-searching is just a big attempt to decide what we spend money on. Who gets to be a scientist, and who gets to become a person with a lot of useless knowledge and an incurable sense of failure.
- infinity0 10y ago
  It could be read either way - it doesn't talk too much about differences between metrics, and groups them as near-equivalent. Anyway the exact intention of the author/article is not so important. My point is to say that one way that you could (and I'm sure some will) interpret it, is not good. Certainly we need to look at our social reward/incentive systems from an adversarial point of view, in science, economics, politics and elsewhere. Game theory is useful for that, and it can help us develop systems that are less-easily susceptible to the flaws that the article mentions.

known 10y ago

https://en.wikipedia.org/wiki/Economic_mobility https://en.wikipedia.org/wiki/Economic_mobility != https://en.wikipedia.org/wiki/Social_mobility https://en.wikipedia.org/wiki/Social_mobility

hedgew 10y ago

Outside of fields with direct penalties for invalid science (think aviation, nuclear reactors, sinking boats, collapsing bridges), we're just relying on the integrity and honor of researchers. On an individual level it's much more effective to cut corners and publish trendy research than it is to make valid science. The situation is only made worse by the fact that universities reward this; because they too are rewarded more for high-profile research than they are for scientific validity. My guess is that most studies in softer sciences are simply erroneous. Many social sciences have already been shown to be unreliable by the replication crisis. My personal experience comes from reviewing hundreds of published studies that evaluated the applied effectiveness of machine learning models. Half had significant statistical errors.

chestervonwinch 10y ago
> My personal experience comes from reviewing hundreds of published studies that evaluated the applied effectiveness of machine learning models. In the social sciences? Just curious. > Half had significant statistical errors. Such as? Have you enumerated the typical errors somewhere, so that other can learn to avoid these common pitfalls?
- Balgair 10y ago
  The stats errors are so common and easy to correct these days that there is a statscheck (ala spellcheck) program (can't find the link, sorry) out of the Belgium that trawls through journals and checks the stats. Mind you, these are peer reviewed and published articles. Still, about half have erroneous stats, sometimes ones that invalidate the paper. Yeah, we all screw up, but it seems like a lot of the screw ups happen much too often. As for a place that has common stats errors and how to fix them? Well, just about every single intro to stats book, the entire R language, just about any library for just about any language, google, etc. The issue is not that the folks are making honest mistakes, that happens, it's that the system is perverse and incentivizes them.
  - c06n 10y ago
    > statscheck You mean this? https://cran.r-project.org/web/packages/statcheck/index.html https://cran.r-project.org/web/packages/statcheck/index.html

hannob 10y ago

The irony is that this is published in Nature, which undoubtedly is a huge contributor to questionable research practices by publishing spectacular findings and ignoring negative results and replications.

tdaltonc 10y ago
I think it's more helpful to blame the incentives and the individual actors. Nature isn't malicious or powerful enough to break science all on it's own. It didn't write the rules, it just plays by them. ie Hate the game, not the player.
- epistasis 10y ago
  In this case, Nature is the game. Nature is far more powerful in science than any individual scientist, or any individual research institution.

nonbel 10y ago

This is presented as if it is a new problem. It stems back to the 1940s with the adoption of NHST (with its arbitrary "statistical significance" metric) first by educational researchers, then by psychologists, followed by spreading like a cancer throughout the research community from there. Things have only been getting worse and worse as the "old guard" in each field retires/dies leaving behind only people trained to think rejecting a strawman hypothesis according to an arbitrary metric, then concluding something about your hypothesis, counts as science.

cuantos 10y ago

You can have science without academia and visa versa. Perhaps it should say that academia has a broken reward system.

Balgair 10y ago

Science has a lot going wrong with it. Here are some I can list: P-values, the h-index (article discusses), the funding crunches and (US specific) the cyclic nature of NIH funding, the massively skewed incentives of publish or perish, the entire idea of trying to assign value to discoveries (article discusses), the 2-body problem of family life and science (at least the Germans expect you to have no life and are clear on that), the work-life non-balance, the relatively very poor pay, the hyper-competition and sabotage (in some fields, labs, or universities), the total ignorance of doctors/politicians that take your work and apply it in appalling ways, the entire racket of scientific publishing, etc. As always, we should point out to newcomers to this discussion of one of the most read scientific paper EVER by John P. Ioannidis "Why Most Published Research Findings Are False" [0]. If you don't have access and don't want to use Sci-Hub or ICanHazPDF, here is a youtube of John discussing his work [1]. You can use the google to really jump into a rabbit hole here. Essentially, flip a coin, heads the paper is right, tails the paper is wrong, it really is that bad these days. Ok, so lets just scrap the whole thing then, right? Here is why not: https://www.youtube.com/watch?v=_Px5sZyxFYc https://www.youtube.com/watch?v=_Px5sZyxFYc https://www.youtube.com/watch?v=XcPuRaSEq1I https://www.youtube.com/watch?v=XcPuRaSEq1I https://www.youtube.com/watch?v=5EL_bLOK8_A https://www.youtube.com/watch?v=5EL_bLOK8_A https://www.youtube.com/watch?v=qXWYSdijCGU https://www.youtube.com/watch?v=qXWYSdijCGU https://www.youtube.com/watch?v=uBh2LxTW0s0 https://www.youtube.com/watch?v=uBh2LxTW0s0 (a good showing of why science is right and helps folks) https://www.youtube.com/watch?v=gs0JQRT6TpY https://www.youtube.com/watch?v=gs0JQRT6TpY So, yeah, science is a mess right now. But if we scientists don't step up and make it not a mess, those scummy money-grubbing scammer dirtbags and their ilk are more than happy to take up the slack. These fucking asshats are going to kill people like you poor uninformed cousins and their kids and take all their money because they think homeopathy is right and the moon landing was faked. So, you young scientist that looks at this pile of garbage that is modern academia, do not despair! Fight the good fight! Yes, you may end up shirtless and ridiculed by the older scientists. But you have to do what you think is right because the rest of this world is depending upon you! Even for that tiny little bullcrap paper you are getting out just to graduate, that matters too. Be in the mud, be in the arena, fight for truth! [0]http://journals.plos.org/plosmedicine/article?id=10.1371/journal.pmed.0020124 http://journals.plos.org/plosmedicine/article?id=10.1371/jou... [1]https://www.youtube.com/watch?v=KOZAV9AvIQE https://www.youtube.com/watch?v=KOZAV9AvIQE

randomsearch 10y ago

I agree that governments should stop using metrics to evaluate science in this way, and instead qualitatively review research output. It's obvious that citation counts are open to manipulation (e.g. through self-citation and encouraging citations from others via various mechanisms) but there's also a very large factor that can't be easily quantified: an enormous amount of good will has been lost in the system. Many scientists who are interested mainly in satisfying their curiosity and contributing to society resent the top-down mismanagement of science in the UK and US. They see a system where those who play political games do well, but the smartest, most dedicated, and passionate researchers are often sidelined or ignored because they do not spend the requisite time playing the political system to artificially boost their reputations. Many times I've seen very talented academics with huge potential leaving academia as they were passed up for promotion, left on a temporary contract indefinitely, or neglected in other ways. Strangely enough, I think that if you don't have pressure to climb the ladder (e.g. a family), then the current system offers great opportunities to do outstanding science. If you choose to stay low on the ladder and spend the vast majority of your time actually focusing on research problems, you can get far more actual research done than those chasing promotion or esteem. And because so many people are focused on their citation count, or some political game, the competition isn't as strong as it should be - you can make real advances if you quietly focus and leave the politicians to fight each other.

jsprogrammer 10y ago

Couldn't this model be used to detect and alert on bad actors?

BurningFrog 10y ago

Where does this broken system come from? Did anybody construct it, or has it just emerged out of the surrounding world? Is it the same across the planet?

danieltillett 10y ago
Like most problems it comes from too many people chasing too few dollars. The ultimate cause was the construction of a system that trains too many people for the money avalible to employ them. The current system was deliberately constructed by politicians after WWII when we had more money (and demand for science) than scientists. They set out to create more scientists by designing a system where each scientist trained 10 to 100s of additional scientists (PhDs). This worked great until around the mid 1970s when the supply of scientists finally caught up with the money. Since then the problem has been getting worse.

foolrush 10y ago

Science hasn't existed forever, and Foucault already smashed to bits of any sort of "evolution with direction" account of science in The Order of Things. The most disturbing thing appears to be that many folks aren't aware of Foucault's work.