9 ms·

This seems like the obvious result of many of the unhealthy forces against science today: replacing professorship with part-timers, rewarding publishing quantit

This seems like the obvious result of many of the unhealthy forces against science today: replacing professorship with part-timers, rewarding publishing quantity over quality, pressure to promise a quick monetizable result over in-depth basic research. I don't buy the "low-handing fruit" argument. I think it vastly overstates the fraction of scientific knowledge that we possess. Contemporary scientific research is in a state of crisis: unhealthy and dysfunctional. Until we pay the most talented researchers to spend decades on in-depth study instead of spending pennies on segmented, p-hacked, post-hoc conclusion garbage papers, major discoveries will continue to disappear.

nonbel 8y ago
From what I have seen it is quite simple. 1) There is this pseudo-scientific methodology called NHST (null hypothesis significance testing) devised pretty much on accident in the 1940-50s that tricks people into thinking they have discovered something interesting. 2) It is so much easier to generate these "significant" results than to study a phenomenon and come up with a model capable of making a precise prediction, then have multiple people/groups carefully collect data to check that prediction. 3) People willing to do the NHST thing can publish papers orders of magnitude faster than people not willing to. 4) Number of publications is the primary metric of success in academia. 5) People willing to do the NHST thing slowly take over the field as those who want to do real science get pushed out, retire, or leave for greener pastures. 6) Eventually the field is filled with people generating worthless papers that cannot be replicated in principle and often contradict each other. - a) Because all that matters is you got a significant result published in a peer reviewed journal - b) The discovery has already been made, the statistics said so, why would anyone need to replicate it? And to preempt a common confusion. This has nothing to do with bayesian vs frequentist, testing a strawman null hypothesis can be done using bayesian math just as well as frequentist. Some decent intros to this problem (yes, it was pointed out long ago but there is no stopping the need to publish apparently): Meehl 1967: https://meehl.dl.umn.edu/sites/g/files/pua1696/f/074theorytestingparadox.pdf https://meehl.dl.umn.edu/sites/g/files/pua1696/f/074theoryte... Gigerenzer 2004: https://library.mpib-berlin.mpg.de/ft/gg/GG_Mindless_2004.pdf https://library.mpib-berlin.mpg.de/ft/gg/GG_Mindless_2004.pd...
- fouc 8y ago
  First time I've heard that NHST could be a source of issues with science. I've heard about p-hacking issues or frequentist statistics getting the blame, but NHST being a root cause seems plausible in retrospect. Thanks!
  - nonbel 8y ago
    Great to hear you are interested in this problem. It is a huge one that goes mostly ignored. Afaict, the p-hacking/etc people usually focus on are red herrings caused by the incompatibility of NHST and scientific research. Eg, it is good thing for a scientist to analyze the data in a bunch of different ways. It is definitely not something that should be punished.
  - kiliantics 8y ago
    Calculating p-values falls under hypothesis testing
    - nonbel 8y ago
      1) Hypothesis testing, significance testing, and NHST are all different things 2) P-values seem ok as a way to check how well your model fits the data. The problem is testing a strawman model/hypothesis.
- kgwgk 8y ago
  The article seems to be mostly about “hard” sciences, where ”discoveries” are not about getting some p<0.05
  - nonbel 8y ago
    Unfortunately, I can see its infection of physics is in progress. And testing 0.05 vs .000005 makes no difference if you are testing a model other than the one you believe. It just makes it more expensive and time consuming to draw an inappropriate conclusion from the p-value. On the other hand, thankfully, in physics areas they usually also test actual predictions of their models. But that is part of the process you can see happened in psych, then medicine. If you sample the literature going back to 1900 or so: Stage 1: Science - Test the predictions of your models Stage 2: Hybrid - Test the predictions of your models and also strawman null/background models Stage 3: Pseudoscience - Test only strawman null/background models My guess is during stage 2 there are still people untrained in NHST in important positions who hold back the wave of BS by demanding actual science (testing predictions of models someone thinks may work). Once they retire you get stage 3. One day I want to write this all up, its really quite interesting to read the old psych and medicine papers where people were coming up with mathematical laws, etc then the slow progression to "is there a significant correlation/difference?".
WhompingWindows 8y ago
I think there are consistencies in your second paragraph. You don't buy the low-hanging fruit argument, however you then state that we need to spend decades on in-depth studies. Well, those are exactly the opposite of low-hanging fruit, those are extremely hard-to-reach and abstract fruits that the vast majority of the scientists of the past would have been unable to grasp, simply because they did not have access to all the foundations we do today. You can't attack Poincare's conjecture without basic arithmetic, nor can you understand antibiotics without a knowledge of infection and germ theory. Look at some of the most sea-changing discoveries in the history of science and you will see it was not the result of decades of laborious and thoughtful calculation and musing. Penicillin was discovered by coincidence or luck; the production of rubber for tires was also lucky. How about chance astronomical observations like an eclipse, whereby a great deal can be learned in a fleeting moment? How about displacement of water, or the pythagorean theorem, or Michael Faraday's cheap and quick experiments into electricity. These are not tours de force, by and large they are little fun divertissements, interesting observations with great consequences. I think there's no doubt that MANY of the low hanging fruits have already been plucked. It's the same way in many other fields, such as music, art, computer science, or anything else. People naturally discover the simplest things first, and then build upon those, with later discoveries taking far more investment of time/learning to even be capable of understanding, let alone discovering.
- dv_dt 8y ago
  I suspect to some extent, performance metrics of research grants steer scientists away from low hanging fruit in unknown areas, and into well known paths that can be justified in grant proposals. Most labs I would suspect have little time to just play around, or even stop and contemplate anomalous behavior like you're describing.
JPKab 8y ago
Agreed, BUT: Sorry, academia is broken, and the professors are a big part of the problem. We don't need more of them, or full timers. Let's stop teaching lectures to undergrads in medieval style in 500 person lecture halls, and instead get them engaged in research as assistants right away. We need more science and less "publish or perish" politicians posing as professors engaged in grant grabbing. I have tremendous resentment against the lazy turds I had to sit and listen to as they wasted my money not teaching and not researching either. Tweed jacket welfare recipients if you ask me.
- throwawayjava 8y ago
  > Let's stop teaching lectures to undergrads in medieval style in 500 person lecture halls Although I agree with this sentiment, you have to realize that the alternatives are either a) worse, or b) don't scale. I prefer the ones that don't scale (small number of students in a class who get real continuous feedback and fully participate in the lecture). It works really well, but this model is enormously expensive compared to the 500 person lecture hall. 500 person lecture halls suck, but what's the scale-able, better alternative? MOOCs aren't the answer -- that's just the old 500 person lecture hall model with a pause button. Woohoo. Plus, it doesn't actually solve the real problem, which is that you need the humans running the course anyway. For a well-oiled machine of a course, maybe 10% of the actual time spent on teaching is spent in the classroom. And for non-well-oiled courses, creating the MOOC and keeping it up to date is a hell of a lot of up-front cost, and with a lot of risk. > and instead get them engaged in research as assistants right away. I can tell that you haven't worked with lots of undergraduate students on research projects. Most undergraduate students aren't ready to participate in research. Of those that are intellectually prepared, few have the grit/persistence required. Of those with the intellectual preparation and persistence, fewer still have the imaginative faculties to actually do research (as opposed to e.g. acting as a tech). This is true even in very applied areas. You'd be surprised how few CS undergraduates would succeed at implementing something as simple as an experimental change to a kernel or a tweak to an ML algorithm. At some point you simplify the research problem enough that it's indistinguishable from a course project or homework assignment. In fact, that's even where some course projects and homework assignments come from. But at that point, you're not "involving students in research", unless "doing homework or programming assignments" = "research" that is. > We need more science and less "publish or perish" politicians posing as professors engaged in grant grabbing. At times I have certainly felt this way about academia! But I could say similar rude things about every profession. > Tweed jacket welfare recipients if you ask me. There are ineffective people in every field, and most of them manage to stay more-or-less employed throughout their working years.
jimjansen1 8y ago
The publish or perish system is obviously a major issue. But in fields like Physics all the low hanging fruit really is gone. To probe deeper into particle physics we need massively more powerful colliders that are economically or technologically unfeasible. Answering fundamental questions like is the Universe finite or is it curved would take experiments so large in scale that we are probably millennia away from being able to tackle them. There are only a handful of open questions left in physics and most of them are similarly intractable.
dnprock 8y ago
I think you need to ask the question about the unhealthy forces in science today. Are these forces breaking science? Or Is the investment model (grants) creating these forces? I think it is the latter. That's the whole point of the article. We need to rethink how we allocate dollars.
- v_lisivka 8y ago
  Any ideas how to fight with grantoeaters?
706f6f70 8y ago
> Until we pay the most talented researchers It's not even that. One of the contributing factors is that we are no longer hiring for talent or merit. If you're a male child interested in science and in high school right now you've been through 8 years telling you that you should not go into science because they need more women of colour who follow the right religions and have the right sexual orientation. You're constantly bombarded with "SCIENCE DOES NOT WANT MEN". And sure, we can say "well if he had the necessary grit, this would not discourage him". No. No amount of grit will get you in when governments are shaping science funding to punish having male staff. No amount of grit will get you in when indoctrination training is mandatory for the selection committees. No amount of grit will get you in when the first filter in a hiring process is to remove all males applicants. I'm sorry but science has to go through a decline in the West for a bit. We still have some remnants to be worked through the system, but the next couple generations of Western scientists are going to be generally pretty low quality. On the bright side, science is not exclusively a Western tradition at this point. Other cultures that at least pretend to still focus on merit and talent will carry the scientific tradition forward.
- Wowfunhappy 8y ago
  > If you're a male child interested in science and in high school right now you've been through 8 years telling you that you should not go into science because they need more women of colour who follow the right religions and have the right sexual orientation. You're constantly bombarded with "SCIENCE DOES NOT WANT MEN". Do you have any examples of teens / young adults being discouraged from going into science because they are white and male? There are certainly some (I'd say good!) programs to encourage women and minorities, but that's quite a different thing from men being actively discouraged. Scientific fields are still significantly more male than female, even among recent graduates.
  - v_lisivka 8y ago
    > Scientific fields are still significantly more male than female, even among recent graduates. Why "still"? Majority of mans are genetically selected to be adventurers, while minority of woman are adventurers. It's impossible to change genetics with advertising, so we will never have 50/50% split, until human genome will change. Why not just accept natural distribution?

nonbel 8y ago

From what I have seen it is quite simple. 1) There is this pseudo-scientific methodology called NHST (null hypothesis significance testing) devised pretty much on accident in the 1940-50s that tricks people into thinking they have discovered something interesting. 2) It is so much easier to generate these "significant" results than to study a phenomenon and come up with a model capable of making a precise prediction, then have multiple people/groups carefully collect data to check that prediction. 3) People willing to do the NHST thing can publish papers orders of magnitude faster than people not willing to. 4) Number of publications is the primary metric of success in academia. 5) People willing to do the NHST thing slowly take over the field as those who want to do real science get pushed out, retire, or leave for greener pastures. 6) Eventually the field is filled with people generating worthless papers that cannot be replicated in principle and often contradict each other. - a) Because all that matters is you got a significant result published in a peer reviewed journal - b) The discovery has already been made, the statistics said so, why would anyone need to replicate it? And to preempt a common confusion. This has nothing to do with bayesian vs frequentist, testing a strawman null hypothesis can be done using bayesian math just as well as frequentist. Some decent intros to this problem (yes, it was pointed out long ago but there is no stopping the need to publish apparently): Meehl 1967: https://meehl.dl.umn.edu/sites/g/files/pua1696/f/074theorytestingparadox.pdf https://meehl.dl.umn.edu/sites/g/files/pua1696/f/074theoryte... Gigerenzer 2004: https://library.mpib-berlin.mpg.de/ft/gg/GG_Mindless_2004.pdf https://library.mpib-berlin.mpg.de/ft/gg/GG_Mindless_2004.pd...

fouc 8y ago
First time I've heard that NHST could be a source of issues with science. I've heard about p-hacking issues or frequentist statistics getting the blame, but NHST being a root cause seems plausible in retrospect. Thanks!
- nonbel 8y ago
  Great to hear you are interested in this problem. It is a huge one that goes mostly ignored. Afaict, the p-hacking/etc people usually focus on are red herrings caused by the incompatibility of NHST and scientific research. Eg, it is good thing for a scientist to analyze the data in a bunch of different ways. It is definitely not something that should be punished.
- kiliantics 8y ago
  Calculating p-values falls under hypothesis testing
  - nonbel 8y ago
    1) Hypothesis testing, significance testing, and NHST are all different things 2) P-values seem ok as a way to check how well your model fits the data. The problem is testing a strawman model/hypothesis.
kgwgk 8y ago
The article seems to be mostly about “hard” sciences, where ”discoveries” are not about getting some p<0.05
- nonbel 8y ago
  Unfortunately, I can see its infection of physics is in progress. And testing 0.05 vs .000005 makes no difference if you are testing a model other than the one you believe. It just makes it more expensive and time consuming to draw an inappropriate conclusion from the p-value. On the other hand, thankfully, in physics areas they usually also test actual predictions of their models. But that is part of the process you can see happened in psych, then medicine. If you sample the literature going back to 1900 or so: Stage 1: Science - Test the predictions of your models Stage 2: Hybrid - Test the predictions of your models and also strawman null/background models Stage 3: Pseudoscience - Test only strawman null/background models My guess is during stage 2 there are still people untrained in NHST in important positions who hold back the wave of BS by demanding actual science (testing predictions of models someone thinks may work). Once they retire you get stage 3. One day I want to write this all up, its really quite interesting to read the old psych and medicine papers where people were coming up with mathematical laws, etc then the slow progression to "is there a significant correlation/difference?".

WhompingWindows 8y ago

I think there are consistencies in your second paragraph. You don't buy the low-hanging fruit argument, however you then state that we need to spend decades on in-depth studies. Well, those are exactly the opposite of low-hanging fruit, those are extremely hard-to-reach and abstract fruits that the vast majority of the scientists of the past would have been unable to grasp, simply because they did not have access to all the foundations we do today. You can't attack Poincare's conjecture without basic arithmetic, nor can you understand antibiotics without a knowledge of infection and germ theory. Look at some of the most sea-changing discoveries in the history of science and you will see it was not the result of decades of laborious and thoughtful calculation and musing. Penicillin was discovered by coincidence or luck; the production of rubber for tires was also lucky. How about chance astronomical observations like an eclipse, whereby a great deal can be learned in a fleeting moment? How about displacement of water, or the pythagorean theorem, or Michael Faraday's cheap and quick experiments into electricity. These are not tours de force, by and large they are little fun divertissements, interesting observations with great consequences. I think there's no doubt that MANY of the low hanging fruits have already been plucked. It's the same way in many other fields, such as music, art, computer science, or anything else. People naturally discover the simplest things first, and then build upon those, with later discoveries taking far more investment of time/learning to even be capable of understanding, let alone discovering.

dv_dt 8y ago
I suspect to some extent, performance metrics of research grants steer scientists away from low hanging fruit in unknown areas, and into well known paths that can be justified in grant proposals. Most labs I would suspect have little time to just play around, or even stop and contemplate anomalous behavior like you're describing.

JPKab 8y ago

Agreed, BUT: Sorry, academia is broken, and the professors are a big part of the problem. We don't need more of them, or full timers. Let's stop teaching lectures to undergrads in medieval style in 500 person lecture halls, and instead get them engaged in research as assistants right away. We need more science and less "publish or perish" politicians posing as professors engaged in grant grabbing. I have tremendous resentment against the lazy turds I had to sit and listen to as they wasted my money not teaching and not researching either. Tweed jacket welfare recipients if you ask me.

throwawayjava 8y ago
> Let's stop teaching lectures to undergrads in medieval style in 500 person lecture halls Although I agree with this sentiment, you have to realize that the alternatives are either a) worse, or b) don't scale. I prefer the ones that don't scale (small number of students in a class who get real continuous feedback and fully participate in the lecture). It works really well, but this model is enormously expensive compared to the 500 person lecture hall. 500 person lecture halls suck, but what's the scale-able, better alternative? MOOCs aren't the answer -- that's just the old 500 person lecture hall model with a pause button. Woohoo. Plus, it doesn't actually solve the real problem, which is that you need the humans running the course anyway. For a well-oiled machine of a course, maybe 10% of the actual time spent on teaching is spent in the classroom. And for non-well-oiled courses, creating the MOOC and keeping it up to date is a hell of a lot of up-front cost, and with a lot of risk. > and instead get them engaged in research as assistants right away. I can tell that you haven't worked with lots of undergraduate students on research projects. Most undergraduate students aren't ready to participate in research. Of those that are intellectually prepared, few have the grit/persistence required. Of those with the intellectual preparation and persistence, fewer still have the imaginative faculties to actually do research (as opposed to e.g. acting as a tech). This is true even in very applied areas. You'd be surprised how few CS undergraduates would succeed at implementing something as simple as an experimental change to a kernel or a tweak to an ML algorithm. At some point you simplify the research problem enough that it's indistinguishable from a course project or homework assignment. In fact, that's even where some course projects and homework assignments come from. But at that point, you're not "involving students in research", unless "doing homework or programming assignments" = "research" that is. > We need more science and less "publish or perish" politicians posing as professors engaged in grant grabbing. At times I have certainly felt this way about academia! But I could say similar rude things about every profession. > Tweed jacket welfare recipients if you ask me. There are ineffective people in every field, and most of them manage to stay more-or-less employed throughout their working years.

jimjansen1 8y ago

The publish or perish system is obviously a major issue. But in fields like Physics all the low hanging fruit really is gone. To probe deeper into particle physics we need massively more powerful colliders that are economically or technologically unfeasible. Answering fundamental questions like is the Universe finite or is it curved would take experiments so large in scale that we are probably millennia away from being able to tackle them. There are only a handful of open questions left in physics and most of them are similarly intractable.

dnprock 8y ago

I think you need to ask the question about the unhealthy forces in science today. Are these forces breaking science? Or Is the investment model (grants) creating these forces? I think it is the latter. That's the whole point of the article. We need to rethink how we allocate dollars.

v_lisivka 8y ago
Any ideas how to fight with grantoeaters?

706f6f70 8y ago

> Until we pay the most talented researchers It's not even that. One of the contributing factors is that we are no longer hiring for talent or merit. If you're a male child interested in science and in high school right now you've been through 8 years telling you that you should not go into science because they need more women of colour who follow the right religions and have the right sexual orientation. You're constantly bombarded with "SCIENCE DOES NOT WANT MEN". And sure, we can say "well if he had the necessary grit, this would not discourage him". No. No amount of grit will get you in when governments are shaping science funding to punish having male staff. No amount of grit will get you in when indoctrination training is mandatory for the selection committees. No amount of grit will get you in when the first filter in a hiring process is to remove all males applicants. I'm sorry but science has to go through a decline in the West for a bit. We still have some remnants to be worked through the system, but the next couple generations of Western scientists are going to be generally pretty low quality. On the bright side, science is not exclusively a Western tradition at this point. Other cultures that at least pretend to still focus on merit and talent will carry the scientific tradition forward.

Wowfunhappy 8y ago
> If you're a male child interested in science and in high school right now you've been through 8 years telling you that you should not go into science because they need more women of colour who follow the right religions and have the right sexual orientation. You're constantly bombarded with "SCIENCE DOES NOT WANT MEN". Do you have any examples of teens / young adults being discouraged from going into science because they are white and male? There are certainly some (I'd say good!) programs to encourage women and minorities, but that's quite a different thing from men being actively discouraged. Scientific fields are still significantly more male than female, even among recent graduates.
- v_lisivka 8y ago
  > Scientific fields are still significantly more male than female, even among recent graduates. Why "still"? Majority of mans are genetically selected to be adventurers, while minority of woman are adventurers. It's impossible to change genetics with advertising, so we will never have 50/50% split, until human genome will change. Why not just accept natural distribution?