5 ms·

Fails to touch on the perverse incentives in academia, "publish or perish" etc. Torturing a dataset to find a p value that a journal will like (or equivalent st

by harry8 8y ago

Fails to touch on the perverse incentives in academia, "publish or perish" etc. Torturing a dataset to find a p value that a journal will like (or equivalent stat measure) is better for your career than not publishing a paper that will be discredited in time. You have no incentive at all to decide "my results are unconvincing at this point, I'm not going to submit them" and every reason to write them up as a useful contribution to human understanding even if you kind of know, deep down, it really isn't. Especially if you're not senior...

marmaduke 8y ago
This is the unfortunate truth. Furthermore, those who take more time to find a general, robust and theoretically sound result (beyond just the p-optimized publishable result), get filtered out of te tenure positions as not have a high impact factor, despite their papers in fact having a higher impact.
- avip 8y ago
  You may have meant "bullshitable result".
  - marmaduke 8y ago
    They aren’t mutually exclusive
polskibus 8y ago
Seems like we need general solutions to overfitting, not just in machine learning.
konschubert 8y ago
Some days ago there was a post here on HN about a post doc who failed to get tenure. And every other comment was like: “What did he expect, he had much less than the usual two papers a year.” So it seems that even here on HN, the mindset of quantity over quality still persists.
- thomasz 8y ago
  Those were not normative statements.
  - konschubert 8y ago
    That's possible of course.
- baxtr 8y ago
  The problem really is: there is no alternative (which I know of). Having a number of papers in well known journals is everything we have to gauge the quality of people that look for a life time position. It’s sad but true
  - pas 8y ago
    No, the number of papers is an okay metric, thr problem is that journals don't like to publish honest negative results
    - Jeff_Brown 8y ago
      There are a few journals that will promise to publish your results based entirely on your pre-registered plan -- e.g. whether or not you find the correlation you were looking for. In the long term, it seems like journals that publish lots of falsified papers should be punished, and journals that don't (e.g. because of a judge-upon-pre-registration policy) should crowd them out.
      - pas 8y ago
        Pre-registration is great, and I think it even helps researchers/scientists/post-docs to submit something (it doesn't have to be perfect, after all, it's just an experiment design, and they don't have to worry about massaging the data to have promising results to report) and then stick to the topic, and then carry out the experiment as well as they can, and gather as much high quality & fidelity data as they can, and do the analysis according to the plan, and then report it. No extra pressure to think about how to frame what you "found". Though this will inevitably lead to the problem that grant boards face, that it'll be a lot harder to differentiate between proposed experiments. And it'll be even harder to do boring stuff. (So if we assume all submitted plans are sound, they have to publish them all. Though then we'll have journals based on how strict they are with experiment design requirements, 1 sigma, 2 sigma, 5 sigma, etc.)
alexgmcm 8y ago
This is the real issue. Getting data is always expensive in any field so it's much easier to analyse data that already exists as you don't need a large grant application. Furthermore, I think the use of certain ML techniques may be akin to resume-driven development particularly for PhD students given that the career prospects in Data Science in industry (and using the HR buzzwords like AI, ML, Deep Learning etc.) are much better than the thin pickings that remain in academia in many fields.
- analog31 8y ago
  I don't blame the students too much. If you get into a PhD program, it's very hard to change fields or even specialties without starting over. A simpler tactic is to weave a field of interest into your existing research project, then market yourself in that field when you graduate. I did the same thing as a physics student, but with electronics and programming, which furnished me with a marketable resume.
Waterluvian 8y ago
We need to find resources to fund "Failures in Science" journals that exclusively seek to publish interesting research that went nowhere. Personally I'd find these far more interesting to study. "Here's some background. Here's a pretty logical, plausible hypothesis we came up with and how, here's our experiment, here's our results, here's our thoughts as to why we were wildly wrong."
- thegabriele 8y ago
  Instead of allows failures (which would be also good btw), i would increase the time at disposal needed to pubblish something worthwhile. The greatest thinkers in history had no pressure outside their own urgence to solve/explore. Surely, not everyone is Galileo but today science need that kind of freedom.
- parmegv 8y ago
  There is a new conference in cryptology that promotes exactly this: CFAIL. From their 2019 website[0]: "Do you have insightful and exciting work sitting in a drawer somewhere because it never quite panned out? Are you willing to share your failed approaches so that others can learn from them without having to re-travel the same road? Are you tired of reading papers that pretend the incremental result they happened to achieve was well-motivated and was their goal all along? We are! That’s why we are founding a new conference: a place for papers that describe instructive failures or not-yet-successes, as they may prefer to be called." [0] https://www.cfail2019.com/ https://www.cfail2019.com/
- hannob 8y ago
  We have that. People don't use it. PLOS e.g. specifically stated multiple times that they'll publish what meets their quality standards regardless of a positive or negative outcome. The problem is: Even if you publish failed research it won't get cited as much. And people still use citation metrics to evaluate "quality" of science. Just having journals that publish your "failed" research is a good start, but it's not changing the incentive structure.
  - Waterluvian 8y ago
astazangasta 8y ago
Publish or perish is gone. The NIH is no longer the funding source of choice, now it is pharma or private donors. These people want large impressive datasets, not publications. You can rent out data to pharma without having to publish anything. These don't even have to be useful datasets, mere size is enough to make them valuable to greedy prospectors with lots of cash. Then they can plumb them for bunk results ad infinitum.
hinkley 8y ago
Didn’t we used to value negative outcomes?

by harry8 8y ago

marmaduke 8y ago

This is the unfortunate truth. Furthermore, those who take more time to find a general, robust and theoretically sound result (beyond just the p-optimized publishable result), get filtered out of te tenure positions as not have a high impact factor, despite their papers in fact having a higher impact.

avip 8y ago
You may have meant "bullshitable result".
- marmaduke 8y ago
  They aren’t mutually exclusive

polskibus 8y ago

Seems like we need general solutions to overfitting, not just in machine learning.

konschubert 8y ago

Some days ago there was a post here on HN about a post doc who failed to get tenure. And every other comment was like: “What did he expect, he had much less than the usual two papers a year.” So it seems that even here on HN, the mindset of quantity over quality still persists.

thomasz 8y ago
Those were not normative statements.
- konschubert 8y ago
  That's possible of course.
baxtr 8y ago
The problem really is: there is no alternative (which I know of). Having a number of papers in well known journals is everything we have to gauge the quality of people that look for a life time position. It’s sad but true
- pas 8y ago
  No, the number of papers is an okay metric, thr problem is that journals don't like to publish honest negative results
  - Jeff_Brown 8y ago
    There are a few journals that will promise to publish your results based entirely on your pre-registered plan -- e.g. whether or not you find the correlation you were looking for. In the long term, it seems like journals that publish lots of falsified papers should be punished, and journals that don't (e.g. because of a judge-upon-pre-registration policy) should crowd them out.
    - pas 8y ago
      Pre-registration is great, and I think it even helps researchers/scientists/post-docs to submit something (it doesn't have to be perfect, after all, it's just an experiment design, and they don't have to worry about massaging the data to have promising results to report) and then stick to the topic, and then carry out the experiment as well as they can, and gather as much high quality & fidelity data as they can, and do the analysis according to the plan, and then report it. No extra pressure to think about how to frame what you "found". Though this will inevitably lead to the problem that grant boards face, that it'll be a lot harder to differentiate between proposed experiments. And it'll be even harder to do boring stuff. (So if we assume all submitted plans are sound, they have to publish them all. Though then we'll have journals based on how strict they are with experiment design requirements, 1 sigma, 2 sigma, 5 sigma, etc.)

alexgmcm 8y ago

This is the real issue. Getting data is always expensive in any field so it's much easier to analyse data that already exists as you don't need a large grant application. Furthermore, I think the use of certain ML techniques may be akin to resume-driven development particularly for PhD students given that the career prospects in Data Science in industry (and using the HR buzzwords like AI, ML, Deep Learning etc.) are much better than the thin pickings that remain in academia in many fields.

analog31 8y ago
I don't blame the students too much. If you get into a PhD program, it's very hard to change fields or even specialties without starting over. A simpler tactic is to weave a field of interest into your existing research project, then market yourself in that field when you graduate. I did the same thing as a physics student, but with electronics and programming, which furnished me with a marketable resume.

Waterluvian 8y ago

We need to find resources to fund "Failures in Science" journals that exclusively seek to publish interesting research that went nowhere. Personally I'd find these far more interesting to study. "Here's some background. Here's a pretty logical, plausible hypothesis we came up with and how, here's our experiment, here's our results, here's our thoughts as to why we were wildly wrong."

thegabriele 8y ago
Instead of allows failures (which would be also good btw), i would increase the time at disposal needed to pubblish something worthwhile. The greatest thinkers in history had no pressure outside their own urgence to solve/explore. Surely, not everyone is Galileo but today science need that kind of freedom.
parmegv 8y ago
There is a new conference in cryptology that promotes exactly this: CFAIL. From their 2019 website[0]: "Do you have insightful and exciting work sitting in a drawer somewhere because it never quite panned out? Are you willing to share your failed approaches so that others can learn from them without having to re-travel the same road? Are you tired of reading papers that pretend the incremental result they happened to achieve was well-motivated and was their goal all along? We are! That’s why we are founding a new conference: a place for papers that describe instructive failures or not-yet-successes, as they may prefer to be called." [0] https://www.cfail2019.com/ https://www.cfail2019.com/
hannob 8y ago
We have that. People don't use it. PLOS e.g. specifically stated multiple times that they'll publish what meets their quality standards regardless of a positive or negative outcome. The problem is: Even if you publish failed research it won't get cited as much. And people still use citation metrics to evaluate "quality" of science. Just having journals that publish your "failed" research is a good start, but it's not changing the incentive structure.
- Waterluvian 8y ago

astazangasta 8y ago

Publish or perish is gone. The NIH is no longer the funding source of choice, now it is pharma or private donors. These people want large impressive datasets, not publications. You can rent out data to pharma without having to publish anything. These don't even have to be useful datasets, mere size is enough to make them valuable to greedy prospectors with lots of cash. Then they can plumb them for bunk results ad infinitum.

hinkley 8y ago

Didn’t we used to value negative outcomes?