4 ms·

The model of science that worked from Newton to Landau seems to be falling apart in today's scale. There are millions of people whose livelihood depends on pub

The model of science that worked from Newton to Landau seems to be falling apart in today's scale. There are millions of people whose livelihood depends on publishing, so they will publish anything they'll get away with. The amount of noise is beyond any researcher's ability to pick through. True incremental improvements in all areas are drowned in a steady flow of bad research. Top tier institutions seem to survive in some sort of bubbles.

thaumasiotes 6y ago
> There are millions of people whose livelihood depends on publishing, so they will publish anything they'll get away with. This is also the problem with the iOS App Store and the Google Play store. They are modeled off Linux repositories, and those do not cause major problems. The phone app stores are riddled with problems, because you can charge for apps in those stores. If you're willing to pay people to lie to you, they will. You have to make the tradeoff between higher participation with lots of fraud, and lower participation with not that much fraud. > Top tier institutions seem to survive in some sort of bubbles. Not really; compare Brian Wansink at Cornell. ( https://en.wikipedia.org/wiki/Brian_Wansink https://en.wikipedia.org/wiki/Brian_Wansink ) Research output is reliable where it is actively relied on by engineers -- and not elsewhere. At this point, fraud is the norm and research is the exception in academia overall.
- unishark 6y ago
  > Research output is reliable where it is actively relied on by engineers -- and not elsewhere. I'd word it this way: if the person producing the output is not responsible for it really working, it almost certainly won't. Even innocently this will be the case with anything complex, people get things backwards, miss a scale factor, etc. Finding that last bug can take more work than the rest of the project combined, much easier to publish what appears to work and move on. Much more so when there's direct career benefits to "hacking" the system over competing honestly. Especially considering the internet is awash with people trying to cheat through every other competition (exam questions, interview questions, etc.)
  - thaumasiotes 6y ago
    > if the person producing the output is not responsible for it really working, it almost certainly won't. Even innocently this will be the case with anything complex Indeed, there's a great example in the article itself, in a totally unrelated area: > I felt these journals generally did their best, and the slowness of the process likely comes from the bureaucracy of the process and the inexperience editors have with that process. In other words, these reasonable journals weren't able to use their retraction process even though they wanted to, because the process never gets used and therefore isn't in a usable state.
    - thaumasiotes 6y ago
      > these reasonable journals weren't able to use their retraction process even though they wanted to, because the process never gets used and therefore isn't in a usable state. Actually, this made me think about a journal purposefully running intentionally spurious papers, with a challenge to the readership to identify which paper was fake. If the system worked, that would cause every paper published in the journal to be investigated adversarially. The obvious problem with the idea seems to be that so much of the process is voluntary; people might be unwilling to submit papers to that journal.
      - esja 6y ago
        Interesting idea... reminds me of Chaos Monkey.
      - unishark 6y ago
        The government pays bounties to whistleblowers who expose grant fraud under the False Claims act, along with big fines for the perpetrators. Not sure how much it extends to research fraud itself, but it certainly seems like something they should do. Perhaps even they might extend it to publishing stuff that can't be reproduced.
- kenjackson 6y ago
  Fraud is the norm? This seems like a likely false statement. I’d love to see data supporting this.
raverbashing 6y ago
I agree (though we're probably looking back with rose-tinted glasses and with all the BS that went at that time filtered through time) There's an over-reliance on p-values and publishing results and "peer-review" that can get pedantic, gate-keeping or useless real quickly.

thaumasiotes 6y ago

> There are millions of people whose livelihood depends on publishing, so they will publish anything they'll get away with. This is also the problem with the iOS App Store and the Google Play store. They are modeled off Linux repositories, and those do not cause major problems. The phone app stores are riddled with problems, because you can charge for apps in those stores. If you're willing to pay people to lie to you, they will. You have to make the tradeoff between higher participation with lots of fraud, and lower participation with not that much fraud. > Top tier institutions seem to survive in some sort of bubbles. Not really; compare Brian Wansink at Cornell. ( https://en.wikipedia.org/wiki/Brian_Wansink https://en.wikipedia.org/wiki/Brian_Wansink ) Research output is reliable where it is actively relied on by engineers -- and not elsewhere. At this point, fraud is the norm and research is the exception in academia overall.

unishark 6y ago
> Research output is reliable where it is actively relied on by engineers -- and not elsewhere. I'd word it this way: if the person producing the output is not responsible for it really working, it almost certainly won't. Even innocently this will be the case with anything complex, people get things backwards, miss a scale factor, etc. Finding that last bug can take more work than the rest of the project combined, much easier to publish what appears to work and move on. Much more so when there's direct career benefits to "hacking" the system over competing honestly. Especially considering the internet is awash with people trying to cheat through every other competition (exam questions, interview questions, etc.)
- thaumasiotes 6y ago
  > if the person producing the output is not responsible for it really working, it almost certainly won't. Even innocently this will be the case with anything complex Indeed, there's a great example in the article itself, in a totally unrelated area: > I felt these journals generally did their best, and the slowness of the process likely comes from the bureaucracy of the process and the inexperience editors have with that process. In other words, these reasonable journals weren't able to use their retraction process even though they wanted to, because the process never gets used and therefore isn't in a usable state.
  - thaumasiotes 6y ago
    > these reasonable journals weren't able to use their retraction process even though they wanted to, because the process never gets used and therefore isn't in a usable state. Actually, this made me think about a journal purposefully running intentionally spurious papers, with a challenge to the readership to identify which paper was fake. If the system worked, that would cause every paper published in the journal to be investigated adversarially. The obvious problem with the idea seems to be that so much of the process is voluntary; people might be unwilling to submit papers to that journal.
    - esja 6y ago
      Interesting idea... reminds me of Chaos Monkey.
    - unishark 6y ago
      The government pays bounties to whistleblowers who expose grant fraud under the False Claims act, along with big fines for the perpetrators. Not sure how much it extends to research fraud itself, but it certainly seems like something they should do. Perhaps even they might extend it to publishing stuff that can't be reproduced.
kenjackson 6y ago
Fraud is the norm? This seems like a likely false statement. I’d love to see data supporting this.

raverbashing 6y ago

I agree (though we're probably looking back with rose-tinted glasses and with all the BS that went at that time filtered through time) There's an over-reliance on p-values and publishing results and "peer-review" that can get pedantic, gate-keeping or useless real quickly.