8 ms·

Your solution is historically inaccurate. Statistical tests of significance were created as a tool to improve the objectivity of evaluating results. Garbage in,

Your solution is historically inaccurate. Statistical tests of significance were created as a tool to improve the objectivity of evaluating results. Garbage in, garbage out still applies. Nor is it relevant. Statistical techniques are a symptom, not an underlying cause. Indeed, everything in the article or in discussions of this that I see is always treated symptoms far from the causes. Let's do a quick five whys: Why are publications not reproducible? Because positive results are being published as fast as possible with little checking on them. Why are they publishing so? Because the people involved need publications to get out of their current transient position and secure another transient position, with hopes of someday being able to settle down somewhere. Why do they need to secure their next transient position? Because they have been inculcated that this is the career to be in by their milieu and that is the only way to stay in it. Why is that the only way to stay in it? Because there are no permanent positions that aren't at the very end of this gauntlet. Why are there no permanent positions except at the end of the gauntlet? Because this is a pyramid scheme built on an apprenticeship system. Don't get distracted by all the other stuff. It's the fundamental structure of academic science that leads to the rest of this.

nonbel 9y ago
So let's go to the beginning: >"Because positive results are being published as fast as possible with little checking on them." Why do you think this is so? It is because they think the stats are telling them the probability their theory is true, etc. A significant p-value means the result is "real", etc.
- IanCal 9y ago
  Replicating a study is not something that is likely to get you published, and putting in twice the work to run an experiment twice is unlikely to give you twice the 'return' on your time investment (assuming the funder even supports spending twice as much).
  - dgfgfdagasdfgfa 9y ago
    Why, though? Assumably the replication would be the interesting part, and presumably both teams work would be equally valuable showing the same result. Innovation is overrated; maybe we should just pay scientists more to do inglorious but immensely valuable work. I was certainly repulsed from lab sciences because I had no desire to do the work that's in demand—pulling results out of your ass to get more funding.
    - IanCal 9y ago
      They are likely to not get cited as much, which would lower a journals impact factor, reducing the incentive to the journal. This makes it worse for the scientist as there's work without a high "impact" paper, and since they get largely to choose the work they do if it's less "fun" it might be done less. The funder also has to choose between funding new work or replicating previous work. Something I'd be very interested in is paying independent labs to replicate studies. This removes the incentive issues surrounding the journals and researchers, leaving the funding issue.
      - nonbel 9y ago
        Why are they less likely to get cited as much? Does your typical researcher have some misunderstanding about how reliable the original results were?
        Fomite 9y ago
        No one gets particularly large amounts of glory for confirmatory research because if it's successful than the first paper is still "The first paper to describe X".
        nonbel 9y ago
        I don't understand what studies you are referring to. Each additional paper should be improving whatever parameter estimates are being made, it makes no sense to only look at one result and ignore all the others.
ethanwillis 9y ago
This is exactly why I am aiming to bootstrap my own lab outside of academia and traditional research positions available private industry. There's a lot of "BS" in these power structures that keep people from being able to dedicate resources and a proper amount of time to generate quality research. I'm fairly "green" in my research career, but I've been in the game for a few years now in the Biology/Bioinformatics realm. I've seen and heard stories from plenty of colleagues about how when they first got started and well into their first P.I. positions (and even until this day when they're 10-20 years into their work) where they get abused by their institutions. Aside from "publish or perish" there's also your institution skimming 50-60% of your hard won grant funding to let you "rent" their facilities where you're fighting over shelf space with other P.I.s Absolutely insane. Seeing this I decided I can bootstrap my own lab for much less and have true autonomy. For example a rough breakdown of my costs for setting up a bio/bioinformatics lab for myself and a few friends: Land Zoned for Light Industrial and Scientific Research? about $1000/acre or less in my area. (hint: buy tax liened property) CO2 incubators? Don't pay $4k-8k new. I can buy refurbed or liquidated lab equipment from the unsustainable institution labs for literally pennies on the dollar. Try $100-200 for this equipment. High RPM centrifuges? Once again instead of $2k-5k I can spend $300 for a 15k RPM centrifuge. and something like $20 for desktop centrifuges. PCR machines? Try < $100 each. And I've even gotten "extras" with them in that price range. Compute power/I.T.? I can get 256 core, 1tb ram blade centers for about $1k. Modern Core i5/Core i7 desktops for $80 each. By my estimates I can buy the land, build a building(admittedly a very utilitarian one), and equip a new lab for sub $30k. Now my only costs are re-agents(if I'm doing research that requires it), power, and cheap land taxes.
- mattb314 9y ago
  Could you link/explain how you get those hardware prices, specifically the servers and desktops? Are the available for the average consumer (ie not making a bulk purchase)? I have very little experience buying hardware, but those prices sound 4-8x lower than what I'd expect (and I'd be super happy to find that I'm wrong).
  - ethanwillis 9y ago
SubiculumCode 9y ago
Very astute. More permanent positions are needed, and grant agencies need to limit number of grants single professor can head. In fact NIH just did so to 3, I think. Good move.

nonbel 9y ago

So let's go to the beginning: >"Because positive results are being published as fast as possible with little checking on them." Why do you think this is so? It is because they think the stats are telling them the probability their theory is true, etc. A significant p-value means the result is "real", etc.

IanCal 9y ago
Replicating a study is not something that is likely to get you published, and putting in twice the work to run an experiment twice is unlikely to give you twice the 'return' on your time investment (assuming the funder even supports spending twice as much).
- dgfgfdagasdfgfa 9y ago
  Why, though? Assumably the replication would be the interesting part, and presumably both teams work would be equally valuable showing the same result. Innovation is overrated; maybe we should just pay scientists more to do inglorious but immensely valuable work. I was certainly repulsed from lab sciences because I had no desire to do the work that's in demand—pulling results out of your ass to get more funding.
  - IanCal 9y ago
    They are likely to not get cited as much, which would lower a journals impact factor, reducing the incentive to the journal. This makes it worse for the scientist as there's work without a high "impact" paper, and since they get largely to choose the work they do if it's less "fun" it might be done less. The funder also has to choose between funding new work or replicating previous work. Something I'd be very interested in is paying independent labs to replicate studies. This removes the incentive issues surrounding the journals and researchers, leaving the funding issue.
    - nonbel 9y ago
      Why are they less likely to get cited as much? Does your typical researcher have some misunderstanding about how reliable the original results were?
      - Fomite 9y ago
        No one gets particularly large amounts of glory for confirmatory research because if it's successful than the first paper is still "The first paper to describe X".
        nonbel 9y ago
        I don't understand what studies you are referring to. Each additional paper should be improving whatever parameter estimates are being made, it makes no sense to only look at one result and ignore all the others.

ethanwillis 9y ago

This is exactly why I am aiming to bootstrap my own lab outside of academia and traditional research positions available private industry. There's a lot of "BS" in these power structures that keep people from being able to dedicate resources and a proper amount of time to generate quality research. I'm fairly "green" in my research career, but I've been in the game for a few years now in the Biology/Bioinformatics realm. I've seen and heard stories from plenty of colleagues about how when they first got started and well into their first P.I. positions (and even until this day when they're 10-20 years into their work) where they get abused by their institutions. Aside from "publish or perish" there's also your institution skimming 50-60% of your hard won grant funding to let you "rent" their facilities where you're fighting over shelf space with other P.I.s Absolutely insane. Seeing this I decided I can bootstrap my own lab for much less and have true autonomy. For example a rough breakdown of my costs for setting up a bio/bioinformatics lab for myself and a few friends: Land Zoned for Light Industrial and Scientific Research? about $1000/acre or less in my area. (hint: buy tax liened property) CO2 incubators? Don't pay $4k-8k new. I can buy refurbed or liquidated lab equipment from the unsustainable institution labs for literally pennies on the dollar. Try $100-200 for this equipment. High RPM centrifuges? Once again instead of $2k-5k I can spend $300 for a 15k RPM centrifuge. and something like $20 for desktop centrifuges. PCR machines? Try < $100 each. And I've even gotten "extras" with them in that price range. Compute power/I.T.? I can get 256 core, 1tb ram blade centers for about $1k. Modern Core i5/Core i7 desktops for $80 each. By my estimates I can buy the land, build a building(admittedly a very utilitarian one), and equip a new lab for sub $30k. Now my only costs are re-agents(if I'm doing research that requires it), power, and cheap land taxes.

mattb314 9y ago
Could you link/explain how you get those hardware prices, specifically the servers and desktops? Are the available for the average consumer (ie not making a bulk purchase)? I have very little experience buying hardware, but those prices sound 4-8x lower than what I'd expect (and I'd be super happy to find that I'm wrong).
- ethanwillis 9y ago

SubiculumCode 9y ago

Very astute. More permanent positions are needed, and grant agencies need to limit number of grants single professor can head. In fact NIH just did so to 3, I think. Good move.