5 ms·

There are a huge number of bureaucratic hoops to jump through before you can get an article published. The flow chart is huge and gets more elaborate year by ye

There are a huge number of bureaucratic hoops to jump through before you can get an article published. The flow chart is huge and gets more elaborate year by year. So if the quality is still poor then I think this simply means that we're doing it wrong. Either science isn't what we think it is or our heart isn't in it (or both). My 'method' of choosing what to read is simple: I follow people whose work I know and enjoy. I trust them. Actually I think that's pretty standard. For example, Leonardo gets published just by having left stuff lying around the place.

Vinnl 9y ago
I'm afraid that science indeed isn't what people usually think it is, and that there's a lot that could be improved. Following people whose work you enjoy generally works, but it also leads to potentially valuable work by lesser-known researchers (e.g. Early-career researchers) to go unnoticed.
Retric 9y ago
I think the expectations for novelty may be to high. Look at the new drugs list every year and you see a lot of churn with minor variations common, but less novelty than many assume. https://www.centerwatch.com/drug-information/fda-approved-drugs/year/2017 https://www.centerwatch.com/drug-information/fda-approved-dr... And this is with 100+ billion in public + private money world wide per year. Sure, there is plenty of stuff being discovered, but we can't expect a steady or increasing rate of progress across all areas.
- Vinnl 9y ago
  That's the common explanation for why journals with a high Impact Factor have more retractions. Funders focus on "excellence", which is measured using the Impact Factor, which is higher when its articles are cited more, which happens more often if research is "novel" and spectacular, also known as not replicated and not in line with expectations, also known as more likely to be unreliable.
- aaavl2821 9y ago
  The relationship between public research funding and drug approvals is actually pretty limited. The bottleneck is actually funding for translational research, i.e. The space after public funding but before the $200B+ in big pharma r&d kicks in. This area is called the "valley of death" and is where most science dies Venture funding is what gets drugs through the valley of death. There's a pretty good correlation between the types of drugs VC funds, the drugs big pharma companies buy from VC, and the drugs that get approved. The kinds of drugs that get approved are less correlated with areas of focus for public research spending Or diseases with most societal impact Wrote something on this last week: http://newbio.tech/blog/vc_basics_1.html http://newbio.tech/blog/vc_basics_1.html
  - Retric 9y ago
    Pharma research is not limited to the US. FDA will approve drugs developed in other countries though their will be some hoops involved they are tiny relative to finding something useful. Anyway, as you say: "Randomly picking 24 early-stage drugs to develop is a losing bet." What I think you are missing is VC's can extract money from failed drugs. VC's are playing with other peoples money and they get a larger chunk of upside while limiting their downsides. This means investing in VC's has poor returns on average which is why the system looks the way it does. PS: Trying to fit that decreasing curve when the actual trend line is clearly positive is ridiculous.
    - aaavl2821 9y ago
      Agreed that pharma research is not limited to the US, but neither is VC investing. US VC firms account for the majority of global biopharma VC investment (though china is catching up). As it stands now most FDA approved drugs are developed by US, European or Japanese firms, and in 5-10 years china will be up there as well. The US is the most profitable drug market, so as a quick and dirty analysis I think using FDA approvals, VC funding and big pharma m&a data makes sense. Not perfect, but it's a decent heuristic VCs do make money on management fees, and VC returns in biotech from 2000-2012 were poor, but biopharma VC has done insanely well the last five years. Better returns than software VC, more IPOs and big m&a exits than tech despite only accounting for 20% of venture funding A lot of money is now chasing these high returns, but there are not enough good biopharma entrepreneurs. So there more money but not more good startups. And yes, that chart about r&d spending is not statistically valid :), but it isn't meant to be. But it illustrates a trend that big pharma is cutting back on r&d, especially early stage research which they outsource to startups. This topic is tangential to the topic of the post, but the article that was the source of that chart has good context around pharma r&d trends
kingkongjaffa 9y ago
The difficulty being you write papers to support grants and funding that some nebulous institution, or govt.dept made, to support research they want to happen, not necessarily the right research.
- jonathanstrange 9y ago
  What do you mean by 'right research'? Proper scientific method will give reliable results no matter what motivations the scientists or their institutions have. Do you perhaps instead have 'right' or 'wrong' priorities of science funding in mind? That's a political issue which should ultimately be decided by the voters and the representatives they choose.
  - aaavl2821 9y ago
    Not always. There are many unknowns in research you can't control for. Some assays / models just aren't that reliable even though they are gold standard. Sometimes the equipment and reagents you use produce unexplainable results. For example, a lab I worked with was trying to make a particular type of immune cell less "aggressive". They tried all kinds of genetic modifications, hit it with different cytokines, changed its nutrients, etc, but nothing worked. Then they used a different flask, and the results were amazing. This was a reproducible phenomenon. They had no way of knowing the equipment they used would change the immunological phenotype of their cells, but it did. There are so many confounding factors like this: reagent source, air quality in animal facilities, how you use a pipette, etc
    - jonathanstrange 9y ago
      I'm glad you've found the error, because you were using proper scientific method. ;)
      - aaavl2821 9y ago
        More like dumb luck, but that certainly seems to be part of the scientific method sometimes
deleted 9y ago
[deleted]
dnomad 9y ago
> Either science isn't what we think it is Science isn't what most people think it is. "Reliability" is not the point of science. It never has been. This whole process started with a bunch of old white guys tooling around in dark medieval cottages trying to turn urine into gold. And yet we've still managed to come pretty far. Science is best understood as a very special form of dialogue. It doesn't matter so much what is being said but it's very important how it is said. It is not a problem at all if one group of scientists are putting out papers claiming X and another group fails to reproduce X. That's the whole point. It's exactly how the dialogue is supposed to go. It would be a problem if one group of scientist published a paper claiming X and then somehow forbid others from trying to reproduce X. Or, when the second group failed to reproduce X the first group sought to discredit or villify or suppress the second group. This would be a big problem and would indicate a termination of the scientific process. The real problem is that this view of science as a neverending, specialized dialogue is not understood by most people, even most scientists, unless they've been exposed to the philosophy of science. That science in itself is so poorly understood leads to silly articles about a "reproduciability crisis" (as if there was ever a time when the majority of scientific papers were shown to be reproducible) but it makes scientific claims easily impersonated ie pseudo-science. It's not clear what the solution really is here but worrying about the "reliability" of certain papers doesn't feel like it would really move the needle.
- Vinnl 9y ago
  It's OK when not reproducing something is a way of validating theories and separating valid theories from the invalid ones. It's less so when something cannot be reproduced because not enough information was shared on how it could be reproduced if it would actually have been reproducible otherwise, or if the information was shared in such a way that is it inaccessible to those who would be able to reproduce it.

Vinnl 9y ago

I'm afraid that science indeed isn't what people usually think it is, and that there's a lot that could be improved. Following people whose work you enjoy generally works, but it also leads to potentially valuable work by lesser-known researchers (e.g. Early-career researchers) to go unnoticed.

Retric 9y ago

I think the expectations for novelty may be to high. Look at the new drugs list every year and you see a lot of churn with minor variations common, but less novelty than many assume. https://www.centerwatch.com/drug-information/fda-approved-drugs/year/2017 https://www.centerwatch.com/drug-information/fda-approved-dr... And this is with 100+ billion in public + private money world wide per year. Sure, there is plenty of stuff being discovered, but we can't expect a steady or increasing rate of progress across all areas.

Vinnl 9y ago
That's the common explanation for why journals with a high Impact Factor have more retractions. Funders focus on "excellence", which is measured using the Impact Factor, which is higher when its articles are cited more, which happens more often if research is "novel" and spectacular, also known as not replicated and not in line with expectations, also known as more likely to be unreliable.
aaavl2821 9y ago
The relationship between public research funding and drug approvals is actually pretty limited. The bottleneck is actually funding for translational research, i.e. The space after public funding but before the $200B+ in big pharma r&d kicks in. This area is called the "valley of death" and is where most science dies Venture funding is what gets drugs through the valley of death. There's a pretty good correlation between the types of drugs VC funds, the drugs big pharma companies buy from VC, and the drugs that get approved. The kinds of drugs that get approved are less correlated with areas of focus for public research spending Or diseases with most societal impact Wrote something on this last week: http://newbio.tech/blog/vc_basics_1.html http://newbio.tech/blog/vc_basics_1.html
- Retric 9y ago
  Pharma research is not limited to the US. FDA will approve drugs developed in other countries though their will be some hoops involved they are tiny relative to finding something useful. Anyway, as you say: "Randomly picking 24 early-stage drugs to develop is a losing bet." What I think you are missing is VC's can extract money from failed drugs. VC's are playing with other peoples money and they get a larger chunk of upside while limiting their downsides. This means investing in VC's has poor returns on average which is why the system looks the way it does. PS: Trying to fit that decreasing curve when the actual trend line is clearly positive is ridiculous.
  - aaavl2821 9y ago
    Agreed that pharma research is not limited to the US, but neither is VC investing. US VC firms account for the majority of global biopharma VC investment (though china is catching up). As it stands now most FDA approved drugs are developed by US, European or Japanese firms, and in 5-10 years china will be up there as well. The US is the most profitable drug market, so as a quick and dirty analysis I think using FDA approvals, VC funding and big pharma m&a data makes sense. Not perfect, but it's a decent heuristic VCs do make money on management fees, and VC returns in biotech from 2000-2012 were poor, but biopharma VC has done insanely well the last five years. Better returns than software VC, more IPOs and big m&a exits than tech despite only accounting for 20% of venture funding A lot of money is now chasing these high returns, but there are not enough good biopharma entrepreneurs. So there more money but not more good startups. And yes, that chart about r&d spending is not statistically valid :), but it isn't meant to be. But it illustrates a trend that big pharma is cutting back on r&d, especially early stage research which they outsource to startups. This topic is tangential to the topic of the post, but the article that was the source of that chart has good context around pharma r&d trends

kingkongjaffa 9y ago

The difficulty being you write papers to support grants and funding that some nebulous institution, or govt.dept made, to support research they want to happen, not necessarily the right research.

jonathanstrange 9y ago
What do you mean by 'right research'? Proper scientific method will give reliable results no matter what motivations the scientists or their institutions have. Do you perhaps instead have 'right' or 'wrong' priorities of science funding in mind? That's a political issue which should ultimately be decided by the voters and the representatives they choose.
- aaavl2821 9y ago
  Not always. There are many unknowns in research you can't control for. Some assays / models just aren't that reliable even though they are gold standard. Sometimes the equipment and reagents you use produce unexplainable results. For example, a lab I worked with was trying to make a particular type of immune cell less "aggressive". They tried all kinds of genetic modifications, hit it with different cytokines, changed its nutrients, etc, but nothing worked. Then they used a different flask, and the results were amazing. This was a reproducible phenomenon. They had no way of knowing the equipment they used would change the immunological phenotype of their cells, but it did. There are so many confounding factors like this: reagent source, air quality in animal facilities, how you use a pipette, etc
  - jonathanstrange 9y ago
    I'm glad you've found the error, because you were using proper scientific method. ;)
    - aaavl2821 9y ago
      More like dumb luck, but that certainly seems to be part of the scientific method sometimes

deleted 9y ago

[deleted]

dnomad 9y ago

> Either science isn't what we think it is Science isn't what most people think it is. "Reliability" is not the point of science. It never has been. This whole process started with a bunch of old white guys tooling around in dark medieval cottages trying to turn urine into gold. And yet we've still managed to come pretty far. Science is best understood as a very special form of dialogue. It doesn't matter so much what is being said but it's very important how it is said. It is not a problem at all if one group of scientists are putting out papers claiming X and another group fails to reproduce X. That's the whole point. It's exactly how the dialogue is supposed to go. It would be a problem if one group of scientist published a paper claiming X and then somehow forbid others from trying to reproduce X. Or, when the second group failed to reproduce X the first group sought to discredit or villify or suppress the second group. This would be a big problem and would indicate a termination of the scientific process. The real problem is that this view of science as a neverending, specialized dialogue is not understood by most people, even most scientists, unless they've been exposed to the philosophy of science. That science in itself is so poorly understood leads to silly articles about a "reproduciability crisis" (as if there was ever a time when the majority of scientific papers were shown to be reproducible) but it makes scientific claims easily impersonated ie pseudo-science. It's not clear what the solution really is here but worrying about the "reliability" of certain papers doesn't feel like it would really move the needle.

Vinnl 9y ago
It's OK when not reproducing something is a way of validating theories and separating valid theories from the invalid ones. It's less so when something cannot be reproduced because not enough information was shared on how it could be reproduced if it would actually have been reproducible otherwise, or if the information was shared in such a way that is it inaccessible to those who would be able to reproduce it.