4 ms·

Science is not broken, but the scientific process is severely misrepresented. I can only really speak about physics, but at least there and I believe also in ot

by yk 11y ago

Science is not broken, but the scientific process is severely misrepresented. I can only really speak about physics, but at least there and I believe also in other fields, people do experiments, observations etc. and then write papers on whatever they did. The papers are intended for a highly specialized audience and contain a lot of detail besides some claim and a p value and there are usually quite a few closely related papers. Actually I sat in more than one conference session where every single talk had the title "The X-ray spectra of NGC XXXX." Then someone will write a review article on the field. And that person is highly specialized, has probably read hundreds of papers in the field over the last decade(s) and has hands on experience. That guy knows that if there is no mention of X in the methodology section, then the authors did not deal with X, which may or may not be justifiable. After that someone writes a textbook, that guy will read several review papers, and he has probably enough experience to know that a sentence like "compare the contradictory results of Y" is a indication that he should look into the actual papers and perhaps use very cautionary language or omit the topic entirely. At that point, when it appears in textbooks, a result is settled science, not when the first indications appear in a paper. On the other hand, science writes want a headline "Something baffles scientists." Well, the scientists are probably baffled because it contradicts their intuition, which is a strong indicator that something is not really working. This something may be a honest mistake by the authors, a poorly understood experimental effect, outright fraud or it may be something worth reporting. The entire process of science writing is geared towards the papers which are most likely to be wrong.

WalterSear 11y ago
IMHO, the process is misrepresented >and< the process is broken. I think the article does a good job of explaining many (but not all) of the reasons I think it is broken.
Retric 11y ago
I agree with what your saying. However, I feel publish or perish has infected many fields resulting in a lot of crap instead of more focused high quality work. Few people can risk spending 10 years on something and have nothing to show for it. It's a negative feedback loop where quality is hard for people outside the field to judge. So, it's just race to see who can lower quality more while still being published.
- cossatot 11y ago
  > Few people can risk spending 10 years on something and have nothing to show for it. This is a common sentiment, and fortunately for scientists I don't think it's a very accurate description of the real problems we face. I really don't think that there are many good reasons to spend 10 years on one thing when there is a substantial risk of failure. Scientists spend 10 years on problems all the time! But it's either part time, or has measurable metrics for progress along the way that can be published. I mean, you try to go to the moon before Pluto, right? Science (as an institution) is really a lot more like Tetris than people think, except some of the blocks (publications) get moved or destroyed later. But if the blocks are smaller, they're much easier to fit together and build upon, and fault tolerant and so forth, so if they blow up it's less of a big deal. Smaller, more well-defined and robust studies are generally better. To change analogies to paths instead of blocks, which is a better description of the decision making a scientist goes through (rather than building the edifice of human knowledge), it's a lot more like finding your way through a new city than choosing two paths in a snowstorm that will not encounter anything for 10 years' walk. At every fork, you have to decide where to go and there will be some reason you have made that choice. After some smaller period, you can if you choose report where you've gotten and why. Other people care! They will be happy to read about it, as long as your logic is pretty good, especially if you make nice observations along the way. You can publish this, and probably no one will steal it from you. Maybe someone will... but if you do it right, do it incrementally and let people know what you're up to, you'll at least have your batshit ideas published as unreviewed abstracts and folks will know who made the first progress.
  - Retric 11y ago
    Let's say you want to study long term impacts of taking Viagra which is clearly going to take some time. Sure, you could look at old data, but that always has issues. So, you get funding, start some research, and want to publish something when your done. Option A: Pick just one thing to look at say cardiovascular health. Option B: Collect a lot of data, and then look for trends. Option B has much stronger risks of false positives simply because your looking at more factors, but those false positives seem much more interesting which helps you get published. Worse you can't use any of that data for verification, so you now need to run another 10 year study. Upside, you got published, downside, your results are almost meaningless. Bigger upside, you then get to publish again in 10 years. Well what about option C use approach B but publish sooner. Now your not only risking false positives, but also extrapolating from more limited data. Hmm, guess what say nutritionists are going to pick... "Chocolate good/bad for you!"

by yk 11y ago

WalterSear 11y ago

IMHO, the process is misrepresented >and< the process is broken. I think the article does a good job of explaining many (but not all) of the reasons I think it is broken.

Retric 11y ago

I agree with what your saying. However, I feel publish or perish has infected many fields resulting in a lot of crap instead of more focused high quality work. Few people can risk spending 10 years on something and have nothing to show for it. It's a negative feedback loop where quality is hard for people outside the field to judge. So, it's just race to see who can lower quality more while still being published.

cossatot 11y ago
> Few people can risk spending 10 years on something and have nothing to show for it. This is a common sentiment, and fortunately for scientists I don't think it's a very accurate description of the real problems we face. I really don't think that there are many good reasons to spend 10 years on one thing when there is a substantial risk of failure. Scientists spend 10 years on problems all the time! But it's either part time, or has measurable metrics for progress along the way that can be published. I mean, you try to go to the moon before Pluto, right? Science (as an institution) is really a lot more like Tetris than people think, except some of the blocks (publications) get moved or destroyed later. But if the blocks are smaller, they're much easier to fit together and build upon, and fault tolerant and so forth, so if they blow up it's less of a big deal. Smaller, more well-defined and robust studies are generally better. To change analogies to paths instead of blocks, which is a better description of the decision making a scientist goes through (rather than building the edifice of human knowledge), it's a lot more like finding your way through a new city than choosing two paths in a snowstorm that will not encounter anything for 10 years' walk. At every fork, you have to decide where to go and there will be some reason you have made that choice. After some smaller period, you can if you choose report where you've gotten and why. Other people care! They will be happy to read about it, as long as your logic is pretty good, especially if you make nice observations along the way. You can publish this, and probably no one will steal it from you. Maybe someone will... but if you do it right, do it incrementally and let people know what you're up to, you'll at least have your batshit ideas published as unreviewed abstracts and folks will know who made the first progress.
- Retric 11y ago
  Let's say you want to study long term impacts of taking Viagra which is clearly going to take some time. Sure, you could look at old data, but that always has issues. So, you get funding, start some research, and want to publish something when your done. Option A: Pick just one thing to look at say cardiovascular health. Option B: Collect a lot of data, and then look for trends. Option B has much stronger risks of false positives simply because your looking at more factors, but those false positives seem much more interesting which helps you get published. Worse you can't use any of that data for verification, so you now need to run another 10 year study. Upside, you got published, downside, your results are almost meaningless. Bigger upside, you then get to publish again in 10 years. Well what about option C use approach B but publish sooner. Now your not only risking false positives, but also extrapolating from more limited data. Hmm, guess what say nutritionists are going to pick... "Chocolate good/bad for you!"