3 ms·

I take issue with grouping non-reproducible (in principle), non-prediction-generating (just "rejects" a null hypothesis without specifying a precise research hy

by nonbel 11y ago

I take issue with grouping non-reproducible (in principle), non-prediction-generating (just "rejects" a null hypothesis without specifying a precise research hypothesis) type of research as science. If anyone has missed Feynman's discussion of cargo cult science, check that out because it describes what is going on pretty well: http://calteches.library.caltech.edu/51/2/CargoCult.htm http://calteches.library.caltech.edu/51/2/CargoCult.htm It is really something very different from the method that gave us the benefits we see all around today. For example, the cancer reproducibility project had to drop 1/4 of the studies before even trying because it was so difficult to get the required protocols and materials: https://news.ycombinator.com/item?id=10687879 https://news.ycombinator.com/item?id=10687879 "Research", ok. But "Science"? I would prefer reserving that word for something better. Something better than a group of people publishing reports lacking 2 weeks worth of full time work worth of necessary methodological information.

stdbrouw 11y ago
Feynman is great, and I'd add Meehl's paradox: http://www.fisme.science.uu.nl/staff/christianb/downloads/meehl1967.pdf http://www.fisme.science.uu.nl/staff/christianb/downloads/me... A lot of the research that relies on null hypothesis significance testing really is lousy. But it is not just a matter of sloppiness, it's also a function of the area of study. For example, there is a lot of evidence that people who had a bad childhood and little economic opportunity are more likely to commit crime. Is the study of that phenomenon not science just because a sociologist is unable to mechanistically predict that if your mother didn't read you bed-time stories, you are guaranteed to rob a neighborhood store, at the age of 23.2 years old? Is research into cancer drugs not science because we can't predict beforehand if the drug will help a little or a lot, even though we know of the causal mechanism that should make the drug effective?
- nonbel 11y ago
  >"Is research into cancer drugs not science because we can't predict beforehand if the drug will help a little or a lot, even though we know of the causal mechanism that should make the drug effective?" It is about the methods used rather than the level of understanding. Science is about people performing independent replications on each others measurements to ensure they are reliable and identify important confounds, then coming up with models that explain these reliable observations. The issue with cancer research right now appears to be it is common to not even publish enough info to do the replications, let alone actually perform them. Once we had some data to "hang our hat on", then it would be time to try to explain this quantitatively. If you can't predict whether a drug will help a little or a lot, I am extremely doubtful the causal mechanism is known.
- closed 11y ago
  To be sure, the type of research that would look at the connection between being read bed-time stories and robbing a store is very likely to use hierarchical linear modeling (HLM), which does produce predictions. For example, in his HLM book Andy Gelman looks at the relationship between radon in homes and lung cancer. Replace radon with being read stories, and lung cancer with robbing a neighborhood store, and you're in business. The key is that he interprets (with, say, confidence intervals) the model rather than emphasize null hypothesis significance testing.

by nonbel 11y ago

stdbrouw 11y ago

Feynman is great, and I'd add Meehl's paradox: http://www.fisme.science.uu.nl/staff/christianb/downloads/meehl1967.pdf http://www.fisme.science.uu.nl/staff/christianb/downloads/me... A lot of the research that relies on null hypothesis significance testing really is lousy. But it is not just a matter of sloppiness, it's also a function of the area of study. For example, there is a lot of evidence that people who had a bad childhood and little economic opportunity are more likely to commit crime. Is the study of that phenomenon not science just because a sociologist is unable to mechanistically predict that if your mother didn't read you bed-time stories, you are guaranteed to rob a neighborhood store, at the age of 23.2 years old? Is research into cancer drugs not science because we can't predict beforehand if the drug will help a little or a lot, even though we know of the causal mechanism that should make the drug effective?

nonbel 11y ago
>"Is research into cancer drugs not science because we can't predict beforehand if the drug will help a little or a lot, even though we know of the causal mechanism that should make the drug effective?" It is about the methods used rather than the level of understanding. Science is about people performing independent replications on each others measurements to ensure they are reliable and identify important confounds, then coming up with models that explain these reliable observations. The issue with cancer research right now appears to be it is common to not even publish enough info to do the replications, let alone actually perform them. Once we had some data to "hang our hat on", then it would be time to try to explain this quantitatively. If you can't predict whether a drug will help a little or a lot, I am extremely doubtful the causal mechanism is known.
closed 11y ago
To be sure, the type of research that would look at the connection between being read bed-time stories and robbing a store is very likely to use hierarchical linear modeling (HLM), which does produce predictions. For example, in his HLM book Andy Gelman looks at the relationship between radon in homes and lung cancer. Replace radon with being read stories, and lung cancer with robbing a neighborhood store, and you're in business. The key is that he interprets (with, say, confidence intervals) the model rather than emphasize null hypothesis significance testing.