10 ms·

Many Researchers Using Same Data and Hypothesis Reveals Universe of Uncertainty

91 points by Ambolia 4y ago

SamBam 4y ago
> Researchers’ expertise, prior beliefs, and expectations barely predict the wide variation in research outcomes. More than 90% of the total variance in numerical results remains unexplained even after accounting for research decisions [including which statistical tests to perform] identified via qualitative coding of each team’s workflow. They're saying that scientists, given the same data, will achieve widely different results, and that they can't even work out how they're getting these different results. That's surprising and concerning. > This reveals a universe of uncertainty that is otherwise hidden when considering a single study in isolation. The idiosyncratic nature of how researchers’ results and conclusions varied is a new explanation for why many scientific hypotheses remain contested.
- Enginerrrd 4y ago
  I've certainly got into the habit of reading the methods and results of social science papers that make headlines and what I've found has been horrifying. More than half of the time, there are serious, serious flaws involved in the data collection or the analysis. Even worse, I've frequently encountered situations where the underlying data really seems to point toward the opposite conclusion than the one made by the researchers who have really twisted and beaten the data into submission to get the conclusion they wanted.
  - DontchaKnowit 4y ago
    Yep, which is why I find the use of "science" as a political battering ram to be one of the most infuriating modern phenomena.
- jltsiren 4y ago
  It's not surprising at all. There are always many arbitrary choices to make and many things that can go wrong if you try to make conclusions from data. Even worse if you try to automate making the conclusions. And because the data you have has often been derived from lower-level data, many arbitrary choices have already been made and many things have already gone wrong. On good days, I find it a small miracle that we can sometimes make justified conclusions from data. On bad days, it feels that making conclusions from data is storytelling, not science.
  - a1369209993 4y ago
    > On good days, I find it a small miracle that we can sometimes make justified conclusions from data. On bad days, it feels that making conclusions from data is storytelling, not science. If you dig far enough down the rabbithole, you eventually find that the numbers are being pulled out of someone's proverbial ass. But pulling numbers out of your ass and using them to make a conclusion is a lot better than pulling a conclusion out of your ass and using it to make numbers.
mellosouls 4y ago
It calls for greater humility and clarity in reporting scientific findings. Hmmm, I think the variance in results is maybe a bit more of an issue in the softer sciences and the conclusions would be better worded to reflect that considering the context here, rather than implying they hold to a similar extent over science generally. There is no doubt human failings also challenge physics etc (see previous threads here) and humility and clarity is appropriate there also, but the significant problems highlighted in this study seem more particular to areas of research that have long been regarded as less sure due to the nature of their domain.
- macintux 4y ago
  Agreed, this seems like they chose a hypothesis destined to be forever unprovable. > they used the same data to independently test the same prominent social science hypothesis: that greater immigration reduces support for social policies among the public.
  - et1337 4y ago
    Their chosen hypothesis doesn’t seem any more unprovable than the countless other “study says” headlines I see every day, even here on HN.
  - scotty79 4y ago
    The fact that they got different numerical results from the same data and the difference wasn't explained by the choice of statistical tools the applied us very concering. I interpret it to mean that different researchers applied same math to same data and got different results.
- uniqueuid 4y ago
  Personally, I prefer the term science of soft reality. After all, the science is as hard as it can be, it's the object of investigation that's fuzzy and shifting and actively refusing to be measured ;)
  - SuoDuanDao 4y ago
cs702 4y ago
> Seventy-three independent research teams used identical cross-country survey data to test an established social science hypothesis: that more immigration will reduce public support for government provision of social policies. Instead of convergence, teams’ numerical results varied greatly, ranging from large negative to large positive effects of immigration on public support. The choices made by the research teams in designing their statistical tests explain very little of this variation: a hidden universe of uncertainty remains. That's... surprising, but keep in mind that we're talking about "soft" social sciences here. Measuring soft quantities like "public support for government provision of social policies in response to increased immigration" is very different from measuring hard quantities like, say, the location of a rocket in space at a point in time using earth as a frame of reference. I'm looking forward to seeing what Andrew Gelman and his colleagues will say about it at https://statmodeling.stat.columbia.edu/ https://statmodeling.stat.columbia.edu/ -- surely they will want to say something about it!
- photochemsyn 4y ago
  Yes, this is a pretty fuzzy 'science' topic - i.e. measuring public opinion. Looking at something a bit more quantitative would make sense. For example, infectious disease epidemiology is kind of mid-way between hard and soft. Looking at Covid transmission and outcomes relative to vaccination and testing, for example, might fit the bill. There's a good deal of uncertainty in the data (lack of reporting, widespread differences in prevalence of testing, efficiency of vaccines relative to different Covid strains, etc.) but ultimately it revolves around a hard science measurement (was person X infected with virus Y).
- deleted 4y ago
  [deleted]
dmichulke 4y ago
I wonder if this study would arrive at a different conclusion if it were conducted by a different team...
- pfortuny 4y ago
  They address this in the abstract (or the introduction, I forget).
hdjjhhvvhga 4y ago
There is another paper by the same author about a similar phenomenon: "Secondary observer effects: idiosyncratic errors in small-N secondary data analysis" - it's available from Sci-Hub.
btrettel 4y ago
I think this effect has been recognized in science for decades, even if it wasn't talked about much in print. Recently I thought about trying to combat this effect myself by using multiple different analysis methods. For example, in my case I'm thinking about trying multiple different regression approaches. My research is in the physical sciences, so I don't think in this case the overall conclusions will be that different because the general trend is clear from a scatter plot (often not true in social science...), but it'll still be an interesting exercise.
taylodl 4y ago
Remember the saying figures don't lie but liars figure? I'm not saying any of these researchers had any malfeasance, but imagine if they did? You can see how easy it is to use "analysis" to say anything you want. The key takeaway is don't don't just look at the results of the analysis, look at the data and how they analyzed it.
raincom 4y ago
There are many problems with this kind of hypotheses: (a) any hypothesis should predict/explain facts that are different from the original data/facts that the hypothesis accounts for; otherwise, it is ad hoc. (b) multiple competing hypotheses can account for the same set of facts. So, how to pick which hypothesis should one pick from the competing hypotheses? Again, here, Philosophy of sciences come to rescue. Pick a hypothesis that explains/solves more facts/problems. Here, the problems/facts should be novel, not the original ones.
adamsmith143 4y ago
Should be careful to differentiate between "Social Science" and Hard Science. One of these is flooded with questionable papers that cannot be replicated and the other has fields where a p value of 3×10^-7 is required for a discovery to even be taken seriously.
- subroutine 4y ago
  Setting the bar at seven sigma just means natural sciences are more biased towards our mathematical preconceptions about the world than accepting physical evidence. There's nothing inherently wrong with that - the same could be done in social science. It just means the null hypothesis (the accepted preconception) will be sustained more often. And more type-II errors will be made. A six sigma result means there is a one in a million chance of getting the result we observed from our experiment if our maths-based theory is correct. And what you are saying is that such a result is just "meh, our math still checks out".
  - adamsmith143 4y ago
    >Setting the bar at seven sigma just means natural sciences are more biased towards our mathematical preconceptions about the world than accepting physical evidence. You used a lot of words despite not saying anything. > And more type-II errors will be made. A six sigma result means there is a one in a million chance of getting the result we observed from our experiment if our maths-based theory is correct. It means nothing of the sort. It means that if the theoretical explanation is wrong there is only a 1 in a million chance that the pattern seen in the data is due to random chance or noise.
    - subroutine 4y ago
      To clarify, instead of "is correct" I should have said "if our math based theory is being treated as the null". But yes, I completely agree with your statement. As for you not understanding the first bit, that might be a personal limitation, or some sort of aphasia. Setting the bar at 7 sigma means we will only reject the null (our math based theory) if the observed experimental data would have only occurred by chance, under the null, with a probability of less than one in ten million. Hence, setting such a high bar to reject the math based theory means we are relatively biased towards preserving our mathematical preconception about the world rather than accepting (unexpected) physical evidence. If we set the bar at say, three sigma, the bias shifts towards accepting the physical evidence. The point is, the p alpha criterion is not what makes an experiment rigorous.
spullara 4y ago
Most subjects that have the word "science" in the name aren't science: social science, computer science, political science, etc. While the real sciences, don't have it: physics, chemistry, biology, etc.
- whatever1 4y ago
  Is chemistry hard science? We mostly memorize a ton of arbitrary rules about how things should react and behave. We still get surprise molecules with surprise properties. The science part is that we identify them and add them to our long catalog of rules, but is it hard science ? I would only classify math and some physics as hard science.
  - spullara 4y ago
    Math isn't science.
- jltsiren 4y ago
  That's a tired argument. Those subjects are called the way they are, because they were named in a culture that preferred plain straightforward names. Computer science could have been computing, informatics, or datalogy. Political science could have been politology. Sociology is already a thing, but it's narrower in scope than social science. Incidentally, I'm writing this in a place called "Physical Sciences Building".
  - spullara 4y ago
    They are named that way to give them some kind of authority when they aren't actually science (i.e. those things studied using the scientific method). You are being way too charitable.
bluejay2387 4y ago
I think the timing of articles like this, the recent alarms over 'reproducibility' and so on... are a sign that we finally have the tools and volume of data to prove that humans are not good with data.

91 points by Ambolia 4y ago

> Researchers’ expertise, prior beliefs, and expectations barely predict the wide variation in research outcomes. More than 90% of the total variance in numerical results remains unexplained even after accounting for research decisions [including which statistical tests to perform] identified via qualitative coding of each team’s workflow. They're saying that scientists, given the same data, will achieve widely different results, and that they can't even work out how they're getting these different results. That's surprising and concerning. > This reveals a universe of uncertainty that is otherwise hidden when considering a single study in isolation. The idiosyncratic nature of how researchers’ results and conclusions varied is a new explanation for why many scientific hypotheses remain contested.

Enginerrrd 4y ago
I've certainly got into the habit of reading the methods and results of social science papers that make headlines and what I've found has been horrifying. More than half of the time, there are serious, serious flaws involved in the data collection or the analysis. Even worse, I've frequently encountered situations where the underlying data really seems to point toward the opposite conclusion than the one made by the researchers who have really twisted and beaten the data into submission to get the conclusion they wanted.
- DontchaKnowit 4y ago
  Yep, which is why I find the use of "science" as a political battering ram to be one of the most infuriating modern phenomena.
jltsiren 4y ago
It's not surprising at all. There are always many arbitrary choices to make and many things that can go wrong if you try to make conclusions from data. Even worse if you try to automate making the conclusions. And because the data you have has often been derived from lower-level data, many arbitrary choices have already been made and many things have already gone wrong. On good days, I find it a small miracle that we can sometimes make justified conclusions from data. On bad days, it feels that making conclusions from data is storytelling, not science.
- a1369209993 4y ago
  > On good days, I find it a small miracle that we can sometimes make justified conclusions from data. On bad days, it feels that making conclusions from data is storytelling, not science. If you dig far enough down the rabbithole, you eventually find that the numbers are being pulled out of someone's proverbial ass. But pulling numbers out of your ass and using them to make a conclusion is a lot better than pulling a conclusion out of your ass and using it to make numbers.

mellosouls 4y ago

It calls for greater humility and clarity in reporting scientific findings. Hmmm, I think the variance in results is maybe a bit more of an issue in the softer sciences and the conclusions would be better worded to reflect that considering the context here, rather than implying they hold to a similar extent over science generally. There is no doubt human failings also challenge physics etc (see previous threads here) and humility and clarity is appropriate there also, but the significant problems highlighted in this study seem more particular to areas of research that have long been regarded as less sure due to the nature of their domain.

macintux 4y ago
Agreed, this seems like they chose a hypothesis destined to be forever unprovable. > they used the same data to independently test the same prominent social science hypothesis: that greater immigration reduces support for social policies among the public.
- et1337 4y ago
  Their chosen hypothesis doesn’t seem any more unprovable than the countless other “study says” headlines I see every day, even here on HN.
- scotty79 4y ago
  The fact that they got different numerical results from the same data and the difference wasn't explained by the choice of statistical tools the applied us very concering. I interpret it to mean that different researchers applied same math to same data and got different results.
uniqueuid 4y ago
Personally, I prefer the term science of soft reality. After all, the science is as hard as it can be, it's the object of investigation that's fuzzy and shifting and actively refusing to be measured ;)
- SuoDuanDao 4y ago

cs702 4y ago

> Seventy-three independent research teams used identical cross-country survey data to test an established social science hypothesis: that more immigration will reduce public support for government provision of social policies. Instead of convergence, teams’ numerical results varied greatly, ranging from large negative to large positive effects of immigration on public support. The choices made by the research teams in designing their statistical tests explain very little of this variation: a hidden universe of uncertainty remains. That's... surprising, but keep in mind that we're talking about "soft" social sciences here. Measuring soft quantities like "public support for government provision of social policies in response to increased immigration" is very different from measuring hard quantities like, say, the location of a rocket in space at a point in time using earth as a frame of reference. I'm looking forward to seeing what Andrew Gelman and his colleagues will say about it at https://statmodeling.stat.columbia.edu/ https://statmodeling.stat.columbia.edu/ -- surely they will want to say something about it!

photochemsyn 4y ago
Yes, this is a pretty fuzzy 'science' topic - i.e. measuring public opinion. Looking at something a bit more quantitative would make sense. For example, infectious disease epidemiology is kind of mid-way between hard and soft. Looking at Covid transmission and outcomes relative to vaccination and testing, for example, might fit the bill. There's a good deal of uncertainty in the data (lack of reporting, widespread differences in prevalence of testing, efficiency of vaccines relative to different Covid strains, etc.) but ultimately it revolves around a hard science measurement (was person X infected with virus Y).
deleted 4y ago
[deleted]

dmichulke 4y ago

I wonder if this study would arrive at a different conclusion if it were conducted by a different team...

pfortuny 4y ago
They address this in the abstract (or the introduction, I forget).

hdjjhhvvhga 4y ago

There is another paper by the same author about a similar phenomenon: "Secondary observer effects: idiosyncratic errors in small-N secondary data analysis" - it's available from Sci-Hub.

btrettel 4y ago

I think this effect has been recognized in science for decades, even if it wasn't talked about much in print. Recently I thought about trying to combat this effect myself by using multiple different analysis methods. For example, in my case I'm thinking about trying multiple different regression approaches. My research is in the physical sciences, so I don't think in this case the overall conclusions will be that different because the general trend is clear from a scatter plot (often not true in social science...), but it'll still be an interesting exercise.

taylodl 4y ago

Remember the saying figures don't lie but liars figure? I'm not saying any of these researchers had any malfeasance, but imagine if they did? You can see how easy it is to use "analysis" to say anything you want. The key takeaway is don't don't just look at the results of the analysis, look at the data and how they analyzed it.

raincom 4y ago

There are many problems with this kind of hypotheses: (a) any hypothesis should predict/explain facts that are different from the original data/facts that the hypothesis accounts for; otherwise, it is ad hoc. (b) multiple competing hypotheses can account for the same set of facts. So, how to pick which hypothesis should one pick from the competing hypotheses? Again, here, Philosophy of sciences come to rescue. Pick a hypothesis that explains/solves more facts/problems. Here, the problems/facts should be novel, not the original ones.

adamsmith143 4y ago

Should be careful to differentiate between "Social Science" and Hard Science. One of these is flooded with questionable papers that cannot be replicated and the other has fields where a p value of 3×10^-7 is required for a discovery to even be taken seriously.

subroutine 4y ago
Setting the bar at seven sigma just means natural sciences are more biased towards our mathematical preconceptions about the world than accepting physical evidence. There's nothing inherently wrong with that - the same could be done in social science. It just means the null hypothesis (the accepted preconception) will be sustained more often. And more type-II errors will be made. A six sigma result means there is a one in a million chance of getting the result we observed from our experiment if our maths-based theory is correct. And what you are saying is that such a result is just "meh, our math still checks out".
- adamsmith143 4y ago
  >Setting the bar at seven sigma just means natural sciences are more biased towards our mathematical preconceptions about the world than accepting physical evidence. You used a lot of words despite not saying anything. > And more type-II errors will be made. A six sigma result means there is a one in a million chance of getting the result we observed from our experiment if our maths-based theory is correct. It means nothing of the sort. It means that if the theoretical explanation is wrong there is only a 1 in a million chance that the pattern seen in the data is due to random chance or noise.
  - subroutine 4y ago
    To clarify, instead of "is correct" I should have said "if our math based theory is being treated as the null". But yes, I completely agree with your statement. As for you not understanding the first bit, that might be a personal limitation, or some sort of aphasia. Setting the bar at 7 sigma means we will only reject the null (our math based theory) if the observed experimental data would have only occurred by chance, under the null, with a probability of less than one in ten million. Hence, setting such a high bar to reject the math based theory means we are relatively biased towards preserving our mathematical preconception about the world rather than accepting (unexpected) physical evidence. If we set the bar at say, three sigma, the bias shifts towards accepting the physical evidence. The point is, the p alpha criterion is not what makes an experiment rigorous.

spullara 4y ago

Most subjects that have the word "science" in the name aren't science: social science, computer science, political science, etc. While the real sciences, don't have it: physics, chemistry, biology, etc.

whatever1 4y ago
Is chemistry hard science? We mostly memorize a ton of arbitrary rules about how things should react and behave. We still get surprise molecules with surprise properties. The science part is that we identify them and add them to our long catalog of rules, but is it hard science ? I would only classify math and some physics as hard science.
- spullara 4y ago
  Math isn't science.
jltsiren 4y ago
That's a tired argument. Those subjects are called the way they are, because they were named in a culture that preferred plain straightforward names. Computer science could have been computing, informatics, or datalogy. Political science could have been politology. Sociology is already a thing, but it's narrower in scope than social science. Incidentally, I'm writing this in a place called "Physical Sciences Building".
- spullara 4y ago
  They are named that way to give them some kind of authority when they aren't actually science (i.e. those things studied using the scientific method). You are being way too charitable.

bluejay2387 4y ago

I think the timing of articles like this, the recent alarms over 'reproducibility' and so on... are a sign that we finally have the tools and volume of data to prove that humans are not good with data.