8 ms·

When social sciences try to use the mathematical tools of physical sciences, I wish they would use the same measure of statistical significance: "the chance th

When social sciences try to use the mathematical tools of physical sciences, I wish they would use the same measure of statistical significance: "the chance that the detections were random events was estimated to be less than one part in 100000"

schoen 10y ago
It does make their results seem unconvincing in comparison... but can you supply them with 4×10¹⁹ research subjects to experiment on?
will_pseudonym 10y ago
I think that the larger problem with social science is that they conduct experiments under extremely specific conditions and then attempt to generalize to populations where those same conditions aren't present at all. And I'm not talking about the media's interpretation and broadcast of the findings. I'm talking about the papers themselves.
- Ntrails 10y ago
  It's also pretty hard/expensive to get 10000 people involved in a long term experiment. So they tend to get tens of people and extrapolate from there. I don't think it's that they're lazy or incompetent - it's just hard to have the sort of scale required for the levels of certainty you can get from firing a bazillion calcium ions at something for months.
  - jdavis703 10y ago
    Why can't they do fewer studies but with larger sizes then? Do you then run into a problem of idled researchers?
    - nonbel 10y ago
      There is a sick culture that thinks being the "idea person" is the most important part of research, and that "idea people" should be in charge of implementing their ideas. A lot of issues stem from that.
    - mathattack 10y ago
      This is the modern physics model. The major discoveries and papers have 100s of authors attached to them. The soft sciences tend to have lots more smaller studies and papers. The damage is that there isn't an "Accepted body of knowledge" that everyone knows to be true. Perhaps this isn't even possible.
  - mathattack 10y ago
    You can get "1 in 10000" significance without 10000 people. (How many Tails in a row would you need to flip a coin to believe it's not random at that level? Somewhere between 13 and 20?) Also, many datasets (think financial data or census data) are much bigger.
nonbel 10y ago
Well, the wikipedia source appears broken so I can't be sure but I bet that is a p-value. In that case it is wrong. The chance (probability) of an observation arising from random events is not the same as the chance (probability) random events explain the observation. In the first case (which is usual) you are assuming the random event model is true. http://rationalwiki.org/wiki/Confusion_of_the_inverse http://rationalwiki.org/wiki/Confusion_of_the_inverse
djsumdog 10y ago
You can't sequester humans in a cage for 50 years, controlling everything except the variable you want to study. Maybe if we could create a parallel universe at a specific point, and have one side try one thing and the other side do another, but there are huge ethical concerns from that as well.
- nerfhammer 10y ago
  If you posit that 50% of people like X, you survey 21 people and find that 0 like X, then the odds that you would get that result due to random sampling error assuming the original hypothesis were true is 1 in 2097152
  - danielvf 10y ago
    However that doesn't take into account correlated sampling error. If you randomly sample college social science majors at a single school, you could easily get results like that, even if the rest of the population is actually split on the matter.
    - dismantlethesun 10y ago
      That's true, but that's also not a random sampling so doesn't really detract from what the above poster is saying. Most social science experiments do not initially handle random samples. The famous ones like the Milgram experiments and others conducted by the same researcher do. If I have my facts correct, he used to mail out letters to people randomly chosen from the phonebook to ask them to participate in his study, then travel to their locales to study them. The Milgram experiment itself was redone multiple times, using different population segments---students, random people in the city, all women, all men, etc. As an anecdote of how poorly correlated a social experiment can be to real life---in my university, econ & social science students were required to take at least 10 experiments per semester. Often we would go there, realize the material tested was identical to what we had studied in class then behave like 'rational market actors' in order to get the best result (econ experiments used small sums of real money, so you'd be paid at the end based on performance). This isn't anything like how a random person would have behaved.
vacri 10y ago
When the social sciences do experiments as useless as this one - spending four months to get one atom for one millisecond - they get roasted for wasting money.

schoen 10y ago

It does make their results seem unconvincing in comparison... but can you supply them with 4×10¹⁹ research subjects to experiment on?

will_pseudonym 10y ago

I think that the larger problem with social science is that they conduct experiments under extremely specific conditions and then attempt to generalize to populations where those same conditions aren't present at all. And I'm not talking about the media's interpretation and broadcast of the findings. I'm talking about the papers themselves.

Ntrails 10y ago
It's also pretty hard/expensive to get 10000 people involved in a long term experiment. So they tend to get tens of people and extrapolate from there. I don't think it's that they're lazy or incompetent - it's just hard to have the sort of scale required for the levels of certainty you can get from firing a bazillion calcium ions at something for months.
- jdavis703 10y ago
  Why can't they do fewer studies but with larger sizes then? Do you then run into a problem of idled researchers?
  - nonbel 10y ago
    There is a sick culture that thinks being the "idea person" is the most important part of research, and that "idea people" should be in charge of implementing their ideas. A lot of issues stem from that.
  - mathattack 10y ago
    This is the modern physics model. The major discoveries and papers have 100s of authors attached to them. The soft sciences tend to have lots more smaller studies and papers. The damage is that there isn't an "Accepted body of knowledge" that everyone knows to be true. Perhaps this isn't even possible.
- mathattack 10y ago
  You can get "1 in 10000" significance without 10000 people. (How many Tails in a row would you need to flip a coin to believe it's not random at that level? Somewhere between 13 and 20?) Also, many datasets (think financial data or census data) are much bigger.

nonbel 10y ago

Well, the wikipedia source appears broken so I can't be sure but I bet that is a p-value. In that case it is wrong. The chance (probability) of an observation arising from random events is not the same as the chance (probability) random events explain the observation. In the first case (which is usual) you are assuming the random event model is true. http://rationalwiki.org/wiki/Confusion_of_the_inverse http://rationalwiki.org/wiki/Confusion_of_the_inverse

djsumdog 10y ago

You can't sequester humans in a cage for 50 years, controlling everything except the variable you want to study. Maybe if we could create a parallel universe at a specific point, and have one side try one thing and the other side do another, but there are huge ethical concerns from that as well.

nerfhammer 10y ago
If you posit that 50% of people like X, you survey 21 people and find that 0 like X, then the odds that you would get that result due to random sampling error assuming the original hypothesis were true is 1 in 2097152
- danielvf 10y ago
  However that doesn't take into account correlated sampling error. If you randomly sample college social science majors at a single school, you could easily get results like that, even if the rest of the population is actually split on the matter.
  - dismantlethesun 10y ago
    That's true, but that's also not a random sampling so doesn't really detract from what the above poster is saying. Most social science experiments do not initially handle random samples. The famous ones like the Milgram experiments and others conducted by the same researcher do. If I have my facts correct, he used to mail out letters to people randomly chosen from the phonebook to ask them to participate in his study, then travel to their locales to study them. The Milgram experiment itself was redone multiple times, using different population segments---students, random people in the city, all women, all men, etc. As an anecdote of how poorly correlated a social experiment can be to real life---in my university, econ & social science students were required to take at least 10 experiments per semester. Often we would go there, realize the material tested was identical to what we had studied in class then behave like 'rational market actors' in order to get the best result (econ experiments used small sums of real money, so you'd be paid at the end based on performance). This isn't anything like how a random person would have behaved.

vacri 10y ago

When the social sciences do experiments as useless as this one - spending four months to get one atom for one millisecond - they get roasted for wasting money.