4 ms·

There's somewhat of a lack of a tautological understanding of null-hypothesis experiments, in that what is really being tested is confined by the scope of the e

There's somewhat of a lack of a tautological understanding of null-hypothesis experiments, in that what is really being tested is confined by the scope of the experiment. What I mean is that the results of the experiment are only true for the sample population. The extrapolation of the sample population to the whole seems problematic and at the root of the ongoing replication crisis. I realize that this interpretation throws a wrench into the empirical approaches taken by the soft sciences and that rejects an important notion that a sample can indeed be considered representative of the whole. But I can't ignore the impact of the experimenters on these kinds of studies, be it bias or just poorly implemented methods, as well as the impact of the subjects chosen for the experiment. There's a lack of a control that is otherwise accounted for in the hard sciences.

spdif899 3y ago
This same group has a lot of insightful exploration into what they call "N=1" studies. I'm no expert and haven't read but a subset of their work, but my impression is that they understand the limitations quite well and believe there is still value in publishing and trying to understand research with small sample sizes. Here's their intro article on the idea, which is only the tip of a massive iceberg: https://slimemoldtimemold.com/2023/01/05/n1-introduction/ https://slimemoldtimemold.com/2023/01/05/n1-introduction/
- williamcotton 3y ago
  I have no problem with small sample sizes. They form the core of qualitative research. They also don't attempt to speak for broad populations as the details are lost with large sample sizes. I'd prefer if the soft sciences had more qualitative research and left most of the quantitative research to sciences that have a hard control.
gwern 3y ago
Isn't the point of this to show that the subjects look like they all differ in their ability to taste, and any single individual's ability (or even an average) wouldn't generalize very well?
- williamcotton 3y ago
  I'm not sure, my mind keeps wondering the answers to questions that would have been better off in interview form! "What kind of cuisine do you normally eat?" "Do you have high blood pressure and/or sodium restrictions"? "Can you describe what tastes differently between the two salts?" "How much cooking do you do?" "How would you classify your sense of smell?" Like, maybe people who cook more have a more refined sense of taste? Maybe certain culture's cuisines lend themselves to subtle differences in salt flavor? Maybe there are physiological reasons? Psychological reasons? It seems that the focus should be on these subjective qualities. If the hard sciences use quantitative methods that do generalize very well, shouldn't the soft sciences focus more on methods that don't generalize very well, like, case studies, etc? Then the reader can see if any of the insights apply to their own novel situations, further research, etc. Tangential, but I sketched out a song about this a couple of years ago! A G D Lady tasting tea A G D Let's see if she can see A G D All the scientists agree A G D It's a likely guarentee
derbOac 3y ago
True, but if you get enough people to do it repeatedly theoretically you could also model accuracy as a random variable. So then you get an overall population estimate of accuracy as well as a model of the heterogeneity among individuals. Still, yes, with this small of a sample and minimal replication, there's issues. But they're not super serious about this scientifically, it's really for entertainment. Taking this a step further, I think to really test the simple discernment hypothesis, they should have the tasters sort the samples into two groups without labeling. The current approach sort of assumes that not only can they distinguish between the samples, but can also identify which is which. I think those are two different, separable issues. Where your concerns become more important for me personally are in ratings of foods etc.; I often wonder how much small cultures develop around these things and influence the ratings. It's interesting to me how much ratings can differ sometimes, and I suspect it's related to what you're mentioning.

spdif899 3y ago

This same group has a lot of insightful exploration into what they call "N=1" studies. I'm no expert and haven't read but a subset of their work, but my impression is that they understand the limitations quite well and believe there is still value in publishing and trying to understand research with small sample sizes. Here's their intro article on the idea, which is only the tip of a massive iceberg: https://slimemoldtimemold.com/2023/01/05/n1-introduction/ https://slimemoldtimemold.com/2023/01/05/n1-introduction/

williamcotton 3y ago
I have no problem with small sample sizes. They form the core of qualitative research. They also don't attempt to speak for broad populations as the details are lost with large sample sizes. I'd prefer if the soft sciences had more qualitative research and left most of the quantitative research to sciences that have a hard control.

gwern 3y ago

Isn't the point of this to show that the subjects look like they all differ in their ability to taste, and any single individual's ability (or even an average) wouldn't generalize very well?

williamcotton 3y ago
I'm not sure, my mind keeps wondering the answers to questions that would have been better off in interview form! "What kind of cuisine do you normally eat?" "Do you have high blood pressure and/or sodium restrictions"? "Can you describe what tastes differently between the two salts?" "How much cooking do you do?" "How would you classify your sense of smell?" Like, maybe people who cook more have a more refined sense of taste? Maybe certain culture's cuisines lend themselves to subtle differences in salt flavor? Maybe there are physiological reasons? Psychological reasons? It seems that the focus should be on these subjective qualities. If the hard sciences use quantitative methods that do generalize very well, shouldn't the soft sciences focus more on methods that don't generalize very well, like, case studies, etc? Then the reader can see if any of the insights apply to their own novel situations, further research, etc. Tangential, but I sketched out a song about this a couple of years ago! A G D Lady tasting tea A G D Let's see if she can see A G D All the scientists agree A G D It's a likely guarentee

derbOac 3y ago

True, but if you get enough people to do it repeatedly theoretically you could also model accuracy as a random variable. So then you get an overall population estimate of accuracy as well as a model of the heterogeneity among individuals. Still, yes, with this small of a sample and minimal replication, there's issues. But they're not super serious about this scientifically, it's really for entertainment. Taking this a step further, I think to really test the simple discernment hypothesis, they should have the tasters sort the samples into two groups without labeling. The current approach sort of assumes that not only can they distinguish between the samples, but can also identify which is which. I think those are two different, separable issues. Where your concerns become more important for me personally are in ratings of foods etc.; I often wonder how much small cultures develop around these things and influence the ratings. It's interesting to me how much ratings can differ sometimes, and I suspect it's related to what you're mentioning.