3 ms·

How could any replication study sample from the same pattern of people, given that there are so many cultures with different ways of living, eating, spending ti

How could any replication study sample from the same pattern of people, given that there are so many cultures with different ways of living, eating, spending time etc. All of these cultures also constantly change. Universal representativity is unattainable, so any replication will have a number of unknown sample variables that changed. This is still not highlighted enough in most papers I read.

dantheman 4y ago
If the result isn't stable across samples how valid is the result?
- samus 4y ago
  If the study otherwise follows best practices, results about differences between well-defined subpopulations, and the reasons behind them, are still worthwhile to investigate and report.
  - leereeves 4y ago
    If the relevant subpopulations are well-defined, then it should be possible to sample from those subpopulations and reproduce the results.
    - Helmut10001 4y ago
      But then, for many studies about (e.g.) Eating, Behavior, Psychology (etc.): Even for well defined populations (lets say, "religious male adults in Texas") there would be so many incidental variables that I would have strong doubts that results are easily reproducible. On the other hand, studies with "well defined" but very narrow subpopulations (e.g. "religious male adults in Texas that work in printing offices and eat granola every morning") are not easily transferrable, which makes it difficult to generalize results (thus, results are less significant).
michaelt 4y ago
Even producing results that reliably replicate across "American psychology college students age 18-21" would be a start.
GeneralMayhem 4y ago
That's the whole point of sampling. If you get a big enough sample, and you have a reasonably fair way of getting it, then all those things wash out as individual variation.
- Helmut10001 4y ago
  I agree. But it is very rare that I see a study that samples people living in more than one country, not to speak of sampling of people from all countries in the world. My point is this: If a study samples only people from the US, it should be highlighted very early (like in the first sentence) that it is only valid(ated) in the US and cannot be applied elsewhere, without testing.

dantheman 4y ago

If the result isn't stable across samples how valid is the result?

samus 4y ago
If the study otherwise follows best practices, results about differences between well-defined subpopulations, and the reasons behind them, are still worthwhile to investigate and report.
- leereeves 4y ago
  If the relevant subpopulations are well-defined, then it should be possible to sample from those subpopulations and reproduce the results.
  - Helmut10001 4y ago
    But then, for many studies about (e.g.) Eating, Behavior, Psychology (etc.): Even for well defined populations (lets say, "religious male adults in Texas") there would be so many incidental variables that I would have strong doubts that results are easily reproducible. On the other hand, studies with "well defined" but very narrow subpopulations (e.g. "religious male adults in Texas that work in printing offices and eat granola every morning") are not easily transferrable, which makes it difficult to generalize results (thus, results are less significant).

michaelt 4y ago

Even producing results that reliably replicate across "American psychology college students age 18-21" would be a start.

GeneralMayhem 4y ago

That's the whole point of sampling. If you get a big enough sample, and you have a reasonably fair way of getting it, then all those things wash out as individual variation.

Helmut10001 4y ago
I agree. But it is very rare that I see a study that samples people living in more than one country, not to speak of sampling of people from all countries in the world. My point is this: If a study samples only people from the US, it should be highlighted very early (like in the first sentence) that it is only valid(ated) in the US and cannot be applied elsewhere, without testing.