5 ms·

You're absolutely right, but the effect size is pretty damn impressive and I expect this to at least partially replicate.

Etheryte 7y ago
Having a small sample and a large effect doesn't mean you're guaranteed at least some effect on a large sample. If anything, a very large effect is more likely to indicate a flaw in your method.
- solveit 7y ago
  Yes, but effect sizes are relatively harder to hack/screw up (knowingly or otherwise) than p-values. My totally anecdotal impression is that findings showing large effect sizes seem to replicate better than findings showing highly significant effects.
  - haffi112 7y ago
    It could be an example of publication bias, from the wiki article on effect size [1]: An example of this is publication bias, which occurs when scientists report results only when the estimated effect sizes are large or are statistically significant. As a result, if many researchers carry out studies with low statistical power, the reported effect sizes will tend to be larger than the true (population) effects, if any. [1] https://en.wikipedia.org/wiki/Effect_size https://en.wikipedia.org/wiki/Effect_size
  - mumblemumble 7y ago
    Giant effect sizes are easy to hack in biological sciences. All you have to do is conduct an in vitro study. Like they did in this paper.
  - jfindley 7y ago
    Excessively large effect sizes are a warning sign in and of themselves. A good article on this is here: [0] -- simply if the effect is too huge, it becomes incumbent on the researcher to explain why no-one has noticed this before. Methane is a gas with a distinct and noticeable smell. Seaweed is plentiful in many places cows are kept. If the effect was really as dramatic as claimed you'd expect that it'd be well known that letting cows eat seaweed made them less smelly, and the authors of the original study make no attempt to explain why this might be the case. 0: http://daniellakens.blogspot.com/2017/07/impossibly-hungry-judges.html http://daniellakens.blogspot.com/2017/07/impossibly-hungry-j...
    - icxa 7y ago
      > "it becomes incumbent on the researcher to explain why no-one has noticed this before" Farmer: huh, we'll I'll be damned. Who in the hell woulda thought to feed them seaweed."
      - VBprogrammer 7y ago
        If I'm not mistaken the original inspiration for this research was precisely that farmers noticed their cattle (or perhaps it was sheep) were happier when allowed to graze in fields which happened to be along the shore, allowing them to naturally eat quantities of sea weed.
    - random_savv 7y ago
      > Methane is a gas with a distinct and noticeable smell Methane is odorless https://en.wikipedia.org/wiki/Methane https://en.wikipedia.org/wiki/Methane
      - jfindley 7y ago
        Interesting, thank you! I was surprised by this, as it's the major component of natural gas (as used widely for cooking here), and this has a distinct smell. I went looking however, and discovered that's actually because Ethyl Mercaptan is added specifically to make it detectable by humans. I never realised this, and does indeed explain why it's plausible no-one would have noticed such a large effect.
        deleted 7y ago
        [deleted]
        brohee 7y ago
        We add mercaptan (an horrible smelling, innocuous gas) to natural gas (methane), so we get a chance to notice leaks before an explosion.
- LifeLiverTransp 7y ago
  My prediction would be the same for humans who have a large salt intake. Cardiac problems for cows.
- sjwright 7y ago
  Maybe? But an effect which relies on a link between diet and farting isn’t exactly remarkable or even surprising. My dog farts the most noxious gas if he eats apple or chicken.
tmalsburg2 7y ago
In a small sample, the observed effect needs to be large in order to cross the threshold for statistical significance. It could still be a fluke and the true effect zero or even negative.
- joombaga 7y ago
  What's the tendency when a study with a small sample size that show a large observed effect get performed on larger samples? Has it been studied?
  - tmalsburg2 7y ago
    It depends. If the original study was published only because it had a positive result (significance filter, usually true), we expect the effect to be smaller in a larger replication. It's called Type M error, with M for magnitude. And, yes, large scale replications typically show smaller effects or no effects. See this paper for details: https://journals.sagepub.com/doi/full/10.1177/1745691614551642 https://journals.sagepub.com/doi/full/10.1177/17456916145516...

Etheryte 7y ago

Having a small sample and a large effect doesn't mean you're guaranteed at least some effect on a large sample. If anything, a very large effect is more likely to indicate a flaw in your method.

solveit 7y ago
Yes, but effect sizes are relatively harder to hack/screw up (knowingly or otherwise) than p-values. My totally anecdotal impression is that findings showing large effect sizes seem to replicate better than findings showing highly significant effects.
- haffi112 7y ago
  It could be an example of publication bias, from the wiki article on effect size [1]: An example of this is publication bias, which occurs when scientists report results only when the estimated effect sizes are large or are statistically significant. As a result, if many researchers carry out studies with low statistical power, the reported effect sizes will tend to be larger than the true (population) effects, if any. [1] https://en.wikipedia.org/wiki/Effect_size https://en.wikipedia.org/wiki/Effect_size
- mumblemumble 7y ago
  Giant effect sizes are easy to hack in biological sciences. All you have to do is conduct an in vitro study. Like they did in this paper.
- jfindley 7y ago
  Excessively large effect sizes are a warning sign in and of themselves. A good article on this is here: [0] -- simply if the effect is too huge, it becomes incumbent on the researcher to explain why no-one has noticed this before. Methane is a gas with a distinct and noticeable smell. Seaweed is plentiful in many places cows are kept. If the effect was really as dramatic as claimed you'd expect that it'd be well known that letting cows eat seaweed made them less smelly, and the authors of the original study make no attempt to explain why this might be the case. 0: http://daniellakens.blogspot.com/2017/07/impossibly-hungry-judges.html http://daniellakens.blogspot.com/2017/07/impossibly-hungry-j...
  - icxa 7y ago
    > "it becomes incumbent on the researcher to explain why no-one has noticed this before" Farmer: huh, we'll I'll be damned. Who in the hell woulda thought to feed them seaweed."
    - VBprogrammer 7y ago
      If I'm not mistaken the original inspiration for this research was precisely that farmers noticed their cattle (or perhaps it was sheep) were happier when allowed to graze in fields which happened to be along the shore, allowing them to naturally eat quantities of sea weed.
  - random_savv 7y ago
    > Methane is a gas with a distinct and noticeable smell Methane is odorless https://en.wikipedia.org/wiki/Methane https://en.wikipedia.org/wiki/Methane
    - jfindley 7y ago
      Interesting, thank you! I was surprised by this, as it's the major component of natural gas (as used widely for cooking here), and this has a distinct smell. I went looking however, and discovered that's actually because Ethyl Mercaptan is added specifically to make it detectable by humans. I never realised this, and does indeed explain why it's plausible no-one would have noticed such a large effect.
      - deleted 7y ago
        [deleted]
      - brohee 7y ago
        We add mercaptan (an horrible smelling, innocuous gas) to natural gas (methane), so we get a chance to notice leaks before an explosion.
LifeLiverTransp 7y ago
My prediction would be the same for humans who have a large salt intake. Cardiac problems for cows.
sjwright 7y ago
Maybe? But an effect which relies on a link between diet and farting isn’t exactly remarkable or even surprising. My dog farts the most noxious gas if he eats apple or chicken.

tmalsburg2 7y ago

In a small sample, the observed effect needs to be large in order to cross the threshold for statistical significance. It could still be a fluke and the true effect zero or even negative.

joombaga 7y ago
What's the tendency when a study with a small sample size that show a large observed effect get performed on larger samples? Has it been studied?
- tmalsburg2 7y ago
  It depends. If the original study was published only because it had a positive result (significance filter, usually true), we expect the effect to be smaller in a larger replication. It's called Type M error, with M for magnitude. And, yes, large scale replications typically show smaller effects or no effects. See this paper for details: https://journals.sagepub.com/doi/full/10.1177/1745691614551642 https://journals.sagepub.com/doi/full/10.1177/17456916145516...