5 ms·

I don't think there is such a thing as abuse of statistics as long as the methodology is published. In this case, N=21 and no control group shows that there is

I don't think there is such a thing as abuse of statistics as long as the methodology is published. In this case, N=21 and no control group shows that there isn't much useful data present in the first place...

beerandt 7y ago
Isn't publishing anything with N<30 that presumes a random distribution straight-up wrong? (I don't know if that's presumed here or not, but without greater detail, it's something most people [who have a basic understanding] will assume.)
- renewiltord 7y ago
  You have 29 people in a room. The people are healthy and well. You give them a pill containing a chemical you want to test. They all die seconds after consuming the pill. The autopsy reveals they did not choke. What is the N you choose before you conclude that chemical is toxic in that dose at some p?
  - beerandt 7y ago
    That's not what I said- you can still make claims against the data. But when people are looking at N=x, most don't realize that N<30 is usually in a completely different category than N>=30. You almost shouldn't look at it as "N". Novice "statisticians" are just (very) limited in the tools/formulas/methods available to use, because so many (entry level/basic/standard) statistical methods are only valid for normal distributions (or other constraints on data, that people typically don't check for). Without details stated up-front, most people will (wrongly) assume that a normal distribution automatically applies, even if they don't know what that means. Similar to situations where people ask for the average/mean when they should be asking for the median. The data isn't useless, but needs strong context and disclaimers.
    - IAmEveryone 7y ago
      I have no idea what you’re getting at here. The parent pointed out, accurately, that for large effect sizes, N=30 is plenty. I’m also fairly certain you don’t understand what a “random distribution” is yourself. That’s because there’s no such thing. There’s a normal distribution, binomial distribution, Poisson distribution etc, and they all involve randomness. But there’s no “random distribution”. So maybe you shouldn’t be quite so quick to judge “novice statisticians”, and what those imaginary people you mention a total of five times may or may not “wrongly assume”, or “don’t know”, or “are limited in”, or “don’t check for”.
      - markroseman 7y ago
        On the other hand, there are so many opportunities for experimental design limiting the applicability of results in mental health experiments, that sample size and distribution doesn't even begin to touch on it. Bottom line: getting a meaningful result that is widely applicable is a horrendously complex undertaking.
      - beerandt 7y ago
        You're right, I meant normal, not random. (Fixed) And I didn't mean to imply that I'm any better than a novice. I just know enough to know to always check that assumptions for any given test needs to be valid. (It's been a few years since I've had to use any of this in a professional capacity.) But is 30 plenty? Or is it sufficient? As I remember, normal distribution usually requires a sample size of N=>30. (As in 30 is the minimum.)
    - renewiltord 7y ago
      Indulge me with the scenario. Do you think that pill is safe or unsafe or do you think we need to give one more guy the pill to test? If you think it depends on something, go ahead and say so and I'll constrain the thought experiment. IAmEveryone (user w/ sibling post) is correct in that I'm referring to the fact that effect size affects statistical power.
- afpx 7y ago
  A lot of times they will do small experiments to gain a sense for the data and see if it's worth doing larger experiments. Not reliable results but can still infer things.
  - beerandt 7y ago
    Yeah, I didn't mean to imply that the knowledge isn't or couldn't be useful. In hindsight, it was too early in the day for me to try and speak statistics.
- ramblenode 7y ago
  (I did not downvote) I have also observed people suggest a statistically special N, but this is a misconception. There is no special N. There is only a relationship between increasing N and a monotonic increase in the confidence in your result. Most scientific standards specify a "confidence" of at least 95%, but if you are willing to accept more uncertainty--perhaps because there is a large upside to a positive result--then you can make due with a smaller sample.

beerandt 7y ago

Isn't publishing anything with N<30 that presumes a random distribution straight-up wrong? (I don't know if that's presumed here or not, but without greater detail, it's something most people [who have a basic understanding] will assume.)

renewiltord 7y ago
You have 29 people in a room. The people are healthy and well. You give them a pill containing a chemical you want to test. They all die seconds after consuming the pill. The autopsy reveals they did not choke. What is the N you choose before you conclude that chemical is toxic in that dose at some p?
- beerandt 7y ago
  That's not what I said- you can still make claims against the data. But when people are looking at N=x, most don't realize that N<30 is usually in a completely different category than N>=30. You almost shouldn't look at it as "N". Novice "statisticians" are just (very) limited in the tools/formulas/methods available to use, because so many (entry level/basic/standard) statistical methods are only valid for normal distributions (or other constraints on data, that people typically don't check for). Without details stated up-front, most people will (wrongly) assume that a normal distribution automatically applies, even if they don't know what that means. Similar to situations where people ask for the average/mean when they should be asking for the median. The data isn't useless, but needs strong context and disclaimers.
  - IAmEveryone 7y ago
    I have no idea what you’re getting at here. The parent pointed out, accurately, that for large effect sizes, N=30 is plenty. I’m also fairly certain you don’t understand what a “random distribution” is yourself. That’s because there’s no such thing. There’s a normal distribution, binomial distribution, Poisson distribution etc, and they all involve randomness. But there’s no “random distribution”. So maybe you shouldn’t be quite so quick to judge “novice statisticians”, and what those imaginary people you mention a total of five times may or may not “wrongly assume”, or “don’t know”, or “are limited in”, or “don’t check for”.
    - markroseman 7y ago
      On the other hand, there are so many opportunities for experimental design limiting the applicability of results in mental health experiments, that sample size and distribution doesn't even begin to touch on it. Bottom line: getting a meaningful result that is widely applicable is a horrendously complex undertaking.
    - beerandt 7y ago
      You're right, I meant normal, not random. (Fixed) And I didn't mean to imply that I'm any better than a novice. I just know enough to know to always check that assumptions for any given test needs to be valid. (It's been a few years since I've had to use any of this in a professional capacity.) But is 30 plenty? Or is it sufficient? As I remember, normal distribution usually requires a sample size of N=>30. (As in 30 is the minimum.)
  - renewiltord 7y ago
    Indulge me with the scenario. Do you think that pill is safe or unsafe or do you think we need to give one more guy the pill to test? If you think it depends on something, go ahead and say so and I'll constrain the thought experiment. IAmEveryone (user w/ sibling post) is correct in that I'm referring to the fact that effect size affects statistical power.
afpx 7y ago
A lot of times they will do small experiments to gain a sense for the data and see if it's worth doing larger experiments. Not reliable results but can still infer things.
- beerandt 7y ago
  Yeah, I didn't mean to imply that the knowledge isn't or couldn't be useful. In hindsight, it was too early in the day for me to try and speak statistics.
ramblenode 7y ago
(I did not downvote) I have also observed people suggest a statistically special N, but this is a misconception. There is no special N. There is only a relationship between increasing N and a monotonic increase in the confidence in your result. Most scientific standards specify a "confidence" of at least 95%, but if you are willing to accept more uncertainty--perhaps because there is a large upside to a positive result--then you can make due with a smaller sample.