5 ms·

> is it common to work with so small sample sizes in psychology? Yes. Even much smaller sample sizes are common. I've seen "results" published in supposedly re

by p-e-w 3y ago

> is it common to work with so small sample sizes in psychology? Yes. Even much smaller sample sizes are common. I've seen "results" published in supposedly reputable journals with sample sizes as small as 9. Not 9 other studies that were meta-analyzed, mind you. 9 individual people. This is just one of several reasons why most of psychology is junk science, if not outright fraudulent. Statistical methods is usually a required course for students in such disciplines, so I do not buy the "they don't know better" excuse. Not that you need to know statistics in order to understand that you can't conclude anything from a sample size of 9.

hackernewds 3y ago
Psychology isn't accepted as a hard science, perhaps an applied science. Since there really isn't a precise follow to the scientific method, hypothesis - inference - conclusion.
- FrustratedMonky 3y ago
  So Astronomy is not a hard science? Because there is only observation, no testing of hypothesis?
  - naillo 3y ago
    They can test it in the limited sense of simulation. Not ideal but it's more than even psychologists can do
    - FrustratedMonky 3y ago
      That is so completely false that I fear you are responding purely by a bias against psychology and have not read anything about it. Of course you can form a hypothesis, and test it on real live humans, gather data and check results.
- willcipriano 3y ago
  I don't think a lot of people know this. The evidence required for a physiological intervention is lower than a surgical or pharmaceutical intervention. The AMA used to lump psychology with things like chiropractic, acupuncture and homeopathy until they were sued by chiropractors under antitrust law in 1987.
  - boppo1 3y ago
    I wish there was a succinct way to communicate this. Lots of people I know revere psychologists' word as end-all be all, beyond doubt. To question it makes them think I'm questioning physics. There are good ideas in the field, to be sure. But on the other hand...
    - willcipriano 3y ago
      For me the scariest part is its the only speciality that we allow to imprison people (involuntary holds) and they can keep people out of prison (not fit to stand trial). They also often influence public policy. That's a lot of power for people who can't do a double blind study most of the time.
      - p-e-w 3y ago
        It's even worse than that: Some jurisdictions (including California IIRC) allow involuntary psychiatric holds based on the assessment of people (like law enforcement officers) who have no psychological qualifications whatsoever. This is without a doubt one of the worst and most normalized human rights violations happening in the world today, and the scale and impact of it is hard to imagine.
        throwaway426079 3y ago
        I have seen police arrest people under section 136 of the UK's Mental Health Act in circumstances where no expertise is needed to see that the person is a danger to themselves or others. The police then take that person to a place of safety, usually a hospital.
FrustratedMonky 3y ago
That is not the reason it is junk. Typically the reason to call something junk is that designing the study itself is difficult, not sample size. There are analysis of small to large groups to determine how small a group can be that will still reflect the whole. 9 does seem on low end, but I think the typically accepted size is something in the 20's of individuals. I'm guessing that if 9 is enough to call it junk, then 20 would not sway your opinion. What is a good number then? 100? 1000?. Guess could do identical study on groups of 10,100,1000, and show that the results replicate across the size. Of course, this has been done, hence why they get away with smaller sizes.
- suslik 3y ago
  This isn't a guessing game, really. You have some prior knowledge of the expected effect size - that allows you to calculate the sample size required for your statistical test to have a certain power. That is an essential and very simple part of a good experimental design which, unfortunately, is very often ignored.
- p-e-w 3y ago
  There is no fixed size of samples that guarantees a result is statistically significant. How many samples are needed depends on the desired p-value among other things, which in turn depends on prior assumptions, and on the quantities being measured. "Typically accepted sizes" are nothing but cargo cult science. That's precisely why psychologists take statistics classes as part of their education. I get that those aren't everyone's cup of tea, but people who don't even understand the basics have no business conducting and publishing studies. In fact, well-designed scientific studies typically start with a discussion of p-values, sample sizes, possible sources of biases, etc. If you don't see any mention of those in a study, you know you're looking at junk.
FrustratedMonky 3y ago
Psychology gets a bad reputation because it is still new. Remember, Astrology turned into Astronomy, Alchemy became Chemistry, and Philosophy (old) became Math and Physics. Psychology is still a newcomer, and study of the mind and the human animal is difficult because it is complex and hard to measure. That doesn’t mean you can’t start somewhere.
- DarkNova6 3y ago
  Psychology gets a bad rep because a lot of bad science is conducted in the field. It’s rate of reproducibility is among the lowest of all sciences. These are structural and methodical issues which are not at all related to it’s age. In many cases it is simply bias, negligence or pure fraud.
  - FrustratedMonky 3y ago
    You are to quick to blame bad actors. I don't think that the typical student or professor has so much evil intention for negligence and fraud. These are examples of a 'new' science. Humans are complex and hard to measure, so yes, there are problems with study design, and re-producibility. Because they are working it out. That is because of the field of study, not evil intent, some dark cabal of researchers putting out false data on purpose. If there is some obvious way to solve these problems, then those people should do it. I was going to throw out some sarcastic example from another field. But guess, when I think about it, maybe it does happen. String Theory in Physics is an example of "group think" leading an entire field down the wrong rabbit hole of stuff that is faulty and can't be measured. So guess it can happen anywhere.
    - sandspar 3y ago
      For what it's worth, during my psychology courses in university, I witnessed professors lie to the class, sometimes in response to direct questions. The topics were things that psychology professors would surely know, so I don't think they were simply misinformed.
- p-e-w 3y ago
ellisv 3y ago
I've seen similar sample sizes in medical journals, but I wouldn't necessarily call those junk. From my personal experience, the problem is that people who study psychology are not interested or skilled in statistics. Unskilled professors teaching uninterested students perpetuates the cycle. The field is slowly improving but still has a long way to go.
- p-e-w 3y ago
  > I've seen similar sample sizes in medical journals, but I wouldn't necessarily call those junk. Unless those small sample sizes are justified by rigorous statistical modeling, showing that the result is still significant even with the small sample size, such studies absolutely are junk. The burden is on the researcher to demonstrate that their experimental design is suitable for obtaining meaningful results. Failing to do so is the definition of junk science. And needless to say, lack of statistical knowledge is not an excuse for not doing this. If a medical scientist is unable to perform an accurate statistical analysis of their data, it means that they cannot publish – not that they can skip that part because it's too difficult for them. The entire study is meaningless without it.
  - weberson 3y ago
    Analysis of statistical power is a super basic thing and is a normal part of research in medicine, psychology, social sciences ... etc.
  - lanstin 3y ago
    Sample size is just one factor. For example, if you are interested in heights and you measure six people and they are all over 10 feet tall, you can go ahead and conclude they are from a group with different height distribution than the rest of humans. Pretending that these papers of “wow, we found a substance that when given to ten people in a cancer group with a 99% six month lethality and three survived more than two years” contribute nothing is a weird belief. Penicillin was what n = 1.
- 2devnull 3y ago
  Psychology attracts people who are interested in working with people than working with things. If you include the undergrads, psychology has a lower IQ (on average!) than say mathematics or engineering, or other soft sciences like economics or data science. But you shouldn’t judge a group by its average. Also, it’s misogynistic in this case, obviously.

by p-e-w 3y ago

hackernewds 3y ago

Psychology isn't accepted as a hard science, perhaps an applied science. Since there really isn't a precise follow to the scientific method, hypothesis - inference - conclusion.

FrustratedMonky 3y ago
So Astronomy is not a hard science? Because there is only observation, no testing of hypothesis?
- naillo 3y ago
  They can test it in the limited sense of simulation. Not ideal but it's more than even psychologists can do
  - FrustratedMonky 3y ago
    That is so completely false that I fear you are responding purely by a bias against psychology and have not read anything about it. Of course you can form a hypothesis, and test it on real live humans, gather data and check results.
willcipriano 3y ago
I don't think a lot of people know this. The evidence required for a physiological intervention is lower than a surgical or pharmaceutical intervention. The AMA used to lump psychology with things like chiropractic, acupuncture and homeopathy until they were sued by chiropractors under antitrust law in 1987.
- boppo1 3y ago
  I wish there was a succinct way to communicate this. Lots of people I know revere psychologists' word as end-all be all, beyond doubt. To question it makes them think I'm questioning physics. There are good ideas in the field, to be sure. But on the other hand...
  - willcipriano 3y ago
    For me the scariest part is its the only speciality that we allow to imprison people (involuntary holds) and they can keep people out of prison (not fit to stand trial). They also often influence public policy. That's a lot of power for people who can't do a double blind study most of the time.
    - p-e-w 3y ago
      It's even worse than that: Some jurisdictions (including California IIRC) allow involuntary psychiatric holds based on the assessment of people (like law enforcement officers) who have no psychological qualifications whatsoever. This is without a doubt one of the worst and most normalized human rights violations happening in the world today, and the scale and impact of it is hard to imagine.
      - throwaway426079 3y ago
        I have seen police arrest people under section 136 of the UK's Mental Health Act in circumstances where no expertise is needed to see that the person is a danger to themselves or others. The police then take that person to a place of safety, usually a hospital.

FrustratedMonky 3y ago

That is not the reason it is junk. Typically the reason to call something junk is that designing the study itself is difficult, not sample size. There are analysis of small to large groups to determine how small a group can be that will still reflect the whole. 9 does seem on low end, but I think the typically accepted size is something in the 20's of individuals. I'm guessing that if 9 is enough to call it junk, then 20 would not sway your opinion. What is a good number then? 100? 1000?. Guess could do identical study on groups of 10,100,1000, and show that the results replicate across the size. Of course, this has been done, hence why they get away with smaller sizes.

suslik 3y ago
This isn't a guessing game, really. You have some prior knowledge of the expected effect size - that allows you to calculate the sample size required for your statistical test to have a certain power. That is an essential and very simple part of a good experimental design which, unfortunately, is very often ignored.
p-e-w 3y ago
There is no fixed size of samples that guarantees a result is statistically significant. How many samples are needed depends on the desired p-value among other things, which in turn depends on prior assumptions, and on the quantities being measured. "Typically accepted sizes" are nothing but cargo cult science. That's precisely why psychologists take statistics classes as part of their education. I get that those aren't everyone's cup of tea, but people who don't even understand the basics have no business conducting and publishing studies. In fact, well-designed scientific studies typically start with a discussion of p-values, sample sizes, possible sources of biases, etc. If you don't see any mention of those in a study, you know you're looking at junk.

FrustratedMonky 3y ago

Psychology gets a bad reputation because it is still new. Remember, Astrology turned into Astronomy, Alchemy became Chemistry, and Philosophy (old) became Math and Physics. Psychology is still a newcomer, and study of the mind and the human animal is difficult because it is complex and hard to measure. That doesn’t mean you can’t start somewhere.

DarkNova6 3y ago
Psychology gets a bad rep because a lot of bad science is conducted in the field. It’s rate of reproducibility is among the lowest of all sciences. These are structural and methodical issues which are not at all related to it’s age. In many cases it is simply bias, negligence or pure fraud.
- FrustratedMonky 3y ago
  You are to quick to blame bad actors. I don't think that the typical student or professor has so much evil intention for negligence and fraud. These are examples of a 'new' science. Humans are complex and hard to measure, so yes, there are problems with study design, and re-producibility. Because they are working it out. That is because of the field of study, not evil intent, some dark cabal of researchers putting out false data on purpose. If there is some obvious way to solve these problems, then those people should do it. I was going to throw out some sarcastic example from another field. But guess, when I think about it, maybe it does happen. String Theory in Physics is an example of "group think" leading an entire field down the wrong rabbit hole of stuff that is faulty and can't be measured. So guess it can happen anywhere.
  - sandspar 3y ago
    For what it's worth, during my psychology courses in university, I witnessed professors lie to the class, sometimes in response to direct questions. The topics were things that psychology professors would surely know, so I don't think they were simply misinformed.
p-e-w 3y ago

ellisv 3y ago

I've seen similar sample sizes in medical journals, but I wouldn't necessarily call those junk. From my personal experience, the problem is that people who study psychology are not interested or skilled in statistics. Unskilled professors teaching uninterested students perpetuates the cycle. The field is slowly improving but still has a long way to go.

p-e-w 3y ago
> I've seen similar sample sizes in medical journals, but I wouldn't necessarily call those junk. Unless those small sample sizes are justified by rigorous statistical modeling, showing that the result is still significant even with the small sample size, such studies absolutely are junk. The burden is on the researcher to demonstrate that their experimental design is suitable for obtaining meaningful results. Failing to do so is the definition of junk science. And needless to say, lack of statistical knowledge is not an excuse for not doing this. If a medical scientist is unable to perform an accurate statistical analysis of their data, it means that they cannot publish – not that they can skip that part because it's too difficult for them. The entire study is meaningless without it.
- weberson 3y ago
  Analysis of statistical power is a super basic thing and is a normal part of research in medicine, psychology, social sciences ... etc.
- lanstin 3y ago
  Sample size is just one factor. For example, if you are interested in heights and you measure six people and they are all over 10 feet tall, you can go ahead and conclude they are from a group with different height distribution than the rest of humans. Pretending that these papers of “wow, we found a substance that when given to ten people in a cancer group with a 99% six month lethality and three survived more than two years” contribute nothing is a weird belief. Penicillin was what n = 1.
2devnull 3y ago
Psychology attracts people who are interested in working with people than working with things. If you include the undergrads, psychology has a lower IQ (on average!) than say mathematics or engineering, or other soft sciences like economics or data science. But you shouldn’t judge a group by its average. Also, it’s misogynistic in this case, obviously.