6 ms·

Your criticism of the sample size is unfounded. You absolutely can have meaningful results with only 258 people provided you have good sampling. If you want to

by p_j_w 5y ago

Your criticism of the sample size is unfounded. You absolutely can have meaningful results with only 258 people provided you have good sampling. If you want to criticize their sampling, then by all means do it, but that's not what you've done here.

enraged_camel 5y ago
Sorry, but what part of "they only surveyed software developers in the United Kingdom" gives you the impression that their sampling might actually be good? Worth noting that the study itself doesn't seem to mention their sampling methodology.
- antris 5y ago
  Studies are never meant to be an ultimate answer to everything. And just because they're not the ultimate answer, doesn't mean that they are therefore worthless, bad or "bullshit".
  - danenania 5y ago
    It actually does mean that in most cases. Having false confidence in a hypothesis based on non-rigorous studies is much worse (and in some cases, a lot more dangerous) than withholding judgment until the data (all the data, not some cherry-picked slice of it) really is conclusive.
  - enraged_camel 5y ago
    No. You perform studies to validate observations or theories and, if you are able to validate them, offer potential factors as contributing causes. A study where the data does not support the conclusion is worthless, and can in fact be harmful.
  - myfavoritedog 5y ago
    I know this refers to actual scientific studies, but the same garbage-in, garbage-out problem applies. If anything, meaningless surveys like this are even less reliable. https://www.bbc.com/news/science-environment-39054778 https://www.bbc.com/news/science-environment-39054778
dragontamer 5y ago
A lot of the COVID19 stuff only has a population size (NOT EVEN sample size) of like, 15. Ex: J&J's blood clot issue. People make conclusions off of tinier sample sizes. There are plenty of scientific studies with just 50 or 100 people, but you need to keep the small numbers in mind. As long as sampling was done correctly, then small sample size won't matter much !!
- bumby 5y ago
  >There are plenty of scientific studies with just 50 or 100 people, but you need to keep the small numbers in mind. The converse also needs to be held in mind as well. Very large studies are more likely to display statistically significant findings. Your p-value is really just a test of your sample size, so as the study grows your p-value tends to shrink. Get a large enough sample and you'll almost always find statistical significance. Just referencing a p-value isn't enough, it needs to be read in the context of the study size (whether large or small). Edit: For those downvoting, here is a much better and eloquent explanation: https://twitter.com/daniela_witten/status/1312180959609974784 https://twitter.com/daniela_witten/status/131218095960997478...
  - loonster 5y ago
    The size of the sample depends on how rare of an event that is being studied. For something that happens 80% of the time, 300 is a very reasonable number. Something that happens 1/1000 or 999/1000 would need a much larger sample size.
    - bumby 5y ago
      Yes, I agree. But that’s to the same point. The p-value is still a measure of whether your sample size is large enough to capture a small probability event in the context of assessing the null hypothesis
  - spywaregorilla 5y ago
    This is a bad explanation of p values. Of the start: > If you’re testing whether mean=0 and actually the truth is that mean=0.000000001, and if you have a large enough sample size, then YOU WILL GET A TINY P-VALUE. Think about this for a second. Does it make sense that the results of my statistical test would change if I measured in terragrams vs. micrograms? No, it doesn't, because it's not the numerical difference between null hypothesis and sample mean, it's the difference relative to standard deviation. If the effect size is 0.000000001 and the standard deviation is 0.0000000000000001 then this is an enormous, important effect. If the standard deviation is 0.1, then this is a really small and unimportant effect. It can still be observed correctly with a sufficient sample size though because you've stated it's a real effect to begin with. You will absolutely not get a tiny p value just because your sample size is large. That is wrong. You will be able to detect very small effects, but if there is no difference, you are more likely to draw the correct conclusion with 1,000,000 samples than with 100 samples. The point the author here is trying to convey but is not getting clearly, is that a large sample size can detect an effect that is practically useless and call it significant. p-values tell you if it seems probable, given your data, if there is a difference between the two datasets. They make no claims about whether the difference is material. Example python code to do prove p values are not likelier to be smaller just due to large sample sizes, given no actual difference: import random from scipy.stats import ttest_ind num_tests = 10000 res = [] for test in range(num_tests): n = 100 a = [random.random() for _ in range(n)] b = [random.random() for _ in range(n)] res = res + [ttest_ind(a,b)[1] <= 0.05] print(f"10,000 tests with n=200 {sum(res)/ num_tests}") num_tests = 1000 res = [] for test in range(num_tests): n = 100000 a = [random.random() for _ in range(n)] b = [random.random() for _ in range(n)] res = res + [ttest_ind(a,b)[1] <= 0.05] print(f"10,000 tests with n=20,000 {sum(res)/ num_tests}")
abc_lisper 5y ago
Yes. 130 people is sufficient for a good sample.
- mateo411 5y ago
  If we use a 95% confidence interval and we have a sample size of 258 then the margin of error is around 4.5%. If the sample size is 130, then the margin of error will be around 6.5%. That's a little high, but probably still good enough.
  - danenania 5y ago
    The sample size you need also depends on variance and the presence of confounding variables. It’s impossible to say whether n is “big enough” in the abstract; it depends completely on what you are measuring and how you measure it.
    - mateo411 5y ago
      We are sampling a Bernoulli distribution for measuring burn out. A software developer could be burnt out/ not burnt out. The variance of a Bernoulli distribution is P(1-P), which is (.83)(.17), since 83% of the respondents answered that they were burnt out.
      - danenania 5y ago
        But why are they "burned out"? Is it because they're a software developer or because they live in an advanced post-industrial civilization in the midst of a global pandemic? Could they feel burned out today but not tomorrow? Has the sun come out in the UK at any time in the last two weeks? Do all the participants even agree on the definition of "burnout"? Were there five other similar studies done that we don't know about that were discarded because they didn't produce the desired click-baity conclusion? I could go on and on.
        mateo411 5y ago
walshemj 5y ago
Normal election poling uses 1k sample size as a minimum and 3k is preferred.
- deleted 5y ago
  [deleted]

by p_j_w 5y ago

enraged_camel 5y ago

Sorry, but what part of "they only surveyed software developers in the United Kingdom" gives you the impression that their sampling might actually be good? Worth noting that the study itself doesn't seem to mention their sampling methodology.

antris 5y ago
Studies are never meant to be an ultimate answer to everything. And just because they're not the ultimate answer, doesn't mean that they are therefore worthless, bad or "bullshit".
- danenania 5y ago
  It actually does mean that in most cases. Having false confidence in a hypothesis based on non-rigorous studies is much worse (and in some cases, a lot more dangerous) than withholding judgment until the data (all the data, not some cherry-picked slice of it) really is conclusive.
- enraged_camel 5y ago
  No. You perform studies to validate observations or theories and, if you are able to validate them, offer potential factors as contributing causes. A study where the data does not support the conclusion is worthless, and can in fact be harmful.
- myfavoritedog 5y ago
  I know this refers to actual scientific studies, but the same garbage-in, garbage-out problem applies. If anything, meaningless surveys like this are even less reliable. https://www.bbc.com/news/science-environment-39054778 https://www.bbc.com/news/science-environment-39054778

dragontamer 5y ago

A lot of the COVID19 stuff only has a population size (NOT EVEN sample size) of like, 15. Ex: J&J's blood clot issue. People make conclusions off of tinier sample sizes. There are plenty of scientific studies with just 50 or 100 people, but you need to keep the small numbers in mind. As long as sampling was done correctly, then small sample size won't matter much !!

bumby 5y ago
>There are plenty of scientific studies with just 50 or 100 people, but you need to keep the small numbers in mind. The converse also needs to be held in mind as well. Very large studies are more likely to display statistically significant findings. Your p-value is really just a test of your sample size, so as the study grows your p-value tends to shrink. Get a large enough sample and you'll almost always find statistical significance. Just referencing a p-value isn't enough, it needs to be read in the context of the study size (whether large or small). Edit: For those downvoting, here is a much better and eloquent explanation: https://twitter.com/daniela_witten/status/1312180959609974784 https://twitter.com/daniela_witten/status/131218095960997478...
- loonster 5y ago
  The size of the sample depends on how rare of an event that is being studied. For something that happens 80% of the time, 300 is a very reasonable number. Something that happens 1/1000 or 999/1000 would need a much larger sample size.
  - bumby 5y ago
    Yes, I agree. But that’s to the same point. The p-value is still a measure of whether your sample size is large enough to capture a small probability event in the context of assessing the null hypothesis
- spywaregorilla 5y ago
  This is a bad explanation of p values. Of the start: > If you’re testing whether mean=0 and actually the truth is that mean=0.000000001, and if you have a large enough sample size, then YOU WILL GET A TINY P-VALUE. Think about this for a second. Does it make sense that the results of my statistical test would change if I measured in terragrams vs. micrograms? No, it doesn't, because it's not the numerical difference between null hypothesis and sample mean, it's the difference relative to standard deviation. If the effect size is 0.000000001 and the standard deviation is 0.0000000000000001 then this is an enormous, important effect. If the standard deviation is 0.1, then this is a really small and unimportant effect. It can still be observed correctly with a sufficient sample size though because you've stated it's a real effect to begin with. You will absolutely not get a tiny p value just because your sample size is large. That is wrong. You will be able to detect very small effects, but if there is no difference, you are more likely to draw the correct conclusion with 1,000,000 samples than with 100 samples. The point the author here is trying to convey but is not getting clearly, is that a large sample size can detect an effect that is practically useless and call it significant. p-values tell you if it seems probable, given your data, if there is a difference between the two datasets. They make no claims about whether the difference is material. Example python code to do prove p values are not likelier to be smaller just due to large sample sizes, given no actual difference: import random from scipy.stats import ttest_ind num_tests = 10000 res = [] for test in range(num_tests): n = 100 a = [random.random() for _ in range(n)] b = [random.random() for _ in range(n)] res = res + [ttest_ind(a,b)[1] <= 0.05] print(f"10,000 tests with n=200 {sum(res)/ num_tests}") num_tests = 1000 res = [] for test in range(num_tests): n = 100000 a = [random.random() for _ in range(n)] b = [random.random() for _ in range(n)] res = res + [ttest_ind(a,b)[1] <= 0.05] print(f"10,000 tests with n=20,000 {sum(res)/ num_tests}")

abc_lisper 5y ago

Yes. 130 people is sufficient for a good sample.

mateo411 5y ago
If we use a 95% confidence interval and we have a sample size of 258 then the margin of error is around 4.5%. If the sample size is 130, then the margin of error will be around 6.5%. That's a little high, but probably still good enough.
- danenania 5y ago
  The sample size you need also depends on variance and the presence of confounding variables. It’s impossible to say whether n is “big enough” in the abstract; it depends completely on what you are measuring and how you measure it.
  - mateo411 5y ago
    We are sampling a Bernoulli distribution for measuring burn out. A software developer could be burnt out/ not burnt out. The variance of a Bernoulli distribution is P(1-P), which is (.83)(.17), since 83% of the respondents answered that they were burnt out.
    - danenania 5y ago
      But why are they "burned out"? Is it because they're a software developer or because they live in an advanced post-industrial civilization in the midst of a global pandemic? Could they feel burned out today but not tomorrow? Has the sun come out in the UK at any time in the last two weeks? Do all the participants even agree on the definition of "burnout"? Were there five other similar studies done that we don't know about that were discarded because they didn't produce the desired click-baity conclusion? I could go on and on.
      - mateo411 5y ago

walshemj 5y ago

Normal election poling uses 1k sample size as a minimum and 3k is preferred.

deleted 5y ago
[deleted]