3 ms·

Even with only a bit over 1,000 total responses, yes you can make such statements. More isn’t necessarily magically better when it comes to sample sizes. In bot

Even with only a bit over 1,000 total responses, yes you can make such statements. More isn’t necessarily magically better when it comes to sample sizes. In both cases, we are making inferences about what the total population thinks about the languages based on a subset of survey responses. Now, is there reason to think that there is bias in these survey results? That is certainly a possibility. To a degree I agree with you in that the framing of “most loved” could be approached better with niche vs widespread languages. There are a lot of people who self select into small languages, vs languages in widespread use, so perhaps noting the relative usage would be helpful in contextualizing what people think.

nonameiguess 4y ago
I believe what the parent is getting at isn't that the sample size for Julia is too small to conclude anything, it's that the differences in sample size between Python and Julia reflect differences in sizes of the total user base. That means Python is more likely to be used by people who have no other choice, whereas more Julia users are using it because they like it. If Julia grew to be as popular as Python, it might no longer be loved by as large a proportion of its users.
- sgt101 4y ago
  well - also self selection. People who respond Julia have (almost certainly) not been made to use it. Python... definitely not so much. Disclaimer: I don't really like Python, but I don't dread it. Disclaimer 2: I quite like Julia but every time I have tried to drop Python to use it I have had to go back to Python.
  - nequo 4y ago
    > I quite like Julia but every time I have tried to drop Python to use it I have had to go back to Python. Is that because of library availability or something to do with the interpreter?
nemetroid 4y ago
The issue is not with sample size but population size.
time_to_smile 4y ago
The issue isn't sample size (nor many of the other things people are pointing out). First off we're ultimately talking about ordinal data here and even the Evan Miller post in another comment misses that ordinal data is fundamentally tricky. Everyone is talking about the p("most loved") but that makes assumptions that "most loved" has a consistent meaning. The problem with ordinal data is that the only thing that's known about it is an ordering of the values, but the distance between values is undefined. That is the difference between a 5 star and a 4 star review is not necessarily that same a between a 2 and 1 star review. This means that you can't meaningfully average these. However there is an even bigger problem, which is what I think parent is feeling here, and that is selection bias. It's very hard to compare the average rating of a film such as Star Wars and a film like Cannibal Holocaust. The first is a general audience film and the later is an extremely niche subset of gore horror. If you show Cannibal Holocaust to the general population it will receive wildly lower rating than Star Wars. However if you were to do a survey of all people who have a DVD of Star Wars and a DVD of Cannibal Holocaust, I wouldn't be at all surprised if Cannibal Holocaust was more loved. There are plenty of people that have Star Wars on their shelf and feel 'meh' about it, but almost no one who owns Cannibal Holocaust on DVD that doesn't feel like it is an essential horror classic. The annoying truth is, there's really no statistical way to solve this. There are some solutions to better modeling ordinal data (you could for example, transform this problem into an ordinal regression problem). But ultimately "loved" is not really a measurable thing, so we're always looking for imperfect proxies.
- ducharmdev 4y ago
  Thanks for the thoughtful reply - I don't have a stats background, but it didn't quite sit right with me. The movie example you give here really puts it in perspective.

nonameiguess 4y ago

I believe what the parent is getting at isn't that the sample size for Julia is too small to conclude anything, it's that the differences in sample size between Python and Julia reflect differences in sizes of the total user base. That means Python is more likely to be used by people who have no other choice, whereas more Julia users are using it because they like it. If Julia grew to be as popular as Python, it might no longer be loved by as large a proportion of its users.

sgt101 4y ago
well - also self selection. People who respond Julia have (almost certainly) not been made to use it. Python... definitely not so much. Disclaimer: I don't really like Python, but I don't dread it. Disclaimer 2: I quite like Julia but every time I have tried to drop Python to use it I have had to go back to Python.
- nequo 4y ago
  > I quite like Julia but every time I have tried to drop Python to use it I have had to go back to Python. Is that because of library availability or something to do with the interpreter?

nemetroid 4y ago

The issue is not with sample size but population size.

time_to_smile 4y ago

The issue isn't sample size (nor many of the other things people are pointing out). First off we're ultimately talking about ordinal data here and even the Evan Miller post in another comment misses that ordinal data is fundamentally tricky. Everyone is talking about the p("most loved") but that makes assumptions that "most loved" has a consistent meaning. The problem with ordinal data is that the only thing that's known about it is an ordering of the values, but the distance between values is undefined. That is the difference between a 5 star and a 4 star review is not necessarily that same a between a 2 and 1 star review. This means that you can't meaningfully average these. However there is an even bigger problem, which is what I think parent is feeling here, and that is selection bias. It's very hard to compare the average rating of a film such as Star Wars and a film like Cannibal Holocaust. The first is a general audience film and the later is an extremely niche subset of gore horror. If you show Cannibal Holocaust to the general population it will receive wildly lower rating than Star Wars. However if you were to do a survey of all people who have a DVD of Star Wars and a DVD of Cannibal Holocaust, I wouldn't be at all surprised if Cannibal Holocaust was more loved. There are plenty of people that have Star Wars on their shelf and feel 'meh' about it, but almost no one who owns Cannibal Holocaust on DVD that doesn't feel like it is an essential horror classic. The annoying truth is, there's really no statistical way to solve this. There are some solutions to better modeling ordinal data (you could for example, transform this problem into an ordinal regression problem). But ultimately "loved" is not really a measurable thing, so we're always looking for imperfect proxies.

ducharmdev 4y ago
Thanks for the thoughtful reply - I don't have a stats background, but it didn't quite sit right with me. The movie example you give here really puts it in perspective.