3 ms·

No I did not come up with it myself. It's a common pitfall in statistics known as selection bias [1]. My problem is that if I ask a researcher about selection

by Kranar 5y ago

No I did not come up with it myself. It's a common pitfall in statistics known as selection bias [1]. My problem is that if I ask a researcher about selection bias, they have no problem defining it for me and even giving me a textbook example of it. But when I present to a researcher a real life scenario that is highly susceptible to selection bias, they will almost all fail to identify it. It's a problem where many people are able to compartmentalize knowledge but fail to actually make use of that knowledge in practice. As a run a quant firm I see this among so many researchers across many disciplines. They fully understand a concept when you ask them directly, but they fail to put those concepts to use outside of a very direct and artificial setting. [1] https://en.wikipedia.org/wiki/Selection_bias https://en.wikipedia.org/wiki/Selection_bias

Cederfjard 5y ago
Do you agree that without proper research backing it up, you can’t say for sure that this discrepancy is due to selection bias, even if on the surface it looks like a nice fit? If you had said that you’re looking for candidates to raise it as a possible explanation, instead of seemingly firmly asserting that it is the one and only correct answer, you’d probably have less replies arguing with you here (or perhaps if you do have such research readily available, to have shared it).
- Kranar 5y ago
  No I don't agree with you. I think my comment is a reasonable statement of the situation. Is it a 100% correct statement that is absolutely categorically true? No, but most people who are not looking to be pedantic and win an Internet argument won't go looking to nitpick the details of my statement. What most level headed people will do is understand that my statement is about how you can not reliably compare a group of people who are predisposed to enjoy a movie to a group of people who review almost every single movie that's released, regardless of their predisposition towards it. People are welcome to argue that and honestly I'd expect nothing less, but it's a basic statistical principle that I expect every single researcher I hire to be well familiar with and if they're not, then I make the choice to not work with them. >you’d probably have less replies arguing with you here... There is also a selection bias among the people who comment on HN and they too are not representative of how most of the community feels towards a topic. The people who want to nitpick the specific wording of my statement are certainly entitled to do so and will likely find something to nitpick regardless of how I framed my statement, but it's not a particularly interesting discussion to follow up on.
  - yakak 5y ago
    Sure, though I think the other implication from the difference stated is that critics have sampled randomly or more often and can therefore recognize derivative work where a target audience that shares a limited interest in film will almost uniformly perceive novelty. I think realizing something is a poor knock off is the basis of snobbery.. so it is strange not to accept that answer.
    - Kranar 5y ago
      That's a very strong assumption to make, I could easily assume that most casual movie viewers watch the same derivative knock-off action/horror/comedy films over and over and over again without issue. You think Adam Sandler is the highest earning actor/producer because his audience perceives a great deal of depth novelty in his work? The answer that requires the fewest assumptions and is consistent with almost all basic statistical analysis is that a group of people predisposed to like a movie will rate that movie higher than a group of people who watch movies regardless of their predisposition towards it.
ashtonbaker 5y ago
If you're using "selection bias" to refer to the fact that these two groups are selecting movies using different, biased methods, doesn't that imply that there's some way of sampling that would not be biased? But no matter what you do, an average critic and an average moviegoer are likely to disagree on many films. I would expect the explanations behind this to be more of a function of differences between the groups, which explains 1. why critics will occasionally enjoy a film more than the general population, and 2. why this disagreement can drift over time. Both of which are not accounted for by your analysis. I'm definitely not saying you're wrong - quite the opposite, just that there are other interesting things to discover here.

by Kranar 5y ago

Cederfjard 5y ago

Do you agree that without proper research backing it up, you can’t say for sure that this discrepancy is due to selection bias, even if on the surface it looks like a nice fit? If you had said that you’re looking for candidates to raise it as a possible explanation, instead of seemingly firmly asserting that it is the one and only correct answer, you’d probably have less replies arguing with you here (or perhaps if you do have such research readily available, to have shared it).

Kranar 5y ago
No I don't agree with you. I think my comment is a reasonable statement of the situation. Is it a 100% correct statement that is absolutely categorically true? No, but most people who are not looking to be pedantic and win an Internet argument won't go looking to nitpick the details of my statement. What most level headed people will do is understand that my statement is about how you can not reliably compare a group of people who are predisposed to enjoy a movie to a group of people who review almost every single movie that's released, regardless of their predisposition towards it. People are welcome to argue that and honestly I'd expect nothing less, but it's a basic statistical principle that I expect every single researcher I hire to be well familiar with and if they're not, then I make the choice to not work with them. >you’d probably have less replies arguing with you here... There is also a selection bias among the people who comment on HN and they too are not representative of how most of the community feels towards a topic. The people who want to nitpick the specific wording of my statement are certainly entitled to do so and will likely find something to nitpick regardless of how I framed my statement, but it's not a particularly interesting discussion to follow up on.
- yakak 5y ago
  Sure, though I think the other implication from the difference stated is that critics have sampled randomly or more often and can therefore recognize derivative work where a target audience that shares a limited interest in film will almost uniformly perceive novelty. I think realizing something is a poor knock off is the basis of snobbery.. so it is strange not to accept that answer.
  - Kranar 5y ago
    That's a very strong assumption to make, I could easily assume that most casual movie viewers watch the same derivative knock-off action/horror/comedy films over and over and over again without issue. You think Adam Sandler is the highest earning actor/producer because his audience perceives a great deal of depth novelty in his work? The answer that requires the fewest assumptions and is consistent with almost all basic statistical analysis is that a group of people predisposed to like a movie will rate that movie higher than a group of people who watch movies regardless of their predisposition towards it.

ashtonbaker 5y ago

If you're using "selection bias" to refer to the fact that these two groups are selecting movies using different, biased methods, doesn't that imply that there's some way of sampling that would not be biased? But no matter what you do, an average critic and an average moviegoer are likely to disagree on many films. I would expect the explanations behind this to be more of a function of differences between the groups, which explains 1. why critics will occasionally enjoy a film more than the general population, and 2. why this disagreement can drift over time. Both of which are not accounted for by your analysis. I'm definitely not saying you're wrong - quite the opposite, just that there are other interesting things to discover here.