6 ms·

Speaking informally, but as a professional analyst here who is unfortunately at a computer with no analysis tools setup :( but who has worked with his fair sha

Speaking informally, but as a professional analyst here who is unfortunately at a computer with no analysis tools setup :( but who has worked with his fair share of social, survey and demographic information. Check your data. My first gut-instinct on seeing data like that is not "Eureka - i found a huge first-birth-order effect", but rather "huh...looks wrong, I wonder why that is?". What do I mean "looks wrong"? Keep in mind I have no real "pony in the race" on first-born effects (although I admit I am one), but intuitively it "doesn't look like real social data looks like". The effect is too big. Its too loud. Its too constant. Its too nice. Something else is driving it and its your job to find out what... The first place I'd look is data quality. Have you proven your data got into your final histogram data set right? I had problems with your csv opening it in libre office...looks like it interpreted it as having some value migration between the variable in whatever you did. Could be my computer, but in my experience, if there's a problem with the csv, these kind of things correlate with other mistakes. The second place i'd look is survey design: was there a default option, are people clicking through by default, are the response frequencies of other variables lining up as would be expected (gender, birthdays, that kind of thing). If not, why not? If you've crossed and ticked all those, only after that would I start the analysis...but again, a pattern and effect of that size and consistency is something that you should default to needing explained in the data. Reach the conclusion that its real only after you've eliminated everything else.

aetherson 9y ago
It wasn't a default option (you were asked to enter into a text field your number of older brothers, older sisters, younger brothers, younger sisters). He can't cross-reference by gender very well because his audience is overwhelmingly male, to the point where there's probably not a lot of statistical significance in the female audience. But yes, I think that these data are very open to an "impossibly hungry judges" critique. If birth order effects were so very visible and so very strong, why haven't we noticed them before?
- ACow_Adonis 9y ago
  Anecdote time/thought experiment: and forgive me, I'm deliberately vague here. Pretend there's a business somewhere. Pretend they implemented a rule that said people had to enter a number/estimate into a box for a certain peice of information before they could proceed to the next box. Pretend as an analyst you're looking at the values resulting some way down the pipe from that data entry point. Do you think it reasonable to assume the distribution of numbers entered into that box that were invalid would be uniformally random? If the answer is no, what numbers do you think people would enter disproportionately?
  - aetherson 9y ago
    It seems tough to me to suggest that people are putting a default 0 in for older siblings but a default non-zero in for younger siblings. Why wouldn't they be consistent? That said, I still don't believe these data. I don't know where the flaw is, but I none the less think there's a flaw. If it were a much smaller effect, it would be much more believable.
    - cabalamat 9y ago
      > If it were a much smaller effect, it would be much more believable. Why do you think that? We know that the SSC readership is in many ways quite distinct from the general population, so why do you find it surprising that it has correlations with birth order? Is it because you find it surprising that anything has correlations with birth order?
      - bildung 9y ago
        The effect would have to be smaller because most effects in psychology and sociology are pretty small. Else they would be obvious (common sense) even without field experiments.
        carlmr 9y ago
        I agree with the other comments here that it is a rather common sense effect, but it has eluded psychological research until now (which is what makes me question it). I know plenty of people who have a common sense view that birth order is important. So that part isn't missing. I can't even grasp how many times I've heard about the middle child being so different. Or being a typical little brother, etc. For me the question is, if it is so obvious and measurable, why is this the first study that finds this obvious effect.
- alien_at_work 9y ago
  >If birth order effects were so very visible and so very strong, why haven't we noticed them before? I'm not sure where you're from but everyone I know (including a psychologist) sees birth order effects as very real with noticable effect. Have you never seen the stats of first borns being overrepresented in management, for example? Am I misunderstanding what you mean by "haven't noticed them before"?
  - graphitezepp 9y ago
    Another example, having older brothers is one of the most consistent correlations with homosexuality for males. Birth order effects are absolutely visible.
  - aetherson 9y ago
    As the article mentions in like the first couple of paragraphs, those results haven't generally withstood replication. But more so, the effects in the Slate Star Codex survey are extremely strong. You'd expect that an effect that strong would be part of the common wisdom, not something that has to be teased out in a psychological study.
    - dclowd9901 9y ago
      Well, to what you just mentioned, it -seems- to be a "common wisdom" that is disregarded by scientific study. So, maybe this is a case where the study was ill-designed, but everyone took it for granted?
      - aetherson 9y ago
        I don't think it is common wisdom. Like, look: which of your coworkers are first-born? How many of them did you intuit that without explicitly asking?
        AstralStorm 9y ago
        That depends on the field. The guesses can be very accurate based on ancillary data like profession.
- pixl97 9y ago
  > why haven't we noticed them before? LMGTFY https://scholar.google.com/scholar?q=birth+order+effects https://scholar.google.com/scholar?q=birth+order+effects
- IshKebab 9y ago
  Was it optional though? He said some people didn't give him enough data, which makes me suspect first-borns were more likely to go "oooo this question is about me being awesome - better fill it in!" and other people to go "eh wtf I'll skip it".
  - aetherson 9y ago
    All questions in the survey were optional. He had 2965 respondents for that question, out of 8077 total respondents, so that's a plausible source of bias.
- JackFr 9y ago
  > It wasn't a default option (you were asked to enter into a text field your number of older brothers, older sisters, younger brothers, younger sisters). Is blank coerced into 0? That would seem to explain the size of the effect.
  - aetherson 9y ago
    No.
ACow_Adonis 9y ago
Just a quick follow up, the reason I suggest looking at things like gender or birthday frequencies is because they're two bits of social data that have brilliant properties for getting to the bottom of this sort of things. In practice, gender is immaculately reported and recorded compared to most other variables. Both gender and birth days have nice reasonableness bounds and statistical properties that can be transparently reasoned about. Deviation from those nice statistical properties not only confirms for you that something is going on in your data, often the way it can deviate gives hints as to the next thread you should start chasing. Obviously it's not perfect, not all data problems will show up there, but in practice it's an extremely low hanging fruit on which to get started, and seeing strong deviations from 50/50 statistical gender splits and predictable day/month/year type distributions should start analytical klaxons going off in your demographic noggin...
- ikeboy 9y ago
  Lots of groups don't have 50/50 gender splits.
  - theptip 9y ago
    It's odd to have to point this out in this forum; the tech community has female participation rates around 25%. That doesn't somehow make statistics about this community suspicious. Sure, if you didn't know that the base rate was skewed, you'd want to check your numbers the first time you made that measurement. But it's well-known (I suppose only by regular readers of SSC) that the SSC community is significantly above the population base rate on a whole host of metrics (self-reported IQ, transgender, % of males, autistic spectrum, etc). Edit: to clarify which point I was referring to
    - ikeboy 9y ago
      My point is that you shouldn't expect gender ratios to be 50/50 unless you're choosing at random, and therefore it's not suspicious
      - 9y ago
deleted 9y ago
[deleted]
afpx 9y ago
Off-topic, but is this a common practice? How do analysts ensure that studies aren’t incorrectly invalidated when bad data ends up ‘looking good’? (BTW: I know little about data, so forgive my ignorance.)
- chewbacha 9y ago
  A few ways come to mind: Randomized double-blind studies peer reviewed question sets longitudinal cohorts The goal is to control for known biases. It’s hard, however, to discern conclusions from a dataset that is already biased. There’s very little in this to conclude effectively that “intellectual curiosity” was caused by birth order. They might be independent variables from each other on this survey born out by another variable. Or are completely spurious in nature.
jerf 9y ago
One possibility that came to mind when I read his post that harmonizes both your point and his data is that the studies that have been done would almost certainly have been looking at very broad effects, on very broad populations, which results in a very averaged, fuzzy view of things. That is not a criticism, because it is a priori both plausible that there will be some effect at that scale, and that there won't be, so it's a worthy test. But it is also possible that while broadly speaking birth order doesn't have any significant effects on things, when you go out to a population as niche as "people who read SSC and like it enough to fill out a form for 15-30 minutes just because Scott asked them too" and you slice the n-dimensional space in question with that much precision, that you will get a very, very particular slice of the world in which there are birth order effects. This would imply that there are birth order effects, but that in general they are too weak to be picked up by a general-purpose study. However, perhaps if you magnify them in such a precise way you can see them, even though they are small. "Member of the SSC community" is a very information-rich cut of the general population and it would not be a surprise to see that poking out of the stats in many other ways. Statistics being what they are, I believe it is completely possible for both "Birth order has no significant impact on whether you will show up in SSC's community" and "Given membership within SSC's community, birth order effects can be witnessed" to be simultaneously statistically true. Previous studies showing birth order to not matter would be testing a hypothesis much closer to the first one, Scott is obviously testing the latter. This is merely a hypothesis that is not very complicated (in Occam's Razor terms, given the fact that "everything sorta correlates to everything" is a basic fact of the world we live in and thus using that does not really count as "multiplying entities") and fits both the facts, but not one I am claiming is certain to be true. A lot more analysis would have to be done, to a degree of rigor I doubt anyone is going to perform since nobody has the full picture and the effort to obtain the full picture is likely larger than the payoff. So let it be clear that I'm just spitballing, and aware of the fact that I'm just spitballing.
- theptip 9y ago
  This hypothesis could well be true, but it's somewhat difficult to falsify; you can always posit an effect that only appears in the small statistically-underpowered cohort that Scott sampled, and which would be swamped by the average of the population as you start to increase the sample size to verify the effect. As a counterpoint, Scott went the other direction in his post, arguing that the true effect was more general, but the metrics previous academics used was broken, and therefore undersold the strength of the effect they were detecting: > I think the most likely explanation is that tests for Openness have limited validity, which makes the correlation look smaller than it really is. If being an eldest sibling increases true underlying Openness by a lot, but your score on psychometric tests for Openness only correlates modestly with true underlying Openness, that would look like being an eldest sibling only increasing test-measured-Openness a little bit. It's simple to falsify this (though it requires some work); just do more experiments on the general population using the metric that Scott likes [1], and see if that detects a stronger birth-order effect. [1]: https://www.researchgate.net/publication/230787941_Construct_validation_using_multitrait-multimethod-twin_data_The_case_of_a_General_Factor_of_Personality https://www.researchgate.net/publication/230787941_Construct...

aetherson 9y ago

It wasn't a default option (you were asked to enter into a text field your number of older brothers, older sisters, younger brothers, younger sisters). He can't cross-reference by gender very well because his audience is overwhelmingly male, to the point where there's probably not a lot of statistical significance in the female audience. But yes, I think that these data are very open to an "impossibly hungry judges" critique. If birth order effects were so very visible and so very strong, why haven't we noticed them before?

ACow_Adonis 9y ago
Anecdote time/thought experiment: and forgive me, I'm deliberately vague here. Pretend there's a business somewhere. Pretend they implemented a rule that said people had to enter a number/estimate into a box for a certain peice of information before they could proceed to the next box. Pretend as an analyst you're looking at the values resulting some way down the pipe from that data entry point. Do you think it reasonable to assume the distribution of numbers entered into that box that were invalid would be uniformally random? If the answer is no, what numbers do you think people would enter disproportionately?
- aetherson 9y ago
  It seems tough to me to suggest that people are putting a default 0 in for older siblings but a default non-zero in for younger siblings. Why wouldn't they be consistent? That said, I still don't believe these data. I don't know where the flaw is, but I none the less think there's a flaw. If it were a much smaller effect, it would be much more believable.
  - cabalamat 9y ago
    > If it were a much smaller effect, it would be much more believable. Why do you think that? We know that the SSC readership is in many ways quite distinct from the general population, so why do you find it surprising that it has correlations with birth order? Is it because you find it surprising that anything has correlations with birth order?
    - bildung 9y ago
      The effect would have to be smaller because most effects in psychology and sociology are pretty small. Else they would be obvious (common sense) even without field experiments.
      - carlmr 9y ago
        I agree with the other comments here that it is a rather common sense effect, but it has eluded psychological research until now (which is what makes me question it). I know plenty of people who have a common sense view that birth order is important. So that part isn't missing. I can't even grasp how many times I've heard about the middle child being so different. Or being a typical little brother, etc. For me the question is, if it is so obvious and measurable, why is this the first study that finds this obvious effect.
alien_at_work 9y ago
>If birth order effects were so very visible and so very strong, why haven't we noticed them before? I'm not sure where you're from but everyone I know (including a psychologist) sees birth order effects as very real with noticable effect. Have you never seen the stats of first borns being overrepresented in management, for example? Am I misunderstanding what you mean by "haven't noticed them before"?
- graphitezepp 9y ago
  Another example, having older brothers is one of the most consistent correlations with homosexuality for males. Birth order effects are absolutely visible.
- aetherson 9y ago
  As the article mentions in like the first couple of paragraphs, those results haven't generally withstood replication. But more so, the effects in the Slate Star Codex survey are extremely strong. You'd expect that an effect that strong would be part of the common wisdom, not something that has to be teased out in a psychological study.
  - dclowd9901 9y ago
    Well, to what you just mentioned, it -seems- to be a "common wisdom" that is disregarded by scientific study. So, maybe this is a case where the study was ill-designed, but everyone took it for granted?
    - aetherson 9y ago
      I don't think it is common wisdom. Like, look: which of your coworkers are first-born? How many of them did you intuit that without explicitly asking?
      - AstralStorm 9y ago
        That depends on the field. The guesses can be very accurate based on ancillary data like profession.
pixl97 9y ago
> why haven't we noticed them before? LMGTFY https://scholar.google.com/scholar?q=birth+order+effects https://scholar.google.com/scholar?q=birth+order+effects
IshKebab 9y ago
Was it optional though? He said some people didn't give him enough data, which makes me suspect first-borns were more likely to go "oooo this question is about me being awesome - better fill it in!" and other people to go "eh wtf I'll skip it".
- aetherson 9y ago
  All questions in the survey were optional. He had 2965 respondents for that question, out of 8077 total respondents, so that's a plausible source of bias.
JackFr 9y ago
> It wasn't a default option (you were asked to enter into a text field your number of older brothers, older sisters, younger brothers, younger sisters). Is blank coerced into 0? That would seem to explain the size of the effect.
- aetherson 9y ago
  No.

ACow_Adonis 9y ago

Just a quick follow up, the reason I suggest looking at things like gender or birthday frequencies is because they're two bits of social data that have brilliant properties for getting to the bottom of this sort of things. In practice, gender is immaculately reported and recorded compared to most other variables. Both gender and birth days have nice reasonableness bounds and statistical properties that can be transparently reasoned about. Deviation from those nice statistical properties not only confirms for you that something is going on in your data, often the way it can deviate gives hints as to the next thread you should start chasing. Obviously it's not perfect, not all data problems will show up there, but in practice it's an extremely low hanging fruit on which to get started, and seeing strong deviations from 50/50 statistical gender splits and predictable day/month/year type distributions should start analytical klaxons going off in your demographic noggin...

ikeboy 9y ago
Lots of groups don't have 50/50 gender splits.
- theptip 9y ago
  It's odd to have to point this out in this forum; the tech community has female participation rates around 25%. That doesn't somehow make statistics about this community suspicious. Sure, if you didn't know that the base rate was skewed, you'd want to check your numbers the first time you made that measurement. But it's well-known (I suppose only by regular readers of SSC) that the SSC community is significantly above the population base rate on a whole host of metrics (self-reported IQ, transgender, % of males, autistic spectrum, etc). Edit: to clarify which point I was referring to
  - ikeboy 9y ago
    My point is that you shouldn't expect gender ratios to be 50/50 unless you're choosing at random, and therefore it's not suspicious
    - 9y ago

deleted 9y ago

[deleted]

afpx 9y ago

Off-topic, but is this a common practice? How do analysts ensure that studies aren’t incorrectly invalidated when bad data ends up ‘looking good’? (BTW: I know little about data, so forgive my ignorance.)

chewbacha 9y ago
A few ways come to mind: Randomized double-blind studies peer reviewed question sets longitudinal cohorts The goal is to control for known biases. It’s hard, however, to discern conclusions from a dataset that is already biased. There’s very little in this to conclude effectively that “intellectual curiosity” was caused by birth order. They might be independent variables from each other on this survey born out by another variable. Or are completely spurious in nature.

jerf 9y ago

One possibility that came to mind when I read his post that harmonizes both your point and his data is that the studies that have been done would almost certainly have been looking at very broad effects, on very broad populations, which results in a very averaged, fuzzy view of things. That is not a criticism, because it is a priori both plausible that there will be some effect at that scale, and that there won't be, so it's a worthy test. But it is also possible that while broadly speaking birth order doesn't have any significant effects on things, when you go out to a population as niche as "people who read SSC and like it enough to fill out a form for 15-30 minutes just because Scott asked them too" and you slice the n-dimensional space in question with that much precision, that you will get a very, very particular slice of the world in which there are birth order effects. This would imply that there are birth order effects, but that in general they are too weak to be picked up by a general-purpose study. However, perhaps if you magnify them in such a precise way you can see them, even though they are small. "Member of the SSC community" is a very information-rich cut of the general population and it would not be a surprise to see that poking out of the stats in many other ways. Statistics being what they are, I believe it is completely possible for both "Birth order has no significant impact on whether you will show up in SSC's community" and "Given membership within SSC's community, birth order effects can be witnessed" to be simultaneously statistically true. Previous studies showing birth order to not matter would be testing a hypothesis much closer to the first one, Scott is obviously testing the latter. This is merely a hypothesis that is not very complicated (in Occam's Razor terms, given the fact that "everything sorta correlates to everything" is a basic fact of the world we live in and thus using that does not really count as "multiplying entities") and fits both the facts, but not one I am claiming is certain to be true. A lot more analysis would have to be done, to a degree of rigor I doubt anyone is going to perform since nobody has the full picture and the effort to obtain the full picture is likely larger than the payoff. So let it be clear that I'm just spitballing, and aware of the fact that I'm just spitballing.

theptip 9y ago
This hypothesis could well be true, but it's somewhat difficult to falsify; you can always posit an effect that only appears in the small statistically-underpowered cohort that Scott sampled, and which would be swamped by the average of the population as you start to increase the sample size to verify the effect. As a counterpoint, Scott went the other direction in his post, arguing that the true effect was more general, but the metrics previous academics used was broken, and therefore undersold the strength of the effect they were detecting: > I think the most likely explanation is that tests for Openness have limited validity, which makes the correlation look smaller than it really is. If being an eldest sibling increases true underlying Openness by a lot, but your score on psychometric tests for Openness only correlates modestly with true underlying Openness, that would look like being an eldest sibling only increasing test-measured-Openness a little bit. It's simple to falsify this (though it requires some work); just do more experiments on the general population using the metric that Scott likes [1], and see if that detects a stronger birth-order effect. [1]: https://www.researchgate.net/publication/230787941_Construct_validation_using_multitrait-multimethod-twin_data_The_case_of_a_General_Factor_of_Personality https://www.researchgate.net/publication/230787941_Construct...