5 ms·

This analysis doesn't make any sense to me. Even by their own intent (i.e., best added value based on salary), their methodology is nonsensical. For example: W

This analysis doesn't make any sense to me. Even by their own intent (i.e., best added value based on salary), their methodology is nonsensical. For example: Why should CalTech get hurt for being near L.A.? They're basically arguing that you're better off going to a school in the middle of nowhere because "hey, for being in such a crappy location, you did pretty well!". In an absolute sense you are better off going to CalTech, it's just that they might not leverage their advantage as well as some other schools. Not that they even show the last point -- it seems unlikely that the true model is linear (I'm guessing they used linear regression). For example, if the true model is closer to a sigmoid, then schools at the high end suddenly get unfairly penalized and schools near the low end get unfairly boosted. Finally, the statistical indicators are equally misleading. I can obtain an R^2 of 1.0 just by including indicators I[is COLLEGE_NAME] for each college. While that might not give you significance, the point is that getting good prediction is meaningless. I think what they really want is to restrict to predictors about the students. So, given that you're a straight A student with a 2400 SAT score, what would you expect to make coming out of each school? This at least tells me something about the added value to me of going to a certain school. (This approach is still prone to bias, but in the opposite direction -- there's a chance that the straight A student with a 2400 SAT score going to community college may have been smart but unmotivated, which might correlate with lower salary.) Edit: Here's another concern. They're claiming to have a model for "expected" earnings: earnings = A * (college covariates) + b + error but they can't distinguish between model error (i.e., error because their model is misspecified) vs. the school variation that they are trying to capture.

krstck 11y ago
> They're basically arguing that you're better off going to a school in the middle of nowhere because "hey, for being in such a crappy location, you did pretty well!". Well, no, not exactly. It's a subtle distinction, but what it's actually ranking is how well that school exceeds expectations, not best outcomes. This is not necessarily a list that will give a student the best school to go to, but rather (what it says on the tin) a scorecard for how well those schools are doing, given their resources.
- obastani 11y ago
  That's my point -- who decides what expectations are? Their results are incredibly dependent on the model specification. I imagine if they changed which indicators they used, the results would vary widely. Here's another way to see my concern. Suppose you had a classifier that achieves 1.0 R^2; then since it perfectly predicts each school's expected value, it'll assign each school a score of 0. I'm pretty suspicious of an approach where the results get worse with better predictive power. Even if you want to do "exceeds expectations", I think you shouldn't include variables that are school specific, only variables that are student specific. In other words, for my expected outcomes, which school is best?
  - haberman 11y ago
    > Suppose you had a classifier that achieves 1.0 R^2; then since it perfectly predicts each school's expected value, it'll assign each school a score of 0. I'm pretty suspicious of an approach where the results get worse with better predictive power. If I'm understanding correctly, that result would indicate a world where the college you attend has no effect on your earning power. ie. choose any college you want, because you'll earn the same amount regardless of which one you choose. This would only apply to colleges that people in your demographic group actually attended though. If the dataset doesn't contain any information about people like you who went to Harvard, then maybe Harvard would indeed increase your earning potential if there was a way for you to actually go there.
    - obastani 11y ago
      I'm not saying that each college you attend has no effect on earning power. It's just that I can perfectly predict the effect of each college on your earning power. Does that make sense? If I have an oracle that tells you "if you got to Harvard, you will make $80,000, if you go to MIT, you will make $86,000", and the oracle is exactly correct, then under this model, The Economist assigns every college a score of 0.
      - deleted 11y ago
        [deleted]
deleted 11y ago
[deleted]

krstck 11y ago

> They're basically arguing that you're better off going to a school in the middle of nowhere because "hey, for being in such a crappy location, you did pretty well!". Well, no, not exactly. It's a subtle distinction, but what it's actually ranking is how well that school exceeds expectations, not best outcomes. This is not necessarily a list that will give a student the best school to go to, but rather (what it says on the tin) a scorecard for how well those schools are doing, given their resources.

obastani 11y ago
That's my point -- who decides what expectations are? Their results are incredibly dependent on the model specification. I imagine if they changed which indicators they used, the results would vary widely. Here's another way to see my concern. Suppose you had a classifier that achieves 1.0 R^2; then since it perfectly predicts each school's expected value, it'll assign each school a score of 0. I'm pretty suspicious of an approach where the results get worse with better predictive power. Even if you want to do "exceeds expectations", I think you shouldn't include variables that are school specific, only variables that are student specific. In other words, for my expected outcomes, which school is best?
- haberman 11y ago
  > Suppose you had a classifier that achieves 1.0 R^2; then since it perfectly predicts each school's expected value, it'll assign each school a score of 0. I'm pretty suspicious of an approach where the results get worse with better predictive power. If I'm understanding correctly, that result would indicate a world where the college you attend has no effect on your earning power. ie. choose any college you want, because you'll earn the same amount regardless of which one you choose. This would only apply to colleges that people in your demographic group actually attended though. If the dataset doesn't contain any information about people like you who went to Harvard, then maybe Harvard would indeed increase your earning potential if there was a way for you to actually go there.
  - obastani 11y ago
    I'm not saying that each college you attend has no effect on earning power. It's just that I can perfectly predict the effect of each college on your earning power. Does that make sense? If I have an oracle that tells you "if you got to Harvard, you will make $80,000, if you go to MIT, you will make $86,000", and the oracle is exactly correct, then under this model, The Economist assigns every college a score of 0.
    - deleted 11y ago
      [deleted]

deleted 11y ago

[deleted]