8 ms·

I don’t see how this analysis identifies the causal factor. Do some college programs select students already on a trajectory to succeed? I would love to see an

I don’t see how this analysis identifies the causal factor. Do some college programs select students already on a trajectory to succeed? I would love to see an analysis comparing ACT/SAT scores and lifetime earnings for those that did and did not attend college. Still, I find the insights of this article important to the college debate

analog31 5y ago
Indeed, even at the college level, the universities in my state give each incoming freshman a math exam, and the results of that exam are for all intents and purposes a "sorting hat" for whether you can even get into particular majors. This is even before the first day of classes.
- grp000 5y ago
  I think that's basically every university. That being said, its just a way to test out of math reqs to hit minimums for starting certain major tracks. No reason someone couldn't take the math courses at university in a general ed. major and then transfer over.
  - ghaff 5y ago
    I don't have a large personal sample size, but 1.) I'm pretty sure giving a math test to incoming students isn't standard in the US. In any case, they generally already have SAT/ACT scores. 2.) Furthermore, in many (though not all) cases, once you're enrolled you can major in whatever you want.
    - analog31 5y ago
      I'm only familiar with the University of Wisconsin system. They use the math test to slot you into levels ranging from calculus to a remedial math course. Don't know why they don't just use SAT/ACT. As for majors, there are limited admissions into some programs such as engineering and business, where you are not formally admitted until after your first year, and have to make a certain GPA. Not all majors had such requirements. Then there were some unique situations. Majoring in music performance required an audition, and the incoming students tended to be playing at a very high level.
madrox 5y ago
You can’t establish causation in an observational study like this one. You’d have to conduct an experiment to do that, and people don’t usually want to leave their careers up to double-blind. There’s still value in measuring the correlation. However, it accounts for things like dropout status, so I don’t think your hypothesis fits the data.
- jostmey 5y ago
  Sometimes it is possible to establish causation from only observational data, but in general, you are correct. Without the ability to conduct an experiment, the best we can do is use college acceptance criteria for screening candidates to compare populations that attend and do not attend. The best we can do is better than nothing
  - madrox 5y ago
    This is fair. I was familiar with methods of evaluating causality, but didn't think it applied here since it hadn't occurred to me to design the study that way. Yet another lesson in "just because I can't think of it doesn't mean it can't be done."
  - _delirium 5y ago
    I can't seem to find it, but there was a report from a few years back that compared an income prediction based solely on incoming student demographics (using a regression model on things like parental income and education levels) with graduates' actual income a few years out. It still doesn't prove causality, but if there's a big difference between the two in the positive direction, it at least suggests the university is providing some kind of actual value-add, vs just passing through students who were already from a high socioeconomic background.
  - sfink 5y ago
    I use this framing quite a bit ("The best we can do is better than nothing"), but I don't really like it. A better framing is that it's not how much we can figure out, it's what we can we figure out. Observational data is great for explaining things after the fact. It must be handled very carefully when used for predicting things before they happen. But if you can identify the most likely confounds and eliminate them, you may still be able to answer a lot of useful questions to a fair degree of certainty.
WORMS_EAT_WORMS 5y ago
Or, what about factoring in long-term and steady stability as being "worth it"? There's definitely a bit of insurance and fallback with having higher education that isn't measurable simply by some ROI analysis. There are a lot of jobs you will never be able to apply to if things go south simply by not meeting the minimum requirements. I was considering a dirt-cheap online MBA once simply for this reason if my biz stuff died (did not pursue).
bumby 5y ago
Do some college programs select students already on a trajectory to succeed? This is interesting and I know there was some studies performed, not on the program, but on the school. IIRC, the results were that it wasn't the school that contributed to success, but the person. Meaning, if you were accepted to a top-tier school but attended a lower-tier one, you had as much success as the people who attended the top-tier school. So, as you say, the schools were already selecting for those who were on a trajectory to succeed. There was a caveat that the school did provide more help if the person was from a low-socioeconomic background, and I've wondered if that is due to network effects.
gianpresq 5y ago
I think it goes deeper than that, although you're making an incisive point. The value is sort of in how the degrees are used as well. Increasingly, they're being seen as licenses of sort, even though that might not be the intent of the program or even reflect the actual experiences of the person in college. That is, the ROI is due as much to the marketplace as it is the degree itself. If HR departments decide that understanding of DL/AI requires a degree in comp sci, even if someone did an honors thesis on that as a psychology major, demonstrating a new proof-of-concept DL model, it doesn't matter what sort of talent is being recruited. Is it the degree holder, the degree, the program, or the employer? I feel like college degrees have become this kind of signalling label, like much of the modern world, like clothing or something. That's not to discount the skills one obtains as part of a major, but it hurts the person who is going outside the box (in a good way). There's also something disingenuous then, about treating degrees as a resource to be obtained, when the value of that resource is entirely dependent on the way it is treated by others. That is, a hammer is more useful than a rotting stick, but if for whatever reasons there's rotting stick mania, the ROI will be greater for the latter. That doesn't mean the ROI analysis is wrong, but it might give a misleading impression about why. I admit I didn't read this in enough detail, but I had other questions as well. For instance, in other similar analyses I've read in other outlets, they've explicitly ignored people who have graduate degrees, as it muddies interpretation. But isn't that important? What about the person who gets an undergrad degree in psychology, but then pursues symbolic logic programming and DL models in their master's program in comp sci? Or who goes on to medical school? Some of these degrees, like philosophy, are notorious for being pre-professional degrees, and comparing a BA-only philosophy grad to a BA-philosophy + JD is a little odd, both because it's unfair, but also because the person who gets a BA in philosophy and then stops is maybe different from the person who gets that degree as part of a longer-term plan that includes law school.

analog31 5y ago

Indeed, even at the college level, the universities in my state give each incoming freshman a math exam, and the results of that exam are for all intents and purposes a "sorting hat" for whether you can even get into particular majors. This is even before the first day of classes.

grp000 5y ago
I think that's basically every university. That being said, its just a way to test out of math reqs to hit minimums for starting certain major tracks. No reason someone couldn't take the math courses at university in a general ed. major and then transfer over.
- ghaff 5y ago
  I don't have a large personal sample size, but 1.) I'm pretty sure giving a math test to incoming students isn't standard in the US. In any case, they generally already have SAT/ACT scores. 2.) Furthermore, in many (though not all) cases, once you're enrolled you can major in whatever you want.
  - analog31 5y ago
    I'm only familiar with the University of Wisconsin system. They use the math test to slot you into levels ranging from calculus to a remedial math course. Don't know why they don't just use SAT/ACT. As for majors, there are limited admissions into some programs such as engineering and business, where you are not formally admitted until after your first year, and have to make a certain GPA. Not all majors had such requirements. Then there were some unique situations. Majoring in music performance required an audition, and the incoming students tended to be playing at a very high level.

madrox 5y ago

You can’t establish causation in an observational study like this one. You’d have to conduct an experiment to do that, and people don’t usually want to leave their careers up to double-blind. There’s still value in measuring the correlation. However, it accounts for things like dropout status, so I don’t think your hypothesis fits the data.

jostmey 5y ago
Sometimes it is possible to establish causation from only observational data, but in general, you are correct. Without the ability to conduct an experiment, the best we can do is use college acceptance criteria for screening candidates to compare populations that attend and do not attend. The best we can do is better than nothing
- madrox 5y ago
  This is fair. I was familiar with methods of evaluating causality, but didn't think it applied here since it hadn't occurred to me to design the study that way. Yet another lesson in "just because I can't think of it doesn't mean it can't be done."
- _delirium 5y ago
  I can't seem to find it, but there was a report from a few years back that compared an income prediction based solely on incoming student demographics (using a regression model on things like parental income and education levels) with graduates' actual income a few years out. It still doesn't prove causality, but if there's a big difference between the two in the positive direction, it at least suggests the university is providing some kind of actual value-add, vs just passing through students who were already from a high socioeconomic background.
- sfink 5y ago
  I use this framing quite a bit ("The best we can do is better than nothing"), but I don't really like it. A better framing is that it's not how much we can figure out, it's what we can we figure out. Observational data is great for explaining things after the fact. It must be handled very carefully when used for predicting things before they happen. But if you can identify the most likely confounds and eliminate them, you may still be able to answer a lot of useful questions to a fair degree of certainty.

WORMS_EAT_WORMS 5y ago

Or, what about factoring in long-term and steady stability as being "worth it"? There's definitely a bit of insurance and fallback with having higher education that isn't measurable simply by some ROI analysis. There are a lot of jobs you will never be able to apply to if things go south simply by not meeting the minimum requirements. I was considering a dirt-cheap online MBA once simply for this reason if my biz stuff died (did not pursue).

bumby 5y ago

Do some college programs select students already on a trajectory to succeed? This is interesting and I know there was some studies performed, not on the program, but on the school. IIRC, the results were that it wasn't the school that contributed to success, but the person. Meaning, if you were accepted to a top-tier school but attended a lower-tier one, you had as much success as the people who attended the top-tier school. So, as you say, the schools were already selecting for those who were on a trajectory to succeed. There was a caveat that the school did provide more help if the person was from a low-socioeconomic background, and I've wondered if that is due to network effects.

gianpresq 5y ago

I think it goes deeper than that, although you're making an incisive point. The value is sort of in how the degrees are used as well. Increasingly, they're being seen as licenses of sort, even though that might not be the intent of the program or even reflect the actual experiences of the person in college. That is, the ROI is due as much to the marketplace as it is the degree itself. If HR departments decide that understanding of DL/AI requires a degree in comp sci, even if someone did an honors thesis on that as a psychology major, demonstrating a new proof-of-concept DL model, it doesn't matter what sort of talent is being recruited. Is it the degree holder, the degree, the program, or the employer? I feel like college degrees have become this kind of signalling label, like much of the modern world, like clothing or something. That's not to discount the skills one obtains as part of a major, but it hurts the person who is going outside the box (in a good way). There's also something disingenuous then, about treating degrees as a resource to be obtained, when the value of that resource is entirely dependent on the way it is treated by others. That is, a hammer is more useful than a rotting stick, but if for whatever reasons there's rotting stick mania, the ROI will be greater for the latter. That doesn't mean the ROI analysis is wrong, but it might give a misleading impression about why. I admit I didn't read this in enough detail, but I had other questions as well. For instance, in other similar analyses I've read in other outlets, they've explicitly ignored people who have graduate degrees, as it muddies interpretation. But isn't that important? What about the person who gets an undergrad degree in psychology, but then pursues symbolic logic programming and DL models in their master's program in comp sci? Or who goes on to medical school? Some of these degrees, like philosophy, are notorious for being pre-professional degrees, and comparing a BA-only philosophy grad to a BA-philosophy + JD is a little odd, both because it's unfair, but also because the person who gets a BA in philosophy and then stops is maybe different from the person who gets that degree as part of a longer-term plan that includes law school.