27 ms·

State education rankings are riddled with methodological flaws

116 points by michael_fine 8y ago

xphilter 8y ago
I don't really understand removal of graduation rates. Obviously that's meant to act as a proxy for learning, especially for those 38 states without proficiency exams.
- bigpicture 8y ago
  Right, surely these states have some sort of standard that must be met for graduation even if it is not passing some exam.
  - saudioger 8y ago
    the standards for graduation vary wildly so isn't really comparable
    - bigpicture 8y ago
      That's a good point
- grandmczeb 8y ago
  The standard to graduate isn’t uniform, so using graduation rates biases towards states with weak requirements.
- ergothus 8y ago
  > Obviously that's meant to act as a proxy for learning That is the INTENT, sure. But does it work? If you have someone that graduated from a school A and someone that failed to graduate from school B that is in an entirely different area, can you draw any meaningful conclusions? Particularly knowing that school A has no particular requirements on why you graduate, while school B does? I'm far more likely to complain about "But when we disaggregate student performance scores by racial categories (white, black, Hispanic, and Asian), the rankings change dramatically." I wouldn't want to go too far with that before ensuring that I wasn't trying to curb swimming deaths by keeping Nick Cage out of movies: http://tylervigen.com/view_correlation?id=359 http://tylervigen.com/view_correlation?id=359
  - kolbe 8y ago
    The data is pretty useful in penalizing states that don't educate their students well enough to even get to the standardized testing stage.
- btilly 8y ago
  First a point about bias. They are quick to point out the bias of traditional scores. But they do not point out that their bias is for libertarian causes. Had they not found that more libertarian states perform well by their measure, they would never have published this. Removing graduation rates is definitely troublesome. Any school system could then improve its numbers by encouraging the worst students to drop out. Problem students both take more resources to educate and produce worse scores. That's bad for efficiency, but it is not obviously bad for society. I am also troubled that they are using measures of performance in grades 4 and 8, but not something like SAT scores. I understand that SAT scores could be hard to get. But a lot of what people care about in an education system is how well prepared you are for college, and not how far along you were in grade 8.
  - the_watcher 8y ago
    > But they do not point out that their bias is for libertarian causes. Had they not found that more libertarian states perform well by their measure, they would never have published this. The first statement is true. The piece would be a bit stronger if the researchers noted that by publishing in Reason, they're at the very least giving the appearance of potential confirmation bias. Your second statement makes an unsupported assumption though. RE: SAT scores - not all students take the SAT, some students take it multiple times, the SAT is taken at different points in time per student, and it's a test that is so influential in college admissions that success on it is strongly correlated with wealth, as scores are best improved by tutoring on test taking strategy, not increasing general knowledge. I agree that grade 4 and 8 are probably not optimal to get a complete picture of education quality, but there is no reward to the student for doing well, so they're less likely to have spent substantial time investing in beating the test.
Loughla 8y ago
There is no reason given to why they take per-student expense out of the picture, other than 'we just don't think it should count'. And that seems to be the entire basis of their finding that rankings are riddled with flaws. There is much research (once you sort through the partisan fluff from research institutions with obvious bias) that supports the positive impact of revenue per student on academic achievement on standardized tests. It's actually funny this article popped up, I legit just read a paper from 2015 called "A Cost-Benefit Analysis for Per-Student Expenditures and Academic Achievement". I genuinely read that this morning. Weird. The authors found that "There was a significant correlation between revenues available per student and ACT scores as one outcome measure of achievement." And just to drive that point home, they replicated the findings from a 2002 study, further solidifying that sentiment. And that's just the most recent one I've read. I'm sure there are more recent. And that is definitely just one in the series of research related to per-pupil expenditures. Also, I'm afraid this piece serves no purpose other than to be self-congratulatory to the 'lower taxes at all costs' group and right-to-work proponents. Why I say this: research pieces probably shouldn't include snide comments like "high-tax, high-spending progressive utopias." "punishing taxes" Maybe that's just me? Am I off base? edit: also, the comment section on that article is awful. Just awful.
- kolbe 8y ago
  >There is no reason given to why they take per-student expense out of the picture What do you think would happen to the regression if per-student expense was both an independent and a dependent variable?
  - Loughla 8y ago
    My experience with educational research is that researchers find per-pupil expense is directly correlated to improved achievement on standardized academic measures across all areas and for all variables in race/ethnicity, gender, age, and (obviously) socioeconomic status. I don't know what variables the authors are talking about when they say "we ran multiple regression analyses on our data, which included several other variables", so I hesitate to make a claim on what their outcomes could be. There are so many things they could be looking at; it's almost impossible to know what they're really evaluating without being able to look at the data.
patorjk 8y ago
I only have 2 data points of experience on this issue. Which I know doesn't mean much, but I'm left scratching my head after looking at their list. I spent the first 14 years of my life in Las Cruces, New Mexico (the state's second biggest city), and the schools I went to were not very good. I did high school in Maryland and the school system was a lot better (I initially struggled because I was so far behind everyone else - a friend of mine who moved to North Carolina for High School told me he had the same problem). In the author's ranking list, they place Maine at #48 and New Mexico at #41. I find this very hard to believe. There are so many problems that New Mexico has that Maine doesn't seem to have (gang problems, drug problems - at least at the magnitude that I saw in New Mexico). Unless Maine has some pretty bad schools, this just doesn't add up. US News lists Maine at #6, a ranking drop of 42 spots seems really significant.
- kolbe 8y ago
  I suggest you read the article if you're curious how they created their list. They explain their methodology pretty thoroughly.
- ericcumbee 8y ago
  I think part of it is the variability between school systems with in the same state. Las Cruces schools maybe below average for NM schools. Off Topic: I went to Las Cruces a few years ago when my alma mater played New Mexico State. The scenery was breathtaking.
  - patorjk 8y ago
    Even though my comment was kind of negative, I love Las Cruses. The scenery and weather are the two things I miss the most. It's certainly possible the rest of the state has better schools, but I never heard anything about them while I lived there.
- CBLT 8y ago
  I'm confused by your comment, would you care to elaborate? Your anecdote has you moving from (the article's) rank 41 in quality to rank 6. You comment that the rank 6 state is much better than the rank 41 in education quality. However, you find their ranking hard to believe?
temp-dude-87844 8y ago
Disaggregating the results by race is real cute but also misses the point: blacks and asians and hispanics are residents of the state too and so leaving them in to evaluate the state's overall achievement (relative to other states) in education is fair. If anything it ought to be more problematic that Texas fails so badly at educating hispanic students when a fifth of them live in Texas. Besides, the race in big-gap states is an unwitting proxy to income, so disaggregate by household per capita income (relative to local cost of living) if you want to disdain. This methodology would work well in all states and expose the challenges of anxiety and the quality of a student's home life in their education attainment.
- theparanoid 8y ago
  > race here is an an unwitting proxy to income, so disaggregate by household per capita income Poor white students do better academically than well off black students. So splitting by race is more informative when discussing academic performance.
  - purple-again 8y ago
    Do you have some evidence to back that up? There are some dumb rich people, no doubt of every color, however there is a clear correlation between income and performance in every reputable source I’ve ever seen. I find it hard to believe that on average rich black kids are doing worse than on average poor white kids.
    - grandmczeb 8y ago
      Here's a study[1] that breaks down SAT scores by race and income in 2003. Take a look at table 2. [1] https://pdfs.semanticscholar.org/e7a0/dfdee8ad5b91e1d179e9e2e838a4952f00be.pdf https://pdfs.semanticscholar.org/e7a0/dfdee8ad5b91e1d179e9e2...
      - ckemere 8y ago
        I believe that study shows that the affect of race on SAT scores when a more complex model of income and parental education is taken into account is halved (i.e., the observed 100 pt difference drops to 50). So I think it still doesn't excuse the original post from taking income into account.
golemiprague 8y ago
Education depends on population, their average IQ and their learning culture, everybody knows it and that's why the main consideration when people with kids buy a house is the population around it. No money or learning methods or good teachers is going to change this basic truth. It doesn't matter in which state you live, what matters is how much money you got to get to the right neighbourhood or the right private school.
dsfyu404ed 8y ago
I have several thoughts on this: The race adjusted performance is interesting and useful but they're basically using it as a proxy for wealth. Why can't they just use some metric that directly represents wealth? If Massachusetts schools are efficiency with money than the system nation wide is far more screwed up than anyone is willing to admit. That state ranking depends in part on money spent is abhorrent. That should be tracked for reasons that should be obvious but not part of a composite score meant to represents results. The bit about unions lines up with my anecdotal experience. I worked with the education department in college and they did not hold teachers unions in high regard. If your unions are so bad that college professors at a state school in a blue state gripe about how they hold back progress then I think it's fair to say your union is pretty bad.
- rayiner 8y ago
  They're not using race as a proxy for wealth, because the two things are different. There are unique challenges faced by minorities compared to white americans of the same income level. (Remember, Trayvon Martin was a middle class black kid whose dad was a white-collar government worker. A big part of #blacklivesmatter is educating people that black folks face unique problems that aren't subsumed within the larger issue of income inequality. And as to Hispanic students the challenges there are obvious--they're much more likely to be in families that are English as a second language, and also much more likely to move around a lot.)
  - dan-robertson 8y ago
    I’m not convinced that ESL families is the main difficulty facing Hispanic children. I’ll accept that there are disadvantages to Hispanics but having ESL parents seems similar to those faced by almost any other child of immigrants. Perhaps one could argue that due to the visa system, non-Hispanic parents of Americans tend to have higher earning potential? I have no idea whether this is true or not
    - rayiner 8y ago
      Due to the visa system, immigrants in the U.S. are not identically situated to each other. Non-Hispanic immigrants are much more likely to already speak English and have advanced degrees because they come here through visa categories that specifically select for those things.
KaoruAoiShiho 8y ago
Is there the per-race data available somewhere?
rb808 8y ago
Ranking is nearly meaningless. eg compare these: School A - full of underachieving kids that are pushed to get avg grades School B - full of avg kids getting above avg grades School C - full of smart kids getting top grades School D - full of smart kids getting nearly top grades, but soft skills, nice personalities and VIP friends School E - Mix of kids getting relevant grades Which is the best school for each kid type? They're impossible to rank. Many schools with top grades are only there because they kick out failing kids, not because they're better at teaching. Many schools are rated good because they can improve grades, but you wouldn't want to send a smart kid there. Many private schools sell themselves on creating well rounded kids, not necessarily good grades.
- ben_jones 8y ago
  I went to school F - full of average kids who knew how to play the system to get top grades, as well as to get into high/elite level colleges.
- darawk 8y ago
  Yes, but all the different 'types of kids' should normalize across a state-level population.
- educationdata 8y ago
  The article is about ranking states, not ranking schools. For the point of ranking schools, there are statistical methods to measure student growth, instead of achievement, such as SGP (student growth percentile) and VAM (value-added modeling), which kind of address part of your question.
lifeisstillgood 8y ago
Is it just me that has a hard time understanding the racial aggregation thing. They claim to have split Iowa and Texas results by race and age. And found that Texas did better than Iowa for whites blacks and hispanics - but then aggregating the results ala USNews, Iowa comes out on top I am struggling to imagine which category is the flip category. If iowa is not doing better for white black or hispanic, which racial category is somlatge as to flip the average?
- alexbecker 8y ago
  This is Simpson's Paradox[0]--Iowa comes out on top because it has more white students, and white students in Iowa do better than non-white students in Texas. [0] https://en.wikipedia.org/wiki/Simpson%27s_paradox https://en.wikipedia.org/wiki/Simpson%27s_paradox
- kolbe 8y ago
  Example using two races and just ACT scores: Texas: 50% Hispanic who average 25 50% white who average 29 State average: 27 Iowa: 10% Hispanic who average 24 90% white who average 28 State average: 27.6
  - lifeisstillgood 8y ago
    texas has 50 % hispanic population ? https://en.m.wikipedia.org/wiki/Demographics_of_Texas https://en.m.wikipedia.org/wiki/Demographics_of_Texas It has 84% white population, not noticeably different to UK makeup at 87% Iowa has 88% White (https://en.m.wikipedia.org/wiki/Iowa https://en.m.wikipedia.org/wiki/Iowa) or maybe 91 not too clear what they mean as difference between white and not hispanic So them is some pretty small numbers to play with - somehow the "whites do better, and iowa has more whites" argument has only got 4-7% to play with I would like to see the actual figures but it smells to me
    - the_watcher 8y ago
      The numbers used in the parent were for illustration only, not necessarily the actuals. Also, student population != general population.
ftwynn 8y ago
The efficiency measure sure seems dangerously misleading. My SC high school didn't have the budget to replace broken tables (which were simply overturned until a handy-ish student managed to unsteadily prop them up) and sent seniors home home early via free periods to cover for a lack of teachers. Was that efficient? I suppose. Was it a desirable education? Not really.
- fjsolwmv 8y ago
  The argument goes that if those cutbacks didn't harm outcomes, they were luxuries not necessary expenses.
abathur 8y ago
I don't have enough familiarity with this sort of research to evaluate it quickly, but I was a little curious about the authors' backgrounds: > Stan J. Liebowitz is the Ashbel Smith Professor of Economics at the Colloquium for the Advancement of Free Enterprise Education in the Jindal School of Management at the University of Texas at Dallas. > Matthew L. Kelly is a research fellow at the Colloquium for the Advancement of Free Enterprise Education in the Jindal School of Management at the University of Texas at Dallas. Here's another piece by them on net neutrality: https://www.insidesources.com/state-governments-drop-net-neutrality/ https://www.insidesources.com/state-governments-drop-net-neu... Here's the site for the "Colloquium for the Advancement of Free Enterprise Education": https://jindal.utdallas.edu/centers-of-excellence/capri/cafe/ https://jindal.utdallas.edu/centers-of-excellence/capri/cafe... Here's a press release about the founding of this entity, which includes some information on where the funding for it comes from: https://jindal.utdallas.edu/news/new-program-at-the-jindal-school-advances-freeenterprise-education https://jindal.utdallas.edu/news/new-program-at-the-jindal-s...
- analog31 8y ago
  "You tell me whar a man gits his corn pone, en I'll tell you what his 'pinions is." -- Mark Twain http://www.paulgraham.com/cornpone.html http://www.paulgraham.com/cornpone.html
  - glup 8y ago
    Here's the some specifics on this particular corn pone: Reason Foundation is a 501(c)(3) nonprofit organization supported by donations and sale of its publications.[15] Its largest donors are the David H. Koch Charitable Foundation ($1,522,212) and the Sarah Scaife Foundation ($2,016,000), according to disclosures.[16] Other major donors are Donors Trust and Donors Capitol Fund, which in turn do not reveal their donors.[17] The Reason Foundation is part of the libertarian Atlas Network,[17][18], the State Policy Network and ALEC.[17] https://en.wikipedia.org/wiki/Reason_(magazine) https://en.wikipedia.org/wiki/Reason_(magazine) Gotta hand it to them, pretty brilliant to call anti-regulatory extremism, antipathy to public institutions, and outright rejection of structural inequality "reason." Guess I am unreasonable.
exabrial 8y ago
This is a headline that raises an eyebrow from me because it seems me odd someone would believe _they are not_ gamed in some manner.
nkassis 8y ago
I'm wondering if there are confounding variables that are affecting this result. For example New York and New Jersey are some of the most regressive states defined as difference in funding levels between rich and poor school districts. I found this chart which shows each states: http://apps.urban.org/features/school-funding-do-poor-kids-get-fair-share/ http://apps.urban.org/features/school-funding-do-poor-kids-g... If race is a mediator for income levels and in turn funding level shouldn't the researchers here include the difference in funding levels between income levels into their model to determine the true performance per spent dollar? I might be wrong as I'm no expert in the field but I think these type of problems are what causal models try to address. I've really enjoyed "The book of why" by Judea Pearl on the topic. It got me interested in learning more about causality.
saitoyeung 8y ago
This is interesting. What I like about it is that it tries to remove any criteria unrelated to a student's outcomes. Too often we see esoteric ranking methodologies that, unfortunately, not many people bother to look at very closely. I'm the founder of PolarisList (https://www.polarislist.com https://www.polarislist.com), a high school ranking based on the number of students sent to Harvard, Princeton, and MIT. This prompted us to take a stab at generating state rankings based on our own dataset, and came up with the following list. We calculated this by looking at the number of students in a state who matriculated to the aforementioned colleges from public high schools divided by the estimated 2017 population: 1 Massachusetts 2 New Jersey 3 Connecticut 4 New York 5 Vermont 6 Maryland 7 Maine 8 New Hampshire 9 California 10 Virginia 11 Alaska 12 Rhode Island 13 Illinois 14 Delaware 15 Pennsylvania 16 Colorado 17 Minnesota 18 Washington 19 Michigan 20 Oregon 21 Florida 22 Georgia 23 Nebraska 24 North Dakota 25 South Dakota 26 Texas 27 Montana 28 Arizona 29 Wisconsin 30 Missouri 31 Idaho 32 West Virginia 33 Indiana 34 North Carolina 35 New Mexico 36 Ohio 37 Kansas 38 Iowa 39 Hawaii 40 Nevada 41 Kentucky 42 Wyoming 43 Oklahoma 44 Utah 45 Tennessee 46 Arkansas 47 South Carolina 48 Washington DC 49 Louisiana 50 Mississippi 51 Alabama A couple thing stand out to me: - All 3 rankings have Massachusetts and New Jersey within the top 10 - All 3 rankings have Oklahoma and Louisiana in the bottom 10 - Our numbers differ from the other datasets the most on: - New York (#4 on our ranking, #30 on Liebowitz/Kelly, #31 on US News) - California (#9 on our ranking, #34 on Liebowitz/Kelly, #44 on US News) - Alaska (#11 on our ranking, #42 on Liebowitz/Kelly, #46 on US News)
- masonic 8y ago
  A metric showing that students going to northeastern universities mostly come from northeastern states isn't very meaningful. Given that it counts only students attending those schools and not all of those accepted by those schools (e.g. the full population of those meeting the actual qualification standard) and doesn't account for ability to pay renders it even less meaningful.
  - dorchadas 8y ago
    It's really a horrible flawed metric. It doesn't mention background of students, nor the connections they might have made in school/life that help them get to those places. It also doesn't take into account students who don't want to necessarily go there (I was valedictorian and had a 4.0, 36 on ACT...I didn't apply Ivy because I didn't want to go there), etc. It's really just a "Hey, these schools have more kids here" ranking, saying nothing about quality of the schools themselves.
Floegipoky 8y ago
This may be unpopular but I really do think that spending should be included in a ranking system. There are a lot of aspects of education that standardized testing can't (or won't) measure: access to extra-curricular programs, funding for art/music departments, technical education, etc. I'm willing to bet a lot of people here have experienced first-hand the difference that having access to a decent computer lab can make. You're not going to see that reflected in tests. Also, based on my own anecdotal experience of standardized testing in grade school, these tests rarely include questions about "controversial" topics in history or scientific ideas that are opposed by Christian fundamentalism. I suspect these are major areas where southern districts would struggle, given their disproportionate implementation of cough alternative textbooks. Overall, this smells of a common trend in American institutions where there's an ideological interest at play. An issue is raised, and it's decided that we need more data to decide. The data collection (in this case standardized testing) is designed in such a way that it produces results that skew toward the ideological position. Then the bad data is used to justify the ideological position. Law enforcement works the same way. Actually now that I'm thinking about it so do our elections.
- amadeuspagel 8y ago
  If the purpose of a ranking system is to help decide questions like how much should be spent on education, then obviously it doesn't make sense to include spending in the ranking. (And this is not a strawman, the article mentions that Paul Krugman for example smartly pointed out that states that spend more have better rankings.) > There are a lot of aspects of education that standardized testing can't (or won't) measure: access to extra-curricular programs, funding for art/music departments, technical education, etc. > these tests rarely include questions about "controversial" topics in history or scientific ideas that are opposed by Christian fundamentalism. These are legitimate criticisms of standardized testings, but not excuses for ignoring its results, if you don't have better data available. > Then the bad data is used to justify the ideological position. Law enforcement works the same way. Maybe instead of using "bad data", like how many crimes are solved, on law enforcement, we should just ask how much is spent on it - the more the better?

116 points by michael_fine 8y ago

I don't really understand removal of graduation rates. Obviously that's meant to act as a proxy for learning, especially for those 38 states without proficiency exams.

bigpicture 8y ago
Right, surely these states have some sort of standard that must be met for graduation even if it is not passing some exam.
- saudioger 8y ago
  the standards for graduation vary wildly so isn't really comparable
  - bigpicture 8y ago
    That's a good point
grandmczeb 8y ago
The standard to graduate isn’t uniform, so using graduation rates biases towards states with weak requirements.
ergothus 8y ago
> Obviously that's meant to act as a proxy for learning That is the INTENT, sure. But does it work? If you have someone that graduated from a school A and someone that failed to graduate from school B that is in an entirely different area, can you draw any meaningful conclusions? Particularly knowing that school A has no particular requirements on why you graduate, while school B does? I'm far more likely to complain about "But when we disaggregate student performance scores by racial categories (white, black, Hispanic, and Asian), the rankings change dramatically." I wouldn't want to go too far with that before ensuring that I wasn't trying to curb swimming deaths by keeping Nick Cage out of movies: http://tylervigen.com/view_correlation?id=359 http://tylervigen.com/view_correlation?id=359
- kolbe 8y ago
  The data is pretty useful in penalizing states that don't educate their students well enough to even get to the standardized testing stage.
btilly 8y ago
First a point about bias. They are quick to point out the bias of traditional scores. But they do not point out that their bias is for libertarian causes. Had they not found that more libertarian states perform well by their measure, they would never have published this. Removing graduation rates is definitely troublesome. Any school system could then improve its numbers by encouraging the worst students to drop out. Problem students both take more resources to educate and produce worse scores. That's bad for efficiency, but it is not obviously bad for society. I am also troubled that they are using measures of performance in grades 4 and 8, but not something like SAT scores. I understand that SAT scores could be hard to get. But a lot of what people care about in an education system is how well prepared you are for college, and not how far along you were in grade 8.
- the_watcher 8y ago
  > But they do not point out that their bias is for libertarian causes. Had they not found that more libertarian states perform well by their measure, they would never have published this. The first statement is true. The piece would be a bit stronger if the researchers noted that by publishing in Reason, they're at the very least giving the appearance of potential confirmation bias. Your second statement makes an unsupported assumption though. RE: SAT scores - not all students take the SAT, some students take it multiple times, the SAT is taken at different points in time per student, and it's a test that is so influential in college admissions that success on it is strongly correlated with wealth, as scores are best improved by tutoring on test taking strategy, not increasing general knowledge. I agree that grade 4 and 8 are probably not optimal to get a complete picture of education quality, but there is no reward to the student for doing well, so they're less likely to have spent substantial time investing in beating the test.

Loughla 8y ago

There is no reason given to why they take per-student expense out of the picture, other than 'we just don't think it should count'. And that seems to be the entire basis of their finding that rankings are riddled with flaws. There is much research (once you sort through the partisan fluff from research institutions with obvious bias) that supports the positive impact of revenue per student on academic achievement on standardized tests. It's actually funny this article popped up, I legit just read a paper from 2015 called "A Cost-Benefit Analysis for Per-Student Expenditures and Academic Achievement". I genuinely read that this morning. Weird. The authors found that "There was a significant correlation between revenues available per student and ACT scores as one outcome measure of achievement." And just to drive that point home, they replicated the findings from a 2002 study, further solidifying that sentiment. And that's just the most recent one I've read. I'm sure there are more recent. And that is definitely just one in the series of research related to per-pupil expenditures. Also, I'm afraid this piece serves no purpose other than to be self-congratulatory to the 'lower taxes at all costs' group and right-to-work proponents. Why I say this: research pieces probably shouldn't include snide comments like "high-tax, high-spending progressive utopias." "punishing taxes" Maybe that's just me? Am I off base? edit: also, the comment section on that article is awful. Just awful.

kolbe 8y ago
>There is no reason given to why they take per-student expense out of the picture What do you think would happen to the regression if per-student expense was both an independent and a dependent variable?
- Loughla 8y ago
  My experience with educational research is that researchers find per-pupil expense is directly correlated to improved achievement on standardized academic measures across all areas and for all variables in race/ethnicity, gender, age, and (obviously) socioeconomic status. I don't know what variables the authors are talking about when they say "we ran multiple regression analyses on our data, which included several other variables", so I hesitate to make a claim on what their outcomes could be. There are so many things they could be looking at; it's almost impossible to know what they're really evaluating without being able to look at the data.

patorjk 8y ago

I only have 2 data points of experience on this issue. Which I know doesn't mean much, but I'm left scratching my head after looking at their list. I spent the first 14 years of my life in Las Cruces, New Mexico (the state's second biggest city), and the schools I went to were not very good. I did high school in Maryland and the school system was a lot better (I initially struggled because I was so far behind everyone else - a friend of mine who moved to North Carolina for High School told me he had the same problem). In the author's ranking list, they place Maine at #48 and New Mexico at #41. I find this very hard to believe. There are so many problems that New Mexico has that Maine doesn't seem to have (gang problems, drug problems - at least at the magnitude that I saw in New Mexico). Unless Maine has some pretty bad schools, this just doesn't add up. US News lists Maine at #6, a ranking drop of 42 spots seems really significant.

kolbe 8y ago
I suggest you read the article if you're curious how they created their list. They explain their methodology pretty thoroughly.
ericcumbee 8y ago
I think part of it is the variability between school systems with in the same state. Las Cruces schools maybe below average for NM schools. Off Topic: I went to Las Cruces a few years ago when my alma mater played New Mexico State. The scenery was breathtaking.
- patorjk 8y ago
  Even though my comment was kind of negative, I love Las Cruses. The scenery and weather are the two things I miss the most. It's certainly possible the rest of the state has better schools, but I never heard anything about them while I lived there.
CBLT 8y ago
I'm confused by your comment, would you care to elaborate? Your anecdote has you moving from (the article's) rank 41 in quality to rank 6. You comment that the rank 6 state is much better than the rank 41 in education quality. However, you find their ranking hard to believe?

temp-dude-87844 8y ago

Disaggregating the results by race is real cute but also misses the point: blacks and asians and hispanics are residents of the state too and so leaving them in to evaluate the state's overall achievement (relative to other states) in education is fair. If anything it ought to be more problematic that Texas fails so badly at educating hispanic students when a fifth of them live in Texas. Besides, the race in big-gap states is an unwitting proxy to income, so disaggregate by household per capita income (relative to local cost of living) if you want to disdain. This methodology would work well in all states and expose the challenges of anxiety and the quality of a student's home life in their education attainment.

theparanoid 8y ago
> race here is an an unwitting proxy to income, so disaggregate by household per capita income Poor white students do better academically than well off black students. So splitting by race is more informative when discussing academic performance.
- purple-again 8y ago
  Do you have some evidence to back that up? There are some dumb rich people, no doubt of every color, however there is a clear correlation between income and performance in every reputable source I’ve ever seen. I find it hard to believe that on average rich black kids are doing worse than on average poor white kids.
  - grandmczeb 8y ago
    Here's a study[1] that breaks down SAT scores by race and income in 2003. Take a look at table 2. [1] https://pdfs.semanticscholar.org/e7a0/dfdee8ad5b91e1d179e9e2e838a4952f00be.pdf https://pdfs.semanticscholar.org/e7a0/dfdee8ad5b91e1d179e9e2...
    - ckemere 8y ago
      I believe that study shows that the affect of race on SAT scores when a more complex model of income and parental education is taken into account is halved (i.e., the observed 100 pt difference drops to 50). So I think it still doesn't excuse the original post from taking income into account.

golemiprague 8y ago

Education depends on population, their average IQ and their learning culture, everybody knows it and that's why the main consideration when people with kids buy a house is the population around it. No money or learning methods or good teachers is going to change this basic truth. It doesn't matter in which state you live, what matters is how much money you got to get to the right neighbourhood or the right private school.

dsfyu404ed 8y ago

I have several thoughts on this: The race adjusted performance is interesting and useful but they're basically using it as a proxy for wealth. Why can't they just use some metric that directly represents wealth? If Massachusetts schools are efficiency with money than the system nation wide is far more screwed up than anyone is willing to admit. That state ranking depends in part on money spent is abhorrent. That should be tracked for reasons that should be obvious but not part of a composite score meant to represents results. The bit about unions lines up with my anecdotal experience. I worked with the education department in college and they did not hold teachers unions in high regard. If your unions are so bad that college professors at a state school in a blue state gripe about how they hold back progress then I think it's fair to say your union is pretty bad.

rayiner 8y ago
They're not using race as a proxy for wealth, because the two things are different. There are unique challenges faced by minorities compared to white americans of the same income level. (Remember, Trayvon Martin was a middle class black kid whose dad was a white-collar government worker. A big part of #blacklivesmatter is educating people that black folks face unique problems that aren't subsumed within the larger issue of income inequality. And as to Hispanic students the challenges there are obvious--they're much more likely to be in families that are English as a second language, and also much more likely to move around a lot.)
- dan-robertson 8y ago
  I’m not convinced that ESL families is the main difficulty facing Hispanic children. I’ll accept that there are disadvantages to Hispanics but having ESL parents seems similar to those faced by almost any other child of immigrants. Perhaps one could argue that due to the visa system, non-Hispanic parents of Americans tend to have higher earning potential? I have no idea whether this is true or not
  - rayiner 8y ago
    Due to the visa system, immigrants in the U.S. are not identically situated to each other. Non-Hispanic immigrants are much more likely to already speak English and have advanced degrees because they come here through visa categories that specifically select for those things.

KaoruAoiShiho 8y ago

Is there the per-race data available somewhere?

rb808 8y ago

Ranking is nearly meaningless. eg compare these: School A - full of underachieving kids that are pushed to get avg grades School B - full of avg kids getting above avg grades School C - full of smart kids getting top grades School D - full of smart kids getting nearly top grades, but soft skills, nice personalities and VIP friends School E - Mix of kids getting relevant grades Which is the best school for each kid type? They're impossible to rank. Many schools with top grades are only there because they kick out failing kids, not because they're better at teaching. Many schools are rated good because they can improve grades, but you wouldn't want to send a smart kid there. Many private schools sell themselves on creating well rounded kids, not necessarily good grades.

ben_jones 8y ago
I went to school F - full of average kids who knew how to play the system to get top grades, as well as to get into high/elite level colleges.
darawk 8y ago
Yes, but all the different 'types of kids' should normalize across a state-level population.
educationdata 8y ago
The article is about ranking states, not ranking schools. For the point of ranking schools, there are statistical methods to measure student growth, instead of achievement, such as SGP (student growth percentile) and VAM (value-added modeling), which kind of address part of your question.

lifeisstillgood 8y ago

Is it just me that has a hard time understanding the racial aggregation thing. They claim to have split Iowa and Texas results by race and age. And found that Texas did better than Iowa for whites blacks and hispanics - but then aggregating the results ala USNews, Iowa comes out on top I am struggling to imagine which category is the flip category. If iowa is not doing better for white black or hispanic, which racial category is somlatge as to flip the average?

alexbecker 8y ago
This is Simpson's Paradox[0]--Iowa comes out on top because it has more white students, and white students in Iowa do better than non-white students in Texas. [0] https://en.wikipedia.org/wiki/Simpson%27s_paradox https://en.wikipedia.org/wiki/Simpson%27s_paradox
kolbe 8y ago
Example using two races and just ACT scores: Texas: 50% Hispanic who average 25 50% white who average 29 State average: 27 Iowa: 10% Hispanic who average 24 90% white who average 28 State average: 27.6
- lifeisstillgood 8y ago
  texas has 50 % hispanic population ? https://en.m.wikipedia.org/wiki/Demographics_of_Texas https://en.m.wikipedia.org/wiki/Demographics_of_Texas It has 84% white population, not noticeably different to UK makeup at 87% Iowa has 88% White (https://en.m.wikipedia.org/wiki/Iowa https://en.m.wikipedia.org/wiki/Iowa) or maybe 91 not too clear what they mean as difference between white and not hispanic So them is some pretty small numbers to play with - somehow the "whites do better, and iowa has more whites" argument has only got 4-7% to play with I would like to see the actual figures but it smells to me
  - the_watcher 8y ago
    The numbers used in the parent were for illustration only, not necessarily the actuals. Also, student population != general population.

ftwynn 8y ago

The efficiency measure sure seems dangerously misleading. My SC high school didn't have the budget to replace broken tables (which were simply overturned until a handy-ish student managed to unsteadily prop them up) and sent seniors home home early via free periods to cover for a lack of teachers. Was that efficient? I suppose. Was it a desirable education? Not really.

fjsolwmv 8y ago
The argument goes that if those cutbacks didn't harm outcomes, they were luxuries not necessary expenses.

abathur 8y ago

I don't have enough familiarity with this sort of research to evaluate it quickly, but I was a little curious about the authors' backgrounds: > Stan J. Liebowitz is the Ashbel Smith Professor of Economics at the Colloquium for the Advancement of Free Enterprise Education in the Jindal School of Management at the University of Texas at Dallas. > Matthew L. Kelly is a research fellow at the Colloquium for the Advancement of Free Enterprise Education in the Jindal School of Management at the University of Texas at Dallas. Here's another piece by them on net neutrality: https://www.insidesources.com/state-governments-drop-net-neutrality/ https://www.insidesources.com/state-governments-drop-net-neu... Here's the site for the "Colloquium for the Advancement of Free Enterprise Education": https://jindal.utdallas.edu/centers-of-excellence/capri/cafe/ https://jindal.utdallas.edu/centers-of-excellence/capri/cafe... Here's a press release about the founding of this entity, which includes some information on where the funding for it comes from: https://jindal.utdallas.edu/news/new-program-at-the-jindal-school-advances-freeenterprise-education https://jindal.utdallas.edu/news/new-program-at-the-jindal-s...

analog31 8y ago
"You tell me whar a man gits his corn pone, en I'll tell you what his 'pinions is." -- Mark Twain http://www.paulgraham.com/cornpone.html http://www.paulgraham.com/cornpone.html
- glup 8y ago
  Here's the some specifics on this particular corn pone: Reason Foundation is a 501(c)(3) nonprofit organization supported by donations and sale of its publications.[15] Its largest donors are the David H. Koch Charitable Foundation ($1,522,212) and the Sarah Scaife Foundation ($2,016,000), according to disclosures.[16] Other major donors are Donors Trust and Donors Capitol Fund, which in turn do not reveal their donors.[17] The Reason Foundation is part of the libertarian Atlas Network,[17][18], the State Policy Network and ALEC.[17] https://en.wikipedia.org/wiki/Reason_(magazine) https://en.wikipedia.org/wiki/Reason_(magazine) Gotta hand it to them, pretty brilliant to call anti-regulatory extremism, antipathy to public institutions, and outright rejection of structural inequality "reason." Guess I am unreasonable.

exabrial 8y ago

This is a headline that raises an eyebrow from me because it seems me odd someone would believe _they are not_ gamed in some manner.

nkassis 8y ago

I'm wondering if there are confounding variables that are affecting this result. For example New York and New Jersey are some of the most regressive states defined as difference in funding levels between rich and poor school districts. I found this chart which shows each states: http://apps.urban.org/features/school-funding-do-poor-kids-get-fair-share/ http://apps.urban.org/features/school-funding-do-poor-kids-g... If race is a mediator for income levels and in turn funding level shouldn't the researchers here include the difference in funding levels between income levels into their model to determine the true performance per spent dollar? I might be wrong as I'm no expert in the field but I think these type of problems are what causal models try to address. I've really enjoyed "The book of why" by Judea Pearl on the topic. It got me interested in learning more about causality.

saitoyeung 8y ago

This is interesting. What I like about it is that it tries to remove any criteria unrelated to a student's outcomes. Too often we see esoteric ranking methodologies that, unfortunately, not many people bother to look at very closely. I'm the founder of PolarisList (https://www.polarislist.com https://www.polarislist.com), a high school ranking based on the number of students sent to Harvard, Princeton, and MIT. This prompted us to take a stab at generating state rankings based on our own dataset, and came up with the following list. We calculated this by looking at the number of students in a state who matriculated to the aforementioned colleges from public high schools divided by the estimated 2017 population: 1 Massachusetts 2 New Jersey 3 Connecticut 4 New York 5 Vermont 6 Maryland 7 Maine 8 New Hampshire 9 California 10 Virginia 11 Alaska 12 Rhode Island 13 Illinois 14 Delaware 15 Pennsylvania 16 Colorado 17 Minnesota 18 Washington 19 Michigan 20 Oregon 21 Florida 22 Georgia 23 Nebraska 24 North Dakota 25 South Dakota 26 Texas 27 Montana 28 Arizona 29 Wisconsin 30 Missouri 31 Idaho 32 West Virginia 33 Indiana 34 North Carolina 35 New Mexico 36 Ohio 37 Kansas 38 Iowa 39 Hawaii 40 Nevada 41 Kentucky 42 Wyoming 43 Oklahoma 44 Utah 45 Tennessee 46 Arkansas 47 South Carolina 48 Washington DC 49 Louisiana 50 Mississippi 51 Alabama A couple thing stand out to me: - All 3 rankings have Massachusetts and New Jersey within the top 10 - All 3 rankings have Oklahoma and Louisiana in the bottom 10 - Our numbers differ from the other datasets the most on: - New York (#4 on our ranking, #30 on Liebowitz/Kelly, #31 on US News) - California (#9 on our ranking, #34 on Liebowitz/Kelly, #44 on US News) - Alaska (#11 on our ranking, #42 on Liebowitz/Kelly, #46 on US News)

masonic 8y ago
A metric showing that students going to northeastern universities mostly come from northeastern states isn't very meaningful. Given that it counts only students attending those schools and not all of those accepted by those schools (e.g. the full population of those meeting the actual qualification standard) and doesn't account for ability to pay renders it even less meaningful.
- dorchadas 8y ago
  It's really a horrible flawed metric. It doesn't mention background of students, nor the connections they might have made in school/life that help them get to those places. It also doesn't take into account students who don't want to necessarily go there (I was valedictorian and had a 4.0, 36 on ACT...I didn't apply Ivy because I didn't want to go there), etc. It's really just a "Hey, these schools have more kids here" ranking, saying nothing about quality of the schools themselves.

Floegipoky 8y ago

This may be unpopular but I really do think that spending should be included in a ranking system. There are a lot of aspects of education that standardized testing can't (or won't) measure: access to extra-curricular programs, funding for art/music departments, technical education, etc. I'm willing to bet a lot of people here have experienced first-hand the difference that having access to a decent computer lab can make. You're not going to see that reflected in tests. Also, based on my own anecdotal experience of standardized testing in grade school, these tests rarely include questions about "controversial" topics in history or scientific ideas that are opposed by Christian fundamentalism. I suspect these are major areas where southern districts would struggle, given their disproportionate implementation of cough alternative textbooks. Overall, this smells of a common trend in American institutions where there's an ideological interest at play. An issue is raised, and it's decided that we need more data to decide. The data collection (in this case standardized testing) is designed in such a way that it produces results that skew toward the ideological position. Then the bad data is used to justify the ideological position. Law enforcement works the same way. Actually now that I'm thinking about it so do our elections.

amadeuspagel 8y ago
If the purpose of a ranking system is to help decide questions like how much should be spent on education, then obviously it doesn't make sense to include spending in the ranking. (And this is not a strawman, the article mentions that Paul Krugman for example smartly pointed out that states that spend more have better rankings.) > There are a lot of aspects of education that standardized testing can't (or won't) measure: access to extra-curricular programs, funding for art/music departments, technical education, etc. > these tests rarely include questions about "controversial" topics in history or scientific ideas that are opposed by Christian fundamentalism. These are legitimate criticisms of standardized testings, but not excuses for ignoring its results, if you don't have better data available. > Then the bad data is used to justify the ideological position. Law enforcement works the same way. Maybe instead of using "bad data", like how many crimes are solved, on law enforcement, we should just ask how much is spent on it - the more the better?