15 ms·

Per the article this came in response to questions raised by a Columbia faculty member critiquing the university's data. I thought that report[1] was actually m

Per the article this came in response to questions raised by a Columbia faculty member critiquing the university's data. I thought that report[1] was actually more interesting than the original article. As per the quote at the top of the report: > Rankings create powerful incentives to manipulate data and distort institutional behavior for the sole or primary purpose of inflating one’s score... It would be shocking to me if Columbia was the only institution that fudged its numbers a bit in pursuit of a higher US News ranking. Given how important that ranking has become in US culture, I wonder if it's time for the methodology to change to require audited/auditable metrics rather than continuing to allow self-reporting. 1: http://www.math.columbia.edu/~thaddeus/ranking/investigation.html http://www.math.columbia.edu/~thaddeus/ranking/investigation...

deleted 4y ago
[deleted]
hackyhacky 4y ago
Your suspicion is correct. I've taught at several US universities, and it's an open secret that the administrative takes steps to manipulate numbers in order to secure better rankings. Typical example: a small, prestigious liberal arts college always has smaller class sizes in the fall semester than in the spring semester. Why? Because classes with less than 30 students are ranked higher by US News, but they only take into consideration the fall semester. No one cares about class sizes in the spring.
- CogitoCogito 4y ago
  > Because classes with less than 30 students are ranked higher by US News, but they only take into consideration the fall semester. No one cares about class sizes in the spring. What a bizarre and totally unnecessary restriction.
  - genewitch 4y ago
    Parents helping/guiding their near-adult children in the final spring of highschool to apply at or choose universities, where they would start university that same fall seem to be the target audience, here. Fall is when "most" incoming freshmen will start.
  - unosama 4y ago
    You add a simple and reasonable (until it gets abused) assumption to your model which allows a lesser data collection burden on thousands of universities. I wouldn't call it a restriction.
    - thaumasiotes 4y ago
      > which allows a lesser data collection burden on thousands of universities. How's it supposed to do that? Universities all already collect the data on enrollment for every class they offer.
      - fshbbdssbbgdd 4y ago
        Spoken like a software engineer! Now imagine every form is being filled out manually by an administrator with a calculator.
        pc86 4y ago
        But it's not.
        deleted 4y ago
        [deleted]
        mwint 4y ago
        Maybe the forms should be made too complex to be filled with a calculator, to encourage doing things the right way…
        geraldwhen 4y ago
        Wouldn’t work. Entire state budgets are accounted for by spreadsheets without formulas, emailed between people in an office with no revision control. A few years ago when 50 million went missing, I wasn’t surprised. I’d seen the budgeting process, and it was worse than you could possibly imagine. There was an attempt to go the software route, but a nepotism deal left the budgeting department with a worse system than manual spreadsheet entry.
  - deleted 4y ago
    [deleted]
- bogota 4y ago
  Great to know many peoples graduations were delayed so someone could game some numbers
  - tenpies 4y ago
    What are they going to do? Transfer out and have only half their credits recognized therefore pushing them back 4 semesters instead of the 1? It's a fantastically anti-consumer market they've managed to create.
  - abakker 4y ago
    Gotta pay for another semester then, right?
  - zerocrates 4y ago
    I think they mean the size of individual classes, so they're paying for more sections/instructors/instruction time in the ranking-relevant periods. Though shrinking the admit size is another well worn tactic, as it's the easiest way to increase how "selective" you are which is another major ranking element, plus it helps you more cheaply achieve smaller average class sizes, though of course you're forgoing some revenue to get there.
    - btilly 4y ago
      Encouraging people to apply is also a common tactic. You can't reject them if they don't apply!
      - javajosh 4y ago
        Applications are also a revenue source.
        dhosek 4y ago
        Apparently there’s a common application thing now where you can apply to large numbers of schools for undergrad without spending large amounts of money.
- nostromo 4y ago
  Gaming and manipulation should absolutely be addressed, but it seems Columbia was just straight up lying about their numbers, not gaming them.
- lumost 4y ago
  Why is small class size a good thing? Anecdotally, you get better class material and TA staff in larger lectures. Why aren't Universities able to innovate here?
  - bbor 4y ago
    Many courses rely on discussion!
  - aquova 4y ago
    Fewer students per teacher is widely seen as a positive thing. A greater percentage of the professor's time and attention can be given to any one student. I've definitely experienced this myself, it's easy to feel like just a number when you're in a large lecture hall of hundreds versus a small classroom of a few dozen.
    - bachmeier 4y ago
      I already said this elsewhere, but I do want to reply to this comment, because it demonstrates a common mistake that may not be clear from my other comment. If you have a given teacher, in a given classroom, with a given amount of TA help, blah blah, so that all else is equal, and then you reduce the class size, the students will do better. The problem is that all else is far from equal in the data used in these rankings. Instructor quality drops off rapidly as you have more sections. TA help falls. The classroom may no longer be appropriate for the class, etc. The rankings are not able to capture the effect you're talking about.
  - ghaff 4y ago
    For liberal arts? It's probably a pretty interactive format and smaller interactive classes are better than large classes that are more broadcast. (Often. I've also had great lecturers in the humanities.) But mostly don't look at many liberal arts classes (or case study, etc.) through the lens of an engineering lecture.
  - poulsbohemian 4y ago
    >Anecdotally, you get better class material and TA staff in larger lectures. Where is this anecdote coming from? A billion years ago when I studied at a small liberal arts college with small class sizes, professors were keen to share with us exactly why they were teaching from particular texts or why they had selected certain materials to make it clear that while the class wasn't being taught by that famous guy from MIT, it was still his book and materials we were using. I knew all my professors by first name, knew where many of them lived, had access to them anytime it was needed, and that's directly counter to my partner's experience at a large university.
- greymalik 4y ago
  Goodhardt’s law in action.
  - adolph 4y ago
    Goodhart's Law The key problem of using a measure as a metric is sometimes referred to as Goodhart’s law. The law is based on a 1975 paper on economic regulation, and is typically paraphrased as “When a measure becomes a metric, it ceases to be a good measure.” When managing a system, this problem is critical, but is deeper than it at first seems, as I’ll argue. https://www.ribbonfarm.com/2016/06/09/goodharts-law-and-why-measurement-is-hard/ https://www.ribbonfarm.com/2016/06/09/goodharts-law-and-why-...
  - dccoolgai 4y ago
    Ctrl+f for "Goodhar" right away to see who called it first +1
- pishpash 4y ago
  What's that adage? Every metric that becomes an optimization target ...
- eduction 4y ago
  I attended a well known and regarded public university. I was admitted for spring, and for the preceding fall entered a program where I could live in a regular university dorm and take classes for full credit taught at the same university’s extension, but special extension classes just for this program (so all spring admit students). Then you stay in the dorm in spring as you become a “regular” student. I always suspected it was a way to fudge numbers, but at the same time I knew I was genuinely on the bubble of being admitted so I took the opportunity gladly. (My memory is that the other spring admits were, like me, from racial groups considered “over represented” so they may have been trying to affect diversity numbers too.)
  - smugma 4y ago
    Berkeley has been doing this for at least 35 years.
    - tobinfricke 4y ago
      Yup, I got into Berkeley EECS this way back in the late 20th century. I'd guess it's legitimate load-balancing rather than an attempt to game the ratings. I assumed this meant that my application was borderline and I was grateful to be able to sneak in. The Extension program is a nice transition to college too.
      - etempleton 4y ago
        Load balancing is the most likely reason. At any given college a lot of first year students decide college isn't for them or they get home sick and drop out within the first semester. After that initial period student population of the class is much more stable.
- bredren 4y ago
  I TA'd in the attendance office at my high school back in the day. The city's tracking of student attendance only lasted a portion of the year or a portion of each quarter. During those times, much more effort went into calling the homes of missing students and finding out why they were not showing up to class. I remember asking about why all the calls were happening or not happening and it was sort of quietly explained by the staffer: measurement of attendance affected how the school was being evaluated by the district. Presumably, budget was hooked to it in some way. I didn't make any waves about it because "working" in this office was super powerful and the staffer always brought in cookies for us. Amazing gig.
  - smugma 4y ago
    In California, schools get paid on daily attendance. I believe it would be considered fraud to not take daily attendance. They suspended this during the heart of the pandemic but it came back last fall. Because daily attendance is down relative to pre pandemic, Public schools are facing a significant revenue gap (5-10% is typical)
    - LeifCarrotson 4y ago
      In contrast, Michigan Public K-12 schools take attendance on specified Count Days to account for the number of students they serve to determine state funding. Fall count day is the 1st Wednesday in October and represents 90% of state funding. Spring count day is the 2nd Wednesday in February and represents 10% of state funding. As you might imagine, getting students in school on these days is a Big Deal. Teachers take attendance on other days for internal use, but Count Day is special and you must not be absent!
      - MaysonL 4y ago
        How often have Count Days been snow days?
        LeifCarrotson 4y ago
        It's almost never a snow day in early October, I only remember one February count day + snow day. They count on the next day: > If instruction is cancelled on count day due to conditions not within the control of school authorities, with the approval of the State Superintendent, the affected instructional programs must use the immediately following day on which the district resumes session for count purposes per the rules at: https://www.michigan.gov/-/media/Project/Websites/mde/Year/2019/02/14/Student_Count_Information.pdf?rev=c14d2498c4b34d2cb519fb9e7acc1bdc https://www.michigan.gov/-/media/Project/Websites/mde/Year/2... I do recall my cousins at a poor rural school in Barry county having a couple snow days on Count Wednesday, and then subsequently calling one on Thursday when it wasn't really required and another on Friday where it really wasn't required, so that counts were as high as possible on Monday. Their snow days were much more frequent due to bad roads and bad road maintenance, and their state funding was much more important compared to my wealthy district.
- Fomite 4y ago
  Similarly, at a large land grant university, we talk a lot about improving our rankings, but acknowledge that one of the major metrics (% of students not accepted) is something that's contrary to our mission. It was an open secret when I was in Boston that Harvard actively encourages people to "give it a shot" when they know full well they're not getting in to boost that metric.
  - joe5150 4y ago
    "Recruit to deny" [1] has been getting more attention but still not enough. 1: https://www.nytimes.com/2019/11/29/us/harvard-admissions-recruit-letter.html https://www.nytimes.com/2019/11/29/us/harvard-admissions-rec...
    - KMag 4y ago
      Michigan also seemed to waitlist me and several of the people I knew at MIT. If they know they're the backup school, and they think you're going to get into your first choice, it hurts their acceptance rate to accept you outright. Though, maybe it's just confirmation bias and ego. Maybe there was something in my application that gave Michigan pause but seemed fine to MIT. At that time, Michigan did ask which other schools to which you were applying. I'm not sure if that's still the case.
      - JohnBooty 4y ago
        I'm confused. How would Michigan know that they were your second choice?
        draz 4y ago
        They expect students of a certain caliber. If the applicant exceeds their expectations, the university assumes the student is using this application as a “safety school.”
        brewdad 4y ago
mbesto 4y ago
Which begs the question - if you want to get top in rankings wouldn't you simply design the school around the objective ranking parameters and nothing else? Ripe for abuse...
- nonameiguess 4y ago
  Not sure if this is still true, but at one point, the biggest factor in the ranking was peer reputation rating. That kind of results in a self-reinforcing vicious cycle of top schools remaining top schools, but short of bribery, there isn't really any way to manipulate that at least.
- colinmhayes 4y ago
  This is sort of what Columbia did… they juiced their faculty numbers by counting the med school which has much lower class sizes and more educated teachers.
nus07 4y ago
I have a friend who works at Duke University’s marketing analytics and performance metrics team. It’s filled with MBAs who’s job is to massage the numbers and metrics that are asked by US News every year and provide them to US News.
- noipv4 4y ago
  username checks out ;) you guya have a collab prog with duke
- onionisafruit 4y ago
  It's probably staffed with MBAs to boost the job placement numbers for their MBA program.
mtlmtlmtlmtl 4y ago
This is a good example of Campbell's Law at work: https://en.wikipedia.org/wiki/Campbell%27s_law https://en.wikipedia.org/wiki/Campbell%27s_law
- ethbr0 4y ago
  An obvious takeaway in this case: US News shouldn't be the primary ranker. If statistics were distributed in open format, transparently, and there were multiple rankers (each with their own weights and mixes), universities would be less incentivized to game "the US News metrics".
  - t_mann 4y ago
    That seems naive to assume. We already live in a world where there are multiple rankings, and many of them (even some of the most important ones, like QS or Shanghai) rely only on independently collected or public data. We already have what you suggested as a solution, and look at where we are. The problem is more fundamental.
  - humanistbot 4y ago
    It already exists and is a free US Dept of Education service. Schools have to report that data to the Dept of Ed already, so they have both an interactive search tool [1] and the raw data available [2]. [1] https://collegescorecard.ed.gov/search/?page=0&sort=threshold_earnings:desc&toggle=institutions https://collegescorecard.ed.gov/search/?page=0&sort=threshol... [2] https://collegescorecard.ed.gov/data https://collegescorecard.ed.gov/data
    - everybodyknows 4y ago
      Looking at [1] just now, it seems overwhelmingly oriented toward answering the question "How do I get in?". Scarcely at all toward "What can the school deliver to the student?" -- only relevant measure is Median Earnings. The fine print: >The median annual earnings of individuals that received federal student aid and began college at this institution 10 years ago, regardless of their completion status. So for the anxious parent, not a lot to go on.
twobitshifter 4y ago
It’s like doping in cycling. Everyone was getting caught but we’re supposed to believe the couple who snuck through weren’t like the rest. Once the tests got good enough there was hardly any one left standing. https://en.m.wikipedia.org/wiki/Doping_at_the_Tour_de_France https://en.m.wikipedia.org/wiki/Doping_at_the_Tour_de_France
gofreddygo 4y ago
Funny how this reminded me of a PG essay[1] The most damaging thing you learned in school wasn't something you learned in any specific class. It was learning to get good grades. [1]:http://paulgraham.com/lesson.html http://paulgraham.com/lesson.html
- bdowling 4y ago
  The problem with that quote is that if you don't learn to get good grades (e.g., because you are actually learning and not studying for the test), then the resulting bad grades can be extremely damaging to your future.
  - kragen 4y ago
    How is that a problem with the quote? It sounds like a problem with the system, a problem the essay correctly identified.
    - brewdad 4y ago
      Ok. Now fix the system in a timeframe that is compatible with your expected working life. That's the problem with the quote. Grades may not be the most important thing but they can, and often are, the most important thing for any given individual. For every Bill Gates that dropped out, there are a million John Doe's drowning in student loan debt with no degree and limited future prospects to show for it.
      - kragen 4y ago
        Who are you asking to fix the system, and why? Me, maybe? Paul Graham? gofreddygo? How is the thing you say a problem with the quote? These can both simultaneously be the case, and often are: 1. The most damaging thing John Doe learned in school was learning to get good grades. 2. Richard Roe didn't get good grades and is drowning in student loan debt with no degree and limited future prospects to show for it. You have offered no reason to believe that if (2) is more true, or true in more cases, (1) is less true, or true in less cases, or less important. In fact, this tighter conjunction can also often be true: 1'. The most damaging thing John Doe learned in school was learning to get good grades. 2'. John Doe didn't get good grades (despite being damaged by learning how, he didn't learn how enough, or other things interfered) and is drowning in student loan debt with no degree and limited future prospects to show for it. You've also offered no reason to believe that if (2') is more true, (1') is less true or less important. I have an idea! Maybe you're misinterpreting the quote as meaning that, if you're in school, it would be a good strategy to stay there but not learn to get good grades. In that case there would be a sort of conflict: by diminishing the damage from (1') you may be increasing the risk of (2'). But that isn't what the essay advises at all, and I'm puzzled as to how you could have read it as suggesting that. It explicitly advises students to do exactly the opposite; it advises them to get good grades by hacking bad tests in the usual ways (cramming before exams, etc.) because everyone will judge them by their grades. But that's not really the point of the essay, and it would be a pretty boring essay if it was. The point of the essay is its advice to people who aren't enrolled as students: unlearn how to get good grades, and instead of focusing on how to make your product look good, actually make a good product. Also, and maybe more importantly, try to work in fields of endeavor where the way to win is to do good work, rather than hacking bad tests like you do in school. (This might suggest not going into school at all, which would at least avoid the student loan debt problem.)
hardtke 4y ago
My kid just went through college selection process and I can assure you that the US News rankings are largely irrelevant in terms of decision making. The revealed preference score[1] or global university rankings[2] are much closer to how students, their counselors, and their parents rank the universities. For instance, any ranking that does not have Stanford first among non specialized US universities (i.e. excluding MIT, Caltech, Julliard, etc.) is suspect. [1] https://www.parchment.com/c/college/college-rankings.php?page=1&perPage=25&thisYear=2021 https://www.parchment.com/c/college/college-rankings.php?pag... [2] https://www.topuniversities.com/university-rankings/world-university-rankings/2023 https://www.topuniversities.com/university-rankings/world-un...
- brianbreslin 4y ago
  I work at a university, and honestly have never heard of this ranking that you referenced in [1]. Also seems super weighted towards military academies. What's their model? Are kids in the states now looking at more global options ?
  - nemothekid 4y ago
    >What's their model? Are kids in the states now looking at more global options ? According to their methodology[1] page, it looks like for student who gets multiple offers they do an Elo style ranking. The first problem; and which probably explains the weight to military academies, is that students tend to self select into tiers of colleges, so a group of students who are interested in the Air Force might not even apply to Harvard (even though the USAFA is a competitive school) and upon admission opt to go there (after all if you even were even considering of joining the air force, why would you reject your one shot there to go any other school)? [1] https://www.parchment.com/c/college/college-rankings.php https://www.parchment.com/c/college/college-rankings.php
    - pyuser583 4y ago
      So this would favor specialized schools in general?
  - 4y ago
aiisjustanif 4y ago
Adam Ruins Everything, Season 2, Episode 7 [1] 1: https://youtu.be/EtQyO93DO-Q https://youtu.be/EtQyO93DO-Q
colinmhayes 4y ago
I took an Econ class with northwestern’s president and he said that they straight up lie about their middle 50 gpa and standardized test scores. Can’t say I was super surprised. Every top 25 school is saying 25% of their students got a 35 or better on the ACT, the numbers just didn’t add up.
- rayiner 4y ago
  Morty? He was getting pretty chatty nearing retirement.
  - colinmhayes 4y ago
    Yea the class was basically him gossiping
- whimsicalism 4y ago
  Maybe it is because they "super score" (ie. take the best of each section if they take it multiple times).
humanistbot 4y ago
See https://en.wikipedia.org/wiki/Goodhart%27s_law https://en.wikipedia.org/wiki/Goodhart%27s_law
- xdavidliu 4y ago
  this is also paraphrased in the paper by the Columbia professor. > Almost any numerical standard, no matter how closely related to academic merit, becomes a malignant force as soon as universities know that it is the standard. A proxy for merit, rather than merit itself, becomes the goal.
lettergram 4y ago
I FOIA requested a lot of universities and man do they fight to keep their stuff secret. https://austingwalters.com/foia-requesting-100-universities/ https://austingwalters.com/foia-requesting-100-universities/ Often I’d find the person in charge on linkedin and email them directly (work email obviously). And dig around until I found the particular person. UIUC put up a particular fight when I FOIA requested information about their post-graduate surveys. They’re definitely manipulating there. I was in the process of filing a lawsuit when covid hit and gave up because I wasn’t sure if I’d need the funds (plus court shutdown at the time).
- dragonwriter 4y ago
  > I FOIA requested a lot of universities FOIA only applies to federal government agencies, universities (other than ones operates by the federal government, like the service academies) aren't subject to it. State universities might be covered by state sunshine laws that are similar to FOIA, and generally the laws and policies around those only require a request from the public and aren't too particular about form or citing a completely wrong and irrelevant legal basis, so in that case an “FOIA” request might actually have some legal effect. But most private universities would just see it as a particularly pompously written discretionary request.
  - Kon-Peki 4y ago
    No, there are definitely FOIA laws that apply to public universities. The post was calling out UIUC and this is their web page about it [1] But I am really curious about the argument against the requested survey data. The poster linked to his web page which specifically lists UIUC as being exemplar and providing data without even the need for a formal FOIA request. [1] https://www.uillinois.edu/erc/foia https://www.uillinois.edu/erc/foia
    - lettergram 4y ago
      With the data I’d be able to prove a degree isn’t worth it. Which was what I was planning on making - enter your degree, high school gpa, we could predict the likely outcome. I’m sure they knew that lol I requested: country of origin, gender, degree, gpa, salary range, field of work and a couple other fields I forget now.
MatthiasPortzel 4y ago
From the conclusion of that professor's article: > No one should try to reform or rehabilitate the ranking. It is irredeemable. In Colin Diver’s memorable formulation, “Trying to rank institutions of higher education is a little like trying to rank religions or philosophies. The entire enterprise is flawed, not only in detail but also in conception.” A lot of HN commenters here trying to "fix" the details of the rankings.
bagels 4y ago
I'm really curious what the personal cost to that faculty member is. Surely this report was not well liked by the administration.

deleted 4y ago

[deleted]

hackyhacky 4y ago

Your suspicion is correct. I've taught at several US universities, and it's an open secret that the administrative takes steps to manipulate numbers in order to secure better rankings. Typical example: a small, prestigious liberal arts college always has smaller class sizes in the fall semester than in the spring semester. Why? Because classes with less than 30 students are ranked higher by US News, but they only take into consideration the fall semester. No one cares about class sizes in the spring.

CogitoCogito 4y ago
> Because classes with less than 30 students are ranked higher by US News, but they only take into consideration the fall semester. No one cares about class sizes in the spring. What a bizarre and totally unnecessary restriction.
- genewitch 4y ago
  Parents helping/guiding their near-adult children in the final spring of highschool to apply at or choose universities, where they would start university that same fall seem to be the target audience, here. Fall is when "most" incoming freshmen will start.
- unosama 4y ago
  You add a simple and reasonable (until it gets abused) assumption to your model which allows a lesser data collection burden on thousands of universities. I wouldn't call it a restriction.
  - thaumasiotes 4y ago
    > which allows a lesser data collection burden on thousands of universities. How's it supposed to do that? Universities all already collect the data on enrollment for every class they offer.
    - fshbbdssbbgdd 4y ago
      Spoken like a software engineer! Now imagine every form is being filled out manually by an administrator with a calculator.
      - pc86 4y ago
        But it's not.
      - deleted 4y ago
        [deleted]
      - mwint 4y ago
        Maybe the forms should be made too complex to be filled with a calculator, to encourage doing things the right way…
        geraldwhen 4y ago
        Wouldn’t work. Entire state budgets are accounted for by spreadsheets without formulas, emailed between people in an office with no revision control. A few years ago when 50 million went missing, I wasn’t surprised. I’d seen the budgeting process, and it was worse than you could possibly imagine. There was an attempt to go the software route, but a nepotism deal left the budgeting department with a worse system than manual spreadsheet entry.
- deleted 4y ago
  [deleted]
bogota 4y ago
Great to know many peoples graduations were delayed so someone could game some numbers
- tenpies 4y ago
  What are they going to do? Transfer out and have only half their credits recognized therefore pushing them back 4 semesters instead of the 1? It's a fantastically anti-consumer market they've managed to create.
- abakker 4y ago
  Gotta pay for another semester then, right?
- zerocrates 4y ago
  I think they mean the size of individual classes, so they're paying for more sections/instructors/instruction time in the ranking-relevant periods. Though shrinking the admit size is another well worn tactic, as it's the easiest way to increase how "selective" you are which is another major ranking element, plus it helps you more cheaply achieve smaller average class sizes, though of course you're forgoing some revenue to get there.
  - btilly 4y ago
    Encouraging people to apply is also a common tactic. You can't reject them if they don't apply!
    - javajosh 4y ago
      Applications are also a revenue source.
      - dhosek 4y ago
        Apparently there’s a common application thing now where you can apply to large numbers of schools for undergrad without spending large amounts of money.
nostromo 4y ago
Gaming and manipulation should absolutely be addressed, but it seems Columbia was just straight up lying about their numbers, not gaming them.
lumost 4y ago
Why is small class size a good thing? Anecdotally, you get better class material and TA staff in larger lectures. Why aren't Universities able to innovate here?
- bbor 4y ago
  Many courses rely on discussion!
- aquova 4y ago
  Fewer students per teacher is widely seen as a positive thing. A greater percentage of the professor's time and attention can be given to any one student. I've definitely experienced this myself, it's easy to feel like just a number when you're in a large lecture hall of hundreds versus a small classroom of a few dozen.
  - bachmeier 4y ago
    I already said this elsewhere, but I do want to reply to this comment, because it demonstrates a common mistake that may not be clear from my other comment. If you have a given teacher, in a given classroom, with a given amount of TA help, blah blah, so that all else is equal, and then you reduce the class size, the students will do better. The problem is that all else is far from equal in the data used in these rankings. Instructor quality drops off rapidly as you have more sections. TA help falls. The classroom may no longer be appropriate for the class, etc. The rankings are not able to capture the effect you're talking about.
- ghaff 4y ago
  For liberal arts? It's probably a pretty interactive format and smaller interactive classes are better than large classes that are more broadcast. (Often. I've also had great lecturers in the humanities.) But mostly don't look at many liberal arts classes (or case study, etc.) through the lens of an engineering lecture.
- poulsbohemian 4y ago
  >Anecdotally, you get better class material and TA staff in larger lectures. Where is this anecdote coming from? A billion years ago when I studied at a small liberal arts college with small class sizes, professors were keen to share with us exactly why they were teaching from particular texts or why they had selected certain materials to make it clear that while the class wasn't being taught by that famous guy from MIT, it was still his book and materials we were using. I knew all my professors by first name, knew where many of them lived, had access to them anytime it was needed, and that's directly counter to my partner's experience at a large university.
greymalik 4y ago
Goodhardt’s law in action.
- adolph 4y ago
  Goodhart's Law The key problem of using a measure as a metric is sometimes referred to as Goodhart’s law. The law is based on a 1975 paper on economic regulation, and is typically paraphrased as “When a measure becomes a metric, it ceases to be a good measure.” When managing a system, this problem is critical, but is deeper than it at first seems, as I’ll argue. https://www.ribbonfarm.com/2016/06/09/goodharts-law-and-why-measurement-is-hard/ https://www.ribbonfarm.com/2016/06/09/goodharts-law-and-why-...
- dccoolgai 4y ago
  Ctrl+f for "Goodhar" right away to see who called it first +1
pishpash 4y ago
What's that adage? Every metric that becomes an optimization target ...
eduction 4y ago
I attended a well known and regarded public university. I was admitted for spring, and for the preceding fall entered a program where I could live in a regular university dorm and take classes for full credit taught at the same university’s extension, but special extension classes just for this program (so all spring admit students). Then you stay in the dorm in spring as you become a “regular” student. I always suspected it was a way to fudge numbers, but at the same time I knew I was genuinely on the bubble of being admitted so I took the opportunity gladly. (My memory is that the other spring admits were, like me, from racial groups considered “over represented” so they may have been trying to affect diversity numbers too.)
- smugma 4y ago
  Berkeley has been doing this for at least 35 years.
  - tobinfricke 4y ago
    Yup, I got into Berkeley EECS this way back in the late 20th century. I'd guess it's legitimate load-balancing rather than an attempt to game the ratings. I assumed this meant that my application was borderline and I was grateful to be able to sneak in. The Extension program is a nice transition to college too.
    - etempleton 4y ago
      Load balancing is the most likely reason. At any given college a lot of first year students decide college isn't for them or they get home sick and drop out within the first semester. After that initial period student population of the class is much more stable.
bredren 4y ago
I TA'd in the attendance office at my high school back in the day. The city's tracking of student attendance only lasted a portion of the year or a portion of each quarter. During those times, much more effort went into calling the homes of missing students and finding out why they were not showing up to class. I remember asking about why all the calls were happening or not happening and it was sort of quietly explained by the staffer: measurement of attendance affected how the school was being evaluated by the district. Presumably, budget was hooked to it in some way. I didn't make any waves about it because "working" in this office was super powerful and the staffer always brought in cookies for us. Amazing gig.
- smugma 4y ago
  In California, schools get paid on daily attendance. I believe it would be considered fraud to not take daily attendance. They suspended this during the heart of the pandemic but it came back last fall. Because daily attendance is down relative to pre pandemic, Public schools are facing a significant revenue gap (5-10% is typical)
  - LeifCarrotson 4y ago
    In contrast, Michigan Public K-12 schools take attendance on specified Count Days to account for the number of students they serve to determine state funding. Fall count day is the 1st Wednesday in October and represents 90% of state funding. Spring count day is the 2nd Wednesday in February and represents 10% of state funding. As you might imagine, getting students in school on these days is a Big Deal. Teachers take attendance on other days for internal use, but Count Day is special and you must not be absent!
    - MaysonL 4y ago
      How often have Count Days been snow days?
      - LeifCarrotson 4y ago
        It's almost never a snow day in early October, I only remember one February count day + snow day. They count on the next day: > If instruction is cancelled on count day due to conditions not within the control of school authorities, with the approval of the State Superintendent, the affected instructional programs must use the immediately following day on which the district resumes session for count purposes per the rules at: https://www.michigan.gov/-/media/Project/Websites/mde/Year/2019/02/14/Student_Count_Information.pdf?rev=c14d2498c4b34d2cb519fb9e7acc1bdc https://www.michigan.gov/-/media/Project/Websites/mde/Year/2... I do recall my cousins at a poor rural school in Barry county having a couple snow days on Count Wednesday, and then subsequently calling one on Thursday when it wasn't really required and another on Friday where it really wasn't required, so that counts were as high as possible on Monday. Their snow days were much more frequent due to bad roads and bad road maintenance, and their state funding was much more important compared to my wealthy district.
Fomite 4y ago
Similarly, at a large land grant university, we talk a lot about improving our rankings, but acknowledge that one of the major metrics (% of students not accepted) is something that's contrary to our mission. It was an open secret when I was in Boston that Harvard actively encourages people to "give it a shot" when they know full well they're not getting in to boost that metric.
- joe5150 4y ago
  "Recruit to deny" [1] has been getting more attention but still not enough. 1: https://www.nytimes.com/2019/11/29/us/harvard-admissions-recruit-letter.html https://www.nytimes.com/2019/11/29/us/harvard-admissions-rec...
  - KMag 4y ago
    Michigan also seemed to waitlist me and several of the people I knew at MIT. If they know they're the backup school, and they think you're going to get into your first choice, it hurts their acceptance rate to accept you outright. Though, maybe it's just confirmation bias and ego. Maybe there was something in my application that gave Michigan pause but seemed fine to MIT. At that time, Michigan did ask which other schools to which you were applying. I'm not sure if that's still the case.
    - JohnBooty 4y ago
      I'm confused. How would Michigan know that they were your second choice?
      - draz 4y ago
        They expect students of a certain caliber. If the applicant exceeds their expectations, the university assumes the student is using this application as a “safety school.”
        brewdad 4y ago

mbesto 4y ago

Which begs the question - if you want to get top in rankings wouldn't you simply design the school around the objective ranking parameters and nothing else? Ripe for abuse...

nonameiguess 4y ago
Not sure if this is still true, but at one point, the biggest factor in the ranking was peer reputation rating. That kind of results in a self-reinforcing vicious cycle of top schools remaining top schools, but short of bribery, there isn't really any way to manipulate that at least.
colinmhayes 4y ago
This is sort of what Columbia did… they juiced their faculty numbers by counting the med school which has much lower class sizes and more educated teachers.

nus07 4y ago

I have a friend who works at Duke University’s marketing analytics and performance metrics team. It’s filled with MBAs who’s job is to massage the numbers and metrics that are asked by US News every year and provide them to US News.

noipv4 4y ago
username checks out ;) you guya have a collab prog with duke
onionisafruit 4y ago
It's probably staffed with MBAs to boost the job placement numbers for their MBA program.

mtlmtlmtlmtl 4y ago

This is a good example of Campbell's Law at work: https://en.wikipedia.org/wiki/Campbell%27s_law https://en.wikipedia.org/wiki/Campbell%27s_law

ethbr0 4y ago
An obvious takeaway in this case: US News shouldn't be the primary ranker. If statistics were distributed in open format, transparently, and there were multiple rankers (each with their own weights and mixes), universities would be less incentivized to game "the US News metrics".
- t_mann 4y ago
  That seems naive to assume. We already live in a world where there are multiple rankings, and many of them (even some of the most important ones, like QS or Shanghai) rely only on independently collected or public data. We already have what you suggested as a solution, and look at where we are. The problem is more fundamental.
- humanistbot 4y ago
  It already exists and is a free US Dept of Education service. Schools have to report that data to the Dept of Ed already, so they have both an interactive search tool [1] and the raw data available [2]. [1] https://collegescorecard.ed.gov/search/?page=0&sort=threshold_earnings:desc&toggle=institutions https://collegescorecard.ed.gov/search/?page=0&sort=threshol... [2] https://collegescorecard.ed.gov/data https://collegescorecard.ed.gov/data
  - everybodyknows 4y ago
    Looking at [1] just now, it seems overwhelmingly oriented toward answering the question "How do I get in?". Scarcely at all toward "What can the school deliver to the student?" -- only relevant measure is Median Earnings. The fine print: >The median annual earnings of individuals that received federal student aid and began college at this institution 10 years ago, regardless of their completion status. So for the anxious parent, not a lot to go on.

twobitshifter 4y ago

It’s like doping in cycling. Everyone was getting caught but we’re supposed to believe the couple who snuck through weren’t like the rest. Once the tests got good enough there was hardly any one left standing. https://en.m.wikipedia.org/wiki/Doping_at_the_Tour_de_France https://en.m.wikipedia.org/wiki/Doping_at_the_Tour_de_France

gofreddygo 4y ago

Funny how this reminded me of a PG essay[1] The most damaging thing you learned in school wasn't something you learned in any specific class. It was learning to get good grades. [1]:http://paulgraham.com/lesson.html http://paulgraham.com/lesson.html

bdowling 4y ago
The problem with that quote is that if you don't learn to get good grades (e.g., because you are actually learning and not studying for the test), then the resulting bad grades can be extremely damaging to your future.
- kragen 4y ago
  How is that a problem with the quote? It sounds like a problem with the system, a problem the essay correctly identified.
  - brewdad 4y ago
    Ok. Now fix the system in a timeframe that is compatible with your expected working life. That's the problem with the quote. Grades may not be the most important thing but they can, and often are, the most important thing for any given individual. For every Bill Gates that dropped out, there are a million John Doe's drowning in student loan debt with no degree and limited future prospects to show for it.
    - kragen 4y ago
      Who are you asking to fix the system, and why? Me, maybe? Paul Graham? gofreddygo? How is the thing you say a problem with the quote? These can both simultaneously be the case, and often are: 1. The most damaging thing John Doe learned in school was learning to get good grades. 2. Richard Roe didn't get good grades and is drowning in student loan debt with no degree and limited future prospects to show for it. You have offered no reason to believe that if (2) is more true, or true in more cases, (1) is less true, or true in less cases, or less important. In fact, this tighter conjunction can also often be true: 1'. The most damaging thing John Doe learned in school was learning to get good grades. 2'. John Doe didn't get good grades (despite being damaged by learning how, he didn't learn how enough, or other things interfered) and is drowning in student loan debt with no degree and limited future prospects to show for it. You've also offered no reason to believe that if (2') is more true, (1') is less true or less important. I have an idea! Maybe you're misinterpreting the quote as meaning that, if you're in school, it would be a good strategy to stay there but not learn to get good grades. In that case there would be a sort of conflict: by diminishing the damage from (1') you may be increasing the risk of (2'). But that isn't what the essay advises at all, and I'm puzzled as to how you could have read it as suggesting that. It explicitly advises students to do exactly the opposite; it advises them to get good grades by hacking bad tests in the usual ways (cramming before exams, etc.) because everyone will judge them by their grades. But that's not really the point of the essay, and it would be a pretty boring essay if it was. The point of the essay is its advice to people who aren't enrolled as students: unlearn how to get good grades, and instead of focusing on how to make your product look good, actually make a good product. Also, and maybe more importantly, try to work in fields of endeavor where the way to win is to do good work, rather than hacking bad tests like you do in school. (This might suggest not going into school at all, which would at least avoid the student loan debt problem.)

hardtke 4y ago

My kid just went through college selection process and I can assure you that the US News rankings are largely irrelevant in terms of decision making. The revealed preference score[1] or global university rankings[2] are much closer to how students, their counselors, and their parents rank the universities. For instance, any ranking that does not have Stanford first among non specialized US universities (i.e. excluding MIT, Caltech, Julliard, etc.) is suspect. [1] https://www.parchment.com/c/college/college-rankings.php?page=1&perPage=25&thisYear=2021 https://www.parchment.com/c/college/college-rankings.php?pag... [2] https://www.topuniversities.com/university-rankings/world-university-rankings/2023 https://www.topuniversities.com/university-rankings/world-un...

brianbreslin 4y ago
I work at a university, and honestly have never heard of this ranking that you referenced in [1]. Also seems super weighted towards military academies. What's their model? Are kids in the states now looking at more global options ?
- nemothekid 4y ago
  >What's their model? Are kids in the states now looking at more global options ? According to their methodology[1] page, it looks like for student who gets multiple offers they do an Elo style ranking. The first problem; and which probably explains the weight to military academies, is that students tend to self select into tiers of colleges, so a group of students who are interested in the Air Force might not even apply to Harvard (even though the USAFA is a competitive school) and upon admission opt to go there (after all if you even were even considering of joining the air force, why would you reject your one shot there to go any other school)? [1] https://www.parchment.com/c/college/college-rankings.php https://www.parchment.com/c/college/college-rankings.php
  - pyuser583 4y ago
    So this would favor specialized schools in general?
- 4y ago

aiisjustanif 4y ago

Adam Ruins Everything, Season 2, Episode 7 [1] 1: https://youtu.be/EtQyO93DO-Q https://youtu.be/EtQyO93DO-Q

colinmhayes 4y ago

I took an Econ class with northwestern’s president and he said that they straight up lie about their middle 50 gpa and standardized test scores. Can’t say I was super surprised. Every top 25 school is saying 25% of their students got a 35 or better on the ACT, the numbers just didn’t add up.

rayiner 4y ago
Morty? He was getting pretty chatty nearing retirement.
- colinmhayes 4y ago
  Yea the class was basically him gossiping
whimsicalism 4y ago
Maybe it is because they "super score" (ie. take the best of each section if they take it multiple times).

humanistbot 4y ago

See https://en.wikipedia.org/wiki/Goodhart%27s_law https://en.wikipedia.org/wiki/Goodhart%27s_law

xdavidliu 4y ago
this is also paraphrased in the paper by the Columbia professor. > Almost any numerical standard, no matter how closely related to academic merit, becomes a malignant force as soon as universities know that it is the standard. A proxy for merit, rather than merit itself, becomes the goal.

lettergram 4y ago

I FOIA requested a lot of universities and man do they fight to keep their stuff secret. https://austingwalters.com/foia-requesting-100-universities/ https://austingwalters.com/foia-requesting-100-universities/ Often I’d find the person in charge on linkedin and email them directly (work email obviously). And dig around until I found the particular person. UIUC put up a particular fight when I FOIA requested information about their post-graduate surveys. They’re definitely manipulating there. I was in the process of filing a lawsuit when covid hit and gave up because I wasn’t sure if I’d need the funds (plus court shutdown at the time).

dragonwriter 4y ago
> I FOIA requested a lot of universities FOIA only applies to federal government agencies, universities (other than ones operates by the federal government, like the service academies) aren't subject to it. State universities might be covered by state sunshine laws that are similar to FOIA, and generally the laws and policies around those only require a request from the public and aren't too particular about form or citing a completely wrong and irrelevant legal basis, so in that case an “FOIA” request might actually have some legal effect. But most private universities would just see it as a particularly pompously written discretionary request.
- Kon-Peki 4y ago
  No, there are definitely FOIA laws that apply to public universities. The post was calling out UIUC and this is their web page about it [1] But I am really curious about the argument against the requested survey data. The poster linked to his web page which specifically lists UIUC as being exemplar and providing data without even the need for a formal FOIA request. [1] https://www.uillinois.edu/erc/foia https://www.uillinois.edu/erc/foia
  - lettergram 4y ago
    With the data I’d be able to prove a degree isn’t worth it. Which was what I was planning on making - enter your degree, high school gpa, we could predict the likely outcome. I’m sure they knew that lol I requested: country of origin, gender, degree, gpa, salary range, field of work and a couple other fields I forget now.

MatthiasPortzel 4y ago

From the conclusion of that professor's article: > No one should try to reform or rehabilitate the ranking. It is irredeemable. In Colin Diver’s memorable formulation, “Trying to rank institutions of higher education is a little like trying to rank religions or philosophies. The entire enterprise is flawed, not only in detail but also in conception.” A lot of HN commenters here trying to "fix" the details of the rankings.

bagels 4y ago

I'm really curious what the personal cost to that faculty member is. Surely this report was not well liked by the administration.