12 ms·
Why a Dutch court stopped high school students from exchanging schools
- sannee 9y agoIt would also be interesting to hear about why there are no entrance exams in the dutch system (or at least the article does not mention them).
- MrQuincle 9y agoThere is a nation wide exam called CITO at the end of primary school: https://www.expatica.com/nl/education/Explaining-the-Cito-exams_100820.html https://www.expatica.com/nl/education/Explaining-the-Cito-ex...
- jorams 9y agoI wanted to give you a brief explanation, but I ended up summarising the entire Dutch school system. I also just found out there's a comprehensive-looking Wikipedia article [10] on the matter, so perhaps this wall of text is unnecessary. I have added sources, primarily on Wikipedia, some in Dutch and some in English. I only used them to check some facts. How the Dutch school system works: The school system in the Netherlands is separated into three stages: primary school, secondary school, and higher education. Primary school is separated into 8 stages, which normally correspond to years. These are "group 1" to "group 8". Children start school roughly when they are 4 years old, but education is compulsory from age 5. These 8 years increasingly introduce different subjects and increase difficulty, with reading, spelling, and arithmetic in group 3, English starting in group 7, etc [1]. During primary school, progress of each student is tracked using tests, allowing the teacher to see how the child develops. Near the end of primary school, the teacher gives a formal "school advice" for the secondary school level. There's also a final test after this, and if the student excels unexpectedly the school needs to reconsider their advice and possibly adjust it. This test is an extra piece of data for the secondary school to consider. The primary school's advice determines which level of secondary schooling the student can enter [2]. The secondary school may not base their acceptance on other tests and may not require any extra tests, unless they provide a "special" kind of education, like bilingual education or education focused at professional sports. In that case, the extra test may only test the student's special abilities required for the school. Secondary school [3] generally takes between 4 and 6 years, and is divided into four levels of increasing difficulty: practical education, vmbo (+ mbo), havo, and vwo. Practical education [4] does not have a specified duration and is meant for students with particularly low intelligence (60 to 75-80) and a learning delay of at least 3 years. Vmbo [5] is preparational secondary vocational education and is again divided into 4 levels, in increasing order of theory-orientedness. It takes 4 years and ends with a final, nationwide exam. It prepares the student for a job, for havo or for mbo. About 60% of all students in secondary education follow vmbo. After vmbo the student can go to mbo [6], which is secondary vocational education. It is focused at joining the job market and, again, divided into 4 levels. The first level does not actually require any previous qualifications and educates the student to become an assistant professional, the second level prepares the student to become a basic professional, the third level prepares the student to become an independent professional, and the fourth level prepares the student to become a specialized professional. The fourth level also allows the student to progress to hbo. Mbo corresponds roughly to community college in the US. Havo [7] is higher general secondary education, takes 5 years, and is followed by about 20% of all secondary education students. Havo prepares the student for hbo, but after 3 years one can also be admitted to mbo. Vwo [8] is preparational scientific education, takes 6 years, and is also followed by about 20% of all secondary education students. It comes in a few variants, primarily atheneum and gymnasium. The difference between those is that gymnasium includes Latin and/or Greek. Vwo prepares the student for wo. Higher education consists of hbo [9], higher vocational education, and wo, scientific education. Hbo is focused at the application of scientific knowledge, while wo is focused at scientific research. These follow the international standards for higher education. As an aside, the terminology around higher education can be a bit confusing when translating between Dutch and English. Hbo is delivered at a "hoge school", which translates literally to English as "high school". The proper English term is "university of applied sciences". Furthermore, in Dutch wo is commonly (even usually) called "universiteit", Dutch for "university". When doing hbo, saying in English that you are a university student may therefore feel like a lie, while being correct. [1] (Dutch): https://nl.wikipedia.org/wiki/Basisonderwijs_in_Nederland https://nl.wikipedia.org/wiki/Basisonderwijs_in_Nederland [2] (Dutch): https://www.rijksoverheid.nl/onderwerpen/toelating-middelbare-school/toelating-voortgezet-onderwijs-gebaseerd-op-schooladvies https://www.rijksoverheid.nl/onderwerpen/toelating-middelbar... [3] (Dutch): https://nl.wikipedia.org/wiki/Voortgezet_onderwijs https://nl.wikipedia.org/wiki/Voortgezet_onderwijs [4] (Dutch): https://nl.wikipedia.org/wiki/Praktijkonderwijs https://nl.wikipedia.org/wiki/Praktijkonderwijs [5] (English): https://en.wikipedia.org/wiki/Voorbereidend_middelbaar_beroepsonderwijs https://en.wikipedia.org/wiki/Voorbereidend_middelbaar_beroe... [6] (Dutch): https://nl.wikipedia.org/wiki/Middelbaar_beroepsonderwijs https://nl.wikipedia.org/wiki/Middelbaar_beroepsonderwijs [7] (English): https://en.wikipedia.org/wiki/Hoger_algemeen_voortgezet_onderwijs https://en.wikipedia.org/wiki/Hoger_algemeen_voortgezet_onde... [8] (English): https://en.wikipedia.org/wiki/Voorbereidend_wetenschappelijk_onderwijs https://en.wikipedia.org/wiki/Voorbereidend_wetenschappelijk... [9] (Dutch): https://nl.wikipedia.org/wiki/Hoger_beroepsonderwijs https://nl.wikipedia.org/wiki/Hoger_beroepsonderwijs [10] (English): https://en.wikipedia.org/wiki/Education_in_the_Netherlands https://en.wikipedia.org/wiki/Education_in_the_Netherlands
- Rainymood 9y agoThere are nation wide tests at the end of primary school (to get into high school) i.e. CITO and NIO I believe. If you score well on these tests you get a good 'advice' which is non-binding which this post is about. I had a near perfect CITO score but a literally retarded NIO score because they explained me how the test worked wrong (I was probably being stupid and WAY took their words LITERALLY [1]). So they basically averaged my scores (best and worst) and said "Well you go to the HAVO" which is average. My parents did NOT sit well with that and just said "No he's going to Gymniasum (highest possible class with Latin/Greek) whether you want it or not. We all know he just screwed up there on that one tests because you explained him the instructions wrong." Anyway, 11 years later and I'm doing a PhD in applied mathematics. I'm really happy my parents fought for me to go to Gymnasium and I'm so glad the advice isn't binding. I'm not sure whether it still isn't binding or they changed it somehow. [1] The instructor told us that "at the end of your test you will have time to finish your questions". So what stupid little me did was I made half the test, then I waited till the time ran out and when they took my paper I asked "can I finish the questions now?" And they just took my paper away.
- jpfed 9y ago>By trading places, students could get another shot at a place. At schools for which they already had a go in the lottery. This would give them better chances. At the expense of students that don’t (or can’t) trade. This is unfair. This is culturally alien to me; the students receiving the benefit are not harming other students. >Another argument, that is related, is about strategizing. Students were told (correctly) that their best strategy was to give their true preferences. If trading were allowed, this would not be true. It would pay off for students to put popular schools high on their list. A spot at these schools would be good trading material. Solution: only allow trades that are in accord with the students' submitted preferences. That is, if Alice prefers Central to Devonshire, and Bob prefers Devonshire to Central, then we could allow Alice and Bob to switch if their submitted rankings indicated that these moves were consistent with their desires. There would be no incentive to lie.
- Sniffnoy 9y ago> This is culturally alien to me; the students receiving the benefit are not harming other students. It's not just "culturally alien"; it's a straight-up error. > Solution: only allow trades that are in accord with the students' submitted preferences. That is, if Alice prefers Central to Devonshire, and Bob prefers Devonshire to Central, then we could allow Alice and Bob to switch if their submitted rankings indicated that these moves were consistent with their desires. There would be no incentive to lie. It's not obvious to me that this is true? Relevant discussion over on /r/math: https://www.reddit.com/r/math/comments/6uyfle/why_a_dutch_court_stopped_high_school_students/dlwtroo/ https://www.reddit.com/r/math/comments/6uyfle/why_a_dutch_co...
- s17n 9y ago>> Solution: only allow trades that are in accord with the students' submitted preferences. That is, if Alice prefers Central to Devonshire, and Bob prefers Devonshire to Central, then we could allow Alice and Bob to switch if their submitted rankings indicated that these moves were consistent with their desires. There would be no incentive to lie. >It's not obvious to me that this is true? Even if you could solve the game theory issues, you wouldn't solve the problem of people selling spots in desirable schools.
- Sniffnoy 9y agoI'm going to repost the same comment I made on /r/math: > By trading places, students could get another shot at a place. At schools for which they already had a go in the lottery. This would give them better chances. At the expense of students that don’t (or can’t) trade. This is unfair. This argument here is not correct. It's not at anyone's expense. The two people trading both benefit, and other students are unaffected. Not everything is zero-sum; not every benefit is at somebody's expense! (The strategy argument -- that people had been explicitly told the system was strategy-free, and it would be wrong to go back on that guarantee by allowing trading -- makes much more sense.) This is an interesting article about social choice theory but the non-mathematical parts seem to be making some implicit assumptions about "fairness" that are, let's say, not exactly uncontroversial. And the argument I point out above is just in error.
- deleted 9y ago[deleted]
- cortesoft 9y agoYou are missing some key points - if they allow trading, they are going to incentivize people choosing popular schools instead of ones they actually want to go to. Since more people are going to choose popular schools in hopes of getting a good trading asset, people who ACTUALLY want to go to those popular schools are going to be less likely to get in. Your assertion that it is an 'error' is only true if they only allow the trading to happen this year (after the preferences were set), and not in future years. The trades are only pareto-superiour if there are no future decisions to be made; since this is a repeated game, allowing pareto-superior trades this year means that next years game will NOT be fair (by their definition of fairness). I agree that their definition of 'fairness' is not objective, but it is one they have decided to go with; they decided that the ability to strategize is not evenly distributed, and so allowing strategy is unfair.
- Sniffnoy 9y agoYes, the incentives this sets up could be bad for subsequent rounds; that is absolutely a real problem, as I mentioned. But this passage seemed to be suggesting that trading, in and of itself, harmed students who were not involved in the trade, and that's not correct. Notice it explicitly says: > This would give them better chances. At the expense of students that don’t (or can’t) trade. But in fact those students are unaffected. In subsequent years, as you say, it is a problem. But I don't see how to read that part as anything other than denying that when just restricted to that year it is a Pareto-improvement. You may say "well that's a self-evidently stupid claim; of course they're not making that claim, because that would be stupid". But there's a lot of people out there who really seem to believe in a zero-sum world, and are convinced that when anyone's position improves, that means that everyone else has been harmed. That Pareto improvements are fundamentally impossible. There are absolutely people out there who would make that claim.
- alkonaut 9y agoI thought the argument was going to be that exchanges would lead to poor students applying for top schools only to "sell" their spot to the highest bidder in a swap. It would seem though that some exchanges could take place within the lottery system itself. If two students were accepted to their second choices, and swapping them would give both their first choices, then that should be an optimization pass run after the lottery pass is done.
- maxerickson 9y agoThe single-DA variant they switched to eliminates the swapped first choices. (when a student is assigned to a school, it is always to the best match available for their position in the lottery)
- air7 9y agoWin-win swaps could be done automatically by the lottery system. This eliminates the non-fairness objection (as this optional upgrade can happen to anyone). It does theoretically create room for strategizing though: If there is a very popular school that I don't want to go to, it makes sense to put in down as option #3 or #4 because if I get it, I might slingshot back into my #1-#3 options.
- emn13 9y ago...and that would reduce efficiency again. It's not at all clear that the loss of efficiency from false preferences is smaller than the loss caused by students getting each others preferences. In fact, the article is a little too kind to the option of trading places. It presents a clean choice between fairness and efficiency: trading harms the former while boosting the latter. In fact, trading will reduce future efficiency due to strategizing; it may well be that trading is a straight-up de-optimization in all but the current round.
- vanderZwan 9y ago> It does theoretically create room for strategizing though And that is the exact issue with it: it would soon be common knowledge that you should do so, and then the people who don't consciously try to game the system would be at a disadvantage. In general that seems to be the goal: devising a system that cannot be gamed in any other way than giving your honest preference. In other words, ensure that the system is a self-enforcing protocol. https://www.schneier.com/blog/archives/2009/08/self-enforcing.html https://www.schneier.com/blog/archives/2009/08/self-enforcin...
- politician 9y agoIf they allowed trading, then there's nothing to stop the lucky winners of spots at popular schools from selling their spots, or for losers of spots to offer bounties for their top choice. As an argument for, "Win-win swaps" comes up a little thin when you consider the sort of economic shenanigans that folks can engage in.
- jseliger 9y agoIf they allowed trading, then there's nothing to stop the lucky winners of spots at popular schools from selling their spots, or for losers of spots to offer bounties for their top choice Who cares? If people prefer cash to a spot at a particular school, let them have it.
- ricardobeat 9y agoWe care, and you should care too: the effect of that policy would be the exact opposite of fairness, the rich would go to the best schools while the poor would get a handful of cash and the worst education.
- chii 9y agoAlso the cash goes to the parent, but the kids gets the shit school.
- valuearb 9y agoReason number ### for why lack of choice in education is bad
- ucaetano 9y agoYeah, the Dutch educational system is clearly failing, being the 9th best in the world...
- smith_winston 9y agoMeasuring and ranking systems like this always involves impossibly circular problems. Ability differs between populations. The best system will produce the best possible result for a population, not necessarily the best overall result.
- maehwasu 9y agoTo what degree do those rankings simply correspond to average population IQ? Genuinely curious.
- ucaetano 9y agoAre you claiming that dutch people have genetic traces that lead to higher IQ? Because a better education leads to higher IQ; so, if the Dutch people have higher IQs on average, it might be because of the better education.
- sweep3r 9y ago>Are you claiming that dutch people have genetic traces that lead to higher IQ? Yes? It's a homogeneous, white society; compare that to the US. http://i.imgur.com/FhEZG1G.png http://i.imgur.com/FhEZG1G.png
- icebraining 9y agoObviously it's not failing, but it could be succeeding despite this. Especially since has it's been pointed out, most of the Dutch educational system does not use this mechanism.
- wolco 9y agoDoes it not make sense for people to go close to where they live. It reduces the carbon footprint and makes the school more local. For the specialist schools have people interview, show there art/skill.
- ricardobeat 9y agoLuckily most students in Amsterdam travel by bicycle :) You can only signup for schools in your city area, which already restricts it to a few km radius. For pre-school neighborhood limits apply.
- icebraining 9y agoCloser to what, though? If, for example, you practice some kind of after-school activity not near to your home, you might be better off choosing a school near it, and the footprint is the same. It's better to discourage the harmful activity directly than to impose indirect restrictions, since those discriminate against specific groups.
- abalone 9y agoSan Francisco United School District uses the multi-DA system, but here's the kicker: it bakes exchanging into the algorithm.[1] They call it "trading up"; most parents here refer to it as "swapping". It works more or less as described, where you list off all the schools you want in order of preference. It runs lotteries for each of them in parallel and you get the highest listed one you win. But then it does one more thing: If it finds mutually advantageous swaps, it does so automatically and instantly. In other words, it's not up to or necessary for the student to do it manually. In fact this feature is detailed by the Ivy League team that put it together, including Alvin Roth, as early as 2010.[2] So while this article is well written it seems to give a less than complete picture. Why did Amsterdam not adopt this feature? P.S. There is much to be said about the SF lottery system which is a fun / highly stressful topic for nerds and parents. If you like these kinds of problems I encourage you to take a look. It also tries to solve for giving preferences to neighborhood schools and disadvantaged families, while also offering special language and K-8 programs that serve the whole city fairly. The goal is to eliminate strategizing as much as possible, but it's very interesting to see people figure out some hacks. [1] http://www.sfusd.edu/en/assets/sfusd-staff/enroll/files/2012-13/Assignment-Transfer%20process%20explained.pdf http://www.sfusd.edu/en/assets/sfusd-staff/enroll/files/2012... [2] http://web.stanford.edu/~niederle/SFUSDBoardPresentationFeb.17.2010.pdf http://web.stanford.edu/~niederle/SFUSDBoardPresentationFeb....
- bigethan 9y agoI disagree about the "fun" part having just gone through this with my first kid. I agree with the Dutch, swapping (at any point in time) adds an edge of unfairness to the process: - In a situation where all decent schools are oversubscribed swapping 100% encourages (and rewards) setting up your school list strategically. Which helps out families that have the time to collect the information needed and sort their schools appropriately (we spent a bunch of time researching / planning here, but knew off other families who _really_ got into gaming the system) - In one of the meetings where the slide you linked in [1] was displayed, they offhand mentioned that the swapping order is done alphabetically. When asked if kids with A* names will get better swaps they kinda ignored the question and moved on. But how are multiple tied swaps resolved? - Also, from what we were told, the swapping doesn't respect the "lottery weight" that locals, poor test score neighborhood kids, or siblings get. If that's important, shouldn't it apply to all phases of the process? (I'm aware the math here might get out of hand) I started out thinking that the swapping was a good idea, but when I saw how much it encouraged and rewarded strategy in building the list of schools I changed my mind. It's not a fair process. It's a system that rewards families that have likely already been rewarded by society.
- gnicholas 9y ago> To allow trading would be to enable strategic behavior in future years...it could also give an unfair advantage to students that are better at strategizing — and to students that have more resources to make good strategic decisions. Similarly, it could enable wealthy students to buy slots from poorer kids. Poorer kids could choose to prioritize schools that are popular, but that they don't actually want to go to, in the hopes of trading places with a rich kid who was willing to pay for the privilege.
- Aaron1011 9y ago> Suppose, for example, that a student’s top choice school is hugely popular. But their number two to five schools are only slightly short on places. Then it could make sense to register for the number two school in the first round. They would have good chances to get into the second best school. Going for the number one choice is risky: if they don’t get in, they have no shot at any school in their top five anymore. Why would not getting into the first-choice school affect the student's chance of getting into any of their top five schools? The article's earlier description of the Boston method makes it sound as though this wouldn't be the case.
- ricardobeat 9y agoBecause the other schools are very likely to have been filled up already in the first round.
- hedora 9y agoThis seems like an extreme example of "good in theory" overriding "good in practice". The multi algorithm outperforms single, so run multi, then greedily swap students at random until there are none that got each other's higher ranked choice. This obviously outperforms multi, and is fair, strategy-resistance be damned.
- skrebbel 9y agoAmsterdam is a very special place. You know how on HN everybody always complains about insane all kinds of legislation in SF? Well in that respect, Amsterdam is the SF of the Netherlands. So please let this taint your opinion of Amsterdam, and not of Europe or the Netherlands. They really like rules and regulations there. More is better. Notably, the whole concept of a high school lottery is culturally alien to most of the country. Let alone a lottery with a 1000-page rule book and court cases.
- paulannesley 9y ago> let this taint your opinion of Amsterdam Why taint? The judge gave good reasons for disallowing trading. In particular; if students know they can trade places later, they'll pile onto the most popular schools (rather than their true preference) knowing that if successful they can easily trade their place for another one (including their preferred school which may not have selected them). Add to this the possibility of paid trades, I think it's great they're keeping it fair and simple.
- rb666 9y agoYes, Amsterdam loves regulations. But Amsterdam is quite uniquely positioned in NL. The popularity of the city with locals as well as tourists is of an entirely different level than the rest of NL. Furthermore, the AMS lottery systems is precisely a regulation that is far from pointless. This article explains it very well.
- Ntrails 9y agoFascinating. As an outsider this reads like a wonderful attempt by those with difficult decisions to make to be as fair as possible to all of the young people for whom they are responsible. The only thing this has done is raise my estimations of the public servants in Amsterdam.
- murphyslab 9y agoLiving in another part of the Netherlands, I haven't seen a significant difference. The Dutch have an altogether practical outlook, valuing regulatory conformity. One cannot even choose to paint the exterior of their house differently. The main difference with Amsterdam is primarily the size of the city: greater size makes for more schools and greater variability between them. Greater variability creates a differential in desirability, thus creating the problem of allocation... to which they have devised a very Dutch solution.
- neilwilson 9y agoYou can always rely upon economists to exclude any human considerations from the process. There is no choice here because it fails the 80/20 rule of any hiring system. Any less than 20% capacity spare and there is no free choice. So since there is no actual choice, why give people one? Is it the same as with toddlers - You can have red juice or blue juice - a way of giving the feeling of control without actually having any. Giving people a choice and then not giving them what they want causes loss aversion responses. That is worse than not having any choice at all. Kids want to go to the nearest school and the schools should be sufficiently equipped to cater for all needs in the area. Every other solution I've seen is more difficult than that one and is based around psychological manipulation rather than solving the problem. And no the 'free market' won't provide. You're supply side limited by the number of people capable and willing to be teachers. That's why you have to have a mass production system for education.
- ralfn 9y agoEverybody does live near a highschool, but that way you end with bad and good schools because of economic indifferences. They are doing this (rather than just basing it on location alone) to spread the pain and more importantly give children born into this world the same chances irregardless of the social economic situation of their parents. Locality still plays an important role, but we are talking high population density neighbourhood. Most will live near at least three schools in an radius of 30m walk (10min bike ride)
- neilwilson 9y agoIt doesn't give people the same chances. It's a way of making the aggregation look good and sod the individual impact. Which is exactly what economists are all about. We should be aiming at better than that.
- ralfn 9y agoOh yes. I was arguing the (from the eyes of the government) optimal allocation isn't perse the same as maximizing the preferences of the individuals. Which is similar to your argument, except i'm emphasizing its intentional -- not a mistake. From their perspective they can help students that are not as well off, but not if they are concentrated in a single school. I wasn't giving an opinion on whether any of this was a good (as in moral) idea or not. Although i understand there is an argument to prevent schools from becoming a political, cultural and racial mono-culture, even if a majority of the students would prefer to be at a school with 'just their kind of people'.
- PeterStuer 9y agoMy main concern would be that there is no way to control the potential hidden transactions included in the trades. I can well imagine less well of parents or parents less concerned with education gambling on the most desirable schools with the sole intention of making a buck on the trading aftermarket, or overly keen parents with less inhibition putting severe pressure on other parents to force a trade.
- cm2187 9y agoFrench universities have adopted a similar system. For some reason they do not like to select students based on merit, and there are more candidates than seats, so they introduced a lottery to determine who gets to go to university. It's very curious from the outside and even being French I can't think of a good reason for this system. Basically telling students it doesn't matter how hard you work in high school, it has no impact on whether you will work in a bank or a factory...
- sannee 9y agoAre French high schools reasonably homogeneous in their quality? Otherwise a merit-based system would be massively discriminating against students which made a poor choice (or live in an area without much of a choice).
- cm2187 9y agoNot really. State controlled schools in theory are supposed to take students based on postcode only, but in practice the most prestigious do their own selection. And you also have lots of private schools that are even more heterogeneous. The French system is very paradoxical. By trying to be very egalitarian it is achieving the opposite result. For high schools, the postcode allocation makes that wealthy areas where parent tend to be all highly educated have much better schools, whereas the poorest suburbs where immigrants live when they first arrive in France, have become educational toxic waste lands. And since teachers do not want to live there or face these students, they also get bottom of the basket teachers and high turnover. The mainstream universities, because of a long standing practice of having no selection, are chronically over-crowded and under-funded and their reputation has gone down the drain, outside of a few disciplines, like medicine, where incompetence has too great consequences, and which maintained a good reputation based on competitive exam selections. In parallel to universities, you have grandes ecoles which are small universities (student count wise) but very selective and very prestigious, and from where the vast majority of company executives come from. In theory these grandes ecoles are very egalitarian too, the entrance is based on a very selective competitive exam in math, physics, economy, etc and are either free or cheap (outside of business schools). To prepare for this exam, students go through a 2/3 years preparatory school, with a very intensive and demanding curriculum. Most students keep a horrible memory of these years and these preparatory schools are known for being a "not for every student" experience. As a result, teachers in poorer areas (who did not go through these schools themselves) discourage good students from enrolling ("it's not for you, too hard"), whereas parents who themselves went through this system apply pressure on their kids to do it anyway. As a result, you end up with a massive endogamy.
- yread 9y agoThey could only allow trading if you didn't get into your top 3 schools. That way you couldn't really count on being able to trade and only the people who were really unlucky could trade.
- tomohawk 9y agoOnly 8000 students? In the US we have high schools that are that large.
- speleding 9y agoI'm the parent of a kid who took part in the 2015 school lottery in Amsterdam, and since I had skin in the game I went as far as studying the original nobel prize winning papers on the DA algorithms to understand exactly what was going. This article is actually a really good summary, the amount of misinformation out there being spread by journalists who did not fully grok the (admittedly complex) system is astonishing. One thing that the article only touches on is the "preferred placement" that some schools managed to bring in from the old system, e.g. Montessori kids get preference for Montessori schools. This ruins the elegance and inherent fairness of the algorithm. In particular it makes it possible to do some form of strategic selection again if you have a preferred placement option that you can trade. This tends to benefit kids with clever parents disproportionately, as the article notes. A "lesson learned" for policy makers should be that even though an algorithm performs better on being "fair" (Multi-DA), it can still be better to choose an algorithm that is easier to explain. (They did end up switching to a different algorithm the next year)