9 ms·

> "[A]lgorithms are made by people and the results ... were reviewed multiple times by people," You got that right. The way they talk about "an algorithm did i

• fairity 6y ago
  I think most people are misunderstanding the likely root cause. > An algorithm was used to assign its first allotment of the vaccine. The algorithm prioritized health care workers at highest risk for COVID infections, along with factors like age and the location or unit where they work in the hospital. Residents did not have an assigned location, and along with their typically young age, they were dropped low on the priority list. Whoever coded up the algorithm for personal COVID-risk probably forgot to take care of null-states for the location input. Residents had their location set to null, and were therefore prioritized lower than they should've been. It's possible that the admins purposefully created this null-state bug so as to have a reasonable fallback story in the case they got caught, but per Occam's Razor, I think it's much more likely it was a dumb, honest mistake.
  • jrochkind1 6y ago
    I don't think I was assuming anything in particular about the intents. it's a rather important thing not to check on actual data, including residents in the sample, isn't it? they just never bothered to test the algorithm, didn't even review the results it spit out, before delivering the results as a plan? (it is quite possible indeed they didn't, I'm not saying that is a coverup for some other motive, I'm saying that is incompetent). What was the QA process like? Who was involved in it? From the article, apparently not any department heads or medical staff in general. That's not a competent design process for something so important, is it? That they never bothered to see how "the algorithm" treated residents demonstrates that they didn't give a shit about residents, not necessarily that they were intentionally screwing them. The administration doesn't say much about how the algorithm was designed, they just say "oh, it was an algorithm." I think many people believe that "algorithms are objective" and they are trying to play that.
    • deleted 6y ago
      [deleted]
    • SpicyLemonZest 6y ago
      You may be misunderstanding what this is a plan for. All Stanford medical staff are expected to be vaccinated over the coming weeks; the allocation in question is for deciding who's getting vaccinated today, with the first available shipment. It does seem like this particular error could have been caught easily, but conducting a thorough, laborious review process rather than getting shots in arms ASAP would have been an even worse decision.
      • function_seven 6y ago
        It wouldn't have required a laborious review process. A quick skim through the list, looking at 20 or so random recipients near the top, middle, and bottom, would have quickly shown that frontline workers were clustered near the end. You would have seen that some people were WFH, but at the top. This wasn't some random HR benefits process that they screwed up. It was a triage process. A medical triage process. That's what they should be good at.
        • SpicyLemonZest 6y ago
          From the article, it sounds like they did notice, but didn't have time to revise the plan before vaccinations started today. And again, I'm not claiming that this was necessarily the right decision, but I can't get behind the idea that it's some transparently evil decision made by uncaring administrators.
    • fairity 6y ago
      Well, we're all agreed that this was a dumb mistake that could've been easily avoided. And, the admins should own up to it and implement a QA process to avoid a second occurrence. That said, I think the reason this story is getting so much attention is because of the assumed selfish intent. And, that assumption is probably wrong.
  • tgsovlerkhgsel 6y ago
    The article addresses this: > While leadership is pointing to an error in an algorithm meant to ensure equity and justice, our understanding is this error was identified on Tuesday and a decision was made not to revise the vaccine allocation scheme before its release today
    • jules 6y ago
      What does that even mean? Surely the utility function being optimised by the algorithm is quality of life adjusted years saved by the vaccine? Surely they do not value different people's lives differently based on their group identity, but then what does that statement mean?
  • lock-free 6y ago
    The issue is not the algorithm but that the issues with it were identified in the days before the allotments finalized. Meaning: administrators saw that the results were dumb, and didn't compensate.
  • bosswipe 6y ago
    I don't like the use of the word algorithm there. I would call it rules or formulas. Algorithm makes it sound like something more complicated when it was probably just a SQL query.
• beamatronic 6y ago
  Select * From employees sort by salary desc
  • edoceo 6y ago
    This is the same algorithm many companies use for benefits programs
  • ishjoh 6y ago
    Small nit-pic. select * from employees sort by total_compensation desc; Otherwise C Level folks who take a $1 salary but receive tons of options would be put at the bottom.
    • deleted 6y ago
      [deleted]
• smnrchrds 6y ago
  "Machine learning is like money laundering for bias" https://twitter.com/pinboard/status/744595961217835008 https://twitter.com/pinboard/status/744595961217835008
  • LeanderK 6y ago
    I am not a native speaker and I don't think I understand the quote. What should "money laundering for bias mean"?
    • bayindirh 6y ago
      Early machine learning / deep learning algorithms have shown bias towards some results or behavior due to problems in training dataset. These problems are tried to be remedied with better training sets but I'm not aware whether it's completely eliminated. "Money laundering for bias" in this context can be translated as "Finding a plausible excuse/cover for a specific bias (your computer/model says the same!)".
    • bombela 6y ago
      Money laundering hides where the money comes from. Bias are subjective assumption you have of the world and people. So this quote is saying that machine learning hides the source of bias.
    • cpeterso 6y ago
      Before machine learning, a system (such as resume screening) that was biased was the fault of the rules and people implementing them. With machine learning, people can blame the “The Algorithm”, a numerical black box that can’t be inspected or interviewed. The system and people now claim they are blameless because “the computer can’t be biased“, even though it was trained on data produced by the previous biased system.
    • MereInterest 6y ago
      To add to the other answers, machine learning based on biased datasets results in biased models. For example, Word2Vec can be used to make analogies. Having it determine "Man is to woman as king is to ___." results in "queen", which is reasonable. On the other hand, determining "Man is to computer programmer as woman is to ___." results in "homemaker". The algorithm wasn't deliberately designed to have sexist bias in it, but there was implicit bias in the dataset from which it learned. https://www.technologyreview.com/2016/07/27/158634/how-vector-space-mathematics-reveals-the-hidden-sexism-in-language/ https://www.technologyreview.com/2016/07/27/158634/how-vecto... As another example, predictive policing tries to place police in places with higher crime rates. Those crime rates are determined by looking at past history of police reports and arrests. That past history has human bias already in it, with disproportionately higher arrest rates in places with racial minorities. The effect of the predictive policing is to justify overpolicing of minorities. https://en.wikipedia.org/wiki/Predictive_policing#Criticisms https://en.wikipedia.org/wiki/Predictive_policing#Criticisms
• PragmaticPulp 6y ago
  In general, blaming the algorithm is an indirect way of blaming software engineers. From Facebook to the VW emissions scandal to Stanford’s vaccine allocation, people love narratives that blame engineers. I suspect this trend will only get worse as the general public realizes that engineers are now a highly compensated professional, similar to how lawyers are the butt of so many jokes.
  • sfvisser 6y ago
    I don’t think people necessarily love narratives that blame engineers. Why would they? Some specific people in positions of responsibility that screw up badly see blaming engineers/tech as an easy way out of their responsibilities. Similar outcome different thing.
    • tedunangst 6y ago
      Yes, people love stories like the space pen vs the pencil. The supposedly educated engineer getting shown up by the clever average joe with common sense. There's always someone smarter than you, but see well actually they're not that smart because they're not street smart.
      • to11mtm 6y ago
        > Yes, people love stories like the space pen vs the pencil. The supposedly educated engineer getting shown up by the clever average joe with common sense. There's always someone smarter than you, but see well actually they're not that smart because they're not street smart. And yet that classic example is also an example of people really not understanding the nuance of larger systems; in that specific case, there were concerns about graphite creating electrical issues in the ship's electrical systems.
        • tedunangst 6y ago
          Surprise twist: the engineers aren't all stupid.
    • 6y ago
• rhino369 6y ago
  I'm not even sure this formula is WRONG. It's just not great for morale. Residents are generally under 34 and in good heath. In the USA, only ~2400 people 34 have died from COVID--and that is mostly people with comorbidites. In comparison about ~250,000 people over 55 have died. You'd obviously have do account for life-years lost, risk of exposure [1], etc. I wouldn't be shocked if giving the vaccine to a 25 year old resident is a sub-optimal choice. [1]Based on my MiL's experience (an ER doc), the risk of infection at a hospital is a lot lower than it used to be because they have PPE now. She's more worried about catching it from her son who works in an office. That said, they need to take into account risk of exposure and risk that the residents spread it to others.
  • XorNot 6y ago
    I mean a surface level analysis would say that you've missed a critical datum if you're saying patient-facing staff should be left at risk of being incapacitated from their duties for up to a month.
    • rhino369 6y ago
      That should definitely be a factor considered, especially in any area where a doctor shortage could occur. But a month sounds like a long time for <35 years olds in good health.
      • XorNot 6y ago
        14 days is the quarantine period to show symptoms. It is not how long the infection takes to run its course if you actually have it, which is generally 3-4 weeks presuming things go well, which they much more frequently do not.
        • deleted 6y ago
          [deleted]
  • deleted 6y ago
    [deleted]
• Spivak 6y ago
  I do think people understand that. Blaming the algorithm is a way of saying that we agreed on the rules by which vaccines would be assigned and that they were fair. And that we can’t just go back and change them because we don’t like the results when the rules are applied. At that point why even have a formal process for assigning vaccines when all we’re doing is coming a post-hoc justification for a particular outcome. If we already know the outcome we want then just use that!
• trianglem 6y ago
  With you right until the end where formulas are made by people. In many NN setups, the “formula” is pretty much a self trained black box so in those cases people can only be blamed for the input data that went into training the system.
  • mattkrause 6y ago
    I disagree—-that’s exactly the the “bias laundering” discussed above. The people/organization deploying a system are responsible for evaluating the system and how it might fail. You can’t just throw up your hands and go “Well, we had that data and this network so...it is what it is.” You could have fit a different model—-or not at all.
  • renewiltord 6y ago
    That's a bit like saying "I didn't burn the food. The stove burned the food. I just put the food on the stove".
• to11mtm 6y ago
  Standing on the shoulders of Giants, I would like to suggest the term 'Distributed Bureaucracy' to reflect when an algorithm you didn't even know existed determines your allocation of resources.
  • joe_the_user 6y ago
    And the thing here is, vaccine distribution in a single institution clearly wouldn't need an elaborate algorithm. But yeah, creating an algorithm and then throwing up your hands to say "who knew!" is very convenient.
• moocow01 6y ago
  I'm more curious why there was an algorithm involved at all. Based upon the number of doses they had, they should have immediately been given to as many staff on first responder units, like ER, ICU, etc as would take it. My wife works at a hospital as a nurse and this is what they have done for the first phase of vaccination. After that I can see a need for an algorithm to determine who is most at need but there is no excuse for the administrators for what happened at Stanford.
• cbanek 6y ago
  “Once men turned their thinking over to machines in the hope that this would set them free. But that only permitted other men with machines to enslave them.” - Dune
• icedistilled 6y ago
  My pet peeve is when people present averages with no care. If the average is is the result they want to see they assume everything is fine under the hood. There's so many lessons why that is terrible, like the fighter cockpit design history, and yet it's super common for people to stop all inquisition when the average shows them what they hope is true across all subgroups.
• adrianmonk 6y ago
  The most charitable interpretation is that they're admitting they failed, but saying where they failed is in basic due diligence of looking at the output of running the algorithm. And that they're claiming that, if they had done so, they wouldn't have been OK with it either. It's sort of like saying, "That's definitely a bug. It was unacceptable for me to put my code into production without ever testing it, and for that I take the blame. But I assure you that I did not intentionally write code designed to delete everyone's data." Obviously this may or may not be what happened. It may even be what they're pretending happened.
• glaugh 6y ago
  Fwiw, the administration said: “”” We are writing to acknowledge the significant concerns expressed by our community regarding the development and execution of our vaccine distribution plan. We take complete responsibility and profusely apologize to all of you. We fully recognize we should have acted more swiftly to address the errors that resulted in an outcome we did not anticipate “”” The chief resident sent an email explaining the nature of the error. The chief resident is, per my limited understanding, a resident from the previous year who stays on in a leadership role. Not obvious to me that someone in that position explaining the root cause is problematic.
  • stephencanon 6y ago
    Re-read it; the letter from the chief residents (who are residents in their final year of training, in some departments it’s everyone in their last year, in others selected leaders) was _protesting_ the allocation. The letter explaining the error was from hospital leadership, which does not include the chief residents. The email sent by a chief resident was simply passing on the administration’s explanation.
    • pdxandi 6y ago
      Chief residents are not always in their final year of training. In some cases, like pediatrics, it's a subsequent leadership year.
      • stephencanon 6y ago
        Ah, true. All my friends are surgeons =)
• JoshTko 6y ago
  It's probably not even a real algorithm and just some janky excel spreadsheet.
• tshaddox 6y ago
  Sounds very similar to “don’t blame the decision maker, because they were just following a moral/legal principle.” Except that, bizarrely, “principles” are usually regarded in high esteem even by people who vehemently disagree with the principles and the decisions excused by them. How often do you hear things (especially in US politics) like “well at least he/she is a principled politician, that’s so rare these days.”
• tomcam 6y ago
  Also “model” is a synonym for “guess”