3 ms·

I don't assume this is easy or even possible to solve. If the model is trying to determine something like the applicant's level of adversity based on whatever i

I don't assume this is easy or even possible to solve. If the model is trying to determine something like the applicant's level of adversity based on whatever info college admissions has, of course the result will have a divergence between races, and historical racism will have shaped some of your input variables (like high school) to some degree that we'll never fully understand. Using some statistical difference between races as a cost function wouldn't be satisfactory either. All I'm saying is at the bare minimum, don't feed race as an input. It's a vague factor that the applicant has zero control over, and a model (or human reviewer) making any assumptions based on race would be inherently racist, which is unjustifiable. Another bad example is how UCSD asks how feminine (rate 0-5) the applicant considers themselves. Only use factors that have some legitimate purpose; at least the high school matters because of grade inflation, but standardized tests are actually fairer. US hiring is this way, where many companies consider it an unwanted risk to even know about protected categories (age, race, religion, marital status, etc). It's far from an airtight solution. This isn't really about science, and the article is off-topic for HN.

godelski 4y ago
> Using some statistical difference between races as a cost function wouldn't be satisfactory either. So how do you expect the expert to de-bias the system without some cost function?
- hot_gril 4y ago
  I meant a cost function that the model trains on. The expert can look at the cost function to double-check whether the inputs are appropriate. Maybe the applicants' names got in there by mistake, and the names serve no purpose other than giving away their ethnicities.
  - godelski 4y ago
    > I meant a cost function that the model trains on. The expert can look at the cost function to double-check whether the inputs are appropriate. Okay, but how is the expert going to judge that the outputs of the model are appropriate? What techniques do they use?
    - hot_gril 4y ago
      If the output seems to strongly favor some races over others, the expert debugs the inputs by temporarily swapping some out and rerunning. This is only a second line of defense beyond checking the inputs manually. Maybe the expert doesn't realize that some factor has a huge correlation with race without saying much about the candidate's merit. Beyond that, if there's a divergence between races, it is what it is. The expert shares what factors are being considered, and it's politics at that point.
      - godelski 4y ago
        > If the output seems to strongly favor some races over others, You'd typically do this with a cost function. > the expert debugs the inputs by temporarily swapping some out and rerunning. The best way to do this is to use that cost function to optimize your parameters. > Of course, could do this without race data just by proofreading. I thought I've adequately demonstrated why this is much easier said than done. > Beyond that, if there's a divergence between races, it is what it is. Except it isn't. It depends if the divergence is because you've trained a model on poisoned data (i.e. the historical racism gives momentum to your model). These things are rather tricky. It really feels like we're going in circles here and I'm trying to get you to understand why we do what we do, but you're avoiding getting specific to support your assumptions. I work with data all day every day. I'm sure you've done some data analysis too, but to do high quality work you have to consider a lot of factors. It's okay to not have this domain expertise. You're a general software engineer and probably far better than I am at that. But it is a bit naive to argue with domain experts about something that isn't in your domain. It would be naive for me to argue with you about your expertise. That's okay though. Our world is highly specialized. No one can know everything. But that's why I wrote so much before, because this is a toxic pattern in our culture that we need to curb.
        hot_gril 4y ago
        Is your domain expertise in college admissions, data science, or a combination? You assuming this model should try and undo some effects of historical racism, but that isn't the engineers' job to decide. If that's the intended behavior, I understand that the difficulties involved are beyond my experience, and I would leave it to the data scientists. That's not what I think the goal of college admissions should be, and our government is making a decision on that now. > The best way to do this is to use that cost function to optimize your parameters. > I thought I've adequately demonstrated why this is much easier said than done. I understand how this works, but that's specifically what I don't want, for race (or other data that serves no purpose but to indicate the race, like name) to be part of the training data. Those inputs are manually selected at some level.

godelski 4y ago

> Using some statistical difference between races as a cost function wouldn't be satisfactory either. So how do you expect the expert to de-bias the system without some cost function?

hot_gril 4y ago
I meant a cost function that the model trains on. The expert can look at the cost function to double-check whether the inputs are appropriate. Maybe the applicants' names got in there by mistake, and the names serve no purpose other than giving away their ethnicities.
- godelski 4y ago
  > I meant a cost function that the model trains on. The expert can look at the cost function to double-check whether the inputs are appropriate. Okay, but how is the expert going to judge that the outputs of the model are appropriate? What techniques do they use?
  - hot_gril 4y ago
    If the output seems to strongly favor some races over others, the expert debugs the inputs by temporarily swapping some out and rerunning. This is only a second line of defense beyond checking the inputs manually. Maybe the expert doesn't realize that some factor has a huge correlation with race without saying much about the candidate's merit. Beyond that, if there's a divergence between races, it is what it is. The expert shares what factors are being considered, and it's politics at that point.
    - godelski 4y ago
      > If the output seems to strongly favor some races over others, You'd typically do this with a cost function. > the expert debugs the inputs by temporarily swapping some out and rerunning. The best way to do this is to use that cost function to optimize your parameters. > Of course, could do this without race data just by proofreading. I thought I've adequately demonstrated why this is much easier said than done. > Beyond that, if there's a divergence between races, it is what it is. Except it isn't. It depends if the divergence is because you've trained a model on poisoned data (i.e. the historical racism gives momentum to your model). These things are rather tricky. It really feels like we're going in circles here and I'm trying to get you to understand why we do what we do, but you're avoiding getting specific to support your assumptions. I work with data all day every day. I'm sure you've done some data analysis too, but to do high quality work you have to consider a lot of factors. It's okay to not have this domain expertise. You're a general software engineer and probably far better than I am at that. But it is a bit naive to argue with domain experts about something that isn't in your domain. It would be naive for me to argue with you about your expertise. That's okay though. Our world is highly specialized. No one can know everything. But that's why I wrote so much before, because this is a toxic pattern in our culture that we need to curb.
      - hot_gril 4y ago
        Is your domain expertise in college admissions, data science, or a combination? You assuming this model should try and undo some effects of historical racism, but that isn't the engineers' job to decide. If that's the intended behavior, I understand that the difficulties involved are beyond my experience, and I would leave it to the data scientists. That's not what I think the goal of college admissions should be, and our government is making a decision on that now. > The best way to do this is to use that cost function to optimize your parameters. > I thought I've adequately demonstrated why this is much easier said than done. I understand how this works, but that's specifically what I don't want, for race (or other data that serves no purpose but to indicate the race, like name) to be part of the training data. Those inputs are manually selected at some level.