8 ms·

> Ban social scoring, not the AI, the tool doesn't matter. Automated human classification without recourse nor transparency is what the EU is against. "You can

> Ban social scoring, not the AI, the tool doesn't matter. Automated human classification without recourse nor transparency is what the EU is against. "You cannot get a loan because the AI says so" is not acceptable. > Would it be good if it was computed by hand on paper instead of through AI? Yes. As a judge can check what information was used and what algorithm applied to get to a conclusion. And it may declare that use of data or algorithm illegal. > They fear the amplification factor of such tech, but the amplification factor works for the good and for the bad Maybe just go slower. Something like an Hippocratic oath can be a good principle to follow for high impact technologies. To do some good may not justify the harm.

concordDance 4y ago
> "You cannot get a loan because the AI says so" is not acceptable. To me this makes it seem like acces to loans is a right,in which case the EU should make a lender of last resort that can loan to people who would otherwise get rejected.
- barrkel 4y ago
  How do you leap from banning black box AI decisions to thinking it means everything is allowed?
  - edgineer 4y ago
    Perhaps the assumption is that humans functioning as black box loan deniers is also a problem.
  - px43 4y ago
    Haven't loan decisions always been black box? Sometimes they'll attribute decisions to credit scores, but those scoring systems have always been proprietary, and they won't even tell you what data your scoring is based on.
    - gopher_space 4y ago
      > Haven't loan decisions always been black box? Not to the people making those decisions. That's brand new.
      - dogma1138 4y ago
        People haven’t been making loan decisions for decades in the EU as well. You put numbers into a computer and what comes out is either an approval with a rate or a decline. No bank EU or not has people manually approving loans.
        ClumsyPilot 4y ago
        You can file a lawsuit 'this bank does not give me loan because I am not white' and check. You can inspect the excell formula in the computer. It does not matter who personally made that decisi9n, the bank is operating this code and paid someone to write it. With 'AI' there is no way to know.
        dogma1138 4y ago
        No, what you are entitled in the EU is to have the decision reviewed by a human, I fail to see how an AI changes that.
        ClumsyPilot 4y ago
        No what? Let's imagine my bank is sexist, the code literally says "if WOMAN, Reject' You can sue my bank for discrimination, and in process of discovery you can request information and documents from the bank, including the code / formulas. If you have a solid argument, let's say my bank never approved a mortgage for a woman, the judge will grant your request. You, or someone appointed on your behalf, will look at the code / excel formula, see that it's discriminatory by design, and you can request massive punitive fines, because this demonstrates contempt for the law. Now let's say I instruct my developers to create a neural network that is as convoluted as possible but is unlikely to approves mortgage for a woman. That can be easily done. There will be no proof that we ever did this on purpose, and even proving that it is discriminatory will be challenging. At best, my bank will get a slap on the wrist. That is why they are banning AI from some areas of life.
  - concordDance 4y ago
    As the other commentators said, humans are also black box. They can, of course, provide vaguely reasonable justifications but these often aren't the actual reasons and ChatGPT can generate reasonable sounding rationalisations on demand.
corbulo 4y ago
>Automated human classification without recourse nor transparency is what the EU is against. "You cannot get a loan because the AI says so" is not acceptable. The credit score system already in place in the US is frequently as arbitrary in some cases already. You get punished for just looking at it. I don't think I need to preach to HN about that (crypto money and all that). Does Europe do it much differently? Not an arguement for AI to do it, but the only reason why the current system is better (at least in the US) is its sluggish incompetence.
- kelnos 4y ago
  > You get punished for just looking at it. Not completely false, but misleading: you get penalized if a lender looks at it with the express purpose of using it to decide whether or not to offer you credit (called a "hard pull"). You looking at your own credit report does not penalize you, and certain other activities (like getting pre-approved for a loan) won't either. The penalty for hard pulls isn't that bad, anyway, and disappears after a relatively short while. I currently have 7 hard pulls (I've been chasing credit card rewards offers lately), but my overall credit score is still over 800.
  - IMTDb 4y ago
    So, GET request not only aren't idempotent but they also affect the party that is read. Who came up with that lovely system ? I gave it roughly 15 seconds of thought and came come up with multiple ways this design fucks up people lives.
    - sokoloff 4y ago
      It’s not a GET request on the credit history; those do not affect the score. It’s a PUT* request on the “considering taking additional credit” record. * perhaps closer to PATCH as multiple requests in short time are combined as one (as in mortgage shopping)
    - fnordpiglet 4y ago
      The key though is this operation only happens when it’s part of a requested credit evaluation for the purposes of expanding credit. The act of requesting new credit diminishes your credit worthiness. The intent is to throttle issuance of credit by providing for a race condition - if I go out and apply for 100 credit cards at the same time, they each get a credit report without the others line of credit established because they’re all in the pending state. But each request to evaluate the worthiness of credit dampens the quality of credit. At a certain point all credit requests thereafter fail. On the flip side it also causes a hole that’s increasingly hard to dig out of because if you have bad credit or borderline credit and apply for a card to see if you’re allowed back in at the capitalist buffet, you enter into a feedback loop of diminishing credit worthiness. It isn’t true however that any form of read changes credit worthiness. There are many routine reads that have no impact. It’s specifically when you’ve directly requested an evaluation for expanded credit. It can’t happen passively, as sometimes banks refresh your credit history and even offer more credit automatically. It also can’t happen for other purposes than expansion of credit.
ComplexSystems 4y ago
> Yes. As a judge can check what information was used and what algorithm applied to get to a conclusion. And it may declare that use of data or algorithm illegal. A ban on "AI social scoring" would be neither necessary nor sufficient to make your scenario a reality.
deleted 4y ago
[deleted]
RC_ITR 4y ago
> Yes. As a judge can check what information was used and what algorithm applied to get to a conclusion. And it may declare that use of data or algorithm illegal. Genuinely, why does process matter? Shouldn’t outcomes? Maybe you’re not from the US, but we have a looooong history of “good unbiased processes” leading to awful and biased outcomes.
- LatteLazy 4y ago
  The process is what we have to judge because in inputs are bias, so even if with a perfectly fair process you will get bias outcomes...
- AnthonyMouse 4y ago
  Outcomes are what the process decides. If you had some other process that could tell you what the outcome should be, you could have just used that as the original process.
- makeitdouble 4y ago
  To note, the discussion is on the EU side. Yes, process matters in that they need to be reviewable and fixable. Them being good or unbiased is a function of how much scrutinity and oversight there is, but before even talking about biases, we need to know what's happening. Even outside of fixing the system, in most countries if your get a credit application refused you can request the info that lead to the decision and act on it (update/fix your credit history for instance). Even if you can't fix specific aspects you at least know what conditions block you and deal with it accordingly (no use in reapplying twelve times if you're banned for life for instance, same way you might want to wait 6 months if it's a matter of prescription) So yes, knowing what, why and how is as critical as the final outcome.
  - mytailorisrich 4y ago
    Sometimes there seems to be an expectation that people are entitled to a loan if they meet some criteria. That's not the case. As long as there is no discrimination based on a legally protected characteristic (e.g. race) you can lend to whoever you want based on whatever criteria you want.
usrbinbash 4y ago
> To do some good may not justify the harm. Bread can be baked to feed the needy, or to fill the bellies of soldiers who invade a peaceful neighbor. Who is to blame? The people inventing industrial bakeries, or the people using them to feed their invasion army?

concordDance 4y ago

> "You cannot get a loan because the AI says so" is not acceptable. To me this makes it seem like acces to loans is a right,in which case the EU should make a lender of last resort that can loan to people who would otherwise get rejected.

barrkel 4y ago
How do you leap from banning black box AI decisions to thinking it means everything is allowed?
- edgineer 4y ago
  Perhaps the assumption is that humans functioning as black box loan deniers is also a problem.
- px43 4y ago
  Haven't loan decisions always been black box? Sometimes they'll attribute decisions to credit scores, but those scoring systems have always been proprietary, and they won't even tell you what data your scoring is based on.
  - gopher_space 4y ago
    > Haven't loan decisions always been black box? Not to the people making those decisions. That's brand new.
    - dogma1138 4y ago
      People haven’t been making loan decisions for decades in the EU as well. You put numbers into a computer and what comes out is either an approval with a rate or a decline. No bank EU or not has people manually approving loans.
      - ClumsyPilot 4y ago
        You can file a lawsuit 'this bank does not give me loan because I am not white' and check. You can inspect the excell formula in the computer. It does not matter who personally made that decisi9n, the bank is operating this code and paid someone to write it. With 'AI' there is no way to know.
        dogma1138 4y ago
        No, what you are entitled in the EU is to have the decision reviewed by a human, I fail to see how an AI changes that.
        ClumsyPilot 4y ago
        No what? Let's imagine my bank is sexist, the code literally says "if WOMAN, Reject' You can sue my bank for discrimination, and in process of discovery you can request information and documents from the bank, including the code / formulas. If you have a solid argument, let's say my bank never approved a mortgage for a woman, the judge will grant your request. You, or someone appointed on your behalf, will look at the code / excel formula, see that it's discriminatory by design, and you can request massive punitive fines, because this demonstrates contempt for the law. Now let's say I instruct my developers to create a neural network that is as convoluted as possible but is unlikely to approves mortgage for a woman. That can be easily done. There will be no proof that we ever did this on purpose, and even proving that it is discriminatory will be challenging. At best, my bank will get a slap on the wrist. That is why they are banning AI from some areas of life.
- concordDance 4y ago
  As the other commentators said, humans are also black box. They can, of course, provide vaguely reasonable justifications but these often aren't the actual reasons and ChatGPT can generate reasonable sounding rationalisations on demand.

corbulo 4y ago

>Automated human classification without recourse nor transparency is what the EU is against. "You cannot get a loan because the AI says so" is not acceptable. The credit score system already in place in the US is frequently as arbitrary in some cases already. You get punished for just looking at it. I don't think I need to preach to HN about that (crypto money and all that). Does Europe do it much differently? Not an arguement for AI to do it, but the only reason why the current system is better (at least in the US) is its sluggish incompetence.

kelnos 4y ago
> You get punished for just looking at it. Not completely false, but misleading: you get penalized if a lender looks at it with the express purpose of using it to decide whether or not to offer you credit (called a "hard pull"). You looking at your own credit report does not penalize you, and certain other activities (like getting pre-approved for a loan) won't either. The penalty for hard pulls isn't that bad, anyway, and disappears after a relatively short while. I currently have 7 hard pulls (I've been chasing credit card rewards offers lately), but my overall credit score is still over 800.
- IMTDb 4y ago
  So, GET request not only aren't idempotent but they also affect the party that is read. Who came up with that lovely system ? I gave it roughly 15 seconds of thought and came come up with multiple ways this design fucks up people lives.
  - sokoloff 4y ago
    It’s not a GET request on the credit history; those do not affect the score. It’s a PUT* request on the “considering taking additional credit” record. * perhaps closer to PATCH as multiple requests in short time are combined as one (as in mortgage shopping)
  - fnordpiglet 4y ago
    The key though is this operation only happens when it’s part of a requested credit evaluation for the purposes of expanding credit. The act of requesting new credit diminishes your credit worthiness. The intent is to throttle issuance of credit by providing for a race condition - if I go out and apply for 100 credit cards at the same time, they each get a credit report without the others line of credit established because they’re all in the pending state. But each request to evaluate the worthiness of credit dampens the quality of credit. At a certain point all credit requests thereafter fail. On the flip side it also causes a hole that’s increasingly hard to dig out of because if you have bad credit or borderline credit and apply for a card to see if you’re allowed back in at the capitalist buffet, you enter into a feedback loop of diminishing credit worthiness. It isn’t true however that any form of read changes credit worthiness. There are many routine reads that have no impact. It’s specifically when you’ve directly requested an evaluation for expanded credit. It can’t happen passively, as sometimes banks refresh your credit history and even offer more credit automatically. It also can’t happen for other purposes than expansion of credit.

ComplexSystems 4y ago

> Yes. As a judge can check what information was used and what algorithm applied to get to a conclusion. And it may declare that use of data or algorithm illegal. A ban on "AI social scoring" would be neither necessary nor sufficient to make your scenario a reality.

deleted 4y ago

[deleted]

RC_ITR 4y ago

> Yes. As a judge can check what information was used and what algorithm applied to get to a conclusion. And it may declare that use of data or algorithm illegal. Genuinely, why does process matter? Shouldn’t outcomes? Maybe you’re not from the US, but we have a looooong history of “good unbiased processes” leading to awful and biased outcomes.

LatteLazy 4y ago
The process is what we have to judge because in inputs are bias, so even if with a perfectly fair process you will get bias outcomes...
AnthonyMouse 4y ago
Outcomes are what the process decides. If you had some other process that could tell you what the outcome should be, you could have just used that as the original process.
makeitdouble 4y ago
To note, the discussion is on the EU side. Yes, process matters in that they need to be reviewable and fixable. Them being good or unbiased is a function of how much scrutinity and oversight there is, but before even talking about biases, we need to know what's happening. Even outside of fixing the system, in most countries if your get a credit application refused you can request the info that lead to the decision and act on it (update/fix your credit history for instance). Even if you can't fix specific aspects you at least know what conditions block you and deal with it accordingly (no use in reapplying twelve times if you're banned for life for instance, same way you might want to wait 6 months if it's a matter of prescription) So yes, knowing what, why and how is as critical as the final outcome.
- mytailorisrich 4y ago
  Sometimes there seems to be an expectation that people are entitled to a loan if they meet some criteria. That's not the case. As long as there is no discrimination based on a legally protected characteristic (e.g. race) you can lend to whoever you want based on whatever criteria you want.

usrbinbash 4y ago

> To do some good may not justify the harm. Bread can be baked to feed the needy, or to fill the bellies of soldiers who invade a peaceful neighbor. Who is to blame? The people inventing industrial bakeries, or the people using them to feed their invasion army?