5 ms·

If you have to be able to review any decision manually that people complain about, that is significantly limiting. It means you can only write software to do th

If you have to be able to review any decision manually that people complain about, that is significantly limiting. It means you can only write software to do things humans can do. I don't think it's alarmist to worry about the GDPR. It's an extremely far-reaching law covering types of technology that are changing very rapidly. Our default reaction should be skepticism, because the default outcome of that combination is disaster.

annabellish 9y ago
I don't think that's quite correct, many (most?) kinds of AI achieve things which are difficult or impossible for humans to do, but that doesn't mean you can't figure out why a decision was made afterwards. There are some scenarios where that isn't possible, but... if nobody knows why decisions impacting people's lives are made and it isn't possible to determine what they were done for, is that something we really want?
- bobcostas55 9y ago
  >if nobody knows why decisions impacting people's lives are made and it isn't possible to determine what they were done for, is that something we really want? We do this every day with human cognition and nobody complains.
  - montrose 9y ago
    It's too bad your comment is buried so far down in the conversation, because this is probably the most insightful point I've seen on the internet today. And not only do we do this, we have to.
    - ethbro 9y ago
      I don't think that's an accurate analogy, at least at the enterprise level. Generally speaking, individuals do not make decisions. They may influence decisions or find novel ways to apply rules, but anything organizationally important is written down somewhere. Typically for legal purposes.
  - danieltillett 9y ago
    Yes we do this everyday, but there is lots of complaining.
  - smsm42 9y ago
    It is not exactly true that nobody complains. There a lot of discrimination complaints to go around. But since we do not have means to open the brain and check what was the reason for this or that decision, deducing motivations is hard (unless you are commenting on internet, where everybody knows everybody else's motivations and is not afraid to explain it voluminously to each other). It would also be hard with algorithms, so basically the approach seems to be "reducing the problem to one we already know" - i.e. switching algorithmic unscrutability for human unscrutability and hoping that humans would do better or at least in won't be worse than what we have already been dealing with for millenia.
  - ThePhysicist 9y ago
    I think one of the reasons that we have a higher standard for algorithmic decision making is the fact that algorithms are used at scale: For example, for a classification task we usually train an algorithm on labeled data (i.e. good debtor vs. bad debtor). When trained, we apply the algorithms to ALL future decisions on whom to give a credit. This makes algorithms so dangerous as compared to humans, because if 10 out of 1000 bankers are racist and discriminate against certain groups of people we still have 990 that don't. If our algorithm is trained on data that contains racism it will discriminate in every single case if it has the data to do so.
ThePhysicist 9y ago
As you said it's often not possible to explain exactly why an AI-based algorithm took a given decision (if it were we wouldn't need such a complicated algorithm in the first place), but this isn't necessary either. What you need to explain to the data subject is the purpose of the automated decision making, the data you use for it and the basic logic structure of your algorithm. Again, I can only recommend to read the guidelines issued by the Article 29 working party, as they will guide the implementation and enforcement of the regulation in the member states. An overview can be found here (the relevant document is "wp251"): http://ec.europa.eu/newsroom/just/item-detail.cfm?item_id=50083 http://ec.europa.eu/newsroom/just/item-detail.cfm?item_id=50...
deleted 9y ago
[deleted]
gaius 9y ago
If you have to be able to review any decision manually that people complain about, that is significantly limiting. It means you can only write software to do things humans can do If you create an algorithm that by pure coincidence happens to discriminate against a protected class then any judge will take a dim view of you claiming to be unable to explain it. I’d like to encourage everyone, even if just for fun, to insist on an explanation for every algorithmic decision they may be subjected to
- londons_explore 9y ago
  The HN ranking algorithm ranked this comment below others. I demand a human explanation.
  - geocar 9y ago
    What significant legal effects do you believe the HN ranking algorithm of that comment has concerning you directly?
    - matte_black 9y ago
      Discrimination.
    - dominotw 9y ago
      Do you need significant legal effects(no quite sure what that means)?
      - geocar 9y ago
        Yes. Article 22[1] gives you the right to not be subject to a decision made solely by automated processing which produces legal effects concerning you or similarly significantly affects you. [1]: https://gdpr-info.eu/art-22-gdpr/ https://gdpr-info.eu/art-22-gdpr/
        omginternets 9y ago
cf498 9y ago
You forget to mention, that this isnt a regulation covering a technology. Its about how a specific technology is utilised in a very specific use case. The entire debate is, like the name suggests, about the regulations of how personal information can be used in an economic context. Unsurprisingly, some of these use cases are illegal now and existed in a loophole before. To simplyfy, framing a question this way is similar to asking if rights of gun ownership are infringed by outlawing shooting people.
mnm1 9y ago
You can develop whatever you want, just don't subject people to arbitrary binding decisions without appeal. The default attitude is one of skepticism, skepticism towards technology that could potentially affect human lives in very nasty ways. If you want your technology adopted instead of banned, it might be a good idea to listen to the people who will ultimately have to deal with the consequences of such technology. Otherwise, why should society adopt a technology that they cannot understand, that possibly no one understands or can explain or will explain? How can society trust you as the creator of such technology when it's a black box?
- amarkov 9y ago
  Your basic premise, that society as a whole ought to approve of new technologies before anyone is allowed to use them, seems very strange.
  - jackvalentine 9y ago
    There is nothing in it about approval, only adoption.
  - mnm1 9y ago
    This isn't a case of ought or ought not to, it's just what happens. Society does indeed either approve or disapprove of certain technologies, not always before they are used. Say you develop ai to replace judges. Don't you think society needs to approve this before we replace human judges? What if someone trained the ai to give all non white people sentences 20% longer than white people? Society will, hopefully, balk at this idea and such a system will not be implemented or scrapped if it has been. If, on the other hand, they don't know the system is biased because its decisions are hidden from the public, it'll be a system that society might accept on false pretenses, leading to mass suffering. For technology that could potentially be used to oppress people, I think people have the right to know how it works and to ask for a human to verify or carry the actual task out. Otherwise it wouldn't be surprising that they would want to ban or severely restrict something so potentially harmful.
  - Thiez 9y ago
raverbashing 9y ago
> to review any decision manually that people complain about, that is significantly limiting. It means you can only write software to do things humans can do So are you happy with a black box that just says "computer says no" without any explanation or way to review this decision? "Sorry, your insurance premium is now 300EUR higher than next year, computer says so"
- geocar 9y ago
  A better example: An insurance claim might use a sophisticated anti-fraud algorithm to reject claims, but they actually have to be rejecting claims for reasons of fraud, and not because "computer says so".
- dominotw 9y ago
  I've asked auto insurance how they arrived at that number, they always give me some vague generic answer.
s73v3r_ 9y ago
I’m not seeing a problem. There should never be a situation where a human cannot review the decision and reverse it.
- will4274 9y ago
  > Why is x 73rd on Google search ranking? It's unlikely that a human would be able to produce the exact same ordering, for any possible search term for all the pages and pages of results that Google shows.
  - dannyw 9y ago
    Since when are search results personal data?
    - manfredo 9y ago
      They are affected by personal data. E.g. if you performed a search, and your friend next to you performed the same search and got different results there is probably no way Google could give a concise or exact explanation as to why your results were different.
    - will4274 9y ago
      I would guess personalized search results went mainstream on 2005 or so, as Google became popular, but perhaps some of the portals Google replaced (ask Jeeves and Yahoo) had personalized results before that
- dominotw 9y ago
  It's not just reversing you should be able to explain how it arrived to that decision in the first place.
Someone 9y ago
”It means you can only write software to do things humans can do.” For the things computers can do, they typically are way faster and way more reliable, but I don’t see how there could be anything software can do that humans can’t (quantum computing may change that) Also, being able to explain what the computer did doesn’t imply being able to do it. As a (silly) example, “We hashed the c.v. you sent us, getting a 150-digit number N. The N-th digit in the hexadecimal expansion of pi isn’t a zero, so you didn’t get invited for an interview” would be a perfectly good explanation of your hiring practices, even if you don’t know how to compute any of those decimals.

annabellish 9y ago

I don't think that's quite correct, many (most?) kinds of AI achieve things which are difficult or impossible for humans to do, but that doesn't mean you can't figure out why a decision was made afterwards. There are some scenarios where that isn't possible, but... if nobody knows why decisions impacting people's lives are made and it isn't possible to determine what they were done for, is that something we really want?

bobcostas55 9y ago
>if nobody knows why decisions impacting people's lives are made and it isn't possible to determine what they were done for, is that something we really want? We do this every day with human cognition and nobody complains.
- montrose 9y ago
  It's too bad your comment is buried so far down in the conversation, because this is probably the most insightful point I've seen on the internet today. And not only do we do this, we have to.
  - ethbro 9y ago
    I don't think that's an accurate analogy, at least at the enterprise level. Generally speaking, individuals do not make decisions. They may influence decisions or find novel ways to apply rules, but anything organizationally important is written down somewhere. Typically for legal purposes.
- danieltillett 9y ago
  Yes we do this everyday, but there is lots of complaining.
- smsm42 9y ago
  It is not exactly true that nobody complains. There a lot of discrimination complaints to go around. But since we do not have means to open the brain and check what was the reason for this or that decision, deducing motivations is hard (unless you are commenting on internet, where everybody knows everybody else's motivations and is not afraid to explain it voluminously to each other). It would also be hard with algorithms, so basically the approach seems to be "reducing the problem to one we already know" - i.e. switching algorithmic unscrutability for human unscrutability and hoping that humans would do better or at least in won't be worse than what we have already been dealing with for millenia.
- ThePhysicist 9y ago
  I think one of the reasons that we have a higher standard for algorithmic decision making is the fact that algorithms are used at scale: For example, for a classification task we usually train an algorithm on labeled data (i.e. good debtor vs. bad debtor). When trained, we apply the algorithms to ALL future decisions on whom to give a credit. This makes algorithms so dangerous as compared to humans, because if 10 out of 1000 bankers are racist and discriminate against certain groups of people we still have 990 that don't. If our algorithm is trained on data that contains racism it will discriminate in every single case if it has the data to do so.

ThePhysicist 9y ago

As you said it's often not possible to explain exactly why an AI-based algorithm took a given decision (if it were we wouldn't need such a complicated algorithm in the first place), but this isn't necessary either. What you need to explain to the data subject is the purpose of the automated decision making, the data you use for it and the basic logic structure of your algorithm. Again, I can only recommend to read the guidelines issued by the Article 29 working party, as they will guide the implementation and enforcement of the regulation in the member states. An overview can be found here (the relevant document is "wp251"): http://ec.europa.eu/newsroom/just/item-detail.cfm?item_id=50083 http://ec.europa.eu/newsroom/just/item-detail.cfm?item_id=50...

deleted 9y ago

[deleted]

gaius 9y ago

If you have to be able to review any decision manually that people complain about, that is significantly limiting. It means you can only write software to do things humans can do If you create an algorithm that by pure coincidence happens to discriminate against a protected class then any judge will take a dim view of you claiming to be unable to explain it. I’d like to encourage everyone, even if just for fun, to insist on an explanation for every algorithmic decision they may be subjected to

londons_explore 9y ago
The HN ranking algorithm ranked this comment below others. I demand a human explanation.
- geocar 9y ago
  What significant legal effects do you believe the HN ranking algorithm of that comment has concerning you directly?
  - matte_black 9y ago
    Discrimination.
  - dominotw 9y ago
    Do you need significant legal effects(no quite sure what that means)?
    - geocar 9y ago
      Yes. Article 22[1] gives you the right to not be subject to a decision made solely by automated processing which produces legal effects concerning you or similarly significantly affects you. [1]: https://gdpr-info.eu/art-22-gdpr/ https://gdpr-info.eu/art-22-gdpr/
      - omginternets 9y ago

cf498 9y ago

You forget to mention, that this isnt a regulation covering a technology. Its about how a specific technology is utilised in a very specific use case. The entire debate is, like the name suggests, about the regulations of how personal information can be used in an economic context. Unsurprisingly, some of these use cases are illegal now and existed in a loophole before. To simplyfy, framing a question this way is similar to asking if rights of gun ownership are infringed by outlawing shooting people.

mnm1 9y ago

You can develop whatever you want, just don't subject people to arbitrary binding decisions without appeal. The default attitude is one of skepticism, skepticism towards technology that could potentially affect human lives in very nasty ways. If you want your technology adopted instead of banned, it might be a good idea to listen to the people who will ultimately have to deal with the consequences of such technology. Otherwise, why should society adopt a technology that they cannot understand, that possibly no one understands or can explain or will explain? How can society trust you as the creator of such technology when it's a black box?

amarkov 9y ago
Your basic premise, that society as a whole ought to approve of new technologies before anyone is allowed to use them, seems very strange.
- jackvalentine 9y ago
  There is nothing in it about approval, only adoption.
- mnm1 9y ago
  This isn't a case of ought or ought not to, it's just what happens. Society does indeed either approve or disapprove of certain technologies, not always before they are used. Say you develop ai to replace judges. Don't you think society needs to approve this before we replace human judges? What if someone trained the ai to give all non white people sentences 20% longer than white people? Society will, hopefully, balk at this idea and such a system will not be implemented or scrapped if it has been. If, on the other hand, they don't know the system is biased because its decisions are hidden from the public, it'll be a system that society might accept on false pretenses, leading to mass suffering. For technology that could potentially be used to oppress people, I think people have the right to know how it works and to ask for a human to verify or carry the actual task out. Otherwise it wouldn't be surprising that they would want to ban or severely restrict something so potentially harmful.
- Thiez 9y ago

raverbashing 9y ago

> to review any decision manually that people complain about, that is significantly limiting. It means you can only write software to do things humans can do So are you happy with a black box that just says "computer says no" without any explanation or way to review this decision? "Sorry, your insurance premium is now 300EUR higher than next year, computer says so"

geocar 9y ago
A better example: An insurance claim might use a sophisticated anti-fraud algorithm to reject claims, but they actually have to be rejecting claims for reasons of fraud, and not because "computer says so".
dominotw 9y ago
I've asked auto insurance how they arrived at that number, they always give me some vague generic answer.

s73v3r_ 9y ago

I’m not seeing a problem. There should never be a situation where a human cannot review the decision and reverse it.

will4274 9y ago
> Why is x 73rd on Google search ranking? It's unlikely that a human would be able to produce the exact same ordering, for any possible search term for all the pages and pages of results that Google shows.
- dannyw 9y ago
  Since when are search results personal data?
  - manfredo 9y ago
    They are affected by personal data. E.g. if you performed a search, and your friend next to you performed the same search and got different results there is probably no way Google could give a concise or exact explanation as to why your results were different.
  - will4274 9y ago
    I would guess personalized search results went mainstream on 2005 or so, as Google became popular, but perhaps some of the portals Google replaced (ask Jeeves and Yahoo) had personalized results before that
dominotw 9y ago
It's not just reversing you should be able to explain how it arrived to that decision in the first place.

Someone 9y ago

”It means you can only write software to do things humans can do.” For the things computers can do, they typically are way faster and way more reliable, but I don’t see how there could be anything software can do that humans can’t (quantum computing may change that) Also, being able to explain what the computer did doesn’t imply being able to do it. As a (silly) example, “We hashed the c.v. you sent us, getting a 150-digit number N. The N-th digit in the hexadecimal expansion of pi isn’t a zero, so you didn’t get invited for an interview” would be a perfectly good explanation of your hiring practices, even if you don’t know how to compute any of those decimals.