4 ms·

In what world do software companies let AI make decisions in a black box without spitting out logs or some code that maps to the action it made?

nottorp 4y ago
Ours. Perhaps you have missed all the posts here on HN where some company or other locks up someone's account without the user being told why and without the possibility of having a human review the ban? Most are Google but they're not the only guilty party.
- rosnd 4y ago
  Is this legal in Europe? https://commission.europa.eu/law/law-topic/data-protection/reform/rules-business-and-organisations/dealing-citizens/are-there-restrictions-use-automated-decision-making_en https://commission.europa.eu/law/law-topic/data-protection/r... >Yes, individuals should not be subject to a decision that is based solely on automated processing (such as algorithms) and that is legally binding or which significantly affects them. I suppose being banned from Uber can very well be something that significantly affects you, especially if you live in a city where it's the primary form of taxi-like transport.
  - Nextgrid 4y ago
    In practice, legality depends on someone willing to investigate the offence and enforce the law. When it comes to the GDPR, that's not happening even for the biggest, blatant offences (Google and Facebook are still around despite breaching the regulation for 4+ years), so do you really think someone is going to bother investigating this?
    - rosnd 4y ago
      It depends, enforcementtracker.com shows that the authorities have been willing to chase after a rather diverse mix of violators. But indeed, leaving GDPR enforcement to government authorities was a huge mistake. Anyone should be able to sue over GDPR violations impacting them.
Bombthecat 4y ago
Because you don't need to. The next big push will be audible AI.
- rosnd 4y ago
  >Because you don't need to. But GDPR indeed requires it for many kinds of decisionmaking, no?
  - thesuitonym 4y ago
    Since when have tech companies cared about pesky little issues like laws and regulations?
  - ceejayoz 4y ago
    No, it requires the ability to appeal an automated decision to a human. (Who will likely consult the computer and say no again, a month or two later.)
    - rosnd 4y ago
      This EU commission page suggests that a human review should be conducted before a client is informed of the decision, not only in the case of an appeal. https://commission.europa.eu/law/law-topic/data-protection/reform/rules-business-and-organisations/dealing-citizens/are-there-restrictions-use-automated-decision-making_en https://commission.europa.eu/law/law-topic/data-protection/r...
      - ceejayoz 4y ago
        "and that is legally binding or which significantly affects them" is important there, and only going to apply to some decisions.
        rosnd 4y ago
        It sounds like banning an Uber account would qualify > In addition, processing can significantly affect an individual if it influences their personal circumstances, their behaviour or their choices (for example an automatic processing may lead to the refusal of an online credit application).
ceejayoz 4y ago
My job involves working with the Facebook API, and it's very clear they have various "computer says no" systems they can't debug effectively. There are tickets in their developer support system that've been around for years now with stuff like "posts with a dollar sign to a Facebook Group fail in the API" (which is almost certainly a bug with their "should this be a Marketplace listing?" detector). Every month or so we get a notification from their bug system that there's an "update on your ticket", which turns out to be an automated "there's no update, sorry" notification.
x86_64Ubuntu 4y ago
The deniability of decision leading to action is a feature, not a bug. Especially as the world tries to move to automated killing devices as weapons of war.
hackerman123469 4y ago
Pretty much everywhere

nottorp 4y ago

Ours. Perhaps you have missed all the posts here on HN where some company or other locks up someone's account without the user being told why and without the possibility of having a human review the ban? Most are Google but they're not the only guilty party.

rosnd 4y ago
Is this legal in Europe? https://commission.europa.eu/law/law-topic/data-protection/reform/rules-business-and-organisations/dealing-citizens/are-there-restrictions-use-automated-decision-making_en https://commission.europa.eu/law/law-topic/data-protection/r... >Yes, individuals should not be subject to a decision that is based solely on automated processing (such as algorithms) and that is legally binding or which significantly affects them. I suppose being banned from Uber can very well be something that significantly affects you, especially if you live in a city where it's the primary form of taxi-like transport.
- Nextgrid 4y ago
  In practice, legality depends on someone willing to investigate the offence and enforce the law. When it comes to the GDPR, that's not happening even for the biggest, blatant offences (Google and Facebook are still around despite breaching the regulation for 4+ years), so do you really think someone is going to bother investigating this?
  - rosnd 4y ago
    It depends, enforcementtracker.com shows that the authorities have been willing to chase after a rather diverse mix of violators. But indeed, leaving GDPR enforcement to government authorities was a huge mistake. Anyone should be able to sue over GDPR violations impacting them.

Bombthecat 4y ago

Because you don't need to. The next big push will be audible AI.

rosnd 4y ago
>Because you don't need to. But GDPR indeed requires it for many kinds of decisionmaking, no?
- thesuitonym 4y ago
  Since when have tech companies cared about pesky little issues like laws and regulations?
- ceejayoz 4y ago
  No, it requires the ability to appeal an automated decision to a human. (Who will likely consult the computer and say no again, a month or two later.)
  - rosnd 4y ago
    This EU commission page suggests that a human review should be conducted before a client is informed of the decision, not only in the case of an appeal. https://commission.europa.eu/law/law-topic/data-protection/reform/rules-business-and-organisations/dealing-citizens/are-there-restrictions-use-automated-decision-making_en https://commission.europa.eu/law/law-topic/data-protection/r...
    - ceejayoz 4y ago
      "and that is legally binding or which significantly affects them" is important there, and only going to apply to some decisions.
      - rosnd 4y ago
        It sounds like banning an Uber account would qualify > In addition, processing can significantly affect an individual if it influences their personal circumstances, their behaviour or their choices (for example an automatic processing may lead to the refusal of an online credit application).

ceejayoz 4y ago

My job involves working with the Facebook API, and it's very clear they have various "computer says no" systems they can't debug effectively. There are tickets in their developer support system that've been around for years now with stuff like "posts with a dollar sign to a Facebook Group fail in the API" (which is almost certainly a bug with their "should this be a Marketplace listing?" detector). Every month or so we get a notification from their bug system that there's an "update on your ticket", which turns out to be an automated "there's no update, sorry" notification.

x86_64Ubuntu 4y ago

The deniability of decision leading to action is a feature, not a bug. Especially as the world tries to move to automated killing devices as weapons of war.

hackerman123469 4y ago

Pretty much everywhere