4 ms·

Teen turned away from roller rink after AI wrongly identifies her

26 points by LinuxBender 5y ago

prepend 5y ago
> One of the arena's managers later called Lamya’s mother to discuss the issue. And in a statement, the biz said: “The software had her daughter at a 97 percent match. This is what we looked at ... if there was a mistake, we apologize for that." A combination of low skill management and poor algorithms is bad. This seems like stupid managers that don’t understand what 97% means. Also funny that they “apologized” but didn’t say the teen was welcome to come roller skate. And they weren’t able to figure out it wasn’t her at the event. So this statement seems to be “the computer says 97%, so you can’t come in, sorry that sucks for you.” I think we need some sort of Uniform Commercial Code for how automated systems hose customers so there is a consistent adjudication method.
- karmakaze 5y ago
  The software shouldn't be showing a percentage value that can't be reasonably interpreted by its operator. 97% on an exam is really high. A 1 in 32 chance that it's incorrect (if that's what this even means) isn't proof. It should only say that the person is flagged and requires a human to check.
  - prepend 5y ago
    I agree in that numbers should not be shown unless it means something. But I’m not sure if it even has a way for a human to check. Is there a mugshot the manager can pull up and compare, it seems like that’s not part of the product. Or the managers are extra stupid and didn’t even try that.
    - karmakaze 5y ago
      I presume the banned person had their name recorded, if they can show a photo ID with a different name, that should do.
  - runnerup 5y ago
    This would indeed be mildly helpful but real problem is: Even a "100% match" shouldn't be grounds for kicking someone out! It should just be a cue to check her ID against a list of banned people. How fucking hard would that have been?
browningstreet 5y ago
I’m perplexed that a roller rink in Michigan is using facial recognition tech. Wow.
- randyrand 5y ago
  I’m sure it’s also for marketing and business decisions. ¯\_(ツ)_/¯
sschueller 5y ago
If a child in some cases can unlock their parents iPhone then I doubt any other face recognition software based on purely a video system can be considered accurate.
ALittleLight 5y ago
This strikes me as similar to the debate about whether censorship should be allowed in social media. On the one hand, it's clear the constitution prohibits the government from censoring free speech. Analogously it's clear that a facial recognition match, or close match, shouldn't be sufficient to arrest somebody. On the other hand, there is no legal prohibiting on private companies limiting free speech. Analogously, a private roller rink banning a customer has and should have a much lower bar for taking action. One complication with the analogy is that the social media companies are giant monopolies whereas presumably this roller rink is not. Though maybe it's the only roller rink in town. What should the standards of evidence be here? Suppose an employee said "I remember that girl, she caused a fight last week." Would you be allowed to ban off of that, even though it may be as or more fallible? I also think it's interesting how the article doesn't explain how we know the face recognition system was wrong. The girl says she was never at the rink before, but that doesn't seem dispositive to me.

26 points by LinuxBender 5y ago

> One of the arena's managers later called Lamya’s mother to discuss the issue. And in a statement, the biz said: “The software had her daughter at a 97 percent match. This is what we looked at ... if there was a mistake, we apologize for that." A combination of low skill management and poor algorithms is bad. This seems like stupid managers that don’t understand what 97% means. Also funny that they “apologized” but didn’t say the teen was welcome to come roller skate. And they weren’t able to figure out it wasn’t her at the event. So this statement seems to be “the computer says 97%, so you can’t come in, sorry that sucks for you.” I think we need some sort of Uniform Commercial Code for how automated systems hose customers so there is a consistent adjudication method.

karmakaze 5y ago
The software shouldn't be showing a percentage value that can't be reasonably interpreted by its operator. 97% on an exam is really high. A 1 in 32 chance that it's incorrect (if that's what this even means) isn't proof. It should only say that the person is flagged and requires a human to check.
- prepend 5y ago
  I agree in that numbers should not be shown unless it means something. But I’m not sure if it even has a way for a human to check. Is there a mugshot the manager can pull up and compare, it seems like that’s not part of the product. Or the managers are extra stupid and didn’t even try that.
  - karmakaze 5y ago
    I presume the banned person had their name recorded, if they can show a photo ID with a different name, that should do.
- runnerup 5y ago
  This would indeed be mildly helpful but real problem is: Even a "100% match" shouldn't be grounds for kicking someone out! It should just be a cue to check her ID against a list of banned people. How fucking hard would that have been?

browningstreet 5y ago

I’m perplexed that a roller rink in Michigan is using facial recognition tech. Wow.

randyrand 5y ago
I’m sure it’s also for marketing and business decisions. ¯\_(ツ)_/¯

sschueller 5y ago

If a child in some cases can unlock their parents iPhone then I doubt any other face recognition software based on purely a video system can be considered accurate.

ALittleLight 5y ago

This strikes me as similar to the debate about whether censorship should be allowed in social media. On the one hand, it's clear the constitution prohibits the government from censoring free speech. Analogously it's clear that a facial recognition match, or close match, shouldn't be sufficient to arrest somebody. On the other hand, there is no legal prohibiting on private companies limiting free speech. Analogously, a private roller rink banning a customer has and should have a much lower bar for taking action. One complication with the analogy is that the social media companies are giant monopolies whereas presumably this roller rink is not. Though maybe it's the only roller rink in town. What should the standards of evidence be here? Suppose an employee said "I remember that girl, she caused a fight last week." Would you be allowed to ban off of that, even though it may be as or more fallible? I also think it's interesting how the article doesn't explain how we know the face recognition system was wrong. The girl says she was never at the rink before, but that doesn't seem dispositive to me.