6 ms·

Trying to detect a crime like shoplifting beforehand seems to open the door for all sorts of biases, racial and otherwise...

vorhp 7y ago
So if say, old white females who wear expensive clothes are more likely to shoplift, should we ignore that?
- ep103 7y ago
  yes?
- anbop 7y ago
  Very droll, very droll. But US law is that even if your policy has no intended racial bias, if it ends up having one, it’s still illegal. And meat is mainly shoplifted by white women.
  - wutbrodo 7y ago
    > But US law is that even if your policy has no intended racial bias, if it ends up having one, it’s still illegal. Can you elaborate on this? As stated, it sounds not just inaccurate, but impossible: it reads like youre saying that any policy that doesn't result in a uniform (or proportional) racial ratio is illegal, which obviously isn't the case. I think you're talking about disparate impact (as a legal concept), but this is 1) IIRC likotes to certain domains, like housing or employment and 2) avoidable if the racially-skewed policy is demonstrably related to the job (in the employment case). As a trivial example, requiring a college degree for a job dramatically shifts the racial distribution of the candidate pool, and obviously isn't illegal. If the shoplifting system is found to be illegal, AIUI, it would probably be because the case is made that it's _directly_ racially biased (disparate treatment).
- learc83 7y ago
  Yes, we should. As a society we've agreed that whatever benefits there may be from racial profiling aren't worth the harm.
  - rlt 7y ago
    But it’s ok if California requires public companies have quotas for board seats, apparently. https://news.ycombinator.com/item?id=19724430 https://news.ycombinator.com/item?id=19724430
- SandersAK 7y ago
  yes
- ziftface 7y ago
  I'm trying to figure out what the relevance of the racial, gender, or class group is to this situation and I can't
  - doublekill 7y ago
    Age discrimination is also illegal, but do you really want to track grannies the same as teenagers for shoplifting? Same group that complains when a TSA agent frisks a five year old white kid. All in the name of fairness.
- doublekill 7y ago
  It is when the quest for fairness goes too far. Predictive policing is close to this, and also causes strong reactions: complaining about negative feedback loops, but seemingly ignoring that these problem neighborhoods need more police attention, if we want to combat violent deaths and youth gangs. Police has a limited capacity. I want that capacity, paid for with taxes, to be optimized. If that means racial profiling, because some races are more prone to crime, then so be it. That is another form of fairness. Humans employing these techniques make use of common sense, a long sought after feat that AI barely mimicks. You do not fix societal problems, such as racism or racial crime statistics with algorithms anyway. These communities should put the blame elsewhere, starting with themselves, before they point the finger at "racist" algorithms. It is not statistics fault when certain groups are more or less likely to commit crimes.
  - learc83 7y ago
    > These communities should put the blame elsewhere, starting with themselves So the communities of people who are direct descendants of people were forcibly brought here, enslaved, and then systematically denied access to the mechanisms the rest of our ancestors used to build wealth, should blame themselves because they have higher crime rates? White families have more than 10 times the wealth of black families on average. This is directly caused by the years of systemic oppression faced by their ancestors. In addition, to this day, we arrest them at a higher rate rate than White people for the same crimes. We are more likely to convict them for the same crimes. And we give them longer sentences for the same crimes. Elementary school teachers even give harsher punishments to Black students for the same infraction. Job seekers with black sounding names are significantly less likely to be called back for interviews. Black renters have a harder time renting than White renters with the same credit score. The list goes on and on. When you combine all of this, you end up with a community with a higher crime rate. The problem is caused by systemic injustice, it's not a problem that black communities can fix themselves. If you apply those same conditions to any demographic, you would see a higher crime rate--it's inevitable. >If that means racial profiling, because some races are more prone to crime, then so be it. And it's absurd to use that higher crime rate to justify racial profiling, which inevitably has a disparate impact on innocent Black people, which erodes trust in police and institutions, which directly leads to even more criminality. In turn that leads to even more racial profiling. You can't just ignore positive feedback loops by mentioning them and pretending they don't exist. >That is another form of fairness Trial by combat, enslaving defeated enemies, and separate but equal are all "other forms of fairness" that our society has rejected.
braunshizzle 7y ago
Exactly, profiling.
- mc32 7y ago
  So profiling in and of itself isn’t automatically bad. It’s bad if it uses illegal methods , characteristics to assign weight to a profile. You can’t say if [some race] then that. But you can use other things (are they loitering, barefoot, poor hygiene, has sunglasses, truckers cap, pulls over hoodie on entering, have a decoy, etc)
  - gbrown 7y ago
    So "legal" is the threshold for "bad"? I disagree. If the proportion of white shoplifters and black shoplifters being caught is different, AI will just scale up that disparity. If reality is biased, we should work to fix that: not make it more efficient.
    - mc32 7y ago
      I think if they want to target shoplifters, so long as their accuracy is within reasonable range of effectiveness, it’s okay. 90% of the Thai, 92% of AngloAmericans, 89% of Nigerians, 88% of South Americans, 87% of locals indeed processed were shoplifting, regardless of absolute numbers, this is okay.
      - gbrown 7y ago
        I think your attitude is really, really dangerous, because it ignores the long term effects these kinds of things have on society. People don't even really understand how humans apply rules in biased ways - when you attache the words "AI", "Machine Learning", or "Statistics", even well educated people tend to just accept the magic truth telling black box at face value (for a small version of this effect, Google "precision bias"). To specifically address your point: "so long as their accuracy is within reasonable range of effectiveness" This is a far too narrow look at what accuracy means - that's why we usually look at sensitivity, specificity, false negative rate, and false positive rates, and why we break those rates up over important subgroups. To the example numbers you give, let's posit (without evidence) that your numbers are correct - they don't tell us what proportion of people visiting the store belong to each group (is the model just learning to identify ethnicity?), and they don't tell us what proportion of each group is really shoplifting. In many areas (drug crimes come to mind), there's good evidence that different ethnic groups commit crimes at similar rates, but arrest rates are wildly, grotesquely disproportionate. Imagine that AI is applied to such a system - it learns, correctly, that black people are more likely to be arrested for particular crimes. It then targets black people, and the accuracy numbers look good on paper. In fact, because of the trust we have in technology, such a system might help normalize or justify this crazy preexisting disparity. If you were building a statistical model to describe predictors of shoplifting, this kind of bias wouldn't make it past a cursory level of peer review. When you're slapping "AI" on a product and implementing it at a huge scale, nobody bats an eye.
anbop 7y ago
Hopefully less than human bias. I was in a CVS and a black teenager came in and was vigorously trying to get the cashiers attention. Eventually he was like “YO” and she looked up and he showed her his soda and told her he was bringing it into the store, so that when he left she wouldn’t think he had stolen it. I can’t imagine having to live life under such perpetual assumption of wrongdoing.
- cheez 7y ago
  That guy's parents had a talk with him.
- bilbo0s 7y ago
  >Hopefully less than human bias... Well you can hope, but it's made by humans, so I wouldn't count on it.
- jerf 7y ago
  I know what you're saying, but that may not be the best example. I'm white, but I've trained my children a couple of times now that they should never bring anything into a store that the store sells, white kids or not. Of course the store has to assume you're stealing it if you walk back out with it and visibly did not pay. A reasonable person (in the legal sense) is going to put the burden of proof on you to prove you had it when you walked in.
  - anbop 7y ago
    I can say that myself, on hot days, often go into stores, grab a drink, drink the entire thing while doing the rest of my shopping, and pay for the empty bottle when I leave. My wife was horrified when she saw this behavior, and made me stop when we had kids to not set an example for them (I now do it only when I’m not with the kids), but not once (in at least 100 instances) has anyone ever questioned me on it. Contrast my experience to whatever experiences that kid had to make him yell to a cashier until she acknowledged his drink. Edit: to people wondering why I do this, I can only say that it’s something that I never even thought of as weird. That’s what growing up rich can do you.
simonh 7y ago
The thing with machine learning systems is that we can't always know what specific characteristics or behaviours they have learned to respond to. All we know is that they have associated some characteristics of shoppers as indicating a high likelihood of imminently engaging in theft, but we might not be able to un-pick what those characteristics or behaviours are from the neural net weightings. I think a potential risk here is from small initial sample sizes. Here's a thought experiment. A system like this is set up and runs successfully for some time. A person with some distinctive characteristic enters the store and steals something and is caught. The system learns that so far 100% of people with that characteristic stole. Next time it detects that characteristic, it flags that person and they're ejected from the store. The system will never learn that this characteristic can be benign, because from now on it will be excluded from it's data set. That's a bit of an artificial example, making assumptions about how the system might work, but something along those lines could be a potential failure mode, and we might never know what that characteristic is because the system can't express the reasons for it's decision in a meaningful way.

vorhp 7y ago

So if say, old white females who wear expensive clothes are more likely to shoplift, should we ignore that?

ep103 7y ago
yes?
anbop 7y ago
Very droll, very droll. But US law is that even if your policy has no intended racial bias, if it ends up having one, it’s still illegal. And meat is mainly shoplifted by white women.
- wutbrodo 7y ago
  > But US law is that even if your policy has no intended racial bias, if it ends up having one, it’s still illegal. Can you elaborate on this? As stated, it sounds not just inaccurate, but impossible: it reads like youre saying that any policy that doesn't result in a uniform (or proportional) racial ratio is illegal, which obviously isn't the case. I think you're talking about disparate impact (as a legal concept), but this is 1) IIRC likotes to certain domains, like housing or employment and 2) avoidable if the racially-skewed policy is demonstrably related to the job (in the employment case). As a trivial example, requiring a college degree for a job dramatically shifts the racial distribution of the candidate pool, and obviously isn't illegal. If the shoplifting system is found to be illegal, AIUI, it would probably be because the case is made that it's _directly_ racially biased (disparate treatment).
learc83 7y ago
Yes, we should. As a society we've agreed that whatever benefits there may be from racial profiling aren't worth the harm.
- rlt 7y ago
  But it’s ok if California requires public companies have quotas for board seats, apparently. https://news.ycombinator.com/item?id=19724430 https://news.ycombinator.com/item?id=19724430
SandersAK 7y ago
yes
ziftface 7y ago
I'm trying to figure out what the relevance of the racial, gender, or class group is to this situation and I can't
- doublekill 7y ago
  Age discrimination is also illegal, but do you really want to track grannies the same as teenagers for shoplifting? Same group that complains when a TSA agent frisks a five year old white kid. All in the name of fairness.
doublekill 7y ago
It is when the quest for fairness goes too far. Predictive policing is close to this, and also causes strong reactions: complaining about negative feedback loops, but seemingly ignoring that these problem neighborhoods need more police attention, if we want to combat violent deaths and youth gangs. Police has a limited capacity. I want that capacity, paid for with taxes, to be optimized. If that means racial profiling, because some races are more prone to crime, then so be it. That is another form of fairness. Humans employing these techniques make use of common sense, a long sought after feat that AI barely mimicks. You do not fix societal problems, such as racism or racial crime statistics with algorithms anyway. These communities should put the blame elsewhere, starting with themselves, before they point the finger at "racist" algorithms. It is not statistics fault when certain groups are more or less likely to commit crimes.
- learc83 7y ago
  > These communities should put the blame elsewhere, starting with themselves So the communities of people who are direct descendants of people were forcibly brought here, enslaved, and then systematically denied access to the mechanisms the rest of our ancestors used to build wealth, should blame themselves because they have higher crime rates? White families have more than 10 times the wealth of black families on average. This is directly caused by the years of systemic oppression faced by their ancestors. In addition, to this day, we arrest them at a higher rate rate than White people for the same crimes. We are more likely to convict them for the same crimes. And we give them longer sentences for the same crimes. Elementary school teachers even give harsher punishments to Black students for the same infraction. Job seekers with black sounding names are significantly less likely to be called back for interviews. Black renters have a harder time renting than White renters with the same credit score. The list goes on and on. When you combine all of this, you end up with a community with a higher crime rate. The problem is caused by systemic injustice, it's not a problem that black communities can fix themselves. If you apply those same conditions to any demographic, you would see a higher crime rate--it's inevitable. >If that means racial profiling, because some races are more prone to crime, then so be it. And it's absurd to use that higher crime rate to justify racial profiling, which inevitably has a disparate impact on innocent Black people, which erodes trust in police and institutions, which directly leads to even more criminality. In turn that leads to even more racial profiling. You can't just ignore positive feedback loops by mentioning them and pretending they don't exist. >That is another form of fairness Trial by combat, enslaving defeated enemies, and separate but equal are all "other forms of fairness" that our society has rejected.

braunshizzle 7y ago

Exactly, profiling.

mc32 7y ago
So profiling in and of itself isn’t automatically bad. It’s bad if it uses illegal methods , characteristics to assign weight to a profile. You can’t say if [some race] then that. But you can use other things (are they loitering, barefoot, poor hygiene, has sunglasses, truckers cap, pulls over hoodie on entering, have a decoy, etc)
- gbrown 7y ago
  So "legal" is the threshold for "bad"? I disagree. If the proportion of white shoplifters and black shoplifters being caught is different, AI will just scale up that disparity. If reality is biased, we should work to fix that: not make it more efficient.
  - mc32 7y ago
    I think if they want to target shoplifters, so long as their accuracy is within reasonable range of effectiveness, it’s okay. 90% of the Thai, 92% of AngloAmericans, 89% of Nigerians, 88% of South Americans, 87% of locals indeed processed were shoplifting, regardless of absolute numbers, this is okay.
    - gbrown 7y ago
      I think your attitude is really, really dangerous, because it ignores the long term effects these kinds of things have on society. People don't even really understand how humans apply rules in biased ways - when you attache the words "AI", "Machine Learning", or "Statistics", even well educated people tend to just accept the magic truth telling black box at face value (for a small version of this effect, Google "precision bias"). To specifically address your point: "so long as their accuracy is within reasonable range of effectiveness" This is a far too narrow look at what accuracy means - that's why we usually look at sensitivity, specificity, false negative rate, and false positive rates, and why we break those rates up over important subgroups. To the example numbers you give, let's posit (without evidence) that your numbers are correct - they don't tell us what proportion of people visiting the store belong to each group (is the model just learning to identify ethnicity?), and they don't tell us what proportion of each group is really shoplifting. In many areas (drug crimes come to mind), there's good evidence that different ethnic groups commit crimes at similar rates, but arrest rates are wildly, grotesquely disproportionate. Imagine that AI is applied to such a system - it learns, correctly, that black people are more likely to be arrested for particular crimes. It then targets black people, and the accuracy numbers look good on paper. In fact, because of the trust we have in technology, such a system might help normalize or justify this crazy preexisting disparity. If you were building a statistical model to describe predictors of shoplifting, this kind of bias wouldn't make it past a cursory level of peer review. When you're slapping "AI" on a product and implementing it at a huge scale, nobody bats an eye.

anbop 7y ago

Hopefully less than human bias. I was in a CVS and a black teenager came in and was vigorously trying to get the cashiers attention. Eventually he was like “YO” and she looked up and he showed her his soda and told her he was bringing it into the store, so that when he left she wouldn’t think he had stolen it. I can’t imagine having to live life under such perpetual assumption of wrongdoing.

cheez 7y ago
That guy's parents had a talk with him.
bilbo0s 7y ago
>Hopefully less than human bias... Well you can hope, but it's made by humans, so I wouldn't count on it.
jerf 7y ago
I know what you're saying, but that may not be the best example. I'm white, but I've trained my children a couple of times now that they should never bring anything into a store that the store sells, white kids or not. Of course the store has to assume you're stealing it if you walk back out with it and visibly did not pay. A reasonable person (in the legal sense) is going to put the burden of proof on you to prove you had it when you walked in.
- anbop 7y ago
  I can say that myself, on hot days, often go into stores, grab a drink, drink the entire thing while doing the rest of my shopping, and pay for the empty bottle when I leave. My wife was horrified when she saw this behavior, and made me stop when we had kids to not set an example for them (I now do it only when I’m not with the kids), but not once (in at least 100 instances) has anyone ever questioned me on it. Contrast my experience to whatever experiences that kid had to make him yell to a cashier until she acknowledged his drink. Edit: to people wondering why I do this, I can only say that it’s something that I never even thought of as weird. That’s what growing up rich can do you.

simonh 7y ago

The thing with machine learning systems is that we can't always know what specific characteristics or behaviours they have learned to respond to. All we know is that they have associated some characteristics of shoppers as indicating a high likelihood of imminently engaging in theft, but we might not be able to un-pick what those characteristics or behaviours are from the neural net weightings. I think a potential risk here is from small initial sample sizes. Here's a thought experiment. A system like this is set up and runs successfully for some time. A person with some distinctive characteristic enters the store and steals something and is caught. The system learns that so far 100% of people with that characteristic stole. Next time it detects that characteristic, it flags that person and they're ejected from the store. The system will never learn that this characteristic can be benign, because from now on it will be excluded from it's data set. That's a bit of an artificial example, making assumptions about how the system might work, but something along those lines could be a potential failure mode, and we might never know what that characteristic is because the system can't express the reasons for it's decision in a meaningful way.