6 ms·

If police harass minorities, it generates the most data in the places they harass them. Then they feed this data into the computer and algorithmically generate

If police harass minorities, it generates the most data in the places they harass them. Then they feed this data into the computer and algorithmically generate "unbiased" results that prove they should be harassing minorities.... that is unless this tendency was explicitly corrected for somehow. Was it? If it was, that's its own can of worms, but is at least debatable as to whether there's an acceptable way to do so. Importantly, these algorithms should not be secret. These are the kinds of choices a democratic society should make.

heartbreak 9y ago
Additional reading: https://arxiv.org/abs/1706.09847 https://arxiv.org/abs/1706.09847
carlmr 9y ago
>Then they feed this data into the computer and algorithmically generate "unbiased" results that prove they should be harassing minorities.... Depends on how you use the data. If you learn success rates then it shouldn't depend on the absolute number of people searched. If you just feed raw data into machine learning algorithms, then nobody knows. E.g. you have 100 stop and frisks, 50 in one and 50 in antother neighborhood. Then you might see that you got 1 criminal in one and 10 in the other neighborhood. Then your algorithm will allocate more forces (shouldn't go below a minimum, otherwise no data) in the 20% criminal neighborhood than in the 2% criminal neighborhood. Obviously this will still lead to more searches in more criminal neighborhoods. But that's what you want. You want to use the police force as efficiently as possible to fight crime.
- tyfon 9y ago
  Unless the police enforce laws differently in different neighbourhoods. In one place you can get off with a warning for violating some traffic rule and in other places you can get arrested and hanged. [1] [1] https://en.wikipedia.org/wiki/Death_of_Sandra_Bland https://en.wikipedia.org/wiki/Death_of_Sandra_Bland
  - carlmr 9y ago
    >Unless the police enforce laws differently in different neighbourhoods. On body cameras and separate evaluation (not by the police department, but by a separate agency with a separate budget).
    - ch4s3 9y ago
      >not by the police department, but by a separate agency with a separate budget Is that a thing that even happens? Most PDs I'm familiar with have a lot of control over their own cams. Even after getting caught using them to plant evidence, many still have some questionable cam practices.
- runako 9y ago
  > E.g. you have 100 stop and frisks, 50 in one and 50 in antother neighborhood. NB: stop and frisk isn't done in the "good" neighborhoods at all. There's not going to be any data about what happens when e.g. stopping and frisking white men in suits in lower Manhattan because it will never be done. The idea of being able to do controls hinges on something for which there is no political will. > Obviously this will still lead to more searches in more criminal neighborhoods. But that's what you want. I would think instead you want more searches in places where there is probable cause, you have gotten warrants, etc. How comfortable are we with the notion that people should be targeted on the basis of the neighborhoods they live in? Is this compatible with our guiding documents?
  - carlmr 9y ago
    >NB: stop and frisk isn't done in the "good" neighborhoods at all. There's not going to be any data about what happens when e.g. stopping and frisking white men in suits in lower Manhattan because it will never be done. The idea of being able to do controls hinges on something for which there is no political will. I already addressed that caveat by forcing the algorithm to stop and frisk a minimum amount in the good neighbourhood. Like a minimum of 10% of frisks taking place in good neighbourhoods. >How comfortable are we with the notion that people should be targeted on the basis of the neighborhoods they live in? If I live in such a neighbourhood, I'd want more police presence. They will stop and frisk and do that to me, too. At the same time they will have the chance to stop some criminals that might kill or rob me and my family.
    - emodendroket 9y ago
      I think a lot of times as engineers we imagine the tech divorced from the reality of the world it exists in. It would likely be politically untenable to start stopping and frisking Wall Street bankers.
      - hackerman12345 9y ago
        And the Wall Street Banker is unlikely to be on his way to a in-person mugging appointment. Rich people usually avoid petty or violent crimes.
omarforgotpwd 9y ago
Yes, this is something we researched and thought about. The idea is "garbage in, garbage out". If the data input into the algorithm reflects a systemic bias, obviously any predictions are going to be a product of the input to some extent. Our predictions were based on the city's crime report data, which is clearly not an accurate picture of crime in the city, but it's what we've got. Sure, it's entirely possible that if police are dramatically over-policing certain poor or minority neighborhoods and generating a lot of crime report data based on those arrests that the predictions could potentially be affected. But poor and minority neighborhoods tend to under-report crimes compared to wealthier areas, so there is a balancing counter-effect to some extent. Another thing we've seen happen sometimes is systematic under-reporting or non-reporting of crimes to make the crime stats for the city look better, as the police department is often judged on these crime stats. Ultimately we have to work with what we have and the police are responsible for maintaining high quality data and ethical standards, but I don't lose any sleep worrying about if my code systematically oppressed minorities for two reasons: 1.First of all, most police departments had separate predictions run per "district" and typically had a fixed number of officers allocated to each district. The idea that we were directing cops away from rich neighborhoods and having them go after the poor or minorities assumes that we had far more power than we actually did. The number of cops assigned to each city district / beat is a political or bureaucratic decision and not something our software is involved in. 2. Furthermore, our app was not directing the cops every move, nor was it trying to. We just told them "when you're not answering a call or doing anything else, when you have a spare minute, check if you're near a prediction box and go figure out why the computer flagged it as a high risk area. Maybe it's a parking lot and the lights have gone out encouraging car burglaries". And even as just a small slice of their time, we really had to push to get them to use the app because the way they saw it was "What does this stupid app know about my city that I, with 20 years of experience, don't know? One of these prediction boxes is in the middle of a lake". Based on what I saw from the analytics and usage logs, I think it would be tough to make a case that anyone was oppressed because the cops barely even used the thing! Not that I take these issues lightly, but I always laugh when people criticize PredPol or someone calls me a murderer on Twitter because it vastly overstates our impact. Maybe I should take it as a compliment that people really think I am powerful enough to direct police to oppress minorities using only a shitty rails app I wrote in my college dorm room. But whatever, you can't do or try anything new in this country today without getting dramatically criticized for it to fuel the internet-rage clickbait economy, so I guess it's just something all of us have to get used to. I haven't had a day to day role at the company for a few years, which makes it a lot easier not to worry about these things as well. But ultimately police using technology and data effectively can REDUCE bias and injustice AND make people safer. I think the questions the Verge presents are certainly worth asking, but based on my experience I do believe that the app I implemented helped more people than it harmed.

heartbreak 9y ago

Additional reading: https://arxiv.org/abs/1706.09847 https://arxiv.org/abs/1706.09847

carlmr 9y ago

>Then they feed this data into the computer and algorithmically generate "unbiased" results that prove they should be harassing minorities.... Depends on how you use the data. If you learn success rates then it shouldn't depend on the absolute number of people searched. If you just feed raw data into machine learning algorithms, then nobody knows. E.g. you have 100 stop and frisks, 50 in one and 50 in antother neighborhood. Then you might see that you got 1 criminal in one and 10 in the other neighborhood. Then your algorithm will allocate more forces (shouldn't go below a minimum, otherwise no data) in the 20% criminal neighborhood than in the 2% criminal neighborhood. Obviously this will still lead to more searches in more criminal neighborhoods. But that's what you want. You want to use the police force as efficiently as possible to fight crime.

tyfon 9y ago
Unless the police enforce laws differently in different neighbourhoods. In one place you can get off with a warning for violating some traffic rule and in other places you can get arrested and hanged. [1] [1] https://en.wikipedia.org/wiki/Death_of_Sandra_Bland https://en.wikipedia.org/wiki/Death_of_Sandra_Bland
- carlmr 9y ago
  >Unless the police enforce laws differently in different neighbourhoods. On body cameras and separate evaluation (not by the police department, but by a separate agency with a separate budget).
  - ch4s3 9y ago
    >not by the police department, but by a separate agency with a separate budget Is that a thing that even happens? Most PDs I'm familiar with have a lot of control over their own cams. Even after getting caught using them to plant evidence, many still have some questionable cam practices.
runako 9y ago
> E.g. you have 100 stop and frisks, 50 in one and 50 in antother neighborhood. NB: stop and frisk isn't done in the "good" neighborhoods at all. There's not going to be any data about what happens when e.g. stopping and frisking white men in suits in lower Manhattan because it will never be done. The idea of being able to do controls hinges on something for which there is no political will. > Obviously this will still lead to more searches in more criminal neighborhoods. But that's what you want. I would think instead you want more searches in places where there is probable cause, you have gotten warrants, etc. How comfortable are we with the notion that people should be targeted on the basis of the neighborhoods they live in? Is this compatible with our guiding documents?
- carlmr 9y ago
  >NB: stop and frisk isn't done in the "good" neighborhoods at all. There's not going to be any data about what happens when e.g. stopping and frisking white men in suits in lower Manhattan because it will never be done. The idea of being able to do controls hinges on something for which there is no political will. I already addressed that caveat by forcing the algorithm to stop and frisk a minimum amount in the good neighbourhood. Like a minimum of 10% of frisks taking place in good neighbourhoods. >How comfortable are we with the notion that people should be targeted on the basis of the neighborhoods they live in? If I live in such a neighbourhood, I'd want more police presence. They will stop and frisk and do that to me, too. At the same time they will have the chance to stop some criminals that might kill or rob me and my family.
  - emodendroket 9y ago
    I think a lot of times as engineers we imagine the tech divorced from the reality of the world it exists in. It would likely be politically untenable to start stopping and frisking Wall Street bankers.
    - hackerman12345 9y ago
      And the Wall Street Banker is unlikely to be on his way to a in-person mugging appointment. Rich people usually avoid petty or violent crimes.

omarforgotpwd 9y ago

Yes, this is something we researched and thought about. The idea is "garbage in, garbage out". If the data input into the algorithm reflects a systemic bias, obviously any predictions are going to be a product of the input to some extent. Our predictions were based on the city's crime report data, which is clearly not an accurate picture of crime in the city, but it's what we've got. Sure, it's entirely possible that if police are dramatically over-policing certain poor or minority neighborhoods and generating a lot of crime report data based on those arrests that the predictions could potentially be affected. But poor and minority neighborhoods tend to under-report crimes compared to wealthier areas, so there is a balancing counter-effect to some extent. Another thing we've seen happen sometimes is systematic under-reporting or non-reporting of crimes to make the crime stats for the city look better, as the police department is often judged on these crime stats. Ultimately we have to work with what we have and the police are responsible for maintaining high quality data and ethical standards, but I don't lose any sleep worrying about if my code systematically oppressed minorities for two reasons: 1.First of all, most police departments had separate predictions run per "district" and typically had a fixed number of officers allocated to each district. The idea that we were directing cops away from rich neighborhoods and having them go after the poor or minorities assumes that we had far more power than we actually did. The number of cops assigned to each city district / beat is a political or bureaucratic decision and not something our software is involved in. 2. Furthermore, our app was not directing the cops every move, nor was it trying to. We just told them "when you're not answering a call or doing anything else, when you have a spare minute, check if you're near a prediction box and go figure out why the computer flagged it as a high risk area. Maybe it's a parking lot and the lights have gone out encouraging car burglaries". And even as just a small slice of their time, we really had to push to get them to use the app because the way they saw it was "What does this stupid app know about my city that I, with 20 years of experience, don't know? One of these prediction boxes is in the middle of a lake". Based on what I saw from the analytics and usage logs, I think it would be tough to make a case that anyone was oppressed because the cops barely even used the thing! Not that I take these issues lightly, but I always laugh when people criticize PredPol or someone calls me a murderer on Twitter because it vastly overstates our impact. Maybe I should take it as a compliment that people really think I am powerful enough to direct police to oppress minorities using only a shitty rails app I wrote in my college dorm room. But whatever, you can't do or try anything new in this country today without getting dramatically criticized for it to fuel the internet-rage clickbait economy, so I guess it's just something all of us have to get used to. I haven't had a day to day role at the company for a few years, which makes it a lot easier not to worry about these things as well. But ultimately police using technology and data effectively can REDUCE bias and injustice AND make people safer. I think the questions the Verge presents are certainly worth asking, but based on my experience I do believe that the app I implemented helped more people than it harmed.