10 ms·

Early evidence on “predictive policing” and civil rights

78 points by miraj 10y ago

gumby 10y ago
I like that they begin by addressing the use of the word "predictive". How we frame an issue deeply effects how it is received, perceived, and accepted.
danso 10y ago
I thought this was an excellent treatment of the garbage-in-garbage-out aspect of this debate. Too often it seems people get wrapped up in the "Is it ethical to use a computer to judge someone?" implications, which are non-trivial, but also seem to be borne out of a general ignorance about the many ways we do allow computers (or rather, computation) to make judgements about people. Also, it's an excellent example of the power of research and tabulation, though it seems the link to the list of departments they researched is broken. I think the use of crime modeling doesn't have to be controversial, but the first step is demanding transparency. It's scary enough when the police are the primary collectors and judgers of data, but even worse when they leave it to a vendor and assume it's hunky-dory out of the circular reasoning that proprietary algorithms are good because they are proprietary. > When the Fresno police briefed the city council about the Beware system, the council president asked a simple question about those color-coded threat levels: “How does a person get to red?” The officer didn’t know, because the vendor wasn’t saying. As the officer delivering the briefing explained: “We don’t know what their algorithm is exactly… We don’t have any kind of a list that will tell us, this is their criteria, this is what would make a person red, this is what would make a person yellow. It’s their own system
- mgleason_3 10y ago
  Government transparency is absolutely necessary, especially financial, but very lacking.
- notahacker 10y ago
  Aren't there numerous ML techniques potentially be applicable to evaluating potential future crimes where even the people responsible for writing the algorithm and feeding the massive dataset in don't really understand how a particular person might "get to red"? Transparency seems to be only part of the problem.
  - M_Grey 10y ago
    Maybe, but I'm not aware of any that have produced predictions on a broad in vivo population that are better than existing methods. It's one thing to do a heat map of a region at a certain time of day, and express that in terms of overall statistical risk. It's quite another to try and read the future intentions of a human being based on what amounts to a wealth of noise and a paucity of signal.
    - jayjay71 10y ago
      Just curious as the name is familiar - do you go to Georgia Tech?
      - M_Grey 10y ago
        I don't sorry.
  - twblalock 10y ago
    The corollary to this: if an algorithm uses AI or machine learning to the extent that nobody precisely understands why it makes the decisions it does, it will be very difficult to change its behavior in specific cases, e.g. "make it stop doing that," especially when the inputs cannot be changed. This is going to come up at some point when a self-driving car does something that appears totally irrational and ends up causing an accident. Engineers will need to come up with some kind of explanation, and I suspect the general public will not be satisfied when they learn that the explanation may be unknowable, or may reduce only to probability instead of certainty. I deal with some of this at work in far more trivial use cases, and non-engineers just can't seem to accept that sometimes you cannot fix the imperfections without introducing worse imperfections in other areas of the system, and that ML generally leads to output that is "good enough" instead of perfect.
  - comex 10y ago
    Maybe. But at least we can demand to know the system's input data and what measure(s) the training process is designed to optimize. For example, in the case of Beware, which public records are being used, and what is the benchmark for red/yellow/green? Also, are there any systems in place to try to reduce things like racial bias or the self-reinforcement effect described in the report? Why or why not?
- rodgerd 10y ago
  > I think the use of crime modeling doesn't have to be controversial Outside of a 50's-style sci-fi utopia, I simply can't see how. What we consider a crime worth pursuing is, alone, a hugely difficult conversation.
tomjen3 10y ago
Possible heretical thought, could it be that predictive policing suggests the same high crime neighborhoods that are normally responsible for most crime, because they are responsible for most crime (at least the kind that the systems track)? That would make them useless, true, but not racists (most of these neighborhoods would probably be predominately african-american, but they would also be predominately poor, which we know correlates with crime).
- noobermin 10y ago
  Their main argument against this is this[0]. See the quote from the expert in that subsection. [0] https://www.teamupturn.com/reports/2016/stuck-in-a-pattern#police-statistics-reflect-enforcement-not-just-crime https://www.teamupturn.com/reports/2016/stuck-in-a-pattern#p...
  - weberc2 10y ago
    I'm confused by this argument; how does predictive policing relate to measuring department performance? The biggest concern I can see is that putting more police in certain neighborhoods will bias the input data for those models (presumably police reports), but this doesn't seem to be what this section of the article is saying. It also doesn't seem like a good reason to avoid computer models, but rather to increase transparency.
    - danso 10y ago
      A predictive policy generally implies an expected outcome, such as: in an area with this kind of demographic, income level, history, etc, etc., we should expect to see X number of drug and weapon possession offenders. If a precinct ends up arresting significantly fewer people for these offenses, then it can be interpreted that the precinct is underpeforming when it comes to patrolling and finding probable cause to stop and search. I referenced Adrian Schoolcraft in another comment, but here's another retelling of his situation: http://www.thisamericanlife.org/radio-archives/episode/414/transcript http://www.thisamericanlife.org/radio-archives/episode/414/t... Ira Glass: The atmosphere at the 81st precinct was set by its commander, Stephen Mauriello. When Mauriello showed up, Adrian Schoolcraft says, things changed. Offices were told to write more tickets, do more stop-and-frisks, arrest more people for low level offenses that they might otherwise let go-- get their numbers up. Adrian Schoolcraft The pressure definitely increased when he arrived and took over as the commanding officer. The analogy I would uses is like having a boot to the back of your heel. It is do this or else. The rent's due. Ira Glass The rent's due? Adrian Schoolcraft The rent is due. Pay the rent. Did you pay the rent last month? Ira Glass Pay the rent means did you get your numbers? Adrian Schoolcraft Correct.
carapace 10y ago
"The target of the Jihad was a machine-attitude as much as the machines," Leto said. "Humans had set those machines to usurp our sense of beauty, our necessary selfdom out of which we make living judgements. Naturally, the machines were destroyed." https://en.wikipedia.org/wiki/Butlerian_Jihad https://en.wikipedia.org/wiki/Butlerian_Jihad Seriously though, how is this not the crudest sort of cargo-cult machine worship? I'm not paying taxes to be judged by a goddamned machine. It's crazy. If the people we're paying to administer the system are so degenerate that they would delegate their most essential function to a machine then we need to vote them out in favor of sober, responsible, educated adults. It's like they're admitting that they can be replaced by a script!!! Knowing what I know about computers, and computer modelling, I'm appalled that people would make, buy, and employ these sorts of "voo-doo" machines. (Thin sans-serif body font means you hate your readers, and making it grey means you really hate them.)
- wyager 10y ago
  > It's like they're admitting that they can be replaced by a script!!! Well yes, evidence suggests that in many cases they can be.
- danso 10y ago
  > I'm not paying taxes to be judged by a goddamned machine I'd be shocked to learn that the IRS does not use some form of mechanical computation to determine probabilistically whether or not a given taxpayer should be audited for possible tax fraud in a given year.
  - carapace 10y ago
    Maybe so, I'm not privy to their methods, but I wouldn't be surprised that they use, e.g. statistical models, computed on machines, to help them figure out whether or not to look more closely at a given taxpayer. But however unpleasant it may be, an audit is not a punishment. (Someone is abusing their power if so.) Not anymore than being selected for jury duty is a punishment. Also, you used the word "probabilistically". We should not be assigning sentences based on the roll of a die, eh? Insurance companies use machines and statistical models to set rates. But they also have strong feedback to improve accuracy. They lose money if they get it wrong. Here we are talking about judgement. I know far too much about computers to ever credit the idea that these machines can replace a human judge. The people who have put these things into practice are deeply incompetent and should be voted out of office and replaced by folks who won't devolve to crude machine-worship.
patcheudor 10y ago
"The fact that we even call these systems “predictive” is itself a telling sign of excessive confidence in the systems." If you are worried about the outcome of the language in police intervention and court cases then "Speculative policing" would seem a more appropriate term for the technology and cary less weight. This might be an area suited for the FTC to step in and ask the industry to fix their marketing to be more inline with what they actually offer.
h4nkoslo 10y ago
The problem is that the police are being asked to do what amounts to population control, of a historic problem population in the US, that has no real desire to be controlled, on the behalf of people with no real desire to be seen as controlling them. The most the algorithm can do is provide a justification for well-known solutions to crime (or the perception of crime), but laundered through math & obfuscated reasoning so as to not be seen as too racist. Somehow, up until like the 1960s-1970s, this was not an issue. What changed, and what reason do we have for believing that any modernized algorithmic solution would be better than now-abandoned social technologies with actual empirical track records? Hell, the vast majority of the time, you can't even accurately describe the problem in eg Chicago without being shouted down or deflected into someone's pet policy proposal.
- panic 10y ago
  The problem is that the police are being asked to do what amounts to population control, of a historic problem population in the US, that has no real desire to be controlled, on the behalf of people with no real desire to be seen as controlling them. The most the algorithm can do is provide a justification for well-known solutions to crime (or the perception of crime), but laundered through math & obfuscated reasoning so as to not be seen as too racist. Considering a group of people a "problem population" due to their race is racist, but I think this perspective is shared by many people in America. How do we get police to stop being agents of racism when so much of the population condones it? Somehow, up until like the 1960s-1970s, this was not an issue. Was it not an issue, or did you just not hear about it? The 1960s civil rights movement happened (and in many ways is still happening) for a reason.
  - h4nkoslo 10y ago
    Considering a group of people a "problem population" due to their race is racist, but I think this perspective is shared by many people in America. How do we get police to stop being agents of racism when so much of the population condones this sort of thing? Blacks have ~4x the murder rate of the combined white / hispanic population, and IIRC something like 6-8x the murder rate of the disaggregated white population, depending on how exactly you segment hispanics. It ends up being approximately half of all murders. Note that this tends to be a geographically concentrated population, often next to an incredibly low-murder urban(e) white area, so you end up with cities like Chicago where the goal is to keep the anarchy from spilling over into your tax base. It's not delusional to think that dealing with facts like this is a major area of concern for law enforcement and local political leadership, especially when people are so quick to denounce any contemplation of reality as racist. I guess if you wanted to put on your PC goggles and studiously avoid any discussion of race, you could do the intellectually dishonest thing and talk about things like "the population of South Chicago", but being intellectually dishonest doesn't seem to have gotten us very far. https://ucr.fbi.gov/crime-in-the-u.s/2013/crime-in-the-u.s.-2013/tables/table-43 https://ucr.fbi.gov/crime-in-the-u.s/2013/crime-in-the-u.s.-...
shmerl 10y ago
Also see "Watchbird" by Robert Sheckley: https://www.gutenberg.org/ebooks/29579 https://www.gutenberg.org/ebooks/29579 By the way, DRM is an example of such approach, where people are judged as guilty by default by digital systems.
Houshalter 10y ago
Remove the scary word "algorithm" and this whole issue seems blown up around something fairly mundane. If the police hand calculated how many crime reports came from each area no one would care. People are so scared of algorithms. There is now scientific evidence confirming that severe distrust of algorithms is a pervasive human bias. Search for "algorithm aversion". They do trust them when they are believed to be infallible, but as soon as they learn they can make mistakes, they vastly underestimate them. And this is a damn shame, because in almost every case even very simple statistical procedures outperform human "experts". Worse is this nonsense that algorithms can be "racist". As if a computer can have prejudice or care about anything other than maximizing accuracy. The media loves to fuel this narrative. E.g. the big hoopla when Google's image tagger tagged a black person as a gorilla. They had to remove it and apologize.
- couchand 10y ago
  If you ignore history it can be really easy to come across as casually racist.
  - Houshalter 10y ago
    What on Earth are you talking about?
deleted 10y ago
[deleted]
deleted 10y ago
[deleted]
omarforgotpwd 10y ago
Wow, cool! I am the technical founder of PredPol and he uses a screenshot of the original version, which I wrote in my college dorm room in his article. Somehow the fact that somebody took so much time to criticize it somehow makes me feel special. (I don't work there an

78 points by miraj 10y ago

gumby 10y ago

I like that they begin by addressing the use of the word "predictive". How we frame an issue deeply effects how it is received, perceived, and accepted.

danso 10y ago

I thought this was an excellent treatment of the garbage-in-garbage-out aspect of this debate. Too often it seems people get wrapped up in the "Is it ethical to use a computer to judge someone?" implications, which are non-trivial, but also seem to be borne out of a general ignorance about the many ways we do allow computers (or rather, computation) to make judgements about people. Also, it's an excellent example of the power of research and tabulation, though it seems the link to the list of departments they researched is broken. I think the use of crime modeling doesn't have to be controversial, but the first step is demanding transparency. It's scary enough when the police are the primary collectors and judgers of data, but even worse when they leave it to a vendor and assume it's hunky-dory out of the circular reasoning that proprietary algorithms are good because they are proprietary. > When the Fresno police briefed the city council about the Beware system, the council president asked a simple question about those color-coded threat levels: “How does a person get to red?” The officer didn’t know, because the vendor wasn’t saying. As the officer delivering the briefing explained: “We don’t know what their algorithm is exactly… We don’t have any kind of a list that will tell us, this is their criteria, this is what would make a person red, this is what would make a person yellow. It’s their own system

mgleason_3 10y ago
Government transparency is absolutely necessary, especially financial, but very lacking.
notahacker 10y ago
Aren't there numerous ML techniques potentially be applicable to evaluating potential future crimes where even the people responsible for writing the algorithm and feeding the massive dataset in don't really understand how a particular person might "get to red"? Transparency seems to be only part of the problem.
- M_Grey 10y ago
  Maybe, but I'm not aware of any that have produced predictions on a broad in vivo population that are better than existing methods. It's one thing to do a heat map of a region at a certain time of day, and express that in terms of overall statistical risk. It's quite another to try and read the future intentions of a human being based on what amounts to a wealth of noise and a paucity of signal.
  - jayjay71 10y ago
    Just curious as the name is familiar - do you go to Georgia Tech?
    - M_Grey 10y ago
      I don't sorry.
- twblalock 10y ago
  The corollary to this: if an algorithm uses AI or machine learning to the extent that nobody precisely understands why it makes the decisions it does, it will be very difficult to change its behavior in specific cases, e.g. "make it stop doing that," especially when the inputs cannot be changed. This is going to come up at some point when a self-driving car does something that appears totally irrational and ends up causing an accident. Engineers will need to come up with some kind of explanation, and I suspect the general public will not be satisfied when they learn that the explanation may be unknowable, or may reduce only to probability instead of certainty. I deal with some of this at work in far more trivial use cases, and non-engineers just can't seem to accept that sometimes you cannot fix the imperfections without introducing worse imperfections in other areas of the system, and that ML generally leads to output that is "good enough" instead of perfect.
- comex 10y ago
  Maybe. But at least we can demand to know the system's input data and what measure(s) the training process is designed to optimize. For example, in the case of Beware, which public records are being used, and what is the benchmark for red/yellow/green? Also, are there any systems in place to try to reduce things like racial bias or the self-reinforcement effect described in the report? Why or why not?
rodgerd 10y ago
> I think the use of crime modeling doesn't have to be controversial Outside of a 50's-style sci-fi utopia, I simply can't see how. What we consider a crime worth pursuing is, alone, a hugely difficult conversation.

tomjen3 10y ago

Possible heretical thought, could it be that predictive policing suggests the same high crime neighborhoods that are normally responsible for most crime, because they are responsible for most crime (at least the kind that the systems track)? That would make them useless, true, but not racists (most of these neighborhoods would probably be predominately african-american, but they would also be predominately poor, which we know correlates with crime).

noobermin 10y ago
Their main argument against this is this[0]. See the quote from the expert in that subsection. [0] https://www.teamupturn.com/reports/2016/stuck-in-a-pattern#police-statistics-reflect-enforcement-not-just-crime https://www.teamupturn.com/reports/2016/stuck-in-a-pattern#p...
- weberc2 10y ago
  I'm confused by this argument; how does predictive policing relate to measuring department performance? The biggest concern I can see is that putting more police in certain neighborhoods will bias the input data for those models (presumably police reports), but this doesn't seem to be what this section of the article is saying. It also doesn't seem like a good reason to avoid computer models, but rather to increase transparency.
  - danso 10y ago
    A predictive policy generally implies an expected outcome, such as: in an area with this kind of demographic, income level, history, etc, etc., we should expect to see X number of drug and weapon possession offenders. If a precinct ends up arresting significantly fewer people for these offenses, then it can be interpreted that the precinct is underpeforming when it comes to patrolling and finding probable cause to stop and search. I referenced Adrian Schoolcraft in another comment, but here's another retelling of his situation: http://www.thisamericanlife.org/radio-archives/episode/414/transcript http://www.thisamericanlife.org/radio-archives/episode/414/t... Ira Glass: The atmosphere at the 81st precinct was set by its commander, Stephen Mauriello. When Mauriello showed up, Adrian Schoolcraft says, things changed. Offices were told to write more tickets, do more stop-and-frisks, arrest more people for low level offenses that they might otherwise let go-- get their numbers up. Adrian Schoolcraft The pressure definitely increased when he arrived and took over as the commanding officer. The analogy I would uses is like having a boot to the back of your heel. It is do this or else. The rent's due. Ira Glass The rent's due? Adrian Schoolcraft The rent is due. Pay the rent. Did you pay the rent last month? Ira Glass Pay the rent means did you get your numbers? Adrian Schoolcraft Correct.

carapace 10y ago

"The target of the Jihad was a machine-attitude as much as the machines," Leto said. "Humans had set those machines to usurp our sense of beauty, our necessary selfdom out of which we make living judgements. Naturally, the machines were destroyed." https://en.wikipedia.org/wiki/Butlerian_Jihad https://en.wikipedia.org/wiki/Butlerian_Jihad Seriously though, how is this not the crudest sort of cargo-cult machine worship? I'm not paying taxes to be judged by a goddamned machine. It's crazy. If the people we're paying to administer the system are so degenerate that they would delegate their most essential function to a machine then we need to vote them out in favor of sober, responsible, educated adults. It's like they're admitting that they can be replaced by a script!!! Knowing what I know about computers, and computer modelling, I'm appalled that people would make, buy, and employ these sorts of "voo-doo" machines. (Thin sans-serif body font means you hate your readers, and making it grey means you really hate them.)

wyager 10y ago
> It's like they're admitting that they can be replaced by a script!!! Well yes, evidence suggests that in many cases they can be.
danso 10y ago
> I'm not paying taxes to be judged by a goddamned machine I'd be shocked to learn that the IRS does not use some form of mechanical computation to determine probabilistically whether or not a given taxpayer should be audited for possible tax fraud in a given year.
- carapace 10y ago
  Maybe so, I'm not privy to their methods, but I wouldn't be surprised that they use, e.g. statistical models, computed on machines, to help them figure out whether or not to look more closely at a given taxpayer. But however unpleasant it may be, an audit is not a punishment. (Someone is abusing their power if so.) Not anymore than being selected for jury duty is a punishment. Also, you used the word "probabilistically". We should not be assigning sentences based on the roll of a die, eh? Insurance companies use machines and statistical models to set rates. But they also have strong feedback to improve accuracy. They lose money if they get it wrong. Here we are talking about judgement. I know far too much about computers to ever credit the idea that these machines can replace a human judge. The people who have put these things into practice are deeply incompetent and should be voted out of office and replaced by folks who won't devolve to crude machine-worship.

patcheudor 10y ago

"The fact that we even call these systems “predictive” is itself a telling sign of excessive confidence in the systems." If you are worried about the outcome of the language in police intervention and court cases then "Speculative policing" would seem a more appropriate term for the technology and cary less weight. This might be an area suited for the FTC to step in and ask the industry to fix their marketing to be more inline with what they actually offer.

h4nkoslo 10y ago

The problem is that the police are being asked to do what amounts to population control, of a historic problem population in the US, that has no real desire to be controlled, on the behalf of people with no real desire to be seen as controlling them. The most the algorithm can do is provide a justification for well-known solutions to crime (or the perception of crime), but laundered through math & obfuscated reasoning so as to not be seen as too racist. Somehow, up until like the 1960s-1970s, this was not an issue. What changed, and what reason do we have for believing that any modernized algorithmic solution would be better than now-abandoned social technologies with actual empirical track records? Hell, the vast majority of the time, you can't even accurately describe the problem in eg Chicago without being shouted down or deflected into someone's pet policy proposal.

panic 10y ago
The problem is that the police are being asked to do what amounts to population control, of a historic problem population in the US, that has no real desire to be controlled, on the behalf of people with no real desire to be seen as controlling them. The most the algorithm can do is provide a justification for well-known solutions to crime (or the perception of crime), but laundered through math & obfuscated reasoning so as to not be seen as too racist. Considering a group of people a "problem population" due to their race is racist, but I think this perspective is shared by many people in America. How do we get police to stop being agents of racism when so much of the population condones it? Somehow, up until like the 1960s-1970s, this was not an issue. Was it not an issue, or did you just not hear about it? The 1960s civil rights movement happened (and in many ways is still happening) for a reason.
- h4nkoslo 10y ago
  Considering a group of people a "problem population" due to their race is racist, but I think this perspective is shared by many people in America. How do we get police to stop being agents of racism when so much of the population condones this sort of thing? Blacks have ~4x the murder rate of the combined white / hispanic population, and IIRC something like 6-8x the murder rate of the disaggregated white population, depending on how exactly you segment hispanics. It ends up being approximately half of all murders. Note that this tends to be a geographically concentrated population, often next to an incredibly low-murder urban(e) white area, so you end up with cities like Chicago where the goal is to keep the anarchy from spilling over into your tax base. It's not delusional to think that dealing with facts like this is a major area of concern for law enforcement and local political leadership, especially when people are so quick to denounce any contemplation of reality as racist. I guess if you wanted to put on your PC goggles and studiously avoid any discussion of race, you could do the intellectually dishonest thing and talk about things like "the population of South Chicago", but being intellectually dishonest doesn't seem to have gotten us very far. https://ucr.fbi.gov/crime-in-the-u.s/2013/crime-in-the-u.s.-2013/tables/table-43 https://ucr.fbi.gov/crime-in-the-u.s/2013/crime-in-the-u.s.-...

shmerl 10y ago

Also see "Watchbird" by Robert Sheckley: https://www.gutenberg.org/ebooks/29579 https://www.gutenberg.org/ebooks/29579 By the way, DRM is an example of such approach, where people are judged as guilty by default by digital systems.

Houshalter 10y ago

Remove the scary word "algorithm" and this whole issue seems blown up around something fairly mundane. If the police hand calculated how many crime reports came from each area no one would care. People are so scared of algorithms. There is now scientific evidence confirming that severe distrust of algorithms is a pervasive human bias. Search for "algorithm aversion". They do trust them when they are believed to be infallible, but as soon as they learn they can make mistakes, they vastly underestimate them. And this is a damn shame, because in almost every case even very simple statistical procedures outperform human "experts". Worse is this nonsense that algorithms can be "racist". As if a computer can have prejudice or care about anything other than maximizing accuracy. The media loves to fuel this narrative. E.g. the big hoopla when Google's image tagger tagged a black person as a gorilla. They had to remove it and apologize.

couchand 10y ago
If you ignore history it can be really easy to come across as casually racist.
- Houshalter 10y ago
  What on Earth are you talking about?

deleted 10y ago

[deleted]

deleted 10y ago

[deleted]

omarforgotpwd 10y ago

Wow, cool! I am the technical founder of PredPol and he uses a screenshot of the original version, which I wrote in my college dorm room in his article. Somehow the fact that somebody took so much time to criticize it somehow makes me feel special. (I don't work there an