12 ms·

Training Computers to Find Future Criminals

90 points by nigrioid 10y ago

niels_olson 10y ago
Can someone just go ahead and inject a blink tag so we can get the full 1994 experience? Oh, my retina...
- cloudjacker 10y ago
  http://www.wonder-tonic.com/geocitiesizer/content.php?theme=1&music=2&url=http://www.bloomberg.com/features/2016-richard-berk-future-crime/ http://www.wonder-tonic.com/geocitiesizer/content.php?theme=... surprisingly more readable
  - cJ0th 10y ago
    On a serious note: They original link looks okayish when you switch to article mode in Firefox.
jamesrom 10y ago
What is bloomberg's MO with these near unreadable articles?
- puddintane 10y ago
  Still rocking paint like it's 1995 - Bloomberg The color choice is just yikes! Did anyone else get reminded of Futurama - Law and Oracle (S06E16 / E104)? [1] https://en.wikipedia.org/wiki/Law_and_Oracle https://en.wikipedia.org/wiki/Law_and_Oracle I do wonder if this type of technology is something we should slowly approach due to the very nature of the outcome of sentencing. We already incarcerate a lot of innocent people and I truly wonder if this is something we should tread lightly on.
  - nickles 10y ago
    Law and Oracle is an adaptation of PKD's short story "Minority Report" in which the Pre-Cog (short for pre-cognition) department polices based on glimpses of future crime. Orwell's "1984" explored the similar, but distinct, notion of 'thoughtcrime'. Both works examine the implications of such policing methods and are certainly worth reading.
    - Jaruzel 10y ago
      Or, if you are time impaired, there is of course Minority Report the film, starring Tom Cruise. Not as in depth, of course, but not a bad adaptation imho.
    - Pamar 10y ago
      To be honest I read 1984 maybe six or seven times and I do not remeber "prediction" to be one of the main themes. See https://en.wikipedia.org/wiki/Thoughtcrime https://en.wikipedia.org/wiki/Thoughtcrime The oppressive regime of 1984 does not use math or computers to look for "suspects". We might argue this was because Orwell had no idea about these methods (book was written in 1948, after all) but personally I doubt he would have used this because advocating decisions to an impersonal algorithm would make the bad guys slightly less bad: in the novel the main baddie says something like "do you want to see the future of Human Race? Think of a boot stomping on a human face, forever...". I.e. power for them is a end in itself, and the only way to use it is to make someone else suffer. I don't see any place for "impersonal algorithms to better adjudicate anything" in this.
  - ladzoppelin 10y ago
    Firefox Reader View makes the so much nicer.
- intrasight 10y ago
  I was actually pleasantly surprised that the weird style and graphics rendered since most of their article just display as plain text due to my uBlock Origin settings. And I sorta like the weird style, mainly because it's out of character with Bloomberg.
- eyelidlessness 10y ago
  I find them more readable than the endless wall of indistinguishable text of pick-a-random-news-site. Of course all the text is readable, and there's innumerable ways of increasing the readability. But their art direction and effort to distinguish particular content helps it stand out and helps overcome the kind of fatigue that causes readers to drop off within a couple paragraphs and has plagued other content-oriented sites with horrifying "minutes to read" estimates.
ccvannorman 10y ago
>Risk scores, generated by algorithms, are an increasingly common factor in sentencing. Computers crunch data—arrests, type of crime committed, and demographic information—and a risk rating is generated. The idea is to create a guide that’s less likely to be subject to unconscious biases, the mood of a judge, or other human shortcomings. Similar tools are used to decide which blocks police officers should patrol, where to put inmates in prison, and who to let out on parole. So, eventually a robot police officer will arrest someone for having the wrong profile. >Berk wants to predict at the moment of birth whether people will commit a crime by their 18th birthday, based on factors such as environment and the history of a new child’s parents. This would be almost impossible in the U.S., given that much of a person’s biographical information is spread out across many agencies and subject to many restrictions. He’s not sure if it’s possible in Norway, either, and he acknowledges he also hasn’t completely thought through how best to use such information. So, we're not sure how dangerous this will be, or how Minority Report thoughtcrime will work, but we're damned sure we want it, because it's the future and careers will be made? This is a very scary trend in the U.S. Eventually, if you're born poor/bad childhood, you will have even less of a chance of making it.
- sliverstorm 10y ago
  On the bright side, if we can pinpoint at risk children with high accuracy, we can also help them make it. Like the attempts to deradicalize individuals at very high risk of flying of to syria, rather than arresting them.
  - seanmcdirmid 10y ago
    We can already pinpoint at risk children with high accuracy, just check out any inner city ghetto. We just lack the caring needed to do anything about the root cause (e.g. and mostly poverty) that causes kids to go bad later. It isn't a mystery.
deleted 10y ago
[deleted]
Smerity 10y ago
As someone who does machine learning, this absolutely terrifies me. The "capstone project" of determining someone's probability of committing a crime by their 18th birthday is beyond ridiculous. Either the author of the article hyped it to the extreme (for the love of everything that's holy, stop freaking hyping machine learning) or the statistician is stark raving mad. The fact that he does this for free is also concerning, primarily as I doubt this has any level of auditing behind it. The only thing I agree with him on is that black box models are even worse as they have even worse audit issues. Given the complexities in making these predictions and the potentially life long impact they might have, there is such a desperately strong need for these systems to have audit guarantees. It's noted that he supposedly shares the code for his systems - if so, I'd love to see it? Is it just shared with the relevant governmental departments who likely have no ability to audit such models? Has it been audited? Would you trust mission critical code that didn't have some level of unit testing? Some level of code review? No? Then why would you potentially destructively change someone's life based on that same level of quality? > "[How risk scores are impacted by race] has not been analyzed yet," she said. "However, it needs to be noted that parole is very different than sentencing. The board is not determining guilt or innocence. We are looking at risk." What? Seriously? Not analyzed? The other worrying assumption is that it isn't used in sentencing. People have a tendency to seek out and misuse information even if they're told not to. This was specifically noted in another article on the misuse of Compas, the black box system. Deciding on parole also doesn't mean you can avoid analyzing bias. If you're denying parole for specific people algorithmically, that can still be insanely destructive. > Berk readily acknowledges this as a concern, then quickly dismisses it. Race isn’t an input in any of his systems, and he says his own research has shown his algorithms produce similar risk scores regardless of race. There are so many proxies for race within the feature set. It's touched on lightly in the article - location, number of arrests, etc - but it gets even more complex when you allow a sufficiently complex machine learning model access to "innocuous" features. Specific ML systems ("deep") can infer hidden variables such as race. Even location is a brilliant proxy for race as seen in redlining[1]. It does appear from his publications that they're shallow models - namely random forests, logistic regression, and boosting[2][3][4]. FOR THE LOVE OF EVERYTHING THAT'S HOLY STOP THROWING MACHINE LEARNING AT EVERYTHING. Think it through. Please. Please please please. I am a big believer that machine learning can enable wonderful things - but it could also enable a destructive feedback loop in so many systems. Resume screening, credit card applications, parole risk classification, ... This is just the tip of the iceberg of potential misuses for machine learning. Edit: I am literally physically feeling ill. He uses logistic regression, random forests, boosting ... standard machine learning algorithms. Fine. Okay ... but you now think the algorithms that might get you okay results on Kaggle competitions can be used to predict a child's future crimes?!?! WTF. What. The actual. ^^^^. Anyone who even knows the hello world of machine learning would laugh at this if the person saying it wasn't literally supplying information to governmental agencies right now. I wrote an article last week on "It's ML, not magic"[5] but I didn't think I'd need to cover this level of stupidity. [1]: https://en.wikipedia.org/wiki/Redlining https://en.wikipedia.org/wiki/Redlining [2]: https://books.google.com/books/about/Criminal_Justice_Forecasts_of_Risk.html?id=Jrlb6Or8YisC&printsec=frontcover&source=kp_read_button&hl=en#v=onepage&q&f=false https://books.google.com/books/about/Criminal_Justice_Foreca... [3]: https://www.semanticscholar.org/paper/Developing-a-Practical-Forecasting-Screener-for-Berk-He/6999981067428dafadd10aa736e4b5c293f89823 https://www.semanticscholar.org/paper/Developing-a-Practical... [4]: https://www.semanticscholar.org/paper/Algorithmic-criminology-Berk/226defcf96d30cf0a17c6caafd60457c9411f458 https://www.semanticscholar.org/paper/Algorithmic-criminolog... [5]: http://smerity.com/articles/2016/ml_not_magic.html http://smerity.com/articles/2016/ml_not_magic.html
anupshinde 10y ago
This is like machine introduced bias/racisim/castism... we need a new term for that.. and its based on statistically induced pseudo-sciences many times similar to astrology. This is the kind of AI everyone should be afraid of.
- benkuykendall 10y ago
  I fail to see what is unscientific about stating conditional probabilities. Astrology is unscientific because the orientation of the heavens is for the most part independent of the observables of a person's life. But the inputs to Berk's system clearly do effect the probability of committing crime. A frequentist would say that a large group of people with this set of characteristics would yield more or fewer criminals; a Bayesian would say we have more knowledge about whether such an individual will commit future crime. These are scientific conclusions. The question "how should we use this data" is a question of ethics, not science.
  - eyelidlessness 10y ago
    > I fail to see what is unscientific about stating conditional probabilities. 1. Create conditions which disadvantage and impoverish a segment of society. 2. Refine those conditions for centuries, continually criminalizing the margins that segment of society is forced to live in. 3. Identify that many of the people in that segment of society are likely to be identified as criminals. 4. Pretend that you're doing math rather than reinforcing generations of deleterious conditioning, completely ignoring the creation of those conditions that led to the probabilities you're identifying. And science can't be divorced from ethics. These are human pursuits.
    - vintermann 10y ago
      This looks like sloppy ML. But human judges do all those things already (susbtitute "just applying the law" for "doing math" in 4) and they can't be inspected - their brains are "closed source". Sure, these humans can come up with wordy justifications for their decisions. But there are plenty of intelligent people who employ their intelligence not to arrive at a conclusion, but to justify the conclusion they already arrived at. Legal professionals aren't merely capable of this, they're explicitly trained to be experts at post-hoc justification. And legal professionals basically ignore all criticism not coming from their own professional class. They are rarely taught any kind of statistics in law school. Nobody wants to discuss math in debate club - answers with hard right and wrong answers are no fun to them. Your pessimism wrt. modeling may be justified, but you're not nearly pessimistic enough about people or the legal system.
phazelift 10y ago
It might be a better idea to first train computers to define criminality objectively, because most people cannot.
- jobigoud 10y ago
  > most people cannot So why do we hold computers to higher standards than humans? Either it's OK to not being able to define criminality objectively and in that case algorithms shouldn't be disqualified for this very reason, or it's not OK, but in that case humans should not be allowed to do the job either.
  - phazelift 10y ago
    Convicting people for criminal behaviour based on a subjective definition of it, is what we already do wrong. Way too many innocent people end up being punished or killed, which I expect to be (objectively) a criminal act in itself. So, I thought we better first have a tool that solves that, instead of a tool that amplifies it.
imh 10y ago
I think the whole idea here is frightening and unjust. We are supposed to give all people equal rights. What people might do is irrelevant. A person whose demographic/conditional expectation is highly criminal should be given an equal opportunity to rise above it, else they might see the system is rigged against them and turn it into a self-fulfilling prophecy.
- Taek 10y ago
  It's frightening depending on how you use the data. A good example perhaps is that I like to horse around when I'm at the beach. I'm more like to get hurt than others who are more cautious. I'm also more likely to hit people accidentally. I had some parents of younger children approach me and ask me to stay on the far side of the beach. On one hand it felt rude, but on the other it allowed me to be rambunctious and it allowed the parents to prioritize their children's safety. The world isn't flat enough for this to be a reality yet, but if you cluster people by their morals, you don't have to throw them in jail. Put all the drug users together. Keep the drugs away from people who don't want anything to do with them. Usually if people are more likely to commit crimes, it's either because they are desperate (which means successful intervention is likely provided you can solve their core problems), or it's because they find that activity/action/crime to be morally or culturally acceptable. To the extent that you can exclude that culture from your own daily life, you don't have to punish/kill that culture. Pollution is a good counter example. You can't really isolate a culture of pollution because it's going to affect everyone else anyway. So there are limits. As long as our methods for dealing with criminals evolve appropriately against our ability to detect them, I am okay. Human history is full of genocide though. I don't think that bodes well for our ability to respect cultures that allow or celebrate things we consider to be crimes.
brillenfux 10y ago
The nonchalance of these people is what really terrifies me. They just laugh any valid criticism off and start using the references "ironically" themselves. I don't understand how they can do that; do they not have a moral compass? are they psychopaths?
- brillenfux 10y ago
  What happened here? There was a whole thread coming after that?!
  - dang 10y ago
    https://news.ycombinator.com/item?id=12123453 https://news.ycombinator.com/item?id=12123453
Dr_tldr 10y ago
I know, it's almost as if they don't consider you the sole and undisputed arbiter of the limits of technology in creating social policy. What a bunch of psychopaths!
- eyelidlessness 10y ago
  If you have (created!) a job that closely resembles a work of dystopian fiction, laughing that off is absolutely lacking in human empathy. That's not even the first problem with this line of work, but since you're also laughing off the problem, it deserves a rebuttal. If I said to you that I was going to create a network of surveillance devices that also serves as mindless entertainment and routinely broadcasts faith routines that non-participants will be punished for, and you told me that sounds like something out of 1984, and I told you were paranoid, you'd think I'm mad. And the advance of technology unhindered is not a universal good. Algorithms only have better judgment than humans according to the constraints they were assigned. If there's a role for automation in criminal justice, that role must be constantly questioned and adjusted for human need, just as the role of human intervention should be. Because it's all human intervention.
  - ionwake 10y ago
    What is a "faith routine" ? Thanks
    - Jtsummers 10y ago
      In context, faith routines would be things like, in the book 1984, the Two Minutes Hate. In reality, it might be a (imlicitly mandatory, if not explicitly) routine such as pledging allegiance to a flag, or mandatory participation in a moment of prayer, or something similar.
  - Dr_tldr 10y ago
    Or it's that non-specialists with only a vague understanding of what you're doing don't know what you're talking about. I don't consider stem cell researchers to be lacking in empathy because some of the hyper-religious accuse them of dangerous experiments and "playing god", even though their work "resembles a work of dystopian fiction." Of course we can question where technology is going, but saying "gee, group x commits crimes at 27 times the rate of group y, on average we should consider them to be more at-risk" isn't 1984, it's how car insurance, the job market, dating, the stock market, almost everything works. It's not optimally fair by a long shot, but neither is the alternative of analyzing no data and making social policy out of how you wish things were instead of how the data tell you they actually are. The solution to data telling you unpleasant facts is to try to change those facts with policy, not put your head in the sand and bleat "totalitarianism" every time a fact enters the conversation.
Houshalter 10y ago
I could not disagree more with these comments. Psychologists are just now starting to study the phenomenon of "algorithm aversion", where people irrationally trust human judgement far more than algorithms. Even after watching an algorithm do far better in many examples. The reality is humans are far worse. We are biased by all sorts of things. Unattractive people were found to get twice as long sentences as attractive ones. Judges were found to give much harsher sentences right before lunch time, when they were hungry. Doing interviews was found to decrease the performance of human judges, in domains like hiring and determining parolle. As opposed to just looking at the facts. Even very simple statistical algorithms far outperform humans in almost every domain. As early as 1928, a simple statistical rule predicted recidivism better than prison psychologists. They predict the success of college students, job applicants, outcomes of medical treatment, etc, far better than human experts. Human experts never even beat the most basic statistical baseline. You should never ever trust human judges. They are neither fair nor accurate. In such an important domain as this, where better predictions reduce the time people spend in prison and crime, there is no excuse not to use them. Anything that gets low risk people out of prison is good. I believe that any rules that apply to algorithms should apply to humans too. We are algorithms too after all. If algorithms have to be blind to race and gender, so should human judges. If economic information is bad to use, humans should be blind to it also. If we have a right to see why an algorithm made a decision the way it did, we should be able to inspect human brains to. Perhaps put judges and parolle officers in an MRI.
- coldtea 10y ago
  >I could not disagree more with these comments. Psychologists are just now starting to study the phenomenon of "algorithm aversion", where people irrationally trust human judgement far more than algorithms. Even after watching an algorithm do far better in many examples. Those algorithms are also designed by humans and codify biases and prejudices and injustices that are considered OK by society into them. Besides, it's like going for a run. Other might might be far better at it, but we want to do it ourselves, even badly, than just be spectators in our own life. Same for humans becoming spectators while algorithms run business "better". One quality that's not accounted for in accessing better is "more participative".
deleted 10y ago
[deleted]
justaaron 10y ago
gee, what could possibly go wrong, Mr. Phrenologist? SOMEONE seems to have viewed Minority Report as an Utopia rather than Dystopia, I'm afraid.
moconnor 10y ago
"between 29 percent and 38 percent of predictions about whether someone is low-risk end up being wrong" Wouldn't win a Kaggle contest with that error rate. What's not disclosed is the percent of predictions about whether someone is high-risk ending up being wrong. These are the ones society should be worried about. And these are the ones that are, if such a system is put into practice, impossible to track. Because all the high-risk people are locked up. The socio-political fallout of randomly letting some high-risk people free to validate the algorithm makes this inevitable. This leaves us in a situation where political pressure is always towards reducing the number of people classified as low-risk who then re-offend. Statistical competence is not prevalent enough in the general population to prevent this. TL;DR our society is either not well-educated enough or is improperly structured to correctly apply algorithms for criminal justice.
- Houshalter 10y ago
  The question is whether human judges do better, and they don't. We have no better method of determining whether someone is low risk or high risk. But keeping everyone locked up forever is just stupid. If these predictions let some people get out of prison sooner, I think that is a net good.
acd 10y ago
Would the following be common risk factors for a child becoming future criminal? Would it not be cheaper for society to invest in this risk children early on rather than dealing with their actions as an adult? Minority report. What are your observations for risk factors? Has there been Social science any interviews of prisoners and their background feed into classification engines? Classification ideas: * Bad parents not raising their child * Living in a poor neighbourhood with lots of crime * Going to a bad school * Parents who are workaholics. * Single parent * Parent who is in jail
nl 10y ago
For those who haven't read it, Propublica article on this is even better (and scarier): https://www.propublica.org/article/machine-bias-risk-assessments-in-criminal-sentencing https://www.propublica.org/article/machine-bias-risk-assessm...
kriro 10y ago
Predictive policing is quite the buzz word these days. IBM (via SPSS) is one of the big players in the field. The most common use case is burglary, I suspect because that's somewhat easy (and also directly actionable). You rarely find other use cases in academic papers (well I only browsed the literature a couple of times preparing for related projects). The basic idea is sending more police patrols to areas that are identified as high thread and thus using your available resources more efficiently. The focus in that area is more on objects/areas than on individuals so you don't try to predict who's a criminal but rather where they'll strike. It sounds like a good enough idea in theory but at least in Germany I know that research projects for predictive policing will be scaled down due to privacy concerns even if the prediction is only area and not person based (noteworthy that that's usually mentioned by the police as a reason why they won't participate in the research). I'm not completely sure and only talked to a couple of state police research people but quite often the data also involves social media in some way and that's the major problem from what I can tell.
- jmngomes 10y ago
  > IBM (via SPSS) is one of the big players in the field They have been pitching "crime prediction" since at least 2010 with no real results so far...
  - antisthenes 10y ago
    The results are that IBM's consulting arm is flourishing from all the crime prediction contracts.
DisgustingRobot 10y ago
I'm curious how good an algorithm would be at identifying future white collar criminals. What would the risk factors be for things like insider trading, political corruption, or other common crimes?
andrewaylett 10y ago
I like the proposal from the EU that automated decisions with a material impact must firstly come with a justification -- so the system must be able to tell you why it came out with the answer it gave -- and must have the right of appeal to a human. The implementation is the difficult bit, of course, but as a principle, I appreciate the ability to sanity-check outputs that currently lack transparency.
- dasboth 10y ago
  Interpretability is going to be a huge area of research in machine learning, what with the advent of deep learning techniques. It's hard enough explaining the output of a random forest, what about a deep net with 100 layers? In some cases it doesn't matter, e.g. you generally don't care why Amazon thinks you should buy book A over book B, but in instances where someone's prison sentence is the output, it will be vital.
sevenless 10y ago
The entire concept of using statistical algorithms to 'predict crime' is wrong. It's just a kind of stereotyping. What needs to happen is a consideration of the social-justice outcomes if 'profiling algorithms' become widely used. Just as in any complicated system, you cannot simply assume reasonable looking rules will translate to desirable emergent properties. It is ethically imperative to aim to eliminate disparities and social inequalities between races, even if, and this is what is usually left unsaid, judgments become less accurate in the process. Facts becoming common knowledge can harm people, even if they are true. Increasingly accurate profiling will have bad effects at the macro scale, and keep marginalized higher-crime groups permanently marginalized. If it were legal to use all the information to hand, it would be totally rational for employers to discriminate against certain groups on the basis of a higher group risk of crime, and that would result in those groups being marginalized even further. We should avoid this kind of societal positive feedback loop. If you accept that government should want to avoid a segregated society, where some groups of people form a permanent underclass, you should avoid any algorithm that results in an increased differential arrest rate for those groups, even if that arrest rate is warranted by actual crimes committed. "The social norm against stereotyping, including the opposition to profiling, has been highly beneficial in creating a more civilized and more equal society. It is useful to remember, however, that neglecting valid stereotypes inevitably results in suboptimal judgments. Resistance to stereotyping is a laudable moral position, but the simplistic idea that the resistance is costless is wrong. The costs are worth paying to achieve a better society, but denying that the costs exist, while satisfying to the soul and politically correct, is not scientifically defensible. Reliance on the affect heuristic is common in politically charged arguments. The positions we favor have no cost and those we oppose have no benefits. We should be able to do better." –Daniel Kahneman, Nobel laureate, in Thinking, Fast and Slow, chapter 16
- deleted 10y ago
  [deleted]
- wtbob 10y ago
  > It is ethically imperative to aim to eliminate disparities and social inequalities between races, even if, and this is what is usually left unsaid, judgments become less accurate in the process. Why? Why is it 'imperative' to be wrong? > Facts becoming common knowledge can harm people, even if they are true. Well, they can harm people who, statistically speaking, are more likely to be bad. If anything, I see accurate statistical profiling being helpful to black folks. Right now, based on FBI arrest data, a random black man is 6.2 times as likely to be a murderer as a random white man; a good statistical profiling algorithm would be able to look at an individual black man and see that he's actually a married, college-educated middle-class recent immigrant from Africa, who lives in a low-crime area — and say that he's less likely than a random white man to be a murderer. Perhaps it could even look at an individual black man, the son of a single mother from the projects, and see that he's actually not like others whom those phrases would describe, because of other factors the algorithm takes into account. > If you accept that government should want to avoid a segregated society, where some groups of people form a permanent underclass, you should avoid any algorithm that results in an increased differential arrest rate for those groups, even if that arrest rate is warranted by actual crimes committed. That statement implies that we should avoid the algorithm 'arrest anyone who has committed a crime, and no-one else,' because that algorithm will necessarily result in increased differential arrest rates. On the contrary, I think that algorithm is obviously ideal, and thus any heuristic which leads to rejecting it should itself be rejected.
peterbonney 10y ago
Here's something I really dislike about all the coverage I've seen about these "risk assessment algorithms": There is absolutely no discussion of the magnitude of the distinctions between classifications. Is "low risk" supposed to be (say) 0.01% likelihood of committing another crime and "high risk" (say) 90%? Or is "low risk" (say) 1% vs. "high risk" of (say) 3%? Having worked on human some predictive modeling of "bad" human events (loan defaults) my gut says it's more like the latter than the former, because prediction of low-frequency human events is really hard, and, well, they're by definition infrequent. If that suspicion is right, then the signal-noise ratio is probably too poor to even consider using them in sentencing, and that's without considering the issues of bias in the training data, etc. But there is never enough detail provided (on either side of the debate) for me to make an informed assessment. It's just a lot of optimism on one side and pessimism on the other. I'd really love to see some concrete, testable claims without having to dive down a rabbit hole to find them.
liberal_arts 10y ago
consider the (fictional) possibility that an AI will be " actively measuring the populace's mental states, personalities, and the probability that individuals will commit crimes " https://en.wikipedia.org/wiki/Psycho-Pass https://en.wikipedia.org/wiki/Psycho-Pass AI may be worth the trade-off if violent crime can be almost eliminated. or consider (non-fiction): body-language/facial detection at airports; what if they actually start catching terrorists?
- vegabook 10y ago
  There is a school of thought that says some crime is necessary for a healthy, functioning society. Personally, while I would hate to be the victim of violent crime (obviously), I actually do agree that cities with very low crime levels are often stultifyingly uncharismatic. https://www.d.umn.edu/~bmork/2111/readings/durkheimrules.htm https://www.d.umn.edu/~bmork/2111/readings/durkheimrules.htm
conjectures 10y ago
What is Berk's model? How well does it do across different risk bands? What variables are fed into it in the states where it is used? How does prediction success vary across types of crimes, versus demographics within crime? This article treats ML like a magic wand, which it isn't. There's not enough information to make a judgement on whether the tools are performing well or not, or whether that performance, or lack of it, is based on discrimination. Where we do have information it is worrying: "Race isn’t an input in any of his systems, and he says his own research has shown his algorithms produce similar risk scores regardless of race." What?!? The appropriate approach would be to include race as a variable, fit the model, and then marginalise out race when providing risk predictions. Confounding is mentioned but the explanation of how it is dealt with, without doing the above isn't given - just a (most likely false) reassurance.
mc32 10y ago
The main question should be, like with autonomous vehicles, is does this system perform better than people (however you want to qualify that)? If so, it's better than what we have. Second, even if it's proven better (fewer false positives, less unduly biased results) it can be improved continuously. There is a danger in that people may not like the results because if we take this and diffuse it, has the potential to shape people's behavior in unintended ways (gaming), on the other hand this system has the potential for objectivity when identifying white collar crime, that is surfacing it better.
Digit-Al 10y ago
I find this really interesting. I think what most people seem to be missing is the wider social context. Think about this. If you exclude white collar financial crime, pre-meditated murder, and organised crime - most other crimes are committed by the socially disadvantaged. So, if the algorithm identifies an area where crime is more likely to be committed, instead of being narrow minded and just putting more police there to arrest people, why not instead try to institute programs to raise the socioeconomic status of the area? People are just concentrating on the crime aspect, but most crime is just a symptom of social inequality.
- darpa_escapee 10y ago
  The American thing to do is to cry "personal responsibility" and treat the symptoms with jail time, fines and a lengthened rap sheet. Suggesting that we treat the cause suggests we all have responsibilities as members of communities to ensure no one is in a place where crime might make sense despite the consequences.

90 points by nigrioid 10y ago

niels_olson 10y ago

Can someone just go ahead and inject a blink tag so we can get the full 1994 experience? Oh, my retina...

cloudjacker 10y ago
http://www.wonder-tonic.com/geocitiesizer/content.php?theme=1&music=2&url=http://www.bloomberg.com/features/2016-richard-berk-future-crime/ http://www.wonder-tonic.com/geocitiesizer/content.php?theme=... surprisingly more readable
- cJ0th 10y ago
  On a serious note: They original link looks okayish when you switch to article mode in Firefox.

jamesrom 10y ago

What is bloomberg's MO with these near unreadable articles?

puddintane 10y ago
Still rocking paint like it's 1995 - Bloomberg The color choice is just yikes! Did anyone else get reminded of Futurama - Law and Oracle (S06E16 / E104)? [1] https://en.wikipedia.org/wiki/Law_and_Oracle https://en.wikipedia.org/wiki/Law_and_Oracle I do wonder if this type of technology is something we should slowly approach due to the very nature of the outcome of sentencing. We already incarcerate a lot of innocent people and I truly wonder if this is something we should tread lightly on.
- nickles 10y ago
  Law and Oracle is an adaptation of PKD's short story "Minority Report" in which the Pre-Cog (short for pre-cognition) department polices based on glimpses of future crime. Orwell's "1984" explored the similar, but distinct, notion of 'thoughtcrime'. Both works examine the implications of such policing methods and are certainly worth reading.
  - Jaruzel 10y ago
    Or, if you are time impaired, there is of course Minority Report the film, starring Tom Cruise. Not as in depth, of course, but not a bad adaptation imho.
  - Pamar 10y ago
    To be honest I read 1984 maybe six or seven times and I do not remeber "prediction" to be one of the main themes. See https://en.wikipedia.org/wiki/Thoughtcrime https://en.wikipedia.org/wiki/Thoughtcrime The oppressive regime of 1984 does not use math or computers to look for "suspects". We might argue this was because Orwell had no idea about these methods (book was written in 1948, after all) but personally I doubt he would have used this because advocating decisions to an impersonal algorithm would make the bad guys slightly less bad: in the novel the main baddie says something like "do you want to see the future of Human Race? Think of a boot stomping on a human face, forever...". I.e. power for them is a end in itself, and the only way to use it is to make someone else suffer. I don't see any place for "impersonal algorithms to better adjudicate anything" in this.
- ladzoppelin 10y ago
  Firefox Reader View makes the so much nicer.
intrasight 10y ago
I was actually pleasantly surprised that the weird style and graphics rendered since most of their article just display as plain text due to my uBlock Origin settings. And I sorta like the weird style, mainly because it's out of character with Bloomberg.
eyelidlessness 10y ago
I find them more readable than the endless wall of indistinguishable text of pick-a-random-news-site. Of course all the text is readable, and there's innumerable ways of increasing the readability. But their art direction and effort to distinguish particular content helps it stand out and helps overcome the kind of fatigue that causes readers to drop off within a couple paragraphs and has plagued other content-oriented sites with horrifying "minutes to read" estimates.

ccvannorman 10y ago

>Risk scores, generated by algorithms, are an increasingly common factor in sentencing. Computers crunch data—arrests, type of crime committed, and demographic information—and a risk rating is generated. The idea is to create a guide that’s less likely to be subject to unconscious biases, the mood of a judge, or other human shortcomings. Similar tools are used to decide which blocks police officers should patrol, where to put inmates in prison, and who to let out on parole. So, eventually a robot police officer will arrest someone for having the wrong profile. >Berk wants to predict at the moment of birth whether people will commit a crime by their 18th birthday, based on factors such as environment and the history of a new child’s parents. This would be almost impossible in the U.S., given that much of a person’s biographical information is spread out across many agencies and subject to many restrictions. He’s not sure if it’s possible in Norway, either, and he acknowledges he also hasn’t completely thought through how best to use such information. So, we're not sure how dangerous this will be, or how Minority Report thoughtcrime will work, but we're damned sure we want it, because it's the future and careers will be made? This is a very scary trend in the U.S. Eventually, if you're born poor/bad childhood, you will have even less of a chance of making it.

sliverstorm 10y ago
On the bright side, if we can pinpoint at risk children with high accuracy, we can also help them make it. Like the attempts to deradicalize individuals at very high risk of flying of to syria, rather than arresting them.
- seanmcdirmid 10y ago
  We can already pinpoint at risk children with high accuracy, just check out any inner city ghetto. We just lack the caring needed to do anything about the root cause (e.g. and mostly poverty) that causes kids to go bad later. It isn't a mystery.

deleted 10y ago

[deleted]

Smerity 10y ago

As someone who does machine learning, this absolutely terrifies me. The "capstone project" of determining someone's probability of committing a crime by their 18th birthday is beyond ridiculous. Either the author of the article hyped it to the extreme (for the love of everything that's holy, stop freaking hyping machine learning) or the statistician is stark raving mad. The fact that he does this for free is also concerning, primarily as I doubt this has any level of auditing behind it. The only thing I agree with him on is that black box models are even worse as they have even worse audit issues. Given the complexities in making these predictions and the potentially life long impact they might have, there is such a desperately strong need for these systems to have audit guarantees. It's noted that he supposedly shares the code for his systems - if so, I'd love to see it? Is it just shared with the relevant governmental departments who likely have no ability to audit such models? Has it been audited? Would you trust mission critical code that didn't have some level of unit testing? Some level of code review? No? Then why would you potentially destructively change someone's life based on that same level of quality? > "[How risk scores are impacted by race] has not been analyzed yet," she said. "However, it needs to be noted that parole is very different than sentencing. The board is not determining guilt or innocence. We are looking at risk." What? Seriously? Not analyzed? The other worrying assumption is that it isn't used in sentencing. People have a tendency to seek out and misuse information even if they're told not to. This was specifically noted in another article on the misuse of Compas, the black box system. Deciding on parole also doesn't mean you can avoid analyzing bias. If you're denying parole for specific people algorithmically, that can still be insanely destructive. > Berk readily acknowledges this as a concern, then quickly dismisses it. Race isn’t an input in any of his systems, and he says his own research has shown his algorithms produce similar risk scores regardless of race. There are so many proxies for race within the feature set. It's touched on lightly in the article - location, number of arrests, etc - but it gets even more complex when you allow a sufficiently complex machine learning model access to "innocuous" features. Specific ML systems ("deep") can infer hidden variables such as race. Even location is a brilliant proxy for race as seen in redlining[1]. It does appear from his publications that they're shallow models - namely random forests, logistic regression, and boosting[2][3][4]. FOR THE LOVE OF EVERYTHING THAT'S HOLY STOP THROWING MACHINE LEARNING AT EVERYTHING. Think it through. Please. Please please please. I am a big believer that machine learning can enable wonderful things - but it could also enable a destructive feedback loop in so many systems. Resume screening, credit card applications, parole risk classification, ... This is just the tip of the iceberg of potential misuses for machine learning. Edit: I am literally physically feeling ill. He uses logistic regression, random forests, boosting ... standard machine learning algorithms. Fine. Okay ... but you now think the algorithms that might get you okay results on Kaggle competitions can be used to predict a child's future crimes?!?! WTF. What. The actual. ^^^^. Anyone who even knows the hello world of machine learning would laugh at this if the person saying it wasn't literally supplying information to governmental agencies right now. I wrote an article last week on "It's ML, not magic"[5] but I didn't think I'd need to cover this level of stupidity. [1]: https://en.wikipedia.org/wiki/Redlining https://en.wikipedia.org/wiki/Redlining [2]: https://books.google.com/books/about/Criminal_Justice_Forecasts_of_Risk.html?id=Jrlb6Or8YisC&printsec=frontcover&source=kp_read_button&hl=en#v=onepage&q&f=false https://books.google.com/books/about/Criminal_Justice_Foreca... [3]: https://www.semanticscholar.org/paper/Developing-a-Practical-Forecasting-Screener-for-Berk-He/6999981067428dafadd10aa736e4b5c293f89823 https://www.semanticscholar.org/paper/Developing-a-Practical... [4]: https://www.semanticscholar.org/paper/Algorithmic-criminology-Berk/226defcf96d30cf0a17c6caafd60457c9411f458 https://www.semanticscholar.org/paper/Algorithmic-criminolog... [5]: http://smerity.com/articles/2016/ml_not_magic.html http://smerity.com/articles/2016/ml_not_magic.html

anupshinde 10y ago

This is like machine introduced bias/racisim/castism... we need a new term for that.. and its based on statistically induced pseudo-sciences many times similar to astrology. This is the kind of AI everyone should be afraid of.

benkuykendall 10y ago
I fail to see what is unscientific about stating conditional probabilities. Astrology is unscientific because the orientation of the heavens is for the most part independent of the observables of a person's life. But the inputs to Berk's system clearly do effect the probability of committing crime. A frequentist would say that a large group of people with this set of characteristics would yield more or fewer criminals; a Bayesian would say we have more knowledge about whether such an individual will commit future crime. These are scientific conclusions. The question "how should we use this data" is a question of ethics, not science.
- eyelidlessness 10y ago
  > I fail to see what is unscientific about stating conditional probabilities. 1. Create conditions which disadvantage and impoverish a segment of society. 2. Refine those conditions for centuries, continually criminalizing the margins that segment of society is forced to live in. 3. Identify that many of the people in that segment of society are likely to be identified as criminals. 4. Pretend that you're doing math rather than reinforcing generations of deleterious conditioning, completely ignoring the creation of those conditions that led to the probabilities you're identifying. And science can't be divorced from ethics. These are human pursuits.
  - vintermann 10y ago
    This looks like sloppy ML. But human judges do all those things already (susbtitute "just applying the law" for "doing math" in 4) and they can't be inspected - their brains are "closed source". Sure, these humans can come up with wordy justifications for their decisions. But there are plenty of intelligent people who employ their intelligence not to arrive at a conclusion, but to justify the conclusion they already arrived at. Legal professionals aren't merely capable of this, they're explicitly trained to be experts at post-hoc justification. And legal professionals basically ignore all criticism not coming from their own professional class. They are rarely taught any kind of statistics in law school. Nobody wants to discuss math in debate club - answers with hard right and wrong answers are no fun to them. Your pessimism wrt. modeling may be justified, but you're not nearly pessimistic enough about people or the legal system.

phazelift 10y ago

It might be a better idea to first train computers to define criminality objectively, because most people cannot.

jobigoud 10y ago
> most people cannot So why do we hold computers to higher standards than humans? Either it's OK to not being able to define criminality objectively and in that case algorithms shouldn't be disqualified for this very reason, or it's not OK, but in that case humans should not be allowed to do the job either.
- phazelift 10y ago
  Convicting people for criminal behaviour based on a subjective definition of it, is what we already do wrong. Way too many innocent people end up being punished or killed, which I expect to be (objectively) a criminal act in itself. So, I thought we better first have a tool that solves that, instead of a tool that amplifies it.

imh 10y ago

I think the whole idea here is frightening and unjust. We are supposed to give all people equal rights. What people might do is irrelevant. A person whose demographic/conditional expectation is highly criminal should be given an equal opportunity to rise above it, else they might see the system is rigged against them and turn it into a self-fulfilling prophecy.

Taek 10y ago
It's frightening depending on how you use the data. A good example perhaps is that I like to horse around when I'm at the beach. I'm more like to get hurt than others who are more cautious. I'm also more likely to hit people accidentally. I had some parents of younger children approach me and ask me to stay on the far side of the beach. On one hand it felt rude, but on the other it allowed me to be rambunctious and it allowed the parents to prioritize their children's safety. The world isn't flat enough for this to be a reality yet, but if you cluster people by their morals, you don't have to throw them in jail. Put all the drug users together. Keep the drugs away from people who don't want anything to do with them. Usually if people are more likely to commit crimes, it's either because they are desperate (which means successful intervention is likely provided you can solve their core problems), or it's because they find that activity/action/crime to be morally or culturally acceptable. To the extent that you can exclude that culture from your own daily life, you don't have to punish/kill that culture. Pollution is a good counter example. You can't really isolate a culture of pollution because it's going to affect everyone else anyway. So there are limits. As long as our methods for dealing with criminals evolve appropriately against our ability to detect them, I am okay. Human history is full of genocide though. I don't think that bodes well for our ability to respect cultures that allow or celebrate things we consider to be crimes.

brillenfux 10y ago

The nonchalance of these people is what really terrifies me. They just laugh any valid criticism off and start using the references "ironically" themselves. I don't understand how they can do that; do they not have a moral compass? are they psychopaths?

brillenfux 10y ago
What happened here? There was a whole thread coming after that?!
- dang 10y ago
  https://news.ycombinator.com/item?id=12123453 https://news.ycombinator.com/item?id=12123453

Dr_tldr 10y ago

I know, it's almost as if they don't consider you the sole and undisputed arbiter of the limits of technology in creating social policy. What a bunch of psychopaths!

eyelidlessness 10y ago
If you have (created!) a job that closely resembles a work of dystopian fiction, laughing that off is absolutely lacking in human empathy. That's not even the first problem with this line of work, but since you're also laughing off the problem, it deserves a rebuttal. If I said to you that I was going to create a network of surveillance devices that also serves as mindless entertainment and routinely broadcasts faith routines that non-participants will be punished for, and you told me that sounds like something out of 1984, and I told you were paranoid, you'd think I'm mad. And the advance of technology unhindered is not a universal good. Algorithms only have better judgment than humans according to the constraints they were assigned. If there's a role for automation in criminal justice, that role must be constantly questioned and adjusted for human need, just as the role of human intervention should be. Because it's all human intervention.
- ionwake 10y ago
  What is a "faith routine" ? Thanks
  - Jtsummers 10y ago
    In context, faith routines would be things like, in the book 1984, the Two Minutes Hate. In reality, it might be a (imlicitly mandatory, if not explicitly) routine such as pledging allegiance to a flag, or mandatory participation in a moment of prayer, or something similar.
- Dr_tldr 10y ago
  Or it's that non-specialists with only a vague understanding of what you're doing don't know what you're talking about. I don't consider stem cell researchers to be lacking in empathy because some of the hyper-religious accuse them of dangerous experiments and "playing god", even though their work "resembles a work of dystopian fiction." Of course we can question where technology is going, but saying "gee, group x commits crimes at 27 times the rate of group y, on average we should consider them to be more at-risk" isn't 1984, it's how car insurance, the job market, dating, the stock market, almost everything works. It's not optimally fair by a long shot, but neither is the alternative of analyzing no data and making social policy out of how you wish things were instead of how the data tell you they actually are. The solution to data telling you unpleasant facts is to try to change those facts with policy, not put your head in the sand and bleat "totalitarianism" every time a fact enters the conversation.

Houshalter 10y ago

I could not disagree more with these comments. Psychologists are just now starting to study the phenomenon of "algorithm aversion", where people irrationally trust human judgement far more than algorithms. Even after watching an algorithm do far better in many examples. The reality is humans are far worse. We are biased by all sorts of things. Unattractive people were found to get twice as long sentences as attractive ones. Judges were found to give much harsher sentences right before lunch time, when they were hungry. Doing interviews was found to decrease the performance of human judges, in domains like hiring and determining parolle. As opposed to just looking at the facts. Even very simple statistical algorithms far outperform humans in almost every domain. As early as 1928, a simple statistical rule predicted recidivism better than prison psychologists. They predict the success of college students, job applicants, outcomes of medical treatment, etc, far better than human experts. Human experts never even beat the most basic statistical baseline. You should never ever trust human judges. They are neither fair nor accurate. In such an important domain as this, where better predictions reduce the time people spend in prison and crime, there is no excuse not to use them. Anything that gets low risk people out of prison is good. I believe that any rules that apply to algorithms should apply to humans too. We are algorithms too after all. If algorithms have to be blind to race and gender, so should human judges. If economic information is bad to use, humans should be blind to it also. If we have a right to see why an algorithm made a decision the way it did, we should be able to inspect human brains to. Perhaps put judges and parolle officers in an MRI.

coldtea 10y ago
>I could not disagree more with these comments. Psychologists are just now starting to study the phenomenon of "algorithm aversion", where people irrationally trust human judgement far more than algorithms. Even after watching an algorithm do far better in many examples. Those algorithms are also designed by humans and codify biases and prejudices and injustices that are considered OK by society into them. Besides, it's like going for a run. Other might might be far better at it, but we want to do it ourselves, even badly, than just be spectators in our own life. Same for humans becoming spectators while algorithms run business "better". One quality that's not accounted for in accessing better is "more participative".

deleted 10y ago

[deleted]

justaaron 10y ago

gee, what could possibly go wrong, Mr. Phrenologist? SOMEONE seems to have viewed Minority Report as an Utopia rather than Dystopia, I'm afraid.

moconnor 10y ago

"between 29 percent and 38 percent of predictions about whether someone is low-risk end up being wrong" Wouldn't win a Kaggle contest with that error rate. What's not disclosed is the percent of predictions about whether someone is high-risk ending up being wrong. These are the ones society should be worried about. And these are the ones that are, if such a system is put into practice, impossible to track. Because all the high-risk people are locked up. The socio-political fallout of randomly letting some high-risk people free to validate the algorithm makes this inevitable. This leaves us in a situation where political pressure is always towards reducing the number of people classified as low-risk who then re-offend. Statistical competence is not prevalent enough in the general population to prevent this. TL;DR our society is either not well-educated enough or is improperly structured to correctly apply algorithms for criminal justice.

Houshalter 10y ago
The question is whether human judges do better, and they don't. We have no better method of determining whether someone is low risk or high risk. But keeping everyone locked up forever is just stupid. If these predictions let some people get out of prison sooner, I think that is a net good.

acd 10y ago

Would the following be common risk factors for a child becoming future criminal? Would it not be cheaper for society to invest in this risk children early on rather than dealing with their actions as an adult? Minority report. What are your observations for risk factors? Has there been Social science any interviews of prisoners and their background feed into classification engines? Classification ideas: * Bad parents not raising their child * Living in a poor neighbourhood with lots of crime * Going to a bad school * Parents who are workaholics. * Single parent * Parent who is in jail

nl 10y ago

For those who haven't read it, Propublica article on this is even better (and scarier): https://www.propublica.org/article/machine-bias-risk-assessments-in-criminal-sentencing https://www.propublica.org/article/machine-bias-risk-assessm...

kriro 10y ago

Predictive policing is quite the buzz word these days. IBM (via SPSS) is one of the big players in the field. The most common use case is burglary, I suspect because that's somewhat easy (and also directly actionable). You rarely find other use cases in academic papers (well I only browsed the literature a couple of times preparing for related projects). The basic idea is sending more police patrols to areas that are identified as high thread and thus using your available resources more efficiently. The focus in that area is more on objects/areas than on individuals so you don't try to predict who's a criminal but rather where they'll strike. It sounds like a good enough idea in theory but at least in Germany I know that research projects for predictive policing will be scaled down due to privacy concerns even if the prediction is only area and not person based (noteworthy that that's usually mentioned by the police as a reason why they won't participate in the research). I'm not completely sure and only talked to a couple of state police research people but quite often the data also involves social media in some way and that's the major problem from what I can tell.

jmngomes 10y ago
> IBM (via SPSS) is one of the big players in the field They have been pitching "crime prediction" since at least 2010 with no real results so far...
- antisthenes 10y ago
  The results are that IBM's consulting arm is flourishing from all the crime prediction contracts.

DisgustingRobot 10y ago

I'm curious how good an algorithm would be at identifying future white collar criminals. What would the risk factors be for things like insider trading, political corruption, or other common crimes?

andrewaylett 10y ago

I like the proposal from the EU that automated decisions with a material impact must firstly come with a justification -- so the system must be able to tell you why it came out with the answer it gave -- and must have the right of appeal to a human. The implementation is the difficult bit, of course, but as a principle, I appreciate the ability to sanity-check outputs that currently lack transparency.

dasboth 10y ago
Interpretability is going to be a huge area of research in machine learning, what with the advent of deep learning techniques. It's hard enough explaining the output of a random forest, what about a deep net with 100 layers? In some cases it doesn't matter, e.g. you generally don't care why Amazon thinks you should buy book A over book B, but in instances where someone's prison sentence is the output, it will be vital.

sevenless 10y ago

The entire concept of using statistical algorithms to 'predict crime' is wrong. It's just a kind of stereotyping. What needs to happen is a consideration of the social-justice outcomes if 'profiling algorithms' become widely used. Just as in any complicated system, you cannot simply assume reasonable looking rules will translate to desirable emergent properties. It is ethically imperative to aim to eliminate disparities and social inequalities between races, even if, and this is what is usually left unsaid, judgments become less accurate in the process. Facts becoming common knowledge can harm people, even if they are true. Increasingly accurate profiling will have bad effects at the macro scale, and keep marginalized higher-crime groups permanently marginalized. If it were legal to use all the information to hand, it would be totally rational for employers to discriminate against certain groups on the basis of a higher group risk of crime, and that would result in those groups being marginalized even further. We should avoid this kind of societal positive feedback loop. If you accept that government should want to avoid a segregated society, where some groups of people form a permanent underclass, you should avoid any algorithm that results in an increased differential arrest rate for those groups, even if that arrest rate is warranted by actual crimes committed. "The social norm against stereotyping, including the opposition to profiling, has been highly beneficial in creating a more civilized and more equal society. It is useful to remember, however, that neglecting valid stereotypes inevitably results in suboptimal judgments. Resistance to stereotyping is a laudable moral position, but the simplistic idea that the resistance is costless is wrong. The costs are worth paying to achieve a better society, but denying that the costs exist, while satisfying to the soul and politically correct, is not scientifically defensible. Reliance on the affect heuristic is common in politically charged arguments. The positions we favor have no cost and those we oppose have no benefits. We should be able to do better." –Daniel Kahneman, Nobel laureate, in Thinking, Fast and Slow, chapter 16

deleted 10y ago
[deleted]
wtbob 10y ago
> It is ethically imperative to aim to eliminate disparities and social inequalities between races, even if, and this is what is usually left unsaid, judgments become less accurate in the process. Why? Why is it 'imperative' to be wrong? > Facts becoming common knowledge can harm people, even if they are true. Well, they can harm people who, statistically speaking, are more likely to be bad. If anything, I see accurate statistical profiling being helpful to black folks. Right now, based on FBI arrest data, a random black man is 6.2 times as likely to be a murderer as a random white man; a good statistical profiling algorithm would be able to look at an individual black man and see that he's actually a married, college-educated middle-class recent immigrant from Africa, who lives in a low-crime area — and say that he's less likely than a random white man to be a murderer. Perhaps it could even look at an individual black man, the son of a single mother from the projects, and see that he's actually not like others whom those phrases would describe, because of other factors the algorithm takes into account. > If you accept that government should want to avoid a segregated society, where some groups of people form a permanent underclass, you should avoid any algorithm that results in an increased differential arrest rate for those groups, even if that arrest rate is warranted by actual crimes committed. That statement implies that we should avoid the algorithm 'arrest anyone who has committed a crime, and no-one else,' because that algorithm will necessarily result in increased differential arrest rates. On the contrary, I think that algorithm is obviously ideal, and thus any heuristic which leads to rejecting it should itself be rejected.

peterbonney 10y ago

Here's something I really dislike about all the coverage I've seen about these "risk assessment algorithms": There is absolutely no discussion of the magnitude of the distinctions between classifications. Is "low risk" supposed to be (say) 0.01% likelihood of committing another crime and "high risk" (say) 90%? Or is "low risk" (say) 1% vs. "high risk" of (say) 3%? Having worked on human some predictive modeling of "bad" human events (loan defaults) my gut says it's more like the latter than the former, because prediction of low-frequency human events is really hard, and, well, they're by definition infrequent. If that suspicion is right, then the signal-noise ratio is probably too poor to even consider using them in sentencing, and that's without considering the issues of bias in the training data, etc. But there is never enough detail provided (on either side of the debate) for me to make an informed assessment. It's just a lot of optimism on one side and pessimism on the other. I'd really love to see some concrete, testable claims without having to dive down a rabbit hole to find them.

liberal_arts 10y ago

consider the (fictional) possibility that an AI will be " actively measuring the populace's mental states, personalities, and the probability that individuals will commit crimes " https://en.wikipedia.org/wiki/Psycho-Pass https://en.wikipedia.org/wiki/Psycho-Pass AI may be worth the trade-off if violent crime can be almost eliminated. or consider (non-fiction): body-language/facial detection at airports; what if they actually start catching terrorists?

vegabook 10y ago
There is a school of thought that says some crime is necessary for a healthy, functioning society. Personally, while I would hate to be the victim of violent crime (obviously), I actually do agree that cities with very low crime levels are often stultifyingly uncharismatic. https://www.d.umn.edu/~bmork/2111/readings/durkheimrules.htm https://www.d.umn.edu/~bmork/2111/readings/durkheimrules.htm

conjectures 10y ago

What is Berk's model? How well does it do across different risk bands? What variables are fed into it in the states where it is used? How does prediction success vary across types of crimes, versus demographics within crime? This article treats ML like a magic wand, which it isn't. There's not enough information to make a judgement on whether the tools are performing well or not, or whether that performance, or lack of it, is based on discrimination. Where we do have information it is worrying: "Race isn’t an input in any of his systems, and he says his own research has shown his algorithms produce similar risk scores regardless of race." What?!? The appropriate approach would be to include race as a variable, fit the model, and then marginalise out race when providing risk predictions. Confounding is mentioned but the explanation of how it is dealt with, without doing the above isn't given - just a (most likely false) reassurance.

mc32 10y ago

The main question should be, like with autonomous vehicles, is does this system perform better than people (however you want to qualify that)? If so, it's better than what we have. Second, even if it's proven better (fewer false positives, less unduly biased results) it can be improved continuously. There is a danger in that people may not like the results because if we take this and diffuse it, has the potential to shape people's behavior in unintended ways (gaming), on the other hand this system has the potential for objectivity when identifying white collar crime, that is surfacing it better.

Digit-Al 10y ago

I find this really interesting. I think what most people seem to be missing is the wider social context. Think about this. If you exclude white collar financial crime, pre-meditated murder, and organised crime - most other crimes are committed by the socially disadvantaged. So, if the algorithm identifies an area where crime is more likely to be committed, instead of being narrow minded and just putting more police there to arrest people, why not instead try to institute programs to raise the socioeconomic status of the area? People are just concentrating on the crime aspect, but most crime is just a symptom of social inequality.

darpa_escapee 10y ago
The American thing to do is to cry "personal responsibility" and treat the symptoms with jail time, fines and a lengthened rap sheet. Suggesting that we treat the cause suggests we all have responsibilities as members of communities to ensure no one is in a place where crime might make sense despite the consequences.