12 ms·
Ask HN: Any open source code/materials on predicting future crimes based on data?
- netrus 9y agoHave you checked kaggle for relevant datasets?
- thisisit 9y agoAre you looking for tools or data?
- febin 9y agoI am looking for tools or algorithms.
- WhitneyLand 9y agoDo you know about the journalist who spent years obsessing about this and supposedly had some predictive success relating to serial killers? If I recall it was kind of a lone wolf effort, so I don’t know the rigor of his techniques, howver you never know if he might want to share results or collaborate. Don’t have a link handy, but that should be enough info to google if you’re interested.
- noisecanceling 9y agoI think you're referring to this article about Thomas Hargrove that was in Bloomberg in February: https://www.bloomberg.com/news/features/2017-02-08/serial-killers-should-fear-this-algorithm https://www.bloomberg.com/news/features/2017-02-08/serial-ki... He's the founder of the Murder Accountability Project: http://www.murderdata.org http://www.murderdata.org
- jjoonathan 9y agoAsk HN: Any open source code/materials on predicting good fall guys based on data?
- zebrafish 9y agoBelieve I heard about a project a UW student did predicting crime in San Francisco based on volume of vulgar tweets in a given area. Not sure if it's on github anywhere but you can always start with that idea. Nothing about specifics of the crimes, just where a high volume of them would be located.
- amigoingtodie 9y agoYou need to watch Person of Interest and Minority Report.
- febin 9y agoI watched POI. I also heard cops using ML to predict crime locations. I am trying to find and reuse some existing code for a different project.
- platz 9y agoDon't be surprised if those predictions are heavily biased against minorities and poor people. Do you care if they do? It's a similar problem to using ML to give people credit scores. If the training data includes a lot of minorities and poor people breaking laws / delinquent payments, then your ML will simply key on race/economic status as a predictor. So you've built a system that simply targets those groups. But you might object and say that this race/economic status targeting gives the highest accuracy! It was only learned in the training data, after all. You can make a great classifier that is extremely unfair. So you have to realize there is a conflict here between accuracy and fairness. This means there is a conflict between observational data (training), and using that data to produce decisions/outcomes. If you make decisions/outcomes that reinforce the training data, you do not give racial groups/low economic status people a chance to improve their lives. That is extremely inhuman, predatory, and unfair.
- febin 9y agoAll I want to predict is time periods/locations which are vulnerable. Nothing more than that.
- jjoonathan 9y agoRacism is morally wrong but not mathematically wrong. P(criminal|black) > P(criminal), but if you observe that someone has black skin and treat them poorly because of it, you've done a bad thing. It doesn't matter that you were just following Bayesian reasoning because you're still hurting someone on the basis of something they can't control. Lady Justice doesn't wear a blindfold as a fashion accessory. Discarding information is a key factor in nearly every established system of justice / morality. Refusing to do so (i.e. "just" running a ML algorithm) places you directly at odds with society's hard-earned best practices.
- thedrake 9y agoA lot of good work by Cynthia Rudin http://online.liebertpub.com/doi/pdf/10.1089/big.2014.0021 http://online.liebertpub.com/doi/pdf/10.1089/big.2014.0021 and her tools are open sourced (her papers https://users.cs.duke.edu/~cynthia/papers.html https://users.cs.duke.edu/~cynthia/papers.html and tools https://users.cs.duke.edu/~cynthia/code.html https://users.cs.duke.edu/~cynthia/code.html)
- michaelmcmillan 9y agoI am currently writing my master thesis on predictive policing using machine learning. Working with local police in Norway. Got a bunch of papers and articles you might find interesting. Hit me up: michaedm@stud.ntnu.no
- febin 9y agoThanks, Just shot an email.
- godelski 9y agoI'd be interested to know how you filter the garbage data? (By questions here I know others are interested, so public response would be great) I know people have done these types of studies before but found that they easily became bias, and thus there is a wariness of using it (like the judge AI who was more likely to convict black people). I'm not sure how it is in Norway, but I don't expect it to be much different from America, where there are places which are disproportionately convicted of crimes, where other areas such crimes are seen as infractions. This is really going to mess with the data and perpetuate the bad system.
- 132121321qewdqw 9y agoI hope you will be an innocent target one day.
- crabl 9y agohttps://github.com/kandluis/crime-prediction https://github.com/kandluis/crime-prediction is a good place to start
- Asdfbla 9y agoIn any case, you shouldn't neglect the subtle but important sources of bias those pre-crime models can have. Here's an interesting talk about it: https://www.youtube.com/watch?v=MfThopD7L1Y https://www.youtube.com/watch?v=MfThopD7L1Y Basically, one instance of bias is the fact that many crime-prediction models are trained on police data, which means they will predict crime in places more often targeted by the police anyway. Then the model predictions even amplify that effect, since more training data may be generated from the places now more often policed, etc. There's lots of resources out there on AI fairness these days. I think everyone who tries stuff like crime prediction should read up on that topic.
- chronic193 9y agoWe must ensure police are continuously allocated to crime-ridden areas, with or without AI, fair or not. If you are hinting at sending fewer police to black neighborhoods, you can bet that will never happen.
- darpa_escapee 9y agoI lived in a town that neighbored another where police presence was about 4 fold compared to the former. You would be stopped while walking down the street in the neighboring town, meanwhile I could look a out my window and watch a few drug deals take place in an afternoon. The difference? The neighboring town had a couple decades long reputation of being a hotbed for crime. That reputation would be cemented in an AI system that was trained off of police data.
- pfarnsworth 9y agoYou must be talking from your white privilege. It's a common known fact that crime-ridden, mostly minority neighborhoods have a complete dearth of services such as police, firefighters, and ambulance. Even back in the 80s, Public Enemy released a song called "911 is a Joke" because if you called 911 from a black neighborhood, they won't respond. It's routinely known that cops will rather stay in rich neighborhoods and let the poorer neighborhoods fester. This happened during the LA riots where Koreantown burned because most of the cops went to Beverly Hill, etc, to protect the rich people. I saw this first hand in Detroit about 8 years ago. I was driving through a bad neighborhood in Detroit with my friend, and we entered the neighborhood of Grosse Point, a rich area. Cops followed us until we left, which is their way of saying "you don't belong here". Meanwhile, among the burnt down houses and broken windows of Detroit proper, you couldn't see a single cop.
- bayesbiol 9y agoAny such system is/would be potentially very dangerous. Crime data is not the same thing as crime. Populations that are over-policed are be disproportionately represented in any such data set, leading to higher prediction of crime, leading in turn more over-policing (feedback loop). I implore anyone attempting to build such a system to consider the serious issue of machine bias and it's implications in the real world. See this tutorial given at this years NIPS machine learning conference: http://mrtz.org/nips17/#/ http://mrtz.org/nips17/#/
- deleted 9y ago[deleted]
- jensv 9y agoPotential dangers of such a film are highlighted in the film Minority Report. https://en.wikipedia.org/wiki/Minority_Report_(film) https://en.wikipedia.org/wiki/Minority_Report_(film)
- chiefalchemist 9y agoFwiw there's some discussion of this in the book Everybody Lies. Look into that. Perhaps follow up with the author. His name escapes me atm.
- jeffmould 9y agoAre you looking for predicting future crimes in an area (i.e. city, neighborhood, state, etc...) or predicting whether an individual will commit future crimes?
- SamReidHughes 9y agoCareful! Your crime predictor might unfairly conclude that men are more likely to commit crimes than women.
- jhiska 9y agoOP wants an open sourced, data-based statistical model of where crime might occur (methodological flaws and all), and not an unasked-for politicized preaching about the supposed virtues of a subset of people over another subset of people.
- platz 9y agoAs long as the methodological flaws are someone else's problem (the user/consumer), we don't need to take responsibility for building the tools that facilitate those methodological flaws, right?
- jhiska 9y agoI agree. I don't want the police using biased tools either. OP has stated elsewhere in the comments that the reason they're interested in the tech is for testing it in some other unrelated area.
- platz 9y agoI don't see the OP making a claim like that. In any case, OP did not specify what the purpose of the tool was/is, and doesn't seem to be willing to explain the purpose of the the tool is.
- michaelmcmillan 9y agoReductio ad absurdum of the bias argument (-:
- dahart 9y agoPeople might conclude the same thing, since more crimes are committed by males than females. Are you saying it's unfair because an innocent male could become a suspect by virtue of only being male? Or for a more subtle reason? Statistics have been consistent in reporting that men commit more criminal acts than women.[1][2] Self-reported delinquent acts are also higher for men than women across many different actions.[3] Burton, et al. (1998) found that low levels of self control are associated with criminal activity.[4] Many professionals have offered explanations for this sex difference. Some differing explanations include men's evolutionary tendency toward risk and violent behavior, sex differences in activity, social support, and gender inequality. https://en.wikipedia.org/wiki/Sex_differences_in_crime https://en.wikipedia.org/wiki/Sex_differences_in_crime
- paulie_a 9y agoFor a source of data: https://data.cityofchicago.org/ https://data.cityofchicago.org/ And in the case of crime, chicago should be a pretty good dataset.
- partycoder 9y agohttps://en.wikipedia.org/wiki/Predictive_policing https://en.wikipedia.org/wiki/Predictive_policing The British series "The Code" speaks a little bit about it in ep 3: https://en.wikipedia.org/wiki/The_Code_(2011_TV_series)#Stage_3:_The_Finale https://en.wikipedia.org/wiki/The_Code_(2011_TV_series)#Stag...
- minimaxir 9y agoInspired by a Kaggle competition (https://www.kaggle.com/c/sf-crime https://www.kaggle.com/c/sf-crime), one of my older blog posts involved predicting the type of arrest in San Francisco (given that an arrest occurred) using data such as location and timing and the relatively new LightGBM machine learning algorithm: http://minimaxir.com/2017/02/predicting-arrests/ http://minimaxir.com/2017/02/predicting-arrests/ The code is open-sourced in an R Notebook: http://minimaxir.com/notebooks/predicting-arrests/ http://minimaxir.com/notebooks/predicting-arrests/ The model performance isn't great enough to usher in precrime, even in the best case. There are likely better approaches nowadays. (e.g. since the location data is spatial, a convolutional neural network might work better.)
- lwansbrough 9y agoThere are much better ways to solve crime than to double down on enforcement that is already happening, which is likely all your model will tell you. “Police the neighbourhoods where people are poor” wow, thanks ML! Palantir already does all this on a massive scale for the US govt. Want to affect future crime in a positive way? Solve the problems that contribute to it. Not that you asked.
- USNetizen 9y agoThis is an area that was explored some years ago, but ultimately determined to have civil rights pitfalls. Crime reporting is only as good (or biased) as the humans that report and input the crime data. Therefore, crime "training" data for AI systems can be very biased and it might only magnify those biases more so using AI - a sort of self-perpetuating negative feedback loop. Having worked in law enforcement at various levels (state and federal) in a prior professional life, I can attest to the differences in what gets reported and how based upon who was working or supervising and where they were assigned. Humans are simply not reliable reporters for this kind of data. No matter how hard we try to make the reports plain and standardized our biases, one way or another, will always seep in.
- YurtleTheTurtle 9y agohttps://www.propublica.org/article/machine-bias-risk-assessments-in-criminal-sentencing https://www.propublica.org/article/machine-bias-risk-assessm... Food for thought on how incredibly biased these effort can be.
- ryanmaynard 9y agoThere is a project[1] + whitepaper[2] on projecting the likelihood of future white collar crimes written by Sam Lavigne, Francis Tseng, and Brian Clifton. [1] https://thenewinquiry.com/white-collar-crime-risk-zones/ https://thenewinquiry.com/white-collar-crime-risk-zones/ [2] https://whitecollar.thenewinquiry.com/static/whitepaper.pdf https://whitecollar.thenewinquiry.com/static/whitepaper.pdf
- 0xdeadbeefbabe 9y agoThe Poisson distribution!
- 0xdeadbeefbabe 9y agoWhy don't you base it on Law data?
- PaulHoule 9y agohttps://en.wikipedia.org/wiki/CompStat https://en.wikipedia.org/wiki/CompStat
- tobylane 9y agoThere's a British tv presenter and scientist called Hannah Fry who has published in this area, including a talk in Germany (received just like many comments on this page), some Numberphile videos and BBC documentaries in other areas of data science.