2 ms·

There are lots of good arguments against the use of these technologies, but disproportionate effect on marginalized communities may be ironically a less powerfu

There are lots of good arguments against the use of these technologies, but disproportionate effect on marginalized communities may be ironically a less powerful one to people who believe in them, because the intent is to find indicators independent of the experiences of individual enforcers. Anyone with experience in marginalized communities knows it is a good bet to predict the self-sustaining effects of marginalization, which are violence, drugs, mental illness and exploitation by people there to take advantage of it. Doesn't take rocket science to predict where that's going to happen and to be there when it does. The real solutions are way upstream of a police encounter, where these predictive models would have a way more viable impact in elementary and high schools where people can be nudged and diverted from police encounters in a more subtle and constructive way. But even then the tech still reinforces and systematize the aggregate bias (or experience) of the data analysts, with the real danger it just becomes a justification, the same way statistics are always used, which is the way a drunk uses a lamp post - for support and not illumination. Whether it's teachers using it to mark and isolate kids from positive peers, or police using it to divert them from a bigger incident, it's the same underlying belief in social engineering that problematizes "other" people and tries to "fix," them. We should begin by recognizing that these ML systems are weapons, not "solutions," and regulate the use of them in the context of weapons against an enemy other, then ensure someone holds personal individual accountability for that decision, including the social and political aspects of it. The second order effect is that more-focused enforcement methods will necessarily mean more confrontations with people prone to violence, which creates a selection bias toward violent encounters with police, which makes these systems all the more dystopian.

3pt14159 6y ago
Eh, I'm not sure I agree. I was contracted by The Government of Canada to help them with a recommendation engine that was built to help people. The problem? Some criminals had reverse engineered the algorithm and were using it for really, really horrible ends. What would you have had me do? Not reverse engineer the criminal's reverse engineering? Did I help GC make a "predictive policing" app because they could now alert the RCMP to these bastards? It's a balance. We need privacy, but we also need security. My take on it is that we should start with good faith and only surveil where it makes sense (embassies, airports, etc) unless there is some sort of signal (in the ML sense) of bad faith. From that point on it should go through the crown's lawyers (like it does in Canada), but ultimately we need to stop murderers, human traffickers, and other people that make the world worse. I agree that some ML systems are weapons, but I don't think they all are. Some are benign. Some start off off as benign then get turned into dual use, like the one I worked on. But not all weapons are necessarily bad. We need a government that can actually protect us. There is real evil in the world.
- motohagiography 6y ago
  I'm saying the tech is a weapon you use to target something, ideally for good reasons, but it's still a weapon and it's the targeting decision that needs accountability. Knowing what I do about tech, I don't think privacy and security are a trade off or zero sum. I do think privacy and surveillance are a trade off, and surveillance is what you have when you don't have security or trustworthy systems. I agree there is actual evil in the world, and it's particularly where you don't expect it. The law isn't designed to fight evil. As a society, a better cultural understanding of evil would do more to improve our ability to deal with it than blunt technical weapons to surveil it.
  - 3pt14159 6y ago
    I agree that the law isn't designed to fight evil, but many laws are designed to fight evil. It's a subtle distinction, but one worth pointing out. Surveillance is part of sense-making for law enforcement and even though I don't think that security and privacy are always at odds, and there are definitely cases where they are not zero-sum, when dealing with persons of interest in criminal investigations it tends to be the case that degrading a suspects privacy improves societies security. But it should be done right. Warrants, probable cause, etc.

3pt14159 6y ago

Eh, I'm not sure I agree. I was contracted by The Government of Canada to help them with a recommendation engine that was built to help people. The problem? Some criminals had reverse engineered the algorithm and were using it for really, really horrible ends. What would you have had me do? Not reverse engineer the criminal's reverse engineering? Did I help GC make a "predictive policing" app because they could now alert the RCMP to these bastards? It's a balance. We need privacy, but we also need security. My take on it is that we should start with good faith and only surveil where it makes sense (embassies, airports, etc) unless there is some sort of signal (in the ML sense) of bad faith. From that point on it should go through the crown's lawyers (like it does in Canada), but ultimately we need to stop murderers, human traffickers, and other people that make the world worse. I agree that some ML systems are weapons, but I don't think they all are. Some are benign. Some start off off as benign then get turned into dual use, like the one I worked on. But not all weapons are necessarily bad. We need a government that can actually protect us. There is real evil in the world.

motohagiography 6y ago
I'm saying the tech is a weapon you use to target something, ideally for good reasons, but it's still a weapon and it's the targeting decision that needs accountability. Knowing what I do about tech, I don't think privacy and security are a trade off or zero sum. I do think privacy and surveillance are a trade off, and surveillance is what you have when you don't have security or trustworthy systems. I agree there is actual evil in the world, and it's particularly where you don't expect it. The law isn't designed to fight evil. As a society, a better cultural understanding of evil would do more to improve our ability to deal with it than blunt technical weapons to surveil it.
- 3pt14159 6y ago
  I agree that the law isn't designed to fight evil, but many laws are designed to fight evil. It's a subtle distinction, but one worth pointing out. Surveillance is part of sense-making for law enforcement and even though I don't think that security and privacy are always at odds, and there are definitely cases where they are not zero-sum, when dealing with persons of interest in criminal investigations it tends to be the case that degrading a suspects privacy improves societies security. But it should be done right. Warrants, probable cause, etc.