5 ms·

Amplify this by 100, and welcome to Minority Report, where you get a arrested for having a "very high score". This is where the "mass surveillance" and "let's

Amplify this by 100, and welcome to Minority Report, where you get a arrested for having a "very high score". This is where the "mass surveillance" and "let's monitor everyone just in case they might do something bad" thinking inevitably ends up.

acqq 12y ago
> Amplify this by 100, and welcome to Minority Report, where you get a arrested for having a "very high score". Or maybe welcome to Oceania of 1984, having to deal with Thinkpol: http://en.wikipedia.org/wiki/Thoughtcrime http://en.wikipedia.org/wiki/Thoughtcrime "The Thought Police (thinkpol in Newspeak) are the secret police of the novel Nineteen Eighty-Four. It is their job to uncover and punish thoughtcrime."
- runiq 12y ago
  >thinkpol Yup, we don't have thinkpol yet (and hopefully we never will), but we do have a pretty damn good analogy to it: https://en.wikipedia.org/wiki/Predictive_policing https://en.wikipedia.org/wiki/Predictive_policing
sliverstorm 12y ago
... you get a arrested for having a "very high score". Considering this is PRECISELY how spam filtering works, it doesn't seem entirely irrational. Much like spam filtering, it would all come down to dialing in your filters and picking a good threshold. Hell, if the system was good enough, it could actually improve freedoms. We currently arrest many innocent people as part of the legal process (who are later exonerated). What if our "arrest filter" outperformed the current system, in terms of percentage of innocent people arrested? It doesn't have to be perfect to be better.
- pyre 12y ago
  The 'arrest filter' is only as good as the inputs. As it currently stands, I'm sure that things like "uses drugs recreationally," "is black," "is Muslim," and "is not Christian," would end up counting towards your 'arrest score.' And also because your arrest score is computed by a machine rather than a human, that will be used as an excuse to call it unimpeachable. E.g. "Machines can't be racist, so the arrest score going after lots of poor, Black men must mean that there's something to it."
  - sliverstorm 12y ago
    Only as good as the inputs, yes. But if it's a halfway decent filter, it will include machine learning, e.g. a Bayesian filter, and if "is Muslim" turns out to have low correspondence with actual criminal activity that input will quickly be deweighted. Or perhaps paired with other aspects- e.g., perhaps "is Muslim" is of no consequence and "Googles Jihad" is also of no consequence, but "is Muslim" && "Googles Jihad" gives you a point. Just as one example of the patterns a good filter could recognize. Machines can't be racist, so the arrest score going after lots of poor, Black men must mean that there's something to it. If a learning Bayesian filter targets a certain demographic, there probably IS something to it. That really would be amusing/pleasing, if all this work we've spent developing spam filters became the lead-up to an accurate, learning crime filter. Perhaps the fork to spamassassin will be known as crimeassassin?
    - me1010 12y ago
      Perhaps you should watch "The Thin Blue Line" a few hundred times... and then repost... I would be seriously concerned about a Bayesian filter being applied as the sole reason for arrests...
    - pyre 12y ago
      > If a learning Bayesian filter targets a certain demographic, there probably IS something to it. I'm pretty sure both Bayes and Laplace would not agree with the categorization of Bayesian probability as some sort of panacea for determining truth in criminal matters.
      - sliverstorm 12y ago
        Yes, it is just probability and not truth. But an arrest is often a "guess" with some degree of confidence, not absolute certainty. Thus is a "weight" or "probability" not perfectly applicable? Trials & convictions is a different matter, more suited to truth-seeking.
        pyre 12y ago
- tesq 12y ago
  The problem with such a filter is it will be perceived as inefficient and broken by those evaluating it if it denies an officer the authority to arrest someone he's already detained or determined to be worthy of arrest. It will be another system which grants permission at the whims of the department, one that absolves individual officers of blame, punishment and consequences for bias and abuse they'll continue to revel in.
  - sliverstorm 12y ago
    There is no reason such a filter need be an authoritative source. Arrest someone it didn't suggest, if you like. Choose not to arrest someone it did suggest. Human discretion would still be applied. But if it proved accurate, officers would trust it, and the population would get upset if it identified a criminal an officer did not arrest. No different than a virus or spam scanner, really. I trust my scanners. Sometimes they are wrong, and I know they are, and I bypass them. But I know they are right most of the time. You seem to have the notion though that officers do not arrest people to "catch the bad guy". It sounds like you are saying you believe most officers do not actually care if the arrestee is guilty (and will thus ignore the filter always), and merely arrest for kicks/pleasure/vengeance? I do not believe the majority are like that.
    - tesq 12y ago
      >You seem to have the notion though that officers do not arrest people to "catch the bad guy". It sounds like you are saying you believe most officers do not actually care if the arrestee is guilty (and will thus ignore the filter always), and merely arrest for kicks/pleasure/vengeance? I do not believe the majority are like that. You seem to have the notion that I believe what you think I believe. No, I do not believe that officers are arresting people for kicks. I do believe that human bias, power, career building/justification and 'gut feelings' about who might be a bad guy perpetuate unequal application of the law and abuse in a policed society. I do believe that if a filter sought to mitigate those very real problems it would be fought by the system it meant to augment. I am giving it the very forgiving assumption that those motivations and skews in perception aren't unknowingly built into the filter by its human creators, nor that biased information isn't fed into the filter. If either of those were true, it would be welcomed with open arms by police departments nationwide. I do believe that such a filter will be perceived as broken or inefficient if it doesn't confirm the officers' preconceived notions about who is a criminal or worthy of arrest. Unfortunately, I do not believe that the problem is 'the bad guys are getting away, and we need a system to find them'. It's rather the opposite, 'innocent people are having their lives interrupted and sometimes ruined because officers think they look suspicious for reasons and need to justify their paychecks; we need a system to mitigate that'. I do not think a system that blinks a little arrest light if a suspect is Muslim and uses Google has any place in society, no matter how many times you cry 'b-but machine learning!! Spam filters!! Bayes!!' Your filter would cement this period's problems into, in the eyes of the public, an infallible machine's instructions and will enable abuse without accountability, because you can always point to the machine and say you followed your best judgment.
- yry4345 12y ago
  > "it doesn't seem entirely irrational." (Yes it does...) > "It doesn't have to be perfect to be better." (Yes it does...) The problem used to be approached by presuming innocence (demanding perfection), rather than with a willingness to accept false positives (20 years ago spam filters weren't available as an analogy...). It is always possible to wrongfully judge someone, but it was never a valid or acceptable outcome ("It is better that ten guilty persons escape than that one innocent suffer" - Blackstone). We accept that spam filters give false positives (not to mention that one person's spam is another person's opportunity), so I think comparing the justice system to detecting spam is a mistake, and more over that a goal of "prevention" itself is a red herring. The goal of prevention encourages us to accept lower thresholds of guilt probability, and that is wrong. In other words, if prevention is an end, then it is worth deliberately (rather than accidentally) restricting innocent people on the basis of virtually any nonzero probability of guilt. 80% "guilty" by association (for using Tor for example), 45%, etc, would all be enough to justify legal action - and the thresholds would certainly depend on whoever is in power and has access to the database that week. This is a very different model than presuming innocence, and having not only a goal of 0 false-positives, but also providing satisfaction when the justice system is in error. I think today we are mostly talking around the fact that a crime has to have been committed in order for it to deserve to be punished, and that, for that reason, prevention cannot be a valid goal in itself (but it's nice when it happens). Rationalizing surveillance as a tool to "prevent" rather than to justly punish wrongdoers (which centralized surveillance does not do because it is centrally operated, due to the conflict of interest; everyone owning a camcorder on the other hand...) implies that the central database needs to go IMHO (and that individuals need to be empowered instead).
  - sliverstorm 12y ago
    Hold on there friend. I was not suggesting we replace the judicial system with a filter. Rather arrests. I.e., make the arrest based on the filter, then run the trial in the same old jury-of-your-peers. Convictions should be false-positive-free. But our system would not work if arrests also needed to be 100% false-positive-free. I'm also not advocating punishment for crimes that have not yet been committed. Rather, think of it as looking for flags for crimes that have already been committed or are in progress. For example, there are all sorts of small flags thrown by embezzlement or salami-slicing that, put together, identify the operation.
- PavlovsCat 12y ago
  Then why does the system not look out for corrupt politicians, or black military budgets? Because the filters are not tuned well enough yet, as if that will ever be an objective? I'd say it's because it's not a spam filter that filters for spam.. more like a spam filter that filters out the spam of the competition, lets yours through, and kills emails warning about this. Call me paranoid, but until the big guns are primarily used to catch the big villains, this is what I see.
- sqrt17 12y ago
  Spam filtering works using actual data that someone once used to train it - if we are taking this at face value, it means that the NSA is not just targeting people who might be terrorists and collects data on other people as a side effect, but that it actively targets people with no ties to terrorism whatsoever, for reasons that both US and international public might find unsavory if they found out about them. Spam filtering also works in a very different context - the spam-to-nonspam ratio is something like 90% spam and 10% nonspam, which means that there is lots and lots of spam to filter out; if an important email slips through, people are bound to notice and either adjust their spam filter or do something about it. In the other setting, you have 99.99% or more of people who have nothing to do with terrorism or criminal activities, and maybe one or two dozen (among tens of millions) who you are actually targeting. First thing, erroneously targeting a substantial chunk of your non-interesting population ties up resources - you're spending your time investigating people who are not terrorists - but since it's difficult anyways, at least you seem like you're doing something with all the money you receive, and nevermind if some of the data is used for industrial espionnage or hunting people that only poultry farmers and fracking magnates would call terrorists. And if you miss one of the two dozen other people, well, they won't do anything harmful this year or the next because they also have to fear regular law enforcement, and when they do it'll be in a moment that's probably suitable for you to ask for more money. tl;dr: Because we don't have a large sample of actual terrorists on hand, it's hard to evaluate activities like the NSA's, which would however be desirable since we're giving large chunks of money to them that could be fruitfully used in making everyone safer if used to fight actual crime and not some fuzzy notion of terrorism.
lucb1e 12y ago
> you get a arrested for having a "very high score". I was thinking more in terms of automated drone strikes, but yeah for the facade of democracy arrests may be the way to go for now.
- lallysingh 12y ago
  Too expensive.
  - judk 12y ago
    Drones get more cost effective as the kill frequency increases. Economy of scale
    - junto 12y ago
      You can make it cheaper still by waiting until more than one target are in the same strike zone. This of course only matters if you really care about the money. I assume that the large black budgets of the CIA and NSA are not really a concern to those that matter. Ideally you spend more every year. That's the usual policy of government agencies, otherwise the following year you get a budget cut.
      - lucb1e 12y ago
        "DronEfficiency (YC2034): better protection of our country and spending less tax money due to improved algorithms!" (Disclaimer: I'm not insinuating that YC supports this)
        TeMPOraL 12y ago
        "Luc Doevil from DronEfficiency: We've collected 10M of initial funding; that's two more rounds and we'll be able to buy our first Hellfire for testing."

acqq 12y ago

> Amplify this by 100, and welcome to Minority Report, where you get a arrested for having a "very high score". Or maybe welcome to Oceania of 1984, having to deal with Thinkpol: http://en.wikipedia.org/wiki/Thoughtcrime http://en.wikipedia.org/wiki/Thoughtcrime "The Thought Police (thinkpol in Newspeak) are the secret police of the novel Nineteen Eighty-Four. It is their job to uncover and punish thoughtcrime."

runiq 12y ago
>thinkpol Yup, we don't have thinkpol yet (and hopefully we never will), but we do have a pretty damn good analogy to it: https://en.wikipedia.org/wiki/Predictive_policing https://en.wikipedia.org/wiki/Predictive_policing

sliverstorm 12y ago

... you get a arrested for having a "very high score". Considering this is PRECISELY how spam filtering works, it doesn't seem entirely irrational. Much like spam filtering, it would all come down to dialing in your filters and picking a good threshold. Hell, if the system was good enough, it could actually improve freedoms. We currently arrest many innocent people as part of the legal process (who are later exonerated). What if our "arrest filter" outperformed the current system, in terms of percentage of innocent people arrested? It doesn't have to be perfect to be better.

pyre 12y ago
The 'arrest filter' is only as good as the inputs. As it currently stands, I'm sure that things like "uses drugs recreationally," "is black," "is Muslim," and "is not Christian," would end up counting towards your 'arrest score.' And also because your arrest score is computed by a machine rather than a human, that will be used as an excuse to call it unimpeachable. E.g. "Machines can't be racist, so the arrest score going after lots of poor, Black men must mean that there's something to it."
- sliverstorm 12y ago
  Only as good as the inputs, yes. But if it's a halfway decent filter, it will include machine learning, e.g. a Bayesian filter, and if "is Muslim" turns out to have low correspondence with actual criminal activity that input will quickly be deweighted. Or perhaps paired with other aspects- e.g., perhaps "is Muslim" is of no consequence and "Googles Jihad" is also of no consequence, but "is Muslim" && "Googles Jihad" gives you a point. Just as one example of the patterns a good filter could recognize. Machines can't be racist, so the arrest score going after lots of poor, Black men must mean that there's something to it. If a learning Bayesian filter targets a certain demographic, there probably IS something to it. That really would be amusing/pleasing, if all this work we've spent developing spam filters became the lead-up to an accurate, learning crime filter. Perhaps the fork to spamassassin will be known as crimeassassin?
  - me1010 12y ago
    Perhaps you should watch "The Thin Blue Line" a few hundred times... and then repost... I would be seriously concerned about a Bayesian filter being applied as the sole reason for arrests...
  - pyre 12y ago
    > If a learning Bayesian filter targets a certain demographic, there probably IS something to it. I'm pretty sure both Bayes and Laplace would not agree with the categorization of Bayesian probability as some sort of panacea for determining truth in criminal matters.
    - sliverstorm 12y ago
      Yes, it is just probability and not truth. But an arrest is often a "guess" with some degree of confidence, not absolute certainty. Thus is a "weight" or "probability" not perfectly applicable? Trials & convictions is a different matter, more suited to truth-seeking.
      - pyre 12y ago
tesq 12y ago
The problem with such a filter is it will be perceived as inefficient and broken by those evaluating it if it denies an officer the authority to arrest someone he's already detained or determined to be worthy of arrest. It will be another system which grants permission at the whims of the department, one that absolves individual officers of blame, punishment and consequences for bias and abuse they'll continue to revel in.
- sliverstorm 12y ago
  There is no reason such a filter need be an authoritative source. Arrest someone it didn't suggest, if you like. Choose not to arrest someone it did suggest. Human discretion would still be applied. But if it proved accurate, officers would trust it, and the population would get upset if it identified a criminal an officer did not arrest. No different than a virus or spam scanner, really. I trust my scanners. Sometimes they are wrong, and I know they are, and I bypass them. But I know they are right most of the time. You seem to have the notion though that officers do not arrest people to "catch the bad guy". It sounds like you are saying you believe most officers do not actually care if the arrestee is guilty (and will thus ignore the filter always), and merely arrest for kicks/pleasure/vengeance? I do not believe the majority are like that.
  - tesq 12y ago
    >You seem to have the notion though that officers do not arrest people to "catch the bad guy". It sounds like you are saying you believe most officers do not actually care if the arrestee is guilty (and will thus ignore the filter always), and merely arrest for kicks/pleasure/vengeance? I do not believe the majority are like that. You seem to have the notion that I believe what you think I believe. No, I do not believe that officers are arresting people for kicks. I do believe that human bias, power, career building/justification and 'gut feelings' about who might be a bad guy perpetuate unequal application of the law and abuse in a policed society. I do believe that if a filter sought to mitigate those very real problems it would be fought by the system it meant to augment. I am giving it the very forgiving assumption that those motivations and skews in perception aren't unknowingly built into the filter by its human creators, nor that biased information isn't fed into the filter. If either of those were true, it would be welcomed with open arms by police departments nationwide. I do believe that such a filter will be perceived as broken or inefficient if it doesn't confirm the officers' preconceived notions about who is a criminal or worthy of arrest. Unfortunately, I do not believe that the problem is 'the bad guys are getting away, and we need a system to find them'. It's rather the opposite, 'innocent people are having their lives interrupted and sometimes ruined because officers think they look suspicious for reasons and need to justify their paychecks; we need a system to mitigate that'. I do not think a system that blinks a little arrest light if a suspect is Muslim and uses Google has any place in society, no matter how many times you cry 'b-but machine learning!! Spam filters!! Bayes!!' Your filter would cement this period's problems into, in the eyes of the public, an infallible machine's instructions and will enable abuse without accountability, because you can always point to the machine and say you followed your best judgment.
yry4345 12y ago
> "it doesn't seem entirely irrational." (Yes it does...) > "It doesn't have to be perfect to be better." (Yes it does...) The problem used to be approached by presuming innocence (demanding perfection), rather than with a willingness to accept false positives (20 years ago spam filters weren't available as an analogy...). It is always possible to wrongfully judge someone, but it was never a valid or acceptable outcome ("It is better that ten guilty persons escape than that one innocent suffer" - Blackstone). We accept that spam filters give false positives (not to mention that one person's spam is another person's opportunity), so I think comparing the justice system to detecting spam is a mistake, and more over that a goal of "prevention" itself is a red herring. The goal of prevention encourages us to accept lower thresholds of guilt probability, and that is wrong. In other words, if prevention is an end, then it is worth deliberately (rather than accidentally) restricting innocent people on the basis of virtually any nonzero probability of guilt. 80% "guilty" by association (for using Tor for example), 45%, etc, would all be enough to justify legal action - and the thresholds would certainly depend on whoever is in power and has access to the database that week. This is a very different model than presuming innocence, and having not only a goal of 0 false-positives, but also providing satisfaction when the justice system is in error. I think today we are mostly talking around the fact that a crime has to have been committed in order for it to deserve to be punished, and that, for that reason, prevention cannot be a valid goal in itself (but it's nice when it happens). Rationalizing surveillance as a tool to "prevent" rather than to justly punish wrongdoers (which centralized surveillance does not do because it is centrally operated, due to the conflict of interest; everyone owning a camcorder on the other hand...) implies that the central database needs to go IMHO (and that individuals need to be empowered instead).
- sliverstorm 12y ago
  Hold on there friend. I was not suggesting we replace the judicial system with a filter. Rather arrests. I.e., make the arrest based on the filter, then run the trial in the same old jury-of-your-peers. Convictions should be false-positive-free. But our system would not work if arrests also needed to be 100% false-positive-free. I'm also not advocating punishment for crimes that have not yet been committed. Rather, think of it as looking for flags for crimes that have already been committed or are in progress. For example, there are all sorts of small flags thrown by embezzlement or salami-slicing that, put together, identify the operation.
PavlovsCat 12y ago
Then why does the system not look out for corrupt politicians, or black military budgets? Because the filters are not tuned well enough yet, as if that will ever be an objective? I'd say it's because it's not a spam filter that filters for spam.. more like a spam filter that filters out the spam of the competition, lets yours through, and kills emails warning about this. Call me paranoid, but until the big guns are primarily used to catch the big villains, this is what I see.
sqrt17 12y ago
Spam filtering works using actual data that someone once used to train it - if we are taking this at face value, it means that the NSA is not just targeting people who might be terrorists and collects data on other people as a side effect, but that it actively targets people with no ties to terrorism whatsoever, for reasons that both US and international public might find unsavory if they found out about them. Spam filtering also works in a very different context - the spam-to-nonspam ratio is something like 90% spam and 10% nonspam, which means that there is lots and lots of spam to filter out; if an important email slips through, people are bound to notice and either adjust their spam filter or do something about it. In the other setting, you have 99.99% or more of people who have nothing to do with terrorism or criminal activities, and maybe one or two dozen (among tens of millions) who you are actually targeting. First thing, erroneously targeting a substantial chunk of your non-interesting population ties up resources - you're spending your time investigating people who are not terrorists - but since it's difficult anyways, at least you seem like you're doing something with all the money you receive, and nevermind if some of the data is used for industrial espionnage or hunting people that only poultry farmers and fracking magnates would call terrorists. And if you miss one of the two dozen other people, well, they won't do anything harmful this year or the next because they also have to fear regular law enforcement, and when they do it'll be in a moment that's probably suitable for you to ask for more money. tl;dr: Because we don't have a large sample of actual terrorists on hand, it's hard to evaluate activities like the NSA's, which would however be desirable since we're giving large chunks of money to them that could be fruitfully used in making everyone safer if used to fight actual crime and not some fuzzy notion of terrorism.

lucb1e 12y ago

> you get a arrested for having a "very high score". I was thinking more in terms of automated drone strikes, but yeah for the facade of democracy arrests may be the way to go for now.

lallysingh 12y ago
Too expensive.
- judk 12y ago
  Drones get more cost effective as the kill frequency increases. Economy of scale
  - junto 12y ago
    You can make it cheaper still by waiting until more than one target are in the same strike zone. This of course only matters if you really care about the money. I assume that the large black budgets of the CIA and NSA are not really a concern to those that matter. Ideally you spend more every year. That's the usual policy of government agencies, otherwise the following year you get a budget cut.
    - lucb1e 12y ago
      "DronEfficiency (YC2034): better protection of our country and spending less tax money due to improved algorithms!" (Disclaimer: I'm not insinuating that YC supports this)
      - TeMPOraL 12y ago
        "Luc Doevil from DronEfficiency: We've collected 10M of initial funding; that's two more rounds and we'll be able to buy our first Hellfire for testing."