11 ms·
They literally named it Skynet. They have an evil sense of humor. Actually using machine learning to detect terrorists isn't a terrible idea. But you are going
by Houshalter 11y ago
They literally named it Skynet. They have an evil sense of humor.
Actually using machine learning to detect terrorists isn't a terrible idea. But you are going to get an error rate, and probably a high one in the noisy real world. Maybe only 50% of the people you detect are actually terrorists. Maybe it's even worse than that. We can't even test it because there is no validation set and unreliable labels.
The reasonable thing to do with that information, would be to surveil them further, search their house, or arrest them. Not assassinate them without a trial.
And the more I read the details, the more alarmed I am. The 50% figure I used above may have been way too high. The base rate of terrorists way too low and they have very little data to begin with.
- milesftl 11y agodid you post this on the corresponding reddit thread as well? if not then someone literally copy pasted your comment, which I have been noting a lot more lately on there.
- Houshalter 11y agoYeah that was me.
- teekert 11y agoGiven that there are very little real terrorists in the data, any sensitive detection mechanism will need insane specificity or it will give you an enormous amount of false positives.
- pwg 11y ago> But you are going to get an error rate, and probably a high one in the noisy real world. ... The reasonable thing to do with that information, would be to surveil them further It is even worse than that. Due to the base rate fallacy (https://en.wikipedia.org/wiki/Base_rate_fallacy https://en.wikipedia.org/wiki/Base_rate_fallacy) the 'machine' is going to have a huge false positive rate, so without that additional surveillance, most will not be guilty of what they are accused. But that surveillance takes money and resources, just the thing the 'machine' system is designed to save...
- moyix 11y agoYes. I feel like anyone considering such a system should be forced to read "The Base-Rate Fallacy and its Implications for the Difficulty of Intrusion Detection": http://www.raid-symposium.org/raid99/PAPERS/Axelsson.pdf http://www.raid-symposium.org/raid99/PAPERS/Axelsson.pdf
- ehudla 11y agoPeople interested in these issues should read the position paper written by Jeff Jonas and Jim Harper in 2006, "Effective Counterterrorism and the Limited Role of Predictive Data Mining". http://www.cato.org/publications/policy-analysis/effective-counterterrorism-limited-role-predictive-data-mining http://www.cato.org/publications/policy-analysis/effective-c...
- flashman 11y agoIf you are going to put surveillance on potential terrorists, you should also be putting surveillance on people not suspected of being terrorists, so you can judge the rate at which your surveillance personnel falsely accuse people being terrorists based on having been selected for surveillance. Single or double blind that process.
- anc84 11y agoThey could put the surveillance on themselves for that good people control group. Oh wait...
- mratzloff 11y agoThat would be true if you were genuinely interested in avoiding innocent deaths. If you placed very little or no value on non-Western lives, then there would be no incentive to take that kind of extra care.
- pavelrub 11y ago> The reasonable thing to do with that information, would be to surveil them further, search their house, or arrest them. Not assassinate them without a trial. Which seems to be exactly the case. Ahmad Zaidan, for example, wasn't assassinated. In fact even if the algorithm was 100% correct - it makes no sense to assassinate people solely based on whether they are terrorists or not. A lot more can be gained from surveilling some of them, etc'. So there is really nothing at all to suggest that those algorithms alone are used to create kill lists.
- qeabc 11y ago> So there is really nothing at all to suggest that those algorithms alone are used to create kill lists. The first line of the article actually suggests this: 'In 2014, the former director of both the CIA and NSA proclaimed that "we kill people based on metadata."' Also, even if something does not make sense does not mean that it does not seem to make sense to a particular person or organization.
- mike_hearn 11y agoYeah but nothing going on in the national security state makes much sense. The CIA drone program needs targets. It devours them. When they ran out of real intelligence they started killing people they didn't even know the name of because they "looked like terrorists", check out signature strikes. Those drones are going to be tasked no matter what. If the quality of the input intelligence is crap, well, good thing it's all top secret right!
- Sir_Substance 11y ago>The reasonable thing to do with that information, would be to surveil them further, search their house, or arrest them. That is in no way a reasonable thing to do if you're only 50% confident in your results. A reasonable thing to do would be scrap the whole system and make something more reliable than a coin flip.
- pmichaud 11y agoCharacterizing it as a coin flip is unfair I think. If this system can promote individuals to attention out of millions of targets, telling us that this person out of all the millions has a 50/50 chance of being a terrorist, then that is hugely valuable information that deserves further inquiry. Now, of course we don't know how good the rate is or what other evidence they already have or what exactly a terrorist is for that matter. But in principle, you'd be throwing out a huge amount of evidence if got rid of a system that changed a target's chance from 1 in a million to 1 in 2.
- Ntrails 11y ago>We can't even test it because there is no validation set and unreliable labels. You can't test it, the NSA surely can and almost certainly can and did.
- nl 11y agoThe reasonable thing to do with that information, would be to surveil them further, search their house, or arrest them. Not assassinate them without a trial. This appears to be what they do? The National Security Agency reportedly tracked phone calls between the courier Abu Ahmed al-Kuwaiti's relatives in the Persian Gulf to all numbers in Pakistan, and NSA surveillance eventually tracked Abu Ahmed al-Kuwaiti's location in Pakistan via one such phone call.... In August 2010 they tracked al-Kuwaiti as he drove from Peshawar to a residence in Abbottabad – and as analysts inventoried the compound's striking security features they became convinced that it housed a high-level al-Qaeda figure... In September 2010, the CIA concluded that the compound was "custom built to hide someone of significance" and that it was very likely that Osama bin Laden was residing there[1] As it turned out, that was where bin Laden was killed, so that process seemed to have worked. I'd note that the arstechnica article gives no evidence at all that this particular program is used to generate kill lists. As you note, identifying possible couriers and just killing them without further investigation is a dumb idea. Fortunately I don't see any evidence in this or other articles that this is what happens. [1] https://en.wikipedia.org/wiki/Abu_Ahmed_al-Kuwaiti https://en.wikipedia.org/wiki/Abu_Ahmed_al-Kuwaiti
- mike_hearn 11y agoThe evidence is that the head of the NSA literally said, "we use metadata to kill people". It doesn't get more direct than that. Signature strikes are a well documented phenomenon by now. Also there's no way the CIA drone program can sustain the kind of long term kill rates they've been having without routinely killing innocent people. Terrorists just aren't that common (although if you define anyone who tries to seek revenge thanks to an errant drone strike as a terrorist, that "problem" can eventually solve itself).
- 15charlimit 11y agoI can't believe I'm actually defending the NSA here, but "we use metadata to kill people" does not mean "we ONLY use metadata and nothing else".
- 11y ago
- purpled_haze 11y agoJust don't try to shut it down. "We love you, Skynet! We just need to do some routine maintenance."
- edge17 11y agoMany other countries have 'Skynet' programs https://en.m.wikipedia.org/wiki/Skynet https://en.m.wikipedia.org/wiki/Skynet
- qeabc 11y agoPerhaps someone who knows the topic can say whether this mathematical phenomena of "adversarial examples" for neural networks can be translated to random forests: http://www.kdnuggets.com/2015/07/deep-learning-adversarial-examples-misconceptions.html http://www.kdnuggets.com/2015/07/deep-learning-adversarial-e... Barely perceptible changes to an image cause it to be misclassified by neural networks which never saw the image before (this is important, because it rules out simple overfitting). As a non-expert, this suggests to me that, at least for some algorithms, the reasoning "statistical algorithm estimates 99.999% probability of guilt" implies "guilty beyond a reasonable doubt" is unreliable at best. There is great potential here for safe, effective government if machine learning output is only ever input for humans with common sense, life experience and the ability to interact with the world.
- pps43 11y agoIt would be difficult to create such persuasive adversarial examples in this case. Humans can recognize images well, so it's obvious for us that those adversarial examples are nowhere near what neural net thinks they are. It would not be as obvious with a list of phone calls.
- jndsn402 11y agoWith respect to adversarial examples for random forests, that link states: Myth: Deep learning is more vulnerable to adversarial examples than other kind of machine learning. Fact: So far we have been able to generate adversarial examples for every model we have tested, including simple traditional machine learning models like nearest neighbor. Deep learning with adversarial training is the most resistant technique we have studied so far. But the larger point is that adversarial examples are just one demonstration that algorithms are still quite primitive. Nobody should be killed on the basis of a statistical algorithm alone (and to be fair, the article does not show that anyone has been).
- marcosdumay 11y agoHow many terrorists are there to put on the List? For something around a ten of thousand people, at a minimum 80% of them will be innocent, just by sheer lack of actual terrorists to fill that many slots. And that's assuming a perfect fitting algorithm, with absolutely no false positives.
- random_upvoter 11y agoReading discussions on drone strikes makes you wonder what ever happened to pacifism. This whole idea that you can just go around other countries killing people -- even if they intend on killing you, and even if you would be able to hit the right target 100% of the time-- it's just old. It's primitive. And that is not to blame the USA. At this point in history humanity, as a collective, has the maturity of an 11 year old. Maybe it's time to evolve a bit.
- mistermann 11y agoCouldn't agree more......just reading the front page of /r/all these days makes me want to drop out of society and move deep into the forest. I mean just take one example, the Flint water crisis - this is so unimaginably corrupt (is that even the right word) that it blows my mind. How do people knowingly do such things, how do we have no checks and balances in the system, how can we justify not sending several people to jail for a very long time?
- rayiner 11y agoI don't see why pacifism is "more evolved." Violence is one of the most fundamental interactions between organisms, right up there with eating and sexual reproduction. Is eating "primitive?" Are you looking forward to a future where humans "evolve" out of needing to eat? Throughout history, prosperous and successful societies have been built on the back of military power (Egypt, Rome, Great Britain).
- Aeolos 11y agoWe, as a human species, are supposed to evolve beyond our basic immediate needs. We created societies, because we need to eat, reproduce, survive, without reverting to base violence. We created medicine, technology to improve our quality of life and reduce the chances that biological evolutionary pressure destroys us. The black plague came close to killing 60% of the European population. It is now eradicated. What is the next step in our evolution as a species? Right now the biggest danger to the survival of humankind is... humankind itself. If we are to survive, not for a hundred years, but for a hundred thousand years into the future, we must learn how to cooperate without annihilating each other in the process.
- eli_gottlieb 11y agoWait wait wait. So they're killing people based on a model that fails Statistics 101!? No accounting for base rates? No cross-validation!? God fucking damnit, why the fuck does our government think killing should be cheaper and easier than healing!? We put new medical treatments through decades of difficult scientific tests, but have none at all for strictly invalid ways to kill people?
- pc86 11y agoI do not think you completely understood the article.
- eli_gottlieb 11y agoI'm pretty sure I did. Random Decision Forest with no regularization, no probabilistic accounting for the base rate of the target class, and with the test and train sets mixed rather than performing proper cross-validation. They've almost definitely overfit, and almost definitely killed someone who wasn't actually a terrorist.
- jndsn402 11y agoThe models as described in the article do seem to have glaring flaws. However, the article did not show that anyone has been killed based on the models alone.
- DyslexicAtheist 11y agoA drone strike typically kills identified terrorist targets but also unidentified targets. The trick is that as long as bystanding casualities (also children yes - or as the drone operators call them "fun sized terrorists") have not been identified they're automatically counted as terrorists. (see also https://theintercept.com/drone-papers/ https://theintercept.com/drone-papers/) I don't think now that the beast has been unleashed it can be controlled or harnessed with peaceful means (the tools of democracy). It will take more than a couple of middle-class people with billboards handing out flyers. America is fuelling its own terrorism and creating hate so they can continue their war on terror. Every terrorist act in the West will further drive the hate and justify surveillance. Pretty sure this is a downward spiral which ultimately has only losers. Which steps can the average Joe take which aren't considered radical by the system? Do you know any please share because I don't.
- hodwik 11y agoWe need a military. All states do -- its part of their duty to their citizenry. The problem in the US is that our hippy-run academia are driving the smart people out of jobs in the military (and police force for that matter), and it's putting us in a terrible position. The only way to fix it is for smart people to start joining the military again.
- arca_vorago 11y agoThis is absolute bullshit. The problem is that we need a military that is actually going to man the fuck up when an O-5+ decides to push some unconstitutional program on the people. What has happened isnt that hippy academia is jacking up the military, but rather that the military has been kicking out the kind of people who arent afraid to tell truth to power, and are increasingly making choices and then only hiring people who tell them those choices are good and what they want to hear. I have spent the majority of my free time since I got out trying to understand what the fuck is going on here, and the reality is that our generals failed our military by allowing the neocons to push us into wars for dubious reasons. Thats even counting the real reasons no one talks about, such as upcoming resource wars and a return to a tripolar world. No, the real problem is that we have barely had a real president since the new world order bunch assassinated JFK (and RFK), because that was a very clear message to any POTUS willing to actually stand up. (Dont get me wrong, JFK was far from perfect...) We now have institutional corruption from the top down in every single branch of government, including the famed fourth estate which has been turned into stenographers weekly. Corruption and incompetence are running rampant, and are acting as a cover for the malicious string pullers (Hanlons razor is a logical fallacy!) So no, its not that the goddamn hippies have fucked up the military, its much closer to the military and its propoganda programs have so far infiltrated academia that its a lifeless shell of what it could and should be in this internet age. Keep in mind though, its the military who has pushed this though. While the three letters have carried out operations similar to mockingbird (thank you church comittee), its the globalist new world order group who have pushed it on their controlled government puppets. (Read up on what Norman Dodd found during the Reese Committee for more info) Yes, you are correct we need a smarter military, but just having "smart people" join up wont do it. You need change from the POTUS down, because beleive you me, we have turned mental deconstruction and reconstruction into an art form.
- jjtheblunt 11y agoDisingenuous title: NSA only collects data (SIGINT) and secures transmissions and identity of friendlies (SIGSEC). What others do with the data is not done by NSA.
- Aeolos 11y agoWhich is of course misdirection through diffusion of responsibility. Stay classy.
- ctlby 11y ago> But you are going to get an error rate, and probably a high one in the noisy real world. They're clearly not implementing a minimum-error rate classifier. Which isn't to defend their choice of loss function (it's not 0-1, but beyond that I have no idea); just to point out the inanity of some of the comments claiming the NSA isn't aware of base rates.
- tremon 11y agothe more I read the details, the more alarmed I am Just wait until the next leak confirms that the system has learned to fly the drones autonomously.