30 ms·
The NSA’s machine learning algorithm may be killing thousands of innocent people
- Aeolos 11y ago"Obliterated a wedding because people where celebrating by shooting in the air." This is the definition of evil. How and why is the Pakistani government allowing this program to operate within their territories?
- skbohra123 11y agoYou are missing the main point of the article.
- Aeolos 11y agoI am not. I am just at a loss why a foreign sovereign would give the US, or any other country, the license to randomly kill their own citizens.
- atroyn 11y agoIf they did not, what could Pakistan do about it? The U.S would do so anyway covertly, and the Pakistani government would be seen to be weak, since they were powerless to stop it. So the Pakistani government 'allows' it to at least appear in control.
- ordbajsare 11y agoAnd Pakistan gets fucktons of foreign aid that the use to buy US made weapons.
- rdancer 11y agoThey have nukes. They could easily institute a defense doctrine of nuclear retaliation tomorrow, and the terror would stop. But the reason they don't is that they are bought and paid for, both as a country via foreign aid and as individual government members via bribes (or "speaker fees", as you guys call those now).
- wrong_variable 11y agoSAVAGE
- meric 11y agoThe U.S. pays the Pakistani government for the right to use drone strikes against innocents who are prescribed as terrorists by a computer algorithm, which encourages hatred and fear and creating the very terrorists justifying U.S. giving the money for the Pakistani government to buy weapons from the U.S. to field an army to fight a war with.
- avar 11y agoLet's run with that thought. Let's say Pakistan committed to using nukes in retaliation for US drones shooting their citizens, what are they going to shoot at exactly? They're supposedly developing longer-range capability capable of hitting some of Europe[1], but right now they might be able to hit some US military bases in the middle-east[2]. I'm not saying Pakistan isn't corrupt, but pretending that they could simply switch from their current US-friendly policy to a strategy that's openly adversarial to US-interests seems to blatantly ignore the realpolitik of the situation. 1. https://en.wikipedia.org/wiki/Taimur_(missile) https://en.wikipedia.org/wiki/Taimur_(missile) 2. http://news.bbc.co.uk/2/hi/south_asia/3545775.stm http://news.bbc.co.uk/2/hi/south_asia/3545775.stm
- rdancer 11y agoYou are right. I haven't considered that. They only care about the standoff with India, so it makes sense that their range only about covers her.
- fsloth 11y agoMy guess would be because Pakistan has had good relations with US since it's independence. "The United States today provides extensive economic, scientific, and military assistance to Pakistan... and is Pakistan's most generous donor of foreign aid"[1] [1] https://en.wikipedia.org/wiki/Pakistan%E2%80%93United_States_relations https://en.wikipedia.org/wiki/Pakistan%E2%80%93United_States...
- dingaling 11y ago> My guess would be because Pakistan has had good relations with US since it's independence. More often not. There was a period from the 1970s when Pakistan was embargoed because of its nuclear power + weapons programmes, mainly in cooperation with France. This pushed Pakistan into the Chinese camp for arms supplies where they have remained firmly lodged ever since. During the 1980s the USA did supply some silver-bullet assets like F-16s, mainly to atagonise the Soviets in Afghanistan, but later deliveries were again embargoed and the USA only recently reimbursed their purchase fees. The USA-Pakistan relationship is one of the text-book examples of realpolitik.
- icegreentea 11y agoTo add to that, DoD just approved sale of 8 F-16s to Pakistan (http://thediplomat.com/2016/02/us-approves-sale-of-8-f-16-block-52-fighters-to-pakistan/ http://thediplomat.com/2016/02/us-approves-sale-of-8-f-16-bl...). The ball is in Congress' court to decide if this actually goes through... but ya. Pretty much the definition of realpolitik.
- bizkeep2 11y agoIt's because we're killing their enemies.These are strikes on the TTP not the Afghani Taliban.
- cmdkeen 11y agoBecause most of the strikes are in the Tribal areas near the border with Afghanistan where the writ of the Pakistani state doesn't really run. The Pakistani military has received some bloody noses when trying to operate in the area. Even Wikipedia is happy to classify the whole thing as a war[0]. Over 6,000 members of the Pakistani security forces have been killed, 20,000 civilians and 35,000 "insurgents". Pakistan gets significant military aid from the US to help put down an insurgency by groups that would overthrow the state. On a realpolitik level they also get to blame the US when a strike goes wrong whereas if they were dropping their own bombs they would have to put up with the blame. Mistakes always happen in war, I'd suggest however that a drone strike has significantly more time to conduct pre-launch checks than an F16 pilot does. [0] https://en.wikipedia.org/wiki/War_in_North-West_Pakistan https://en.wikipedia.org/wiki/War_in_North-West_Pakistan
- mcv 11y agoBut the end result of all that is just that it generates even more hostility towards the US and feeds terrorism. The US is creating its own enemies this way. Again.
- Aeolos 11y agoCynically speaking, this is win-win for everyone involved. Except for the innocents who are killed. The US gets to debug and optimize their assassination program "on brown people that noone cares about". Pakistan and US also get to claim that they are "fighting terrorism". This breeds resentment and new generations of terrorists to be "fought" in the future (divide and conquer 101). Plus, who knows when and where the droid program might come in handy in the future...
- IIAOPSW 11y agodroid program?
- venomsnake 11y agoThey could always not allow and be left to deal with Afgani collapse on their own. And with soaring and increasingly nationalistic India under Modi.
- coldtea 11y ago>How and why is the Pakistani government allowing this program to operate within their territories? Because it's a powerless country, and as always those are at the mercy of the top dogs. At best they can ask for some bargains (foreign aid, some favorable diplomatic stance, etc).
- qrendel 11y agoAmong other reasons, Pakistan security forces receive a lot of military aid from the US State Dept, including weapons and materials (e.g. bulletproof limousines) used by the same Pakistani officials for their own protection. I expect US support also helps as a deterrent in their problems with India as well.
- massemphasis 11y agoThey don't actually care if the person is innocent. The point is to intimidate their population. They use this metadata to fit into the system of law so they can cover their ass.
- Houshalter 11y agoThey literally named it Skynet. They have an evil sense of humor. Actually using machine learning to detect terrorists isn't a terrible idea. But you are going to get an error rate, and probably a high one in the noisy real world. Maybe only 50% of the people you detect are actually terrorists. Maybe it's even worse than that. We can't even test it because there is no validation set and unreliable labels. The reasonable thing to do with that information, would be to surveil them further, search their house, or arrest them. Not assassinate them without a trial. And the more I read the details, the more alarmed I am. The 50% figure I used above may have been way too high. The base rate of terrorists way too low and they have very little data to begin with.
- milesftl 11y agodid you post this on the corresponding reddit thread as well? if not then someone literally copy pasted your comment, which I have been noting a lot more lately on there.
- Houshalter 11y agoYeah that was me.
- teekert 11y agoGiven that there are very little real terrorists in the data, any sensitive detection mechanism will need insane specificity or it will give you an enormous amount of false positives.
- pwg 11y ago> But you are going to get an error rate, and probably a high one in the noisy real world. ... The reasonable thing to do with that information, would be to surveil them further It is even worse than that. Due to the base rate fallacy (https://en.wikipedia.org/wiki/Base_rate_fallacy https://en.wikipedia.org/wiki/Base_rate_fallacy) the 'machine' is going to have a huge false positive rate, so without that additional surveillance, most will not be guilty of what they are accused. But that surveillance takes money and resources, just the thing the 'machine' system is designed to save...
- nl 11y agoThis story is stupid. I'm sorry. No fan of the NSA, but the premise behind it is completely ridiculous. There is zero evidence of the repeatably asserted idea that the list this tool generates is any kind of kill list. It's a tool that generates indicators of people that may be worth looking at when trying to find couriers. That's a very specific subgroup of terrorists, and I find it entirely unsurprising that a journalist would be falsely flagged as journalists have statistically unusual travel habits (Clearly labeling him as "member of Al Qaida" is unjustified by this evidence though). Also, criticizing the NSA on their knowledge of statistics seems unwise. The NSA is many things, but "bad at Math" isn't one of them. Read the information yourself, and come to your own conclusions.
- SeanDav 11y agoYou may of course be right, I would find it surprising in the extreme if people were killed based purely on output of an algorithm. However we have no assurance that this tool was not used to generate kill targets without any human input. The NSA operates without any meaningful oversight and if they decided to use algorithms to generate kill targets, they would go right ahead and do it.
- pavelrub 11y ago> However we have no assurance that this tool was not used to generate kill targets without any human input. That's absurd. We also have no assurance that the NSA isn't communicating with aliens to generate kill lists of people who will oppose the alien invasion of 2021, though we do have common sense to filter out such absurd scenarios. "We have no assurance that this isn't so" isn't an argument.
- titzer 11y agoI'll bite. The difference is that communicating with aliens requires positing the existence of aliens, whereas simply generating a kill list is already something we know they do, though we don't know how it works. It isn't "common sense" to filter out entirely obvious actions the NSA would take, and given the scale of disclosures from the Snowden files you ought to have already expanded your imagination away from "they'd never do that" to "yes, they already have the technical capability and have done similar things in the past, lied to Congress about it, dismantled oversight, lobbied against oversight, and otherwise deceived every mechanism of oversight put in place." The fact that 2,500 to 4,000 people have been killed in Pakistan according to a completely opaque process for classifying targets and blasting them from the sky--a process that operates entirely outside any law--ought to be considered absurd, shocking, frightening, and soul-crushingly inhuman. But it's not. We're arguing about with each other about bullshit. Focus.
- karmacondon 11y agoI was wondering exactly how the NSA trained models to detect terrorists, and was surprised by the level of detail in this article. So the NSA performs classification using a Random Forest and about 80 input features? Huh. That actually sounds a little too similar to a Kaggle contest for my liking, but I wasn't expecting that much quasi-technical information anyway. It seems a little silly to write this whole article based on a few powerpoint slides from years ago. Even as an amateur practitioner, I can see several obvious things that could be changed about the presented methods. I'm sure what the NSA is doing now in this regard is much further along than what was portrayed in those documents. Whether or not they should be doing it is a discussion for others to have. But it is interesting to learn a little more about the details of truly high stakes machine learning.
- andrewclunn 11y agoWait, this isn't an Onion article? Please be hyperbolic bullshit. PLEASE BE HYPERBOLIC BULLSHIT!
- mtrycz 11y agoThe title actually reads like your usual clickbait. Unfortunately, it is also tragically accurate, wrt the info in the article.
- agd 11y agoBig data analysis + mass surveillance is a frightening prospect. Of course you can train software to look for 'terrorists', but you could also train it to look for: - whistleblowers - minority groups (e.g. gay people, particular religious beliefs, political affiliation) - political dissidents - journalists whose behaviour changes - personal vulnerabilities (affairs, mental health issues etc) Think what authoritarian governments could (and are) doing with this capability.
- Swizec 11y ago>- minority groups (e.g. gay people So there was this infamous incident of a Netflix machine learning competition for movie recommendations ... Guess what somebody was able to do.
- mike_hearn 11y agoI can guess what you must mean but I can't find any references about it. Got a link?
- amelius 11y agoI would call it "mass discrimination". Because, in principle, that is what this amounts to: you're using a few traits of a person to generalize into a whole group. I find it strange that nobody seems to identify this. It is almost as if, because it is done on a big computer and no human is involved in the processing, suddenly it is not discriminating anymore. But humans programmed these computers in the first place.
- nickff 11y agoThis has been identified, posted to HN, and discussed.[1] [1] https://news.ycombinator.com/item?id=10872852 https://news.ycombinator.com/item?id=10872852
- realusername 11y agoTo be honest, being someone interested in lots of things, just based on my browser history you could also put me in the terrorist box.
- andygambles 11y agoThe "terrorists" are just taking the piss. Drawing a cock and balls on screen.
- mtrn 11y agoReminds me of this dialog from Spectre (http://www.imdb.com/title/tt2379713/ http://www.imdb.com/title/tt2379713/): > Have you ever had to kill a man, Max? Have you? To pull that trigger, you have to be sure. Yes, you investigate, analyze, assess, target. And then you have to look him in the eye. And you make the call. And all the drones, bugs, cameras, transcripts, all the surveillance in the world can't tell you what to do next.
- spazmaster 11y agoThe thing that stood out for me is "Somewhere between 2,500 and 4,000 people have been killed by drone strikes in Pakistan since 2004". WHAT!? That is so wrong! This stinks and we're making fuss about mathematics. Just read that sentence again.
- Aeolos 11y agoExactly. In which freaking world are assassinations without due process legal or even desirable? The mere fact that we are debating whether someone was or wasn't assassinated is a perversion in itself. Arrest them and try them in a court of justice. End of story.
- oafitupa 11y agoArrest them? They were in freaking Pakistan. Why would you arrest them? You guys are going crazy. Don't be surprised when you travel and realize the world hates you.
- binarymax 11y agoI think the parent means arrest the criminals responsible for killing thousands of people with drones based on a faulty algorithm.
- Aeolos 11y agoI meant both, in fact.
- MikeNomad 11y agoI already know parts of the world hate me, because I come from the country that engages in drone strikes.
- mike_hearn 11y agoHe obviously means "send intelligence to the government of Pakistan and ask them to investigate".
- anon4 11y agoThe important part is that it's helping them kill more people, so it means they're more productive. /s
- luisjgomez 11y ago"Under the random selection of a tiny subset of less than 0.1 percent of the total population, the density of the social graph of the citizens is massively reduced, while the "terrorist" cluster remains strongly interconnected." There is no indication in the slides that the feature calculations are done on the smaller subgraph. The above comment doesn't hold if the calculations are done on the entire graph and then the 100k are sampled for training.
- known 11y agoI've developed a DRONE that can TRACK each and every BRAHMIN in the World;
- YashN 11y agoFunny how in the title, the culprit is the algorithm and not the organisation.
- klean92 11y agoThe argument was that in the US, they "only" collected metadata. No real phone calls. No big deal... The power of their use of "only" metadata in Pakistan is frightening..
- scotty79 11y ago> ... possibly resulting in their untimely demise. I get increasingly annoyed by these kinds of euphemisms in articles discussing US actions. As if it was something humorous. Should be something like: ... possibly resulting in them getting assassinated by US military.
- stevetrewick 11y agoEven before reading TFA that headline stands out like poke in the eye. An algorithm has no agency - moral or otherwise. Algorithms don't kill people, people kill people.
- mcv 11y agoThey automated the decision to kill people and called it Skynet? That sounds awfully familiar. When can we expect assistance from the future to take the NSA people out?
- Cacti 11y agoWould have appreciated a "top secret" flag in the title. The slides in the article are not declassified...
- appleflaxen 11y agoThe founding fathers put due process into the constitution / bill of rights because it's a fundamental check on government bureaucracy run amok, and prevents physical harm from coming to people as the result of a capricious executive whim. Now, we learn that the executive has excused themselves from following this process (which applies to Americans, some of which these victims may be, even when they are not in America). The farther open we pull the NSA lid, the more revolting discoveries we find. The individuals behind these decisions should be identified, and tried in the court of law.
- afsina 11y agoOh come on "machine learning algorithm may be killing thousands.." just put a "may be" and your BS assertion becomes more plausible? By looking at this, people are having and idea that attack drones (it is rich that article puts one picture of it in the beginning) are loaded with such software and killing by looking at the result of a classifier. Or some super computer gives you a name and says "exterminate". Apparently these tools only gives operators some clues and save their time. If it is a false positive, probably they just ignore it. Of course, obtaining the information is a different story.
- graycat 11y ago> If it is a false positive, probably they just ignore it. And how do they know it's a false positive? Indeed, if they really knew that, then they would have some more training data.
- afsina 11y agoI am not defending peoples lousy/callous decision processes for fatal actions. But telling that bad machine learning algorithms are responsible for killings is just stupid.
- vladtaltos 11y agothe problem is you don't know that 'If it is a false positive, probably they just ignore it.' -> 'probably' is a fucking optimistic view. AND how the hell do you know this is a false positive ? some algorithm tells a soldier that this person is probably a terrorist - what happens then ? do you think they bother to verify this by spending more money ? In my opinion it is most likely that, the people using this software send the output of this algorithm as a recommendation up the chain and some idiot decides to be 'safe' and recommends execution. because they don't have any accountability. I don't think you would be this relaxed in your opinion if these f-ing drones fly over your head - it is just that most people in the US do not care what happens to some idiot in Pakistan. from the reports disseminated by the state department they are not even sure how many people they killed ? it is reported as between 2500 and 4000. Isn't this insane ? I have no idea in my mind that US is killing people indiscriminately using drones - and the word is not probably - it should be 'definitely'. i'm sure drones are turning people into terrorists more than they are killing terrorists. don't tell me this is a bs assertion before US can give the name of every body they killed with a drone and the justification of it.
- daviddumenil 11y agoA related approach that identified suspects based on their banking history was outlined in SuperFreakonomics. The chapter is reproduced in this review: https://www.goodreads.com/review/show/90622011 https://www.goodreads.com/review/show/90622011
- botterworkshop 11y agoIt's ignorant to assume any targets revealed by this program wouldn't be given a human intelligence analyst to verify accuracy before risking millions of dollars on a predator strike and the potential risk it was bad information. Enough said.
- georgeek 11y agoThere are two issues here, firstly some dramatic overfitting of the model, and secondly that all-too-familiar garbage in, garbage out fact, which is even more relevant when dealing with data.
- donatj 11y agoWe're killing people in a country were not even at war with over statistics. Not evidence, but statistics. People who statistical might have something against us. Since when has this been grounds for killing someone? This is utterly terrifying. Just imagine if someone decided to do this to us. How does Pakistan feel about all this? This is approaching Auschwitz levels of evil.
- deleted 11y ago[deleted]
- DKnol 11y agoI see lots of doubt about the ridiculous nature of this program described in this article. People saying it's not possible. Are you sure? https://theintercept.com/2014/02/10/the-nsas-secret-role/ https://theintercept.com/2014/02/10/the-nsas-secret-role/
- visarga 11y agoWhy are terrorists still using SMS if it is lethal?
- alanwatts 11y agoSo yes, a state would not be much of a state without some form of military. But why do we need separate states? Edit: this is a genuine question about the practicality of a concept (the "nation-state") that was invented during the Gutenberg era of the printing press, and how such a concept has become impractical in the age of instantaneous international interconnectivity and economic globalization. Downvoters: How are arbitrary divisions drawn on a map anything but counterproductive to synergy and efficiency in the age of globalization?
- Aeolos 11y agoYou really want to know? Because of regression towards the mean. Having a single state would mean that the US couldn't afford to have a significantly higher standard of living compared to the rest of the world. In fact, 1st world countries would have to meet 3rd world countries somewhere in the middle. (Assuming you are living in the states) Would you be willing to live on a wage of $5000 a year? Edit: see here https://en.wikipedia.org/wiki/List_of_countries_by_GDP_%28nominal%29_per_capita https://en.wikipedia.org/wiki/List_of_countries_by_GDP_%28no...
- alanwatts 11y agoYou're still thinking in terms of 19th century paper-based processes when you attempt to estimate globally integrated wealth in terms of a nationally fragmented paper-based information system ($). Also, GDP is not indicative of net wealth, as it is a measurement which implicitly asserts the broken window fallacy. Over the long term, breaking down these legacy barriers to trade and freedom of movement would necessarily uplift everyone economically by principle of synergy. Synergy: the interaction of elements that when combined produce a total effect that is greater than the sum of the individual elements, contributions, etc. >I'm going to give you an important kind of a picture. I hear a lot of people say "I don't like machinery and technology, it's making a lot of trouble." So we're going to take all the machinery away from all the countries of the world, all machinery, all the tracks and the wires, and the works and we're going to dump it all in the ocean. And you will discover that within 6 months, 2 billion people will die of starvation having gone through great pain. So we say "That's not a very good idea lets put all the machinery back where it was." Then, we're going to take all the politicians from all the countries around the world and we're going to send them on a trip around the sun, and you'll find we keep right on eating. And the political barriers now... scientists say very clear you could make the world work and take care of 100% of the people at a higher standard of living that anyone has ever known despite the increasing population, but you can't do it with the barriers, any more than you can try run a human organism with a wall between the ear, the eye, and the stomach. It is an organic whole, it is total industrialization. -Buckminster Fuller >Human history can be viewed as a slowly dawning awareness that we are members of a larger group. Initially our loyalties were to ourselves and our immediate family, next, to bands of wandering hunter-gatherers, then to tribes, small settlements, city-states, nations. We have broadened the circle of those we love. We have now organized what are modestly described as super-powers, which include groups of people from divergent ethnic and cultural backgrounds working in some sense together — surely a humanizing and character building experience. If we are to survive, our loyalties must be broadened further, to include the whole human community, the entire planet Earth. Many of those who run the nations will find this idea unpleasant. They will fear the loss of power. We will hear much about treason and disloyalty. Rich nation-states will have to share their wealth with poor ones. But the choice, as H. G. Wells once said in a different context, is clearly the universe or nothing. -Carl Sagan, Cosmos
- tobbyb 11y agoThe US, UK response to terrorism is so disproportional, odious and sinister one wonders whether this is infant a 'response' or something far more evil cooking below the surface. This kind of technology necessitates mass surveillance, and offers you little more beyond the faint possibility that crime can be predicted. This is bogus science. If you let your mind entertain the idea you can predict crime, then you are already at the thresh hold of a stifling surveillance state. And why are we even assassinating people. What happens to due process? One by one all these fundamental principles are set aside, massive and dubious mass surveillance infrastructures are being built, doublespeak is comically rampant. One can begin to understand Snowden's urgent need to act, but what about the moral compass of all the people in the NSA and who support the US and UK security apparatus? People need to act. Surely they cannot suddenly subscribe to values we have aggressively demonized for more than 100 years. Who would have thought that the US and UK are now the rogue states and other countries who are far more secure and sure about their politics need to begin to isolate themselves from these dangerous totalitarian instincts.
- EGreg 11y agoThe more I read about US activities abroad, the more I realize Chomsky's right. Are sovereign countries supposed to tolerate a certain level of drone strikes by the USA (4500 people in Pakistan killed in the last decade) based on -- now revealed -- machine learning hunches?
- anovikov 11y agoIf that system skill kills enough terrorists to make them scared and limit inflow if new recruits into that system than why not? Obviously rest of the population of Pakistan pays their price, but what they expect if they can't (or really, don't want to) shut down terrorist activity in their own country themselves?
- foxyv 11y agoNon-Bayes analysis kills.
- andremendes 11y agoI feel infuriated while reading that they named it Skynet. They are clearly the ones creating terror, that's so absurd! I can't believe that they do not have the understanding that this very act is what propagates wars and keep them going on forever. As it cannot pass as ignorance, I take as malicious the people who feed this system.
- dcposch 11y agoThis story is hugely important. It gets to a deep question everyone here should ask themselves -- beyond immediate concerns of salary, equity, and learning value, is my specific work making the world a better place or a worse place? This story goes to the core of ethics in engineering. It's all of 9 hours old and has 341 points -- yet it's already off of the front page where nobody will see it. You could check HN literally every day and still easily miss this story. Meanwhile, the front page is full of unimportant links to obscure tech trivia, many of which have less than 20 points. We know that HN automatically penalizes submissions containing certain words, including "NSA", in the title. Certain prolific HN users have also said that they "automatically flag" submissions they consider "political". But I really think HN would be a better place if the front page cycled out a little slower, if stories like this were not suppressed, if they got at least one day's worth of attention and discussion.
- kafkaesq 11y agoHmm: Behavior-based Analytics: - Low [cell phone] use, incoming calls only - Frequent handset changes - Frequent detach / power-down Visits to other countries - overnight trips - permanent move I guess that makes me an extremist militant, also.