10 ms·
Xkeyscorerules100.txt
- v64 12y agoDoes anyone have any context or information as to what this is?
- deleted 12y ago[deleted]
- anateus 12y agohttp://en.wikipedia.org/wiki/XKeyscore http://en.wikipedia.org/wiki/XKeyscore
- acqq 12y agohttps://news.ycombinator.com/item?id=7983124 https://news.ycombinator.com/item?id=7983124 The article commented has the whole context. For us programmers it's interesting, among other bigger issues, to see that the rules contain the pieces of code in C++.
- GHFigs 12y agoThe article commented has the whole context. It really doesn't. In spite of repeatedly claiming things like that people searching for Tor are "monitored" or users are "tracked", it's completely vague about what that those terms actually mean and provides zero examples.
- acqq 12y agoDo we read the same article? It actually has 5 pages and it attempts to explain the different sections of the file. http://daserste.ndr.de/panorama/aktuell/NSA-targets-the-privacy-conscious,nsa230.html http://daserste.ndr.de/panorama/aktuell/NSA-targets-the-priv... http://daserste.ndr.de/panorama/aktuell/nsa230_page-2.html http://daserste.ndr.de/panorama/aktuell/nsa230_page-2.html http://daserste.ndr.de/panorama/aktuell/nsa230_page-3.html http://daserste.ndr.de/panorama/aktuell/nsa230_page-3.html ... Maybe you've read only the first page and missed the remaining four?
- GHFigs 12y agoYes, I read it, and I stand by what I said. The article explains a set of rules used to filter a set of information out of another set, but it does not support claims about what is done with that set of information. There is a lot of insinuation, but no example of any individual user of Tor or reader of Linux Journal, etc. being monitored or tracked simply for doing so.
- acqq 12y agoYou can always demand even more proofs, but it's not probable that they'll come easily -- it's obvious that such leaking is extremely dangerous. This is still significantly more that public knows now than it knew before the article.
- influx 12y agoIt makes more sense to me that they actually use these in AND statements. For example, uses TOR and searches for JIHAD could be traffic that would be interesting. If I had to guess, the Linux Journal stuff was just something a geek put in there during testing.
- runiq 12y agoThat's what I thought—these rules obviously aren't the entire pipeline, and the results obtained from them may or may not be interesting in and of themselves. That said, using Tails probably does increase my XKeyScore rating. Is there anything published as to the scale of the rating? Something along the lines of "Once you get a rating of (say) 500, we're gonna come and beat down your door, wife and dog, and not necessarily in that order".
- GHFigs 12y agoThere's no indication that there is such a rating exists, or that any such decisions are made based on automated rules. It's a tool for selecting some traffic out of all traffic, not for replacing human analysis or decision-making. http://en.wikipedia.org/wiki/XKEYSCORE http://en.wikipedia.org/wiki/XKEYSCORE
- WestCoastJustin 12y agoRelated to https://news.ycombinator.com/item?id=7983124 https://news.ycombinator.com/item?id=7983124 Additionally, /. has a pretty good summary of what this is [1]. -- "If you search the web for communications security information, or read online tech publications like Linux Journal or BoingBoing, you might be a terrorist. The German publication Das Erste disclosed a crumb of alleged XKeyScore configuration, with the vague suggestion of more source code to come, showing that Tor directory servers and their users, and as usual the interested and their neighbor's dogs due to overcapture, were flagged for closer monitoring. Linux Journal, whose domain is part of a listed selector, has a few choice words on their coveted award. Would it be irresponsible not to speculate further?" [1] http://yro.slashdot.org/story/14/07/03/1846215/nsa-considers-linux-journal-readers-tor-and-linux-users-extremists http://yro.slashdot.org/story/14/07/03/1846215/nsa-considers...
- tlrobinson 12y agoLinux Journal seems out of place. They must have had an article about Tails?
- aw3c2 12y agohttp://www.linuxjournal.com/content/nsa-linux-journal-extremist-forum-and-its-readers-get-flagged-extra-surveillance http://www.linuxjournal.com/content/nsa-linux-journal-extrem... I saw it on the HN homepage some moments ago, I guess the mods hid it for some reason?
- cyberbully 12y agoMore than one link to the Linux Journal article with comment thread has been locked and hidden, and a previous link to the link we're commenting on now. I'm sure this one will be dead soon too. For example: https://news.ycombinator.com/item?id=7984456 https://news.ycombinator.com/item?id=7984456. hckrnews.com shows dead threads in gray. Weird, no?
- MatthiasP 12y agoYes they did[1], and a link to that article was published on an 'extremist' forum, triggering the inclusion of the domain in the filter. [1]http://www.linuxjournal.com/content/linux-distro-tales-you-can-never-be-too-paranoid http://www.linuxjournal.com/content/linux-distro-tales-you-c...
- sp332 12y agoWhat language is this?
- acqq 12y agoObviously the custom language (which uses $ like Bash or Perl) but which allows the programmer to include the pieces of C++. Hints: static std::string and boost! The Boost, for which Google writes: http://google-styleguide.googlecode.com/svn/trunk/cppguide.xml http://google-styleguide.googlecode.com/svn/trunk/cppguide.x... "Cons: Some Boost libraries encourage coding practices which can hamper readability, such as metaprogramming and other advanced template techniques, and an excessively "functional" style of programming. Decision: In order to maintain a high level of readability for all contributors who might read and maintain code, we only allow an approved subset of Boost features."
- Centigonal 12y ago/* These variables define terms and websites relating to the TAILs (The Amnesic Incognito Live System) software program, a comsec mechanism advocated by extremists on extremist forums. */ That's an interesting definition. It might cover legitimate extremists, but a quick look at Wikipedia tells me that TAILs has also been used by some pretty respectable Pulitzer Prize winners. These comments provide an interesting and ultimately disheartening insight to how the people designing these surveillance systems view privacy software (and, by extension, privacy?).
- opendais 12y agoI'm sure a legitimate extremist would want to use something like TAILs. I'm more worried about the fact they are tracking people who access articles about tails by the media than I am about that particular component. If you read one article on linuxjournal containing 'tails' you are flagged: $TAILS_terms=word('tails' or 'Amnesiac Incognito Live System') and word('linux' or ' USB ' or ' CD ' or 'secure desktop' or ' IRC ' or 'truecrypt' or ' tor '); $TAILS_websites=('tails.boum.org/') or ('linuxjournal.com/content/linux*'); So they aren't really "monitoring extremists" they are monitoring anyone with an interest in security [even if its just a hobby].
- GHFigs 12y agoWe have no context for what "flagged" or "monitoring" means, though. There's no indication that these rules are taken to mean anything in isolation beyond what is frankly common sense. The idea that just Googling Tails means you're monitored is a narrative leap. If I write an email about Viagra, the recipient's spam filter is undoubtedly going to consider it to have a non-zero probability of being spam. That doesn't mean it will be blocked, in the end. Nor does it mean that the spam filter is wrong to take notice.
- opendais 12y agoYes. But do you honestly expect them to provide that context so we can be sure?
- api 12y agoIn other news: the NSA developed a DSL with embedded C++. Is this the most horrific revelation yet?
- acqq 12y agoNot compared to the possibly in-house developed object-oriented event-driven C framework for async communication: http://www.securityweek.com/kaspersky-lab-duqu-framework-likely-written-unknown-programming-language http://www.securityweek.com/kaspersky-lab-duqu-framework-lik... "“It is possible that its authors used an in-house framework to generate intermediary C code, or they used another completely different programming language,” Soumenkov explained. For reference, Stuxnet was written entirely in Microsoft Visual C++. The Kaspersky researchers say certain “slices” of code in the Payload DLL may have been initially compiled in separate object files before being linked in a single DLL, but the slice in question is different. “This slice is different from others, because it was not compiled from C++ sources. It contains no references to any standard or user-written C++ functions.” But there a few things the researchers do know about the mystery code: It’s object-oriented and event driven, and performs its own set of related activities ideal for network applications. The highly event driven architecture points to code which was designed to be used in variety of conditions, including asynchronous commutations." For the reference, New York Times, June 2012: http://www.nytimes.com/2012/06/01/world/middleeast/obama-ordered-wave-of-cyberattacks-against-iran.html http://www.nytimes.com/2012/06/01/world/middleeast/obama-ord... "Eventually the beacon would have to “phone home” — literally send a message back to the headquarters of the National Security Agency"
- Phlarp 12y agoWould I be wrong to gather that they also built an in house map reduce implementation? What year is this code from? Most of the other documents have been from 2007-2009, when did Google first implement map reduce?
- acqq 12y agoGoogle's paper from 2004: http://static.googleusercontent.com/media/research.google.com/en//archive/mapreduce-osdi04.pdf http://static.googleusercontent.com/media/research.google.co... "We wrote the first version of the MapReduce library in February of 2003"
- ogijaoijfawje 12y agoI believe this "source code" is made up, invented for the masses. In fact, the more and more I see of these surveillance reports and reveals, the more I believe this is all purposeful deception, and that while it may be true they're doing all these things, they aren't leaks, but announcements. On a board where most of us should be familiar with the concept of not trusting user input, I think we should all take a step backward and treat these "leaks" as just that: input from an untrusted source. This could all be a fabrication we're buying into.
- mschuster91 12y agoPoor enough you use a throwaway, but I see what you're trying to tell. I personally believe the NDR/Spiegel to be reputable, trustworthy media. It may be that we have another Hitler's diary scandal there, but the other Snowden/NSA material they have published so far has held up examinations, the NSA iirc even acknowledging some of the docs as actually valid.
- ogijaoijfawje 12y agoI'm not disputing that the NSA stuff is real - of course they're doing it, it's what the agency is expected to do. I'm disputing whether these are leaks or announcements. The media might not even know they're part of the plan. Is it that farfetched for the NSA to say "Hey, we want to make this information public for some reason, and we're going to do so by using a whistleblower. You're going to release all these documents to a bunch of media sources and live abroad for a while and be hailed as a hero"? There's a long history of leaking information, factual or otherwise, for a number of purposes. There's potentially no end to the rabbit hole. Maybe Snowden doesn't even know he's part of the plan. It's my opinion that the NSA, some group, or some individual, is letting us know they're watching closely. Maybe they're acclimating us to the idea of being spied on, or worse, distracting us from or preparing us for something else. I think the big picture has yet to be revealed.
- sroerick 12y agoSomething worth considering is the conditions necessary for having both subjective and objective expectation of privacy. If nobody expects to be able to avoid the NSA, nobody can expect to have subjective expectation of privacy. NSA will certainly know everybody's reaction to the Snowden leaks. I figure the absolute worst case scenario imaginable is that USGOV is producing a list of sysadmins and privacy conscious individuals for extermination. Most "coding" seems to be happening now several layers up from linux systems. If the Government wanted, they could kill everyone who knew how to actually use a computer. Then they could "Teach low income Americans to program" and then just conveniently forget to teach them the full stack. Note, I don't think this is happening, but it isn't unimaginable. In addition, the narrative has been very pro CIA since the very beginning. There have been a lot of people making the clear distinction between machine intelligence and human intelligence, which is definately a CIA line.
- deleted 12y ago[deleted]
- frisco 12y agoI don't have a good reason to believe that this is real, but if it is the most surprising part to me is the "mapreduce" rule definition in there. As far as I know the only group with a C++ mapreduce implementation called "mapreduce" that also uses protocol buffers (the "proto:" block is protocol buffers) is Google. This seems to say to me that the NSA is using a Google implementation of map reduce. That can't possibly be right, can it?
- jmillikin 12y agoThey're not using the Google implementation; it looks like they rolled their own.
- frisco 12y agoYour profile says you work at Google... are you saying that the code snippet here is inconsistent with Google's mapreduce?
- nostrademons 12y agoI used to work at Google - this looks absolutely nothing like a Google MapReduce specification or code would look like.
- elliotanderson 12y agoNot a Google employee - but the linked file looks more like a simplified DSL for analysts.
- drunkcatsdgaf 12y agoInteresting video about them using openstack https://www.youtube.com/watch?v=NgahKksMZis https://www.youtube.com/watch?v=NgahKksMZis
- Zaephyr 12y agoWhile mapreduce isn't my area of expertise, my guess is it's probably easier for them to use mapreduce/ and Hadoop than invent their own wheel in some cases.
- nickporter 12y agoFrom the filename, I thought this was some kind of X window config file for key remapping.
- jevinskie 12y ago// START_DEFINITION /* The fingerprint identifies sessions visiting the Tor Project website from non-fvey countries. */ fingerprint('anonymizer/tor/torpoject_visit')=http_host('www.torproject.org') and not(xff_cc('US' OR 'GB' OR 'CA' OR 'AU' OR 'NZ')); // END_DEFINITION I was surprised to see that they actually tried to exclude Five Eyes countries. The cynic in me wonders if there is "bug" that neuters the restriction.
- sp332 12y agoI'm wondering if "FVEY" meant something before it was rendered as "five eyes"?
- Mandatum 12y agoThe "Five Eyes" term has been in use since 2006. Strangely enough the original Wikipedia document archived on the Wayback Machine seems to be unavailable, however we do know it pointed to "USAUK_Community" whose relationship was extended to include other allies which is now known as the "Five Eyes". Before 2007 the amount of searches for "FVEY" was near zero. The relationship between these five nations wasn't named publicly until 2006 or so, before that the relationship was there - just under another name. However there are documents dated to 2001 available on mors.org which have FVEY references: http://www.mors.org/UserFiles/file/82nd-Symposium/Form%20712%20A.pdf http://www.mors.org/UserFiles/file/82nd-Symposium/Form%20712... These are public so you can assume internal naming was used well before then.
- dsl 12y agoFVEY is the accepted shorthand for AUSCANNZUKUS classification, and has been used since the 50's when UKUSA was expanded.
- revelation 12y agoWell, we have already learned that their idea of origin country is IP ranges, even if the data clearly indicates otherwise. That said, they have a variable for directory servers in FVEY countries and one for non-FVEY countries. So I guess they collect everything and the distinction is only here because theres different legal boilerplate for the analysts to follow.
- makomk 12y agoThere's a nice bit in here that automatically collates a list of Tor bridge nodes from snooped e-mails. The full list of bridge nodes isn't public, and one of the ways the Tor project attempted to prevent someone from building a complete list was by requiring people to use a valid GMail address to request them, effectively piggy-backing on Google's account verification to stop people from using a swathe of fake accounts to request nodes. Unfortunately, that failed to take account of the fact that the NSA had completely compromised Google's internal network.
- sp332 12y agoActually it was Britain's GCHQ that tapped Google's datacenter links and shared the data with the NSA. I only mention this to remind peeople that it's more than just one country and agency that's doing this.
- mschuster91 12y agoThe problem I have with this selector source code is that it is incredibly complex to execute. How is the NSA able to do this in realtime for all their interception points?!
- AlyssaRowan 12y agoXKeyScore runs nearline, not in-line - and it's distributed.
- dthunt 12y agoIt probably would have been a lot smarter to post this somewhere inside the continental US.
- deleted 12y ago[deleted]
- coderholic 12y agoHere are details for all of the IPs in that doc: $ curl -s http://daserste.ndr.de/panorama/xkeyscorerules100.txt | grep -Eo "([0-9]+\.?){4}" | xargs -I% curl -s http://ipinfo.io/% { "ip": "193.23.244.244", "hostname": "No Hostname", "city": null, "region": null, "country": "DE", "loc": "51.0000,9.0000", "org": "AS50472 Chaos Computer Club e.V." }{ "ip": "194.109.206.212", "hostname": "tor.dizum.com", "city": null, "region": null, "country": "NL", "loc": "52.5000,5.7500", "org": "AS3265 XS4ALL Internet BV" }{ "ip": "86.59.21.38", "hostname": "No Hostname", "city": null, "region": null, "country": "AT", "loc": "47.3333,13.3333", "org": "AS3248 Tele2 Telecommunication GmbH" }{ "ip": "213.115.239.118", "hostname": "No Hostname", "city": null, "region": null, "country": "SE", "loc": "62.0000,15.0000", "org": "AS2119 Telenor Norge AS" }{ "ip": "212.112.245.170", "hostname": "No Hostname", "city": null, "region": null, "country": "DE", "loc": "51.0000,9.0000", "org": "AS24900 QSC AG" }{ "ip": "128.31.0.39", "hostname": "belegost.csail.mit.edu", "city": "Cambridge", "region": "Massachusetts", "country": "US", "loc": "42.3646,-71.1028", "org": "AS3 Massachusetts Institute of Technology", "postal": "02139" }{ "ip": "216.224.124.114", "hostname": "No Hostname", "city": "Aptos", "region": "California", "country": "US", "loc": "37.0082,-121.8777", "org": "AS40231 Ethr.Net LLC", "postal": "95003" }{ "ip": "208.83.223.34", "hostname": "No Hostname", "city": "San Francisco", "region": "California", "country": "US", "loc": "37.7749,-122.4194", "org": "AS40475 Applied Operations, LLC", "postal": "94159" }{ "ip": "128.31.0.34", "hostname": "moria.csail.mit.edu", "city": "Cambridge", "region": "Massachusetts", "country": "US", "loc": "42.3646,-71.1028", "org": "AS3 Massachusetts Institute of Technology", "postal": "02139" }
- maximumoverload 12y agoOK, I am going to play a devil's advocate. - The job of three-letters agencies is to find out all various enemies of the state. That includes terrorists, but also various gangsters, officials of sort-of-enemy states (like Russia). That's why they exist. - Following US nation interests has higher priority than rights of citizens, especially in other countries. - People, that US have a reason to spy on, probably will use Tor/Tails, or at least try to find something about it. It makes sense for NSA to filter those people and focus their spying on them specially. - Not all Tor interested folks will be evil, but the percentage there will be much higher than in just random internet. So it makes sense to focus on them. Just like it makes sense for a local police to be in a neighbourhood, that's known for a higher criminality. So I understand why NSA does this, and why do they single out Tor-interested folks.
- chroem 12y ago>The job of three-letters agencies is to find out all various enemies of the state. Within the confines of the law and US constitution. If all you care about is rooting out enemies of the state, then you're left with organizations like the SS or Stasi.
- maximumoverload 12y agoDoes the US constitution matter with respect to non-US entities though? Why should US-based agency care about rights of German citizens, for example?
- chroem 12y agoWhile I think it should apply to non-US citizens, the fact that it technically doesn't is irrelevant as long as they're still willfully violating the rights of US citizens too.
- chippy 12y agoThe job of a government is towards it's country. A country is also made up of multinational companies and corporations. Modern say spying is commercial and industrial in nature also... it's not just used to catch bad guys, it's used to get an advantage. Whilst industrial spying does not trigger emotions as much and infringe their own citizens rights as much, it's arguably an even worse applications of our intelligence agencies.
- philip1209 12y agoSo the score may not be IPv6 compatible?
- xwintermutex 12y agoThe file is dissected here: http://blog.erratasec.com/2014/07/reading-xkeyscore-rules-source.html http://blog.erratasec.com/2014/07/reading-xkeyscore-rules-so...
- DarkIye 12y agolol, nobody cares. flails arms in hysteria
- andy_ppp 12y agoPresumably reading hacker news also puts you on one of their lists. Just remember should there ever be a problem between us, we know everything about you.