10 ms·

The Surveillance Engine: How the NSA Built Its Own Secret Google

356 points by aburan28 12y ago

us0r 12y ago
Somewhat OT but its amazing out of 3 "FTEs" 1 was for design ("GUI") and this is what they came up with. [1] pg 22 https://firstlook.org/theintercept/document/2014/08/25/sharing-communications-metadata-across-u-s-intelligence-community https://firstlook.org/theintercept/document/2014/08/25/shari...
- seiji 12y ago
  Also notice the entire screen shot. Netscape on Windows XP in, best we can tell, sometime after March 2006. Netscape. With an AIM launcher in the status bar. Accessing TOP SECRET//COMINT//NOFORN resources.
  - deleted 12y ago
    [deleted]
  - apsec112 12y ago
    This looks like Netscape 7 (http://en.wikipedia.org/wiki/Netscape_(version_7)#mediaviewer/File:Netscape_7_En.PNG http://en.wikipedia.org/wiki/Netscape_(version_7)#mediaviewe...), which had its final release in August 2004, not the original Netscape. Would it have been more secure if they used IE 6?
  - C7E0F338E42448 12y ago
    Maybe all software released after this is infested with spyware and exploits, and we are just led to believe it's the other way around
  - Amezarak 12y ago
    It would be on a totally isolated network. Classified networks don't touch unclassified networks.
    - seiji 12y ago
      Except by way of überadmin USB drives?
MalcolmDiggs 12y ago
It's kind of remarkable that they kept the lid on this on long as they did. With so many local/loosely-related agencies involved, I'm surprised somebody didn't leak this much sooner.
- danvayn 12y ago
  My guess is a lack of understanding.
- d1str0 12y ago
  It clearly says only a few intelligence officers from the different agencies had access. So it's not like any ol' police officer of FBI agent had access to this at all times.
  - chiph 12y ago
    Or maybe they did, and the search page had been rebranded so the officers didn't know it was operated by the NSA. Perhaps "Fusion Center Search 9000" or similar.
- Jimbotron 12y ago
  Yes, I find it VERY hard to believe that this went on and nobody mentioned it at all. Not one disgruntled employee, not anyone mentioning it to a friend at a bar? No way.
  - sentenza 12y ago
    Uhm. Just because you didn't believe it doesn't mean that it wasn't known. The CCC was talking about this stuff years ago, some of it conjecture, some of it information from old sources/anonymous sources and the like. Now we have proof. So that's that. Also, given the strong "disincentivization" for whistleblowing that is going on at intelligence agencies, I am not at all surprised that it took a few years before somebody went public with it. Hell, it used to take decades before this kind of stuff became public. Just this year we were made aware that the West German spy agencies opened _all_ cross-border inter-German mail. You know, the kind of thing that we in West Germany were told only the East Germans would do. That was kept secret for more than 20 years after it stopped being done, even though thousands of people were involved.
eyeareque 12y ago
People always assumed they did this but now we have proof. I thought it was amazing that the government spent so much time discussing the call records being logged.. when they are doing so much worse. Maybe that's how they keep people focusing on what the government wants to talk about? (aka look over here, nevermind that thing over there...) This find is way worse than call detail records..
- higherpurpose 12y ago
  It was pretty obvious that's why they were doing it. The FISA Amendments Act is also much worse than the 215 section of the Patriot Act - or the Executive Order 12333, which is even less talked about. That doesn't mean the call records issue wasn't important. After all, they pick their torture and drone assassination targets based on that alone, in many cases. But considering the Internet is such a big part of everyone's lives these days, I think NSA getting a firehose from the big carriers and cable providers is a bigger deal, yet I've heard very little talk about denying NSA access to the undersea cables.
- knowaveragejoe 12y ago
  I think it's a combination of things. On the one hand, the phone records issue is the most easily understood by the general public, and so it already had traction there. It sort of fell into the government's lap that they could focus all of their press' attention on that issue in particular. I agree doubly so however - the phone records are near the very bottom of the list when it comes to the severity of what Snowden's(and possibly other's) revelations have brought to light.
- deleted 12y ago
  [deleted]
wyck 12y ago
Brought to you by meta data commodities and pattern recognition limited - don't be evil. just be data.
beartime 12y ago
wouldn't we rather have more efficient tools like this than hacky-sketchy tools that can go too far?
deleted 12y ago
[deleted]
nowarninglabel 12y ago
I don't think it remained "under the radar" like some are commenting. There's mention of it in Snowden's disclosures and if you search around, lots of government recruiting related info, e.g., http://www.socnet.com/archive/index.php/t-108034.html http://www.socnet.com/archive/index.php/t-108034.html
eli 12y ago
I guess it's always cool to learn about top secret stuff that spies do, but I don't get why the specifics of how their search engine works is significant. Am I just missing something?
- IBM 12y ago
  Glenn Greenwald got a treasure trove and it's time to cash in.
  - yen223 12y ago
    Glenn Greenwald has a lot of Powerpoint slides. Am I the only one who wants to see the source code for ICREACH, and all the other ALLCAPS projects?
    - scintill76 12y ago
      I wonder if that's a line even Snowden wouldn't cross. It risks those tools falling into the "wrong hands", and is the most extreme way he could be said to be aiding terrorists by alerting them of our intelligence capabilities (which he is already accused of.)
      - HoochTHX 12y ago
        Its not new news that Google was contracted to build the search engine[1]. "Google was also contracted to provide the search technology and servers which makes up “Intellipedia”, a massive, incredibly secure database through which over 100,000 United States spies and intelligence professionals from 16 government agencies share information. And the NSA has purchased numerous servers from Google, utilizing their search technology to analyze and organize massive amounts of secure data all over the world." At this point I think it safe to assume its the exact same code Google used in 2010, just a bit dumber, since Googles search engine only learns by being used, and since there are far fewer using it, it learns slower. [1]http://questioneverything2012.wordpress.com/2012/03/14/googles-connection-to-intelligence-agencies-exposed-darpa-director-becomes-senior-exec/ http://questioneverything2012.wordpress.com/2012/03/14/googl...
curiousDog 12y ago
Wonder how and where they recruit their top-talent. Pretty much every top scorer in my school went to Facebook/Google/Msft/Amzn. The mediocre ones went to Northrop Grumman/Raytheon/Rockwell and the like.
- anoothrowaway 12y ago
  My experience in school (and grad school) was similar, so I've been curious about the same thing. Perhaps, they only recruit folks from the industry after they've worked for a while and have specialized?
  - visarga 12y ago
    Maybe they recruit people while they are employed at FB/GG/Msft/Amz, you know, to help with national security while also receiving a second wage.
    - junto 12y ago
      I imagine that you'll get downvoted to hell for saying that, but if I was the NSA I'd do exactly this. I'd also threaten you if you refused to comply. I would already know exactly which employee to target based on your dubious interests gleaned from your internet searches.
      - cmdkeen 12y ago
        And not a single one of them has spoken up about being coerced into compromising their employer (a big one, with lots of lawyers) by the government?
        junto 12y ago
        If someone has been approached by a powerful government agency that threatens them with embarrassing information concerning their private life, would they volunteer to share that coercion in public (and be believed)? You'd have to have pretty big balls to fuck with a three letter agency like that.
pdkl95 12y ago
What I want to know is if this search tool is the backend for the "parallel construction" application forms[1] from earlier this year? [1] https://www.techdirt.com/articles/20140203/11143926078/parallel-construction-revealed-how-dea-is-trained-to-launder-classified-surveillance-info.shtml https://www.techdirt.com/articles/20140203/11143926078/paral...
- atmosx 12y ago
  Possibly, among other things.
niels_olson 12y ago
Is it just me, or is it tragicomical that I can't access firstlook from a government network?
- Tangokat 12y ago
  According to the Intercept, the U.S Military has banned employees from visiting First Look because there could be classified information on the site. [1] [2] (extra source since you can't see the real one..) Maybe that is what you are experiencing? [1] https://firstlook.org/theintercept/2014/08/20/u-s-military-bans-the-intercept/ https://firstlook.org/theintercept/2014/08/20/u-s-military-b... [2] https://thebulletwire.com/u-s-military-bans-glenn-greenwalds-intercept/ https://thebulletwire.com/u-s-military-bans-glenn-greenwalds...
  - atmosx 12y ago
    That's similar to the wikileaks ban. I don't understand this policy: If something is accessible virtually from everywhere, what do you gain by not allowing your employees to access the info?! The only thing I can think of, is internal issues at political level (who said what lies to whom to achieve more funding about his work, etc.) which even that won't take long before the interested party eventually finds out.
    - dublinben 12y ago
      It perpetuates the mythology of classification and information control. If you have clearance, you can lose it by accessing classified information improperly. That would probably mean losing your job. If you don't have clearance, you should be considering all the information you can get your hands on. Not doing so, would be poorly doing your job.
      - atmosx 12y ago
        nicely explained, thanks :-)
    - mpyne 12y ago
Twirrim 12y ago
I think I'm more surprised that this was a surprise to people. Of course they built a search engine. Wouldn't you? Don't you have similar at your workplace? We use them all the time. Think about web interfaces built on top of ElasticSearch, for example. Is that not a 'search engine'?
- dredmorbius 12y ago
  It's not the building a search engine part. It's the opening it up to LEO. Taking homeland security-justified warrantless searches and turning those over to criminal investigations is far beyond unconstitutional.
  - mpyne 12y ago
    > Taking homeland security-justified warrantless searches and turning those over to criminal investigations is far beyond unconstitutional. Except, that it's not. It's all about how the government comes into possession of information that makes it the fruit (or not) of the "poisonous tree". Once the government legally learns of some bit of information, there's no requirement by itself that they have to "pretend not to know" something is going on. That's why a third party voluntarily divulging information to the government doesn't cause that evidence to get thrown out of court: There may be a cause for civil or even criminal action against that third party, but that doesn't invalidate the evidence. Now, Congress has passed statute laws limiting information sharing between intelligence agencies and law enforcement precisely because of the threat of having this all-seeing eye subvert democratic government, but the limits were never complete exclusions, even before 9/11. And the reason statute laws were needed is because there was no barrier to this activity from the judicial side alone. Parallel construction came in not because evidence was illegally gathered, but because giving the (fully legal!) chain of custody for evidence derived from intelligence sources would have quite naturally have "burned" that source or method, so the NSA would require law enforcement agencies using that data to use alternate (again though, legal) means to make a case in court, in order to protect their intelligence source. It's the old apocryphal dilemma about whether to let Coventry be bombed to protect ULTRA, applied to the post-9/11 world.
cryptolect 12y ago
To me, this explains why the five eyes nations are pushing for (meta)data retention legislation. It's a condition of participating in the scheme.
DigitalSea 12y ago
Imagine if hackers were to somehow to find out where this tool is being hosted and then brute force their way in? Presumably such a tool is heavily secured and off the grid, but if it is somehow accessible from the Internet and an attacker were to find out where, it's only a matter of time now that the cat is out of the bag. Could you imagine hackers having access to troves of metadata and information like that? It's a scary thought. I'm surprised such a thing took so long to be revealed. If you've got as much data as the NSA has, wouldn't you want a Google like search engine to be able to search through it? It makes so much sense which is why I am surprised some people are surprised about this.
- ChrisAntaki 12y ago
  Furthermore, think of how profitable surveillance on certain individuals could be. Could a military contractor not have a thriving side business funneling out intelligence on innocent American targets? Could multiple people not be hired just to clear tracks and make justifiable excuses for accessing the records? If Snowden, as one person, accomplished what he did with (the majority of Americans would say) good intentions, imagine what a team of people, who were just as smart if not smarter, could do if they didn't have morals guiding their actions. Imagine how well a team could cover their tracks, dot all the I's, and cross all the T's, compared to just one guy. This is all common sense, but it bears repeating.
  - smokeyj 12y ago
    I'd just play the market. Find dirt on CEOs, short their stock and pass the dirt to the media.
BorisMelnik 12y ago
I am just not surprised about this at all. They have more information than I could ever imagine, wouldn't any rational person assume there is a search engine to index, sort, parse, and return results? seems like a lot until you consider how many indexed pages Google has: http://i.imgur.com/EqIJAoL.jpg http://i.imgur.com/EqIJAoL.jpg why not throw in grains of sand or atoms in the universe?
- Perseids 12y ago
  Nice comparison, though I hate it when graphics look like they were drawn to scale, but actually aren't. (The Google indexed pages bar would have to be more than 10 times larger.) More on topic: Whether you can compare the index sizes or not mostly depends on their signal to noise ration. Given the number of Google search request per month and assuming and average of 5 viewed results, you can put a lower bound of 99% of pages that are never seen by anyone. As the search results should greatly overlap the actual number of relevant sites is probably orders of magnitude less than 1%.
fit2rule 12y ago
I believe its really telling that, rather than take the tactic of fully promoting open society, the inclination is towards more and more secrecy. Like, dire, utter secrecy. Kill someone-style, secrecy. Imagine the other end of the scale - where in fact every detail about everyones lives is wide open and available for everyone and anyone to access. Willingly. Freely. A new order of celebrity: total telepathy. Do you think we'd be dealing with terrorism, then? Would there be the idealist, killing souls, for a little private time?
Dolimiter 12y ago
Is anyone else puzzled that stories about the NSA are accepted as gospel fact when posted on Hacker News? The Reality Distortion Field appears, and people believe, because they want to believe. It depresses me, the lack of intelligent discourse. Most tech people I meet actually believe that the NSA records and stores all telephone calls. It's depressingly stupid, but I have given up arguing, logic and sense are not welcome when the NSA is the topic.
- shiven 12y ago
  Perhaps proof to the contrary would be more useful than the veracity or logic of an argument? Unfortunately, with the NSA, independently verifiable proof of anything, especially doing the right thing, is impossible to attain.
  - mpyne 12y ago
    You're asking the parent commenter to prove a negative? Is this what logic has come to at HN?
    - just1comment677 12y ago
      Yes, ever since June 2013.
- glitchdout 12y ago
  They do record and store (foreign and domestic) phone calls. Your comment proves you're simply an ignorant or a shill. Proof: * NSA Collects ‘Word for Word’ Every Domestic Communication, Says Former Analyst http://www.pbs.org/newshour/bb/government_programs-july-dec13-whistleblowers_08-01/ http://www.pbs.org/newshour/bb/government_programs-july-dec1... * NSA collecting phone records of millions of Verizon customers daily http://www.theguardian.com/world/2013/jun/06/nsa-phone-records-verizon-court-order http://www.theguardian.com/world/2013/jun/06/nsa-phone-recor... * NSA surveillance program reaches ‘into the past’ to retrieve, replay phone calls http://www.washingtonpost.com/world/national-security/nsa-surveillance-program-reaches-into-the-past-to-retrieve-replay-phone-calls/2014/03/18/226d2646-ade9-11e3-a49e-76adc9210f19_story.html http://www.washingtonpost.com/world/national-security/nsa-su... * NSA spying flap extends to contents of U.S. phone calls http://www.cnet.com/news/nsa-spying-flap-extends-to-contents-of-u-s-phone-calls/ http://www.cnet.com/news/nsa-spying-flap-extends-to-contents...
DanielBMarkham 12y ago
This has been released and is use by almost two dozen federal agencies and now is the first time we're hearing about it? When folks tell us crazy things, like the government is tracking every place you go and your opinions through your cell phone and social networks, we're supposed to say something like "That's extraordinary. With extraordinary claims, we require extraordinary proof" Then, if they persist, we're supposed to say something like "Such a program would require far too many people to keep a secret. We couldn't even keep the atom bomb a secret. The government is terrible at keeping secrets. Such a claim is just too far-fetched." These are the traditional things taught to people who are supposed to be clear-headed and rational. It's the way we engage crackpots without taking them too seriously. These responses seem to have failed us miserably in the current circumstances. As it turns out, yes, that's what they were doing, and yes, it was extraordinary and required lots of people to keep incredible secrets. But it still happened. These things keep happening in the realm of automated surveillance, both by the government and corporations (and worse, when corps do it and the govt scoops it all up later) that would have been considered completely whacked just ten years ago. The stuff of paranoid fantasies. Our tools of rational inquiry have failed us.
- knowaveragejoe 12y ago
  Just curious, where is the bit about keeping track of your opinions? Surely it's been proposed(so has everything and the kitchen sink), but that doesn't mean it's effective or useful in any way.
- mikegioia 12y ago
  You're up against a growing population of people who inherently think that "the government is tracking every place you go and your opinions through your cell phone and social networks" is NOT crazy.
- Squarel 12y ago
  Tracking of information has been known about since the initial NSA whistleblowers after 9/11 I have not heard a single conspiracy theorist suggest any of the things disclosed by Snowden, and I have yet to see evidence of RFID chips and the government monitoring us through our microwaves, or that Person of Interest is based on a true story. Extraordinary claims still require extraordinary evidence, and that evidence usually comes from someone involved, not from a random blog/site on the internet, or a youtube channel. The one thing everyone misses with this, which is my main concern, is less that the entire NSA/Insert Alphabet Soup here can track me, but that any one individual within these organisations who decides they don't like you for any reason can majorly screw you over. It is not the whole that concerns me, it is the parts.
srj 12y ago
Maybe there was an earlier release I missed but this report looks to be the first to show the extent of the call record data being collected. SIM, IMEI, Lat/Lng and more. Logged for every American, every day, all since 2007. And now shared outside of the military context it was gathered and authorized under to civilian agencies. It's possible the program has been reigned in since these slides were authored but that seems unlikely.

356 points by aburan28 12y ago

us0r 12y ago

Somewhat OT but its amazing out of 3 "FTEs" 1 was for design ("GUI") and this is what they came up with. [1] pg 22 https://firstlook.org/theintercept/document/2014/08/25/sharing-communications-metadata-across-u-s-intelligence-community https://firstlook.org/theintercept/document/2014/08/25/shari...

seiji 12y ago
Also notice the entire screen shot. Netscape on Windows XP in, best we can tell, sometime after March 2006. Netscape. With an AIM launcher in the status bar. Accessing TOP SECRET//COMINT//NOFORN resources.
- deleted 12y ago
  [deleted]
- apsec112 12y ago
  This looks like Netscape 7 (http://en.wikipedia.org/wiki/Netscape_(version_7)#mediaviewer/File:Netscape_7_En.PNG http://en.wikipedia.org/wiki/Netscape_(version_7)#mediaviewe...), which had its final release in August 2004, not the original Netscape. Would it have been more secure if they used IE 6?
- C7E0F338E42448 12y ago
  Maybe all software released after this is infested with spyware and exploits, and we are just led to believe it's the other way around
- Amezarak 12y ago
  It would be on a totally isolated network. Classified networks don't touch unclassified networks.
  - seiji 12y ago
    Except by way of überadmin USB drives?

MalcolmDiggs 12y ago

It's kind of remarkable that they kept the lid on this on long as they did. With so many local/loosely-related agencies involved, I'm surprised somebody didn't leak this much sooner.

danvayn 12y ago
My guess is a lack of understanding.
d1str0 12y ago
It clearly says only a few intelligence officers from the different agencies had access. So it's not like any ol' police officer of FBI agent had access to this at all times.
- chiph 12y ago
  Or maybe they did, and the search page had been rebranded so the officers didn't know it was operated by the NSA. Perhaps "Fusion Center Search 9000" or similar.
Jimbotron 12y ago
Yes, I find it VERY hard to believe that this went on and nobody mentioned it at all. Not one disgruntled employee, not anyone mentioning it to a friend at a bar? No way.
- sentenza 12y ago
  Uhm. Just because you didn't believe it doesn't mean that it wasn't known. The CCC was talking about this stuff years ago, some of it conjecture, some of it information from old sources/anonymous sources and the like. Now we have proof. So that's that. Also, given the strong "disincentivization" for whistleblowing that is going on at intelligence agencies, I am not at all surprised that it took a few years before somebody went public with it. Hell, it used to take decades before this kind of stuff became public. Just this year we were made aware that the West German spy agencies opened _all_ cross-border inter-German mail. You know, the kind of thing that we in West Germany were told only the East Germans would do. That was kept secret for more than 20 years after it stopped being done, even though thousands of people were involved.

eyeareque 12y ago

People always assumed they did this but now we have proof. I thought it was amazing that the government spent so much time discussing the call records being logged.. when they are doing so much worse. Maybe that's how they keep people focusing on what the government wants to talk about? (aka look over here, nevermind that thing over there...) This find is way worse than call detail records..

higherpurpose 12y ago
It was pretty obvious that's why they were doing it. The FISA Amendments Act is also much worse than the 215 section of the Patriot Act - or the Executive Order 12333, which is even less talked about. That doesn't mean the call records issue wasn't important. After all, they pick their torture and drone assassination targets based on that alone, in many cases. But considering the Internet is such a big part of everyone's lives these days, I think NSA getting a firehose from the big carriers and cable providers is a bigger deal, yet I've heard very little talk about denying NSA access to the undersea cables.
knowaveragejoe 12y ago
I think it's a combination of things. On the one hand, the phone records issue is the most easily understood by the general public, and so it already had traction there. It sort of fell into the government's lap that they could focus all of their press' attention on that issue in particular. I agree doubly so however - the phone records are near the very bottom of the list when it comes to the severity of what Snowden's(and possibly other's) revelations have brought to light.
deleted 12y ago
[deleted]

wyck 12y ago

Brought to you by meta data commodities and pattern recognition limited - don't be evil. just be data.

beartime 12y ago

wouldn't we rather have more efficient tools like this than hacky-sketchy tools that can go too far?

deleted 12y ago

[deleted]

nowarninglabel 12y ago

I don't think it remained "under the radar" like some are commenting. There's mention of it in Snowden's disclosures and if you search around, lots of government recruiting related info, e.g., http://www.socnet.com/archive/index.php/t-108034.html http://www.socnet.com/archive/index.php/t-108034.html

eli 12y ago

I guess it's always cool to learn about top secret stuff that spies do, but I don't get why the specifics of how their search engine works is significant. Am I just missing something?

IBM 12y ago
Glenn Greenwald got a treasure trove and it's time to cash in.
- yen223 12y ago
  Glenn Greenwald has a lot of Powerpoint slides. Am I the only one who wants to see the source code for ICREACH, and all the other ALLCAPS projects?
  - scintill76 12y ago
    I wonder if that's a line even Snowden wouldn't cross. It risks those tools falling into the "wrong hands", and is the most extreme way he could be said to be aiding terrorists by alerting them of our intelligence capabilities (which he is already accused of.)
    - HoochTHX 12y ago
      Its not new news that Google was contracted to build the search engine[1]. "Google was also contracted to provide the search technology and servers which makes up “Intellipedia”, a massive, incredibly secure database through which over 100,000 United States spies and intelligence professionals from 16 government agencies share information. And the NSA has purchased numerous servers from Google, utilizing their search technology to analyze and organize massive amounts of secure data all over the world." At this point I think it safe to assume its the exact same code Google used in 2010, just a bit dumber, since Googles search engine only learns by being used, and since there are far fewer using it, it learns slower. [1]http://questioneverything2012.wordpress.com/2012/03/14/googles-connection-to-intelligence-agencies-exposed-darpa-director-becomes-senior-exec/ http://questioneverything2012.wordpress.com/2012/03/14/googl...

curiousDog 12y ago

Wonder how and where they recruit their top-talent. Pretty much every top scorer in my school went to Facebook/Google/Msft/Amzn. The mediocre ones went to Northrop Grumman/Raytheon/Rockwell and the like.

anoothrowaway 12y ago
My experience in school (and grad school) was similar, so I've been curious about the same thing. Perhaps, they only recruit folks from the industry after they've worked for a while and have specialized?
- visarga 12y ago
  Maybe they recruit people while they are employed at FB/GG/Msft/Amz, you know, to help with national security while also receiving a second wage.
  - junto 12y ago
    I imagine that you'll get downvoted to hell for saying that, but if I was the NSA I'd do exactly this. I'd also threaten you if you refused to comply. I would already know exactly which employee to target based on your dubious interests gleaned from your internet searches.
    - cmdkeen 12y ago
      And not a single one of them has spoken up about being coerced into compromising their employer (a big one, with lots of lawyers) by the government?
      - junto 12y ago
        If someone has been approached by a powerful government agency that threatens them with embarrassing information concerning their private life, would they volunteer to share that coercion in public (and be believed)? You'd have to have pretty big balls to fuck with a three letter agency like that.

pdkl95 12y ago

What I want to know is if this search tool is the backend for the "parallel construction" application forms[1] from earlier this year? [1] https://www.techdirt.com/articles/20140203/11143926078/parallel-construction-revealed-how-dea-is-trained-to-launder-classified-surveillance-info.shtml https://www.techdirt.com/articles/20140203/11143926078/paral...

atmosx 12y ago
Possibly, among other things.

niels_olson 12y ago

Is it just me, or is it tragicomical that I can't access firstlook from a government network?

Tangokat 12y ago
According to the Intercept, the U.S Military has banned employees from visiting First Look because there could be classified information on the site. [1] [2] (extra source since you can't see the real one..) Maybe that is what you are experiencing? [1] https://firstlook.org/theintercept/2014/08/20/u-s-military-bans-the-intercept/ https://firstlook.org/theintercept/2014/08/20/u-s-military-b... [2] https://thebulletwire.com/u-s-military-bans-glenn-greenwalds-intercept/ https://thebulletwire.com/u-s-military-bans-glenn-greenwalds...
- atmosx 12y ago
  That's similar to the wikileaks ban. I don't understand this policy: If something is accessible virtually from everywhere, what do you gain by not allowing your employees to access the info?! The only thing I can think of, is internal issues at political level (who said what lies to whom to achieve more funding about his work, etc.) which even that won't take long before the interested party eventually finds out.
  - dublinben 12y ago
    It perpetuates the mythology of classification and information control. If you have clearance, you can lose it by accessing classified information improperly. That would probably mean losing your job. If you don't have clearance, you should be considering all the information you can get your hands on. Not doing so, would be poorly doing your job.
    - atmosx 12y ago
      nicely explained, thanks :-)
  - mpyne 12y ago

Twirrim 12y ago

I think I'm more surprised that this was a surprise to people. Of course they built a search engine. Wouldn't you? Don't you have similar at your workplace? We use them all the time. Think about web interfaces built on top of ElasticSearch, for example. Is that not a 'search engine'?

dredmorbius 12y ago
It's not the building a search engine part. It's the opening it up to LEO. Taking homeland security-justified warrantless searches and turning those over to criminal investigations is far beyond unconstitutional.
- mpyne 12y ago
  > Taking homeland security-justified warrantless searches and turning those over to criminal investigations is far beyond unconstitutional. Except, that it's not. It's all about how the government comes into possession of information that makes it the fruit (or not) of the "poisonous tree". Once the government legally learns of some bit of information, there's no requirement by itself that they have to "pretend not to know" something is going on. That's why a third party voluntarily divulging information to the government doesn't cause that evidence to get thrown out of court: There may be a cause for civil or even criminal action against that third party, but that doesn't invalidate the evidence. Now, Congress has passed statute laws limiting information sharing between intelligence agencies and law enforcement precisely because of the threat of having this all-seeing eye subvert democratic government, but the limits were never complete exclusions, even before 9/11. And the reason statute laws were needed is because there was no barrier to this activity from the judicial side alone. Parallel construction came in not because evidence was illegally gathered, but because giving the (fully legal!) chain of custody for evidence derived from intelligence sources would have quite naturally have "burned" that source or method, so the NSA would require law enforcement agencies using that data to use alternate (again though, legal) means to make a case in court, in order to protect their intelligence source. It's the old apocryphal dilemma about whether to let Coventry be bombed to protect ULTRA, applied to the post-9/11 world.

cryptolect 12y ago

To me, this explains why the five eyes nations are pushing for (meta)data retention legislation. It's a condition of participating in the scheme.

DigitalSea 12y ago

Imagine if hackers were to somehow to find out where this tool is being hosted and then brute force their way in? Presumably such a tool is heavily secured and off the grid, but if it is somehow accessible from the Internet and an attacker were to find out where, it's only a matter of time now that the cat is out of the bag. Could you imagine hackers having access to troves of metadata and information like that? It's a scary thought. I'm surprised such a thing took so long to be revealed. If you've got as much data as the NSA has, wouldn't you want a Google like search engine to be able to search through it? It makes so much sense which is why I am surprised some people are surprised about this.

ChrisAntaki 12y ago
Furthermore, think of how profitable surveillance on certain individuals could be. Could a military contractor not have a thriving side business funneling out intelligence on innocent American targets? Could multiple people not be hired just to clear tracks and make justifiable excuses for accessing the records? If Snowden, as one person, accomplished what he did with (the majority of Americans would say) good intentions, imagine what a team of people, who were just as smart if not smarter, could do if they didn't have morals guiding their actions. Imagine how well a team could cover their tracks, dot all the I's, and cross all the T's, compared to just one guy. This is all common sense, but it bears repeating.
- smokeyj 12y ago
  I'd just play the market. Find dirt on CEOs, short their stock and pass the dirt to the media.

BorisMelnik 12y ago

I am just not surprised about this at all. They have more information than I could ever imagine, wouldn't any rational person assume there is a search engine to index, sort, parse, and return results? seems like a lot until you consider how many indexed pages Google has: http://i.imgur.com/EqIJAoL.jpg http://i.imgur.com/EqIJAoL.jpg why not throw in grains of sand or atoms in the universe?

Perseids 12y ago
Nice comparison, though I hate it when graphics look like they were drawn to scale, but actually aren't. (The Google indexed pages bar would have to be more than 10 times larger.) More on topic: Whether you can compare the index sizes or not mostly depends on their signal to noise ration. Given the number of Google search request per month and assuming and average of 5 viewed results, you can put a lower bound of 99% of pages that are never seen by anyone. As the search results should greatly overlap the actual number of relevant sites is probably orders of magnitude less than 1%.

fit2rule 12y ago

I believe its really telling that, rather than take the tactic of fully promoting open society, the inclination is towards more and more secrecy. Like, dire, utter secrecy. Kill someone-style, secrecy. Imagine the other end of the scale - where in fact every detail about everyones lives is wide open and available for everyone and anyone to access. Willingly. Freely. A new order of celebrity: total telepathy. Do you think we'd be dealing with terrorism, then? Would there be the idealist, killing souls, for a little private time?

Dolimiter 12y ago

Is anyone else puzzled that stories about the NSA are accepted as gospel fact when posted on Hacker News? The Reality Distortion Field appears, and people believe, because they want to believe. It depresses me, the lack of intelligent discourse. Most tech people I meet actually believe that the NSA records and stores all telephone calls. It's depressingly stupid, but I have given up arguing, logic and sense are not welcome when the NSA is the topic.

shiven 12y ago
Perhaps proof to the contrary would be more useful than the veracity or logic of an argument? Unfortunately, with the NSA, independently verifiable proof of anything, especially doing the right thing, is impossible to attain.
- mpyne 12y ago
  You're asking the parent commenter to prove a negative? Is this what logic has come to at HN?
  - just1comment677 12y ago
    Yes, ever since June 2013.
glitchdout 12y ago
They do record and store (foreign and domestic) phone calls. Your comment proves you're simply an ignorant or a shill. Proof: * NSA Collects ‘Word for Word’ Every Domestic Communication, Says Former Analyst http://www.pbs.org/newshour/bb/government_programs-july-dec13-whistleblowers_08-01/ http://www.pbs.org/newshour/bb/government_programs-july-dec1... * NSA collecting phone records of millions of Verizon customers daily http://www.theguardian.com/world/2013/jun/06/nsa-phone-records-verizon-court-order http://www.theguardian.com/world/2013/jun/06/nsa-phone-recor... * NSA surveillance program reaches ‘into the past’ to retrieve, replay phone calls http://www.washingtonpost.com/world/national-security/nsa-surveillance-program-reaches-into-the-past-to-retrieve-replay-phone-calls/2014/03/18/226d2646-ade9-11e3-a49e-76adc9210f19_story.html http://www.washingtonpost.com/world/national-security/nsa-su... * NSA spying flap extends to contents of U.S. phone calls http://www.cnet.com/news/nsa-spying-flap-extends-to-contents-of-u-s-phone-calls/ http://www.cnet.com/news/nsa-spying-flap-extends-to-contents...

DanielBMarkham 12y ago

This has been released and is use by almost two dozen federal agencies and now is the first time we're hearing about it? When folks tell us crazy things, like the government is tracking every place you go and your opinions through your cell phone and social networks, we're supposed to say something like "That's extraordinary. With extraordinary claims, we require extraordinary proof" Then, if they persist, we're supposed to say something like "Such a program would require far too many people to keep a secret. We couldn't even keep the atom bomb a secret. The government is terrible at keeping secrets. Such a claim is just too far-fetched." These are the traditional things taught to people who are supposed to be clear-headed and rational. It's the way we engage crackpots without taking them too seriously. These responses seem to have failed us miserably in the current circumstances. As it turns out, yes, that's what they were doing, and yes, it was extraordinary and required lots of people to keep incredible secrets. But it still happened. These things keep happening in the realm of automated surveillance, both by the government and corporations (and worse, when corps do it and the govt scoops it all up later) that would have been considered completely whacked just ten years ago. The stuff of paranoid fantasies. Our tools of rational inquiry have failed us.

knowaveragejoe 12y ago
Just curious, where is the bit about keeping track of your opinions? Surely it's been proposed(so has everything and the kitchen sink), but that doesn't mean it's effective or useful in any way.
mikegioia 12y ago
You're up against a growing population of people who inherently think that "the government is tracking every place you go and your opinions through your cell phone and social networks" is NOT crazy.
Squarel 12y ago
Tracking of information has been known about since the initial NSA whistleblowers after 9/11 I have not heard a single conspiracy theorist suggest any of the things disclosed by Snowden, and I have yet to see evidence of RFID chips and the government monitoring us through our microwaves, or that Person of Interest is based on a true story. Extraordinary claims still require extraordinary evidence, and that evidence usually comes from someone involved, not from a random blog/site on the internet, or a youtube channel. The one thing everyone misses with this, which is my main concern, is less that the entire NSA/Insert Alphabet Soup here can track me, but that any one individual within these organisations who decides they don't like you for any reason can majorly screw you over. It is not the whole that concerns me, it is the parts.

srj 12y ago

Maybe there was an earlier release I missed but this report looks to be the first to show the extent of the call record data being collected. SIM, IMEI, Lat/Lng and more. Logged for every American, every day, all since 2007. And now shared outside of the military context it was gathered and authorized under to civilian agencies. It's possible the program has been reigned in since these slides were authored but that seems unlikely.