5 ms·

If it didn't, the NSA would not be so interested in collecting it. Paranoid people who believe they might be being listened in on are unlikely to reveal much di

If it didn't, the NSA would not be so interested in collecting it. Paranoid people who believe they might be being listened in on are unlikely to reveal much directly in the conversation itself anyway, so in those cases the metadata is more important. Also, the metadata can reveal anomalous behavior, which they look for mainly because it's easy to find, but also because it can reveal important information assuming the targets are correctly selected. Anyway, the only reason they aren't collecting the calls themselves is because the storage required is not yet available, so their begging off about "but we aren't storing the content" is disingenuous. They can, at any moment, capture any call going over the long-distance network (which would include pretty much all cell phone calls) so the only thing they're unable to do is to retroactively listen in on calls. If you are flagged for whatever reason (you know someone who knows someone who knows someone they suspect of sending money to a bad charity), you may well be being monitored.

slashdotaccount 13y ago
The amount of storage required for storing all of the call recordings (GSM, VoIP, land-line) are currently available. For example, Speex [1] can compress voice even in 2kbps. So storing everything e.g. in 8kbps you can store 916259 hours (104 years) of voice in just one 3TB disk. [1] http://speex.org/ http://speex.org/
- Cthulhu_ 13y ago
  Let's take the US. 317 million people, assuming they call for an average of 10 minutes (based on nothing whatsoever, btw), gives approx. (10317) / 60 = 53 million hours of phone conversations a day. Given 916.259 hours = 3 TB, 53M / 916.259 = 57 3 TB = 172 TB of data /a day/. And that's just the US. Even if you adjust the average to just one minute a day, you're still looking at 17 TB / day, which should be sorta manageable I reckon. But it's not just the US. Let's assume they want to track all voice communications globally, rounding it to an even 7 billion, ten minutes a day. I'm counting 3819 TB/day. That's a lotta 3 TB hard drives. tl;dr: big data is big. disclaimer: I suck at basic arithmetic, I probably made a miscalculation.
  - slashdotaccount 13y ago
    I think they can handle it in Utah: "An article by Forbes estimates the storage capacity as between 3 and 12 exabytes in the near term" [1] [1] https://en.wikipedia.org/wiki/Utah_Data_Center https://en.wikipedia.org/wiki/Utah_Data_Center
    - arg01 13y ago
      Which gives you between 2-10 years of storage if it was recording everyone in the world. (Based on the assumptions above).
  - dictum 13y ago
    ~200TB/day is entirely within the room the NSA's budget allows. You don't need to record all phones in the world if you have the metadata of all calls in the world. The NSA or another spy agency could record only calls that match a given pattern: place from which the call originates, who is being called, time of call, previous calls made or received by the line, when the line or phone was purchased, whether this phone's call patterns resemble another phone's call patterns, etc. They wouldn't achieve 100% coverage, but efficacy would probably be 99%+. I would guess the problem with this kind of semi-targeted collection is processing power to decide who is a target and schedule the line taps.
    - gknoy 13y ago
      Or, record all of it that you can into a daily or weekly cache, and then keep in indefinite (expensive) storage the things which are Statistically Interesting but outside our current budget/capabilities to store forever.
      - dictum 13y ago
        Yes, good point. You don't have to store everything forever, you can have tiers of interestingness — capture everything, read it looking for certain patterns, store anything that matches those patterns forever, store stuff that matches [secondary, less important but still useful pattern] for a few years, and store all other calls for a few months or a year. Phone calls are quite low quality audio, but I don't expect the NSA to be limited to consumer grade text-to-speech technology, so at least for calls in some languages, they could store the transcripts forever. EDIT: Apart from processing power, another expensive problem with such a setup is memory to store the firehose temporarily. EDIT 2: If you were wondering, 200TB/day would run at $7500 for 50 4TB external hard drives at $150 each, assuming you wanted to use a Backblaze-like setup. In a year, that's $2.7 million. (This doesn't account for redundancy.)
- parkaboy 13y ago
  You can actually do much better than 2 kbps. MELP-E (unfortunately patented), actually can do 600 bps (and is fairly robust to noise I might add). With the advent of great speech-to-text ML, you could actually reduce this even further, to the 100-200 bps range).
dTal 13y ago
Are you quite sure they aren't collecting content? The storage is feasible, and several remarks (slips?) by people in a position to know suggest that they are in fact doing this, if not to everyone, then to hundreds of thousands or even millions of people. http://blog.rubbingalcoholic.com/post/52913031241/its-not-just-metadata-the-nsa-is-getting-everything http://blog.rubbingalcoholic.com/post/52913031241/its-not-ju... https://www.schneier.com/blog/archives/2013/06/evidence_that_t.html https://www.schneier.com/blog/archives/2013/06/evidence_that...
- mpyne 13y ago
  > Are you quite sure they aren't collecting content? It seems to depend entirely on the program in question, and the legal authorities for each program. For foreign intelligence actually collected overseas under EO 12333 they have programs that collect content (e.g. SMS), which have existed in one form or another since the beginnings of the Cold War. For foreign intelligence conducted on U.S. soil they have to fall within the boundaries of Fourth Amendment reasonable search (and related legal rulings like Smith v. Maryland) so they wouldn't capture content. But on the other hand they can still capture content in targeted fashion against non-U.S. persons in certain scenarios, and that content might itself involve conversations involving a U.S. person and still be a legal search.
  - rhizome 13y ago
    It seems to depend entirely on the program in question Accessibility does, but it has nothing to do with the agency. "Does the government store the content of US calls?" can be answered in the aggregate without reference to specific programs. If the content is there, an agency only has to acquire the authorities required to access what is already there. So, the question remains: is the content there to access, given proper authorities?
    - mpyne 13y ago
      There's multiple levels of authorities though. Collecting the calls at all would itself require some legal authority. Searching the calls you collected would then need yet another legal authority.
fit2rule 13y ago
I think you need to pay a little more attention: http://www.youtube.com/watch?feature=player_detailpage&v=vILAlhwUgIU#t=1717 http://www.youtube.com/watch?feature=player_detailpage&v=vIL... The ARE storing everything. 15 years worth.
bostik 13y ago
> If it didn't, the NSA would not be so interested in collecting it. I'm going to quote Schneier. Stop calling it metadata. Call it what cryptographers and security professionals have called it since forever: traffic analysis. Traffic analysis is a powerful tool by itself. Combined with practically unlimited access to source material, and the ability to unmask almost every communication node, you don't even NEED to care about the contents. Timing, frequency and direction of communication is more than enough.
glitchdout 13y ago
They are already storing the phone calls... * http://www.pbs.org/newshour/bb/government_programs-july-dec13-whistleblowers_08-01/ http://www.pbs.org/newshour/bb/government_programs-july-dec1... * http://www.theguardian.com/commentisfree/2013/may/04/telephone-calls-recorded-fbi-boston http://www.theguardian.com/commentisfree/2013/may/04/telepho... I'll quote the slip up of Tim Clemente, a former FBI agent: > All digital communications are uh uh... There's a way to look at digital communications in the past. And I can't go into detail of how that's done or what's done but I can tell you that no digital communication is secure.
- legutierr 13y ago
  And even if they are not storing the audio recordings, wouldn't it be reasonable to expect that automatic transcription is run against the audio for search indexing and long-term storage purposes? Text transcriptions like that, even if they are imperfect, can preserve conversations with minimal storage requirements almost indefinitely.

slashdotaccount 13y ago

The amount of storage required for storing all of the call recordings (GSM, VoIP, land-line) are currently available. For example, Speex [1] can compress voice even in 2kbps. So storing everything e.g. in 8kbps you can store 916259 hours (104 years) of voice in just one 3TB disk. [1] http://speex.org/ http://speex.org/

Cthulhu_ 13y ago
Let's take the US. 317 million people, assuming they call for an average of 10 minutes (based on nothing whatsoever, btw), gives approx. (10317) / 60 = 53 million hours of phone conversations a day. Given 916.259 hours = 3 TB, 53M / 916.259 = 57 3 TB = 172 TB of data /a day/. And that's just the US. Even if you adjust the average to just one minute a day, you're still looking at 17 TB / day, which should be sorta manageable I reckon. But it's not just the US. Let's assume they want to track all voice communications globally, rounding it to an even 7 billion, ten minutes a day. I'm counting 3819 TB/day. That's a lotta 3 TB hard drives. tl;dr: big data is big. disclaimer: I suck at basic arithmetic, I probably made a miscalculation.
- slashdotaccount 13y ago
  I think they can handle it in Utah: "An article by Forbes estimates the storage capacity as between 3 and 12 exabytes in the near term" [1] [1] https://en.wikipedia.org/wiki/Utah_Data_Center https://en.wikipedia.org/wiki/Utah_Data_Center
  - arg01 13y ago
    Which gives you between 2-10 years of storage if it was recording everyone in the world. (Based on the assumptions above).
- dictum 13y ago
  ~200TB/day is entirely within the room the NSA's budget allows. You don't need to record all phones in the world if you have the metadata of all calls in the world. The NSA or another spy agency could record only calls that match a given pattern: place from which the call originates, who is being called, time of call, previous calls made or received by the line, when the line or phone was purchased, whether this phone's call patterns resemble another phone's call patterns, etc. They wouldn't achieve 100% coverage, but efficacy would probably be 99%+. I would guess the problem with this kind of semi-targeted collection is processing power to decide who is a target and schedule the line taps.
  - gknoy 13y ago
    Or, record all of it that you can into a daily or weekly cache, and then keep in indefinite (expensive) storage the things which are Statistically Interesting but outside our current budget/capabilities to store forever.
    - dictum 13y ago
      Yes, good point. You don't have to store everything forever, you can have tiers of interestingness — capture everything, read it looking for certain patterns, store anything that matches those patterns forever, store stuff that matches [secondary, less important but still useful pattern] for a few years, and store all other calls for a few months or a year. Phone calls are quite low quality audio, but I don't expect the NSA to be limited to consumer grade text-to-speech technology, so at least for calls in some languages, they could store the transcripts forever. EDIT: Apart from processing power, another expensive problem with such a setup is memory to store the firehose temporarily. EDIT 2: If you were wondering, 200TB/day would run at $7500 for 50 4TB external hard drives at $150 each, assuming you wanted to use a Backblaze-like setup. In a year, that's $2.7 million. (This doesn't account for redundancy.)
parkaboy 13y ago
You can actually do much better than 2 kbps. MELP-E (unfortunately patented), actually can do 600 bps (and is fairly robust to noise I might add). With the advent of great speech-to-text ML, you could actually reduce this even further, to the 100-200 bps range).

dTal 13y ago

Are you quite sure they aren't collecting content? The storage is feasible, and several remarks (slips?) by people in a position to know suggest that they are in fact doing this, if not to everyone, then to hundreds of thousands or even millions of people. http://blog.rubbingalcoholic.com/post/52913031241/its-not-just-metadata-the-nsa-is-getting-everything http://blog.rubbingalcoholic.com/post/52913031241/its-not-ju... https://www.schneier.com/blog/archives/2013/06/evidence_that_t.html https://www.schneier.com/blog/archives/2013/06/evidence_that...

mpyne 13y ago
> Are you quite sure they aren't collecting content? It seems to depend entirely on the program in question, and the legal authorities for each program. For foreign intelligence actually collected overseas under EO 12333 they have programs that collect content (e.g. SMS), which have existed in one form or another since the beginnings of the Cold War. For foreign intelligence conducted on U.S. soil they have to fall within the boundaries of Fourth Amendment reasonable search (and related legal rulings like Smith v. Maryland) so they wouldn't capture content. But on the other hand they can still capture content in targeted fashion against non-U.S. persons in certain scenarios, and that content might itself involve conversations involving a U.S. person and still be a legal search.
- rhizome 13y ago
  It seems to depend entirely on the program in question Accessibility does, but it has nothing to do with the agency. "Does the government store the content of US calls?" can be answered in the aggregate without reference to specific programs. If the content is there, an agency only has to acquire the authorities required to access what is already there. So, the question remains: is the content there to access, given proper authorities?
  - mpyne 13y ago
    There's multiple levels of authorities though. Collecting the calls at all would itself require some legal authority. Searching the calls you collected would then need yet another legal authority.

fit2rule 13y ago

I think you need to pay a little more attention: http://www.youtube.com/watch?feature=player_detailpage&v=vILAlhwUgIU#t=1717 http://www.youtube.com/watch?feature=player_detailpage&v=vIL... The ARE storing everything. 15 years worth.

bostik 13y ago

> If it didn't, the NSA would not be so interested in collecting it. I'm going to quote Schneier. Stop calling it metadata. Call it what cryptographers and security professionals have called it since forever: traffic analysis. Traffic analysis is a powerful tool by itself. Combined with practically unlimited access to source material, and the ability to unmask almost every communication node, you don't even NEED to care about the contents. Timing, frequency and direction of communication is more than enough.

glitchdout 13y ago

They are already storing the phone calls... * http://www.pbs.org/newshour/bb/government_programs-july-dec13-whistleblowers_08-01/ http://www.pbs.org/newshour/bb/government_programs-july-dec1... * http://www.theguardian.com/commentisfree/2013/may/04/telephone-calls-recorded-fbi-boston http://www.theguardian.com/commentisfree/2013/may/04/telepho... I'll quote the slip up of Tim Clemente, a former FBI agent: > All digital communications are uh uh... There's a way to look at digital communications in the past. And I can't go into detail of how that's done or what's done but I can tell you that no digital communication is secure.

legutierr 13y ago
And even if they are not storing the audio recordings, wouldn't it be reasonable to expect that automatic transcription is run against the audio for search indexing and long-term storage purposes? Text transcriptions like that, even if they are imperfect, can preserve conversations with minimal storage requirements almost indefinitely.