4 ms·

I recall some months ago a seemingly fairly technical and generally informed commenter -- here on HN, I believe -- laying out their back of the envelop calculat

I recall some months ago a seemingly fairly technical and generally informed commenter -- here on HN, I believe -- laying out their back of the envelop calculation of what it might take to archive all the U.S. voice traffic. The resulting figures were currently readily achievable, and they became increasingly... "trivial" (my interpretation) with the already announced and in progress data center expansions. And here, in this article, we have a description of Brewster Khale coming up with what is truly a trivial dollar amount to accomplish this. It's increasingly apparent that there is probably no technical limitation to their accomplishing this. The only question remains, is any other limitation stopping them?

niels_olson 13y ago
The real problem is that once a government program is started, good luck shutting it down. Look at the GITMO detainee program.
- hownottowrite 13y ago
  Correct. If there's one thing that governments are good at, it's storage. Long term storage.
vidarh 13y ago
You can record GSM encoded (9600bps) voice of all americans 24/7 for a year and still only fill up a couple of percent of the speculated storage capacity of the NSA's new Utah data centre. I don't know how long the average person spends on the phone each day, but clearly recording the phone calls will be just a small fraction of that again. So keeping a few years of recordings of all phone calls they can get their hands on would not be a storage problem.
alephnil 13y ago
A phone call can be compressed to 10 kbits/sec or less given a good speech codec and still be understandable, if not of very good quality. That means that if you are on the phone 20 minutes every day, it will be around a half a GB per year. If every American is on phone that much, it will be bit less than 143 petabytes a year, or about 36000 4GB hardisks. That is a bit, but the datacenter in Utah presumably have a capacity of a yottabyte, or nearly seven order of magnitude more than what is needed to record all phonecalls for a given year. They would have capacity to store all phone calls for nearly a million years if the yottabyte figure is right.
- lazyjones 13y ago
  > it will be bit less than 143 petabytes a year, or about 36000 4GB hardisks 4TB harddisks you mean ... But why stop at Americans? It's obviously feasible to record all phone calls made anywhere in the world, forever. Hooking into large network operators won't be such a big problem either.
- jedbrown 13y ago
  The "yottabyte" estimate is absurd, but individual scientific computing centers have about half an exabyte of storage, so the NSA wouldn't have the slightest trouble storing a few exabytes (some GBs per person per year). In contrast, total IP traffic is in the tens of exabytes per month [1], but much of it is re-transmission of content such as movies, that the NSA is not interested in storing. I wonder what sort of steganography detection they have in place. [1] http://www.cisco.com/en/US/solutions/collateral/ns341/ns525/ns537/ns705/ns827/white_paper_c11-481360_ns827_Networking_Solutions_White_Paper.html http://www.cisco.com/en/US/solutions/collateral/ns341/ns525/...
- Amadou 13y ago
  I tracked down the yottabyte claim to, what I think is, its origin and it seems to refer to raw uncompressed data. http://www.nybooks.com/articles/archives/2009/nov/05/whos-in-big-brothers-database/ http://www.nybooks.com/articles/archives/2009/nov/05/whos-in...

niels_olson 13y ago

The real problem is that once a government program is started, good luck shutting it down. Look at the GITMO detainee program.

hownottowrite 13y ago
Correct. If there's one thing that governments are good at, it's storage. Long term storage.

vidarh 13y ago

You can record GSM encoded (9600bps) voice of all americans 24/7 for a year and still only fill up a couple of percent of the speculated storage capacity of the NSA's new Utah data centre. I don't know how long the average person spends on the phone each day, but clearly recording the phone calls will be just a small fraction of that again. So keeping a few years of recordings of all phone calls they can get their hands on would not be a storage problem.

alephnil 13y ago

A phone call can be compressed to 10 kbits/sec or less given a good speech codec and still be understandable, if not of very good quality. That means that if you are on the phone 20 minutes every day, it will be around a half a GB per year. If every American is on phone that much, it will be bit less than 143 petabytes a year, or about 36000 4GB hardisks. That is a bit, but the datacenter in Utah presumably have a capacity of a yottabyte, or nearly seven order of magnitude more than what is needed to record all phonecalls for a given year. They would have capacity to store all phone calls for nearly a million years if the yottabyte figure is right.

lazyjones 13y ago
> it will be bit less than 143 petabytes a year, or about 36000 4GB hardisks 4TB harddisks you mean ... But why stop at Americans? It's obviously feasible to record all phone calls made anywhere in the world, forever. Hooking into large network operators won't be such a big problem either.
jedbrown 13y ago
The "yottabyte" estimate is absurd, but individual scientific computing centers have about half an exabyte of storage, so the NSA wouldn't have the slightest trouble storing a few exabytes (some GBs per person per year). In contrast, total IP traffic is in the tens of exabytes per month [1], but much of it is re-transmission of content such as movies, that the NSA is not interested in storing. I wonder what sort of steganography detection they have in place. [1] http://www.cisco.com/en/US/solutions/collateral/ns341/ns525/ns537/ns705/ns827/white_paper_c11-481360_ns827_Networking_Solutions_White_Paper.html http://www.cisco.com/en/US/solutions/collateral/ns341/ns525/...
Amadou 13y ago
I tracked down the yottabyte claim to, what I think is, its origin and it seems to refer to raw uncompressed data. http://www.nybooks.com/articles/archives/2009/nov/05/whos-in-big-brothers-database/ http://www.nybooks.com/articles/archives/2009/nov/05/whos-in...