4 ms·

Math is fun! You can buy a 2 TB HD for $99.99 right now. The population of the United States is 311,591,917. That's 7057 bytes per citizen, for a penny under

Math is fun! You can buy a 2 TB HD for $99.99 right now. The population of the United States is 311,591,917. That's 7057 bytes per citizen, for a penny under a Franklin. The Complete Works of William Shakespeare can be downloaded from Project Gutenberg in EPUB format, in 2.3 MB. Therefore, if you wanted to have William Shakespeare write as much text about every single citizen in the United States, as he wrote in all of his Complete Works, you would need 341.74 of those HDs. Therefore, the raw storage would cost you $34,196.58. The NSA's annual budget is around $3.6 billion. Every year, the NSA could buy enough hard drive space to store 105,273 times as much text as the Complete Works of William Shakespeare, on every single U.S. citizen.

thereallurch 14y ago
While you make a good point, your end number is wildly inaccurate. 1) I highly doubt the NSA is buying $100 dollar low grade hard drives. I bet this data is backed up at least twice per site, at multiple sites. 2) The data is not unstructured, it will be structured in some sort of database. Storing your facebook photos alone will be potentially hundreds of megabytes. 3) I doubt NSA is spending it's entire budget every year on hard drives.
- VikingCoder 14y ago
  Yes, that's why I quite carefully said, "Every year, the NSA COULD..." Emphasis on the COULD. It's to emphasize how cheap storage is, and how relatively few U.S. Citizens there are. In our day-to-day lives, we think 311 Million is an enormous number. But in the scale of computing, it's actually quite small. My real point is: storing the data is no longer a challenge. If you want someone (businesses, governments, etc.) to not store data about every citizen of the U.S., you need laws, regulations, and checks and balances.
- pessimizer 14y ago
  1) If he's three or four orders of magnitude off, his point still stands. 2) Also, the data could be almost completely unstructured. You mine the data that you have to create structure. If you toss anything, or prematurely optimize things, you're missing out on what the algorithms that you come up with 10 years from now will give you. 3) If you're using enough redundancy to multiply $35,000 into $3,000,000,000, you're insane.
dman 14y ago
Storing the text is easy - deciding what text is relevant enough to store is hard.
- VikingCoder 14y ago
  If you had the NSA's budget, you probably wouldn't need to decide what text is relevant to store.
  - dman 14y ago
    Deciding what to store is highly correlated with what information are we most likely going to find useful.
    - VikingCoder 14y ago
      When I finally bought an iPod, it had 60 GB of storage. I thought, "Which albums of mine should I upload to it?" Then I remembered, "Oh right - all of them!" My point is that deciding what to store is expensive and error-prone, with false-positives and false-negatives. Storage is so cheap, that there's an argument to be made that they can just store everything they want to.
sliverstorm 14y ago
Why does this matter?

thereallurch 14y ago

While you make a good point, your end number is wildly inaccurate. 1) I highly doubt the NSA is buying $100 dollar low grade hard drives. I bet this data is backed up at least twice per site, at multiple sites. 2) The data is not unstructured, it will be structured in some sort of database. Storing your facebook photos alone will be potentially hundreds of megabytes. 3) I doubt NSA is spending it's entire budget every year on hard drives.

VikingCoder 14y ago
Yes, that's why I quite carefully said, "Every year, the NSA COULD..." Emphasis on the COULD. It's to emphasize how cheap storage is, and how relatively few U.S. Citizens there are. In our day-to-day lives, we think 311 Million is an enormous number. But in the scale of computing, it's actually quite small. My real point is: storing the data is no longer a challenge. If you want someone (businesses, governments, etc.) to not store data about every citizen of the U.S., you need laws, regulations, and checks and balances.
pessimizer 14y ago
1) If he's three or four orders of magnitude off, his point still stands. 2) Also, the data could be almost completely unstructured. You mine the data that you have to create structure. If you toss anything, or prematurely optimize things, you're missing out on what the algorithms that you come up with 10 years from now will give you. 3) If you're using enough redundancy to multiply $35,000 into $3,000,000,000, you're insane.

dman 14y ago

Storing the text is easy - deciding what text is relevant enough to store is hard.

VikingCoder 14y ago
If you had the NSA's budget, you probably wouldn't need to decide what text is relevant to store.
- dman 14y ago
  Deciding what to store is highly correlated with what information are we most likely going to find useful.
  - VikingCoder 14y ago
    When I finally bought an iPod, it had 60 GB of storage. I thought, "Which albums of mine should I upload to it?" Then I remembered, "Oh right - all of them!" My point is that deciding what to store is expensive and error-prone, with false-positives and false-negatives. Storage is so cheap, that there's an argument to be made that they can just store everything they want to.

sliverstorm 14y ago

Why does this matter?