10 ms·

Funny thing is, they started the surveillance in late-2001 to suck all the records up. Where did they get the computers and storage devices to do that all at on

by ok_dad 4y ago

Funny thing is, they started the surveillance in late-2001 to suck all the records up. Where did they get the computers and storage devices to do that all at once? They must have been installing equipment for months or years, especially since I am pretty sure they even built a new data center on Fort Mead for it around that time. Now, this is 2001, when building a datacenter didn't just involve spinning up 1000 AWS EC2's and opening the spigot to S3, so this type of thing would have taken some time. So the question is: did they pass a law to allow data collection because of 9/11 and other attacks, or did they pass a law because they wanted the NSA to be able to collect this data using computer systems they had been planning for years, and used those attacks as a pretext?

lstodd 4y ago
Remember the Clipper chip from 1993? It's not like they suddenly started it in 2001.
vorpalhex 4y ago
I legit don't believe they can store all that data. Youtube alone creates too much data for them to process and handle. They must be storing either a subset or only partial metadata.
- specto 4y ago
  likely netflow data.
- calvinmorrison 4y ago
  with large datasets like bioinformatics you often compress the data by extracting features and building matrixes, then only keeping things that score above a threshold
  - Scoundreller 4y ago
    Sounds right. I like to say we’re all on a list and it just matters where you rank on it. And the precision is only good when you rank high on it. No different than google search results. The accuracy of the 7th page of results just doesn’t matter.
- getcrunk 4y ago
  I mean more than a few x00 someones have to actually know the answer to that question. Shocked it hasn't been leaked.
- perrylaj 4y ago
  I'm on the other side of the fence - I believe they can store all they need to. Backdoors into Google, AWS, and others were strongly hinted at if not fully exposed with all the leaks back in the Snowden days, so US Agencies may not even need to store some of the bulky stuff. But given the vast size and number of datacenters operated by US agencies, it seems likely they have a LOT of storage capacity. http://worldstopdatacenters.com/government-data-centers/ http://worldstopdatacenters.com/government-data-centers/
- eointierney 4y ago
  I legit believe they store everything they can because they've been doing exactly this for longer than anyone else. They are _the_ experts in metadata management.
- StanislavPetrov 4y ago
  The NSA has an unlimited budget to store all the data they want. They have massive, billion-dollar data centers around the country (and the world?). The one in Utah is perhaps the best known. https://en.wikipedia.org/wiki/Utah_Data_Center https://en.wikipedia.org/wiki/Utah_Data_Center
  - ReactiveJelly 4y ago
    Nothing is unlimited. If it cut into our tank budget, or anything old retirees care about, something would happen.
    - StanislavPetrov 4y ago
      >Nothing is unlimited. If it cut into our tank budget, or anything old retirees care about, something would happen. Nothing is unlimited, especially inflation. The idea of a "budget" when the FED can (and does) create trillions of dollars out of thin air with a few keystrokes is antiquated. They can - and do - devote whatever resources they desire for "national security". https://www.usdebtclock.org/ https://www.usdebtclock.org/
- ReactiveJelly 4y ago
  If I was them, I would not back up YouTube, but I might carefully scrape and discard. Hell, if I was a _benevolent_ surveillance program, I'd probably run routine searches for illegal stuff on YouTube, both to find it myself, and to make sure YouTube's tripwires are working. There is so much low-hanging fruit in terms of "interesting secrets per byte" Like, I could believe all SMS messages are stored for a year or so. Some random source says, "Over 6 billion texts are sent every day". If a text is about 140 characters, and you use a dumb image classifier to transcribe photos as "Nude woman", "nude man", "dick pic", "image macro", "guns", etc., that's only about 1 TB per day, right? 365 TB to keep all US text messages for a year? Maybe my source is wrong. That sounds low. But, it's just text. Maybe it's right. In fact, the upper bound for all US keyboard input for a year must be below 4.6 petabytes. (350 million people typing 365 days a year, 16 hours a day, 40 words per minute, 1 bit of entropy per character after compression, 8 bits per word)
- carapace 4y ago
  > While the actual capacity is classified for NATIONAL SECURITY REASONS, we can say this: The Utah Data Center was built with future expansion in mind and the ultimate capacity will definitely be "alottabytes"! https://nsa.gov1.info/utah-data-center/ https://nsa.gov1.info/utah-data-center/
bombcar 4y ago
I would suspect it was already being done (in part, or in whole who knows) and the attacks were an excuse to "legitimize" it going forward.
- kevin_thibedeau 4y ago
  It was definitely being done before 9/11. I know of one data broker who was under contract with a certain TLA in the 90's to perform hardware assisted rapid data scanning.
StanislavPetrov 4y ago
>Funny thing is, they started the surveillance in late-2001 to suck all the records up. It started long, long before 2001. Here is a 2000 press release from the ACLU about global surveillance done by Echelon, which had already been ongoing for years (back when the ACLU cared about privacy issues): https://www.aclu.org/press-releases/privacy-advocates-concerned-about-echelon https://www.aclu.org/press-releases/privacy-advocates-concer... By 1999 privacy advocates at already gotten together for an official "jam Echelon day". >It was the latest in a long line of apparently futile online protest movements. On Thursday October 21 1999, internet users were urged to drop trigger words into their email so as to overwhelm Echelon, the massive surveillance project which is said to keep track of all electronic communications into and out of the USA, the UK, Canada, Australia and New Zealand. https://techmonitor.ai/technology/did_jam_echelon_day_do_more_than_make_the_nsa_laugh https://techmonitor.ai/technology/did_jam_echelon_day_do_mor...
- amatecha 4y ago
  Ah right, I remember when Echelon Watch[0] was launched, and made the rounds in chat rooms I frequented at the time... Of course most people dismissed it as conspiracy theory stuff :) [0] https://web.archive.org/web/20001018103147/http://www.echelonwatch.org/ https://web.archive.org/web/20001018103147/http://www.echelo...
colordrops 4y ago
I recall some natural disaster in Thailand being blamed for a hard drive shortage at the time, when it ended up being the NSA buying them all up for their Utah data center. Edit: anyone downvoting want to comment on why this is implausible or not noteworthy to mention? It was a topic of discussion at the time.
- ReactiveJelly 4y ago
  Source? Would love to hear about this. I searched for "thailand hard drive shortage nsa" and only got... 2013 Nov, Backblaze's annual hard drive report still only blamed the flooding: https://www.backblaze.com/blog/farming-hard-drives-2-years-and-1m-later/ https://www.backblaze.com/blog/farming-hard-drives-2-years-a... 2015, Tech Power-Up has a story about NSA malware hidden in hard drive firmware: https://www.techpowerup.com/209925/nsa-hides-spying-backdoors-into-hard-drive-firmware?cp=5 https://www.techpowerup.com/209925/nsa-hides-spying-backdoor...
  - colordrops 4y ago
    I don't recall where I heard it but found this in a quick search: https://www.reddit.com/r/conspiracy/comments/li1pep/a_global_hard_drive_shortage_occurred_at_the_same/ https://www.reddit.com/r/conspiracy/comments/li1pep/a_global...
landemva 4y ago
Insider Bill Binney tried to do responsible data collection. He got harassed. https://m.youtube.com/watch?v=JvAflFcpJFQ https://m.youtube.com/watch?v=JvAflFcpJFQ

by ok_dad 4y ago

lstodd 4y ago

Remember the Clipper chip from 1993? It's not like they suddenly started it in 2001.

vorpalhex 4y ago

I legit don't believe they can store all that data. Youtube alone creates too much data for them to process and handle. They must be storing either a subset or only partial metadata.

specto 4y ago
likely netflow data.
calvinmorrison 4y ago
with large datasets like bioinformatics you often compress the data by extracting features and building matrixes, then only keeping things that score above a threshold
- Scoundreller 4y ago
  Sounds right. I like to say we’re all on a list and it just matters where you rank on it. And the precision is only good when you rank high on it. No different than google search results. The accuracy of the 7th page of results just doesn’t matter.
getcrunk 4y ago
I mean more than a few x00 someones have to actually know the answer to that question. Shocked it hasn't been leaked.
perrylaj 4y ago
I'm on the other side of the fence - I believe they can store all they need to. Backdoors into Google, AWS, and others were strongly hinted at if not fully exposed with all the leaks back in the Snowden days, so US Agencies may not even need to store some of the bulky stuff. But given the vast size and number of datacenters operated by US agencies, it seems likely they have a LOT of storage capacity. http://worldstopdatacenters.com/government-data-centers/ http://worldstopdatacenters.com/government-data-centers/
eointierney 4y ago
I legit believe they store everything they can because they've been doing exactly this for longer than anyone else. They are _the_ experts in metadata management.
StanislavPetrov 4y ago
The NSA has an unlimited budget to store all the data they want. They have massive, billion-dollar data centers around the country (and the world?). The one in Utah is perhaps the best known. https://en.wikipedia.org/wiki/Utah_Data_Center https://en.wikipedia.org/wiki/Utah_Data_Center
- ReactiveJelly 4y ago
  Nothing is unlimited. If it cut into our tank budget, or anything old retirees care about, something would happen.
  - StanislavPetrov 4y ago
    >Nothing is unlimited. If it cut into our tank budget, or anything old retirees care about, something would happen. Nothing is unlimited, especially inflation. The idea of a "budget" when the FED can (and does) create trillions of dollars out of thin air with a few keystrokes is antiquated. They can - and do - devote whatever resources they desire for "national security". https://www.usdebtclock.org/ https://www.usdebtclock.org/
ReactiveJelly 4y ago
If I was them, I would not back up YouTube, but I might carefully scrape and discard. Hell, if I was a _benevolent_ surveillance program, I'd probably run routine searches for illegal stuff on YouTube, both to find it myself, and to make sure YouTube's tripwires are working. There is so much low-hanging fruit in terms of "interesting secrets per byte" Like, I could believe all SMS messages are stored for a year or so. Some random source says, "Over 6 billion texts are sent every day". If a text is about 140 characters, and you use a dumb image classifier to transcribe photos as "Nude woman", "nude man", "dick pic", "image macro", "guns", etc., that's only about 1 TB per day, right? 365 TB to keep all US text messages for a year? Maybe my source is wrong. That sounds low. But, it's just text. Maybe it's right. In fact, the upper bound for all US keyboard input for a year must be below 4.6 petabytes. (350 million people typing 365 days a year, 16 hours a day, 40 words per minute, 1 bit of entropy per character after compression, 8 bits per word)
carapace 4y ago
> While the actual capacity is classified for NATIONAL SECURITY REASONS, we can say this: The Utah Data Center was built with future expansion in mind and the ultimate capacity will definitely be "alottabytes"! https://nsa.gov1.info/utah-data-center/ https://nsa.gov1.info/utah-data-center/

bombcar 4y ago

I would suspect it was already being done (in part, or in whole who knows) and the attacks were an excuse to "legitimize" it going forward.

kevin_thibedeau 4y ago
It was definitely being done before 9/11. I know of one data broker who was under contract with a certain TLA in the 90's to perform hardware assisted rapid data scanning.

StanislavPetrov 4y ago

>Funny thing is, they started the surveillance in late-2001 to suck all the records up. It started long, long before 2001. Here is a 2000 press release from the ACLU about global surveillance done by Echelon, which had already been ongoing for years (back when the ACLU cared about privacy issues): https://www.aclu.org/press-releases/privacy-advocates-concerned-about-echelon https://www.aclu.org/press-releases/privacy-advocates-concer... By 1999 privacy advocates at already gotten together for an official "jam Echelon day". >It was the latest in a long line of apparently futile online protest movements. On Thursday October 21 1999, internet users were urged to drop trigger words into their email so as to overwhelm Echelon, the massive surveillance project which is said to keep track of all electronic communications into and out of the USA, the UK, Canada, Australia and New Zealand. https://techmonitor.ai/technology/did_jam_echelon_day_do_more_than_make_the_nsa_laugh https://techmonitor.ai/technology/did_jam_echelon_day_do_mor...

amatecha 4y ago
Ah right, I remember when Echelon Watch[0] was launched, and made the rounds in chat rooms I frequented at the time... Of course most people dismissed it as conspiracy theory stuff :) [0] https://web.archive.org/web/20001018103147/http://www.echelonwatch.org/ https://web.archive.org/web/20001018103147/http://www.echelo...

colordrops 4y ago

I recall some natural disaster in Thailand being blamed for a hard drive shortage at the time, when it ended up being the NSA buying them all up for their Utah data center. Edit: anyone downvoting want to comment on why this is implausible or not noteworthy to mention? It was a topic of discussion at the time.

ReactiveJelly 4y ago
Source? Would love to hear about this. I searched for "thailand hard drive shortage nsa" and only got... 2013 Nov, Backblaze's annual hard drive report still only blamed the flooding: https://www.backblaze.com/blog/farming-hard-drives-2-years-and-1m-later/ https://www.backblaze.com/blog/farming-hard-drives-2-years-a... 2015, Tech Power-Up has a story about NSA malware hidden in hard drive firmware: https://www.techpowerup.com/209925/nsa-hides-spying-backdoors-into-hard-drive-firmware?cp=5 https://www.techpowerup.com/209925/nsa-hides-spying-backdoor...
- colordrops 4y ago
  I don't recall where I heard it but found this in a quick search: https://www.reddit.com/r/conspiracy/comments/li1pep/a_global_hard_drive_shortage_occurred_at_the_same/ https://www.reddit.com/r/conspiracy/comments/li1pep/a_global...

landemva 4y ago

Insider Bill Binney tried to do responsible data collection. He got harassed. https://m.youtube.com/watch?v=JvAflFcpJFQ https://m.youtube.com/watch?v=JvAflFcpJFQ