6 ms·

I'm curious where someone just gets 56.7 tb of storage that quickly.

ckdarby 6y ago
AWS. If not them because in you're worried they'd also shut you down than probably BackBlaze. Could also just buy a bunch of fairly cheap 100 mbit unmetered boxes off OVH/Kimsufi for a total cost probably of ~$300/m.
ikiris 6y ago
You can get that in 5 drives from best buy these days. Not exactly a huge leap for cloud storage.
j2bax 6y ago
I believe the person that pulled it down is a digital archivist. I’m sure she has plenty of storage laying around for such occasions.
- adwww 6y ago
  Presumably just another s3 bucket? Do all your transferring from an EC2 instance in the same region and it never needs to waste bandwidth going over the public internet anyway.
  - jackweirdy 6y ago
    Or local storage. The DataHoarder subreddit where a lot of similar efforts are coordinated has a lot of info about building dense home storage on the cheap
- polar 6y ago
  > laying around ITYM lying around, unless this is a quirk of US English. Sorry to be pedantic!
  - nkrisc 6y ago
    > unless this is a quirk of US English It is indeed. Very common in colloquial speech around here.
asdff 6y ago
work gave us google drive accounts with "unlimited" capacity
- compsciphd 6y ago
  except you can only upload 750GB a day without somewhat workarounds
  - imvictor 6y ago
    You can. Create multiple service accounts, add them to the Shared drives and connect the service accounts using Rclone, a CLI tool that allows you to perform I/O operations on multiple cloud storage. Once a service account reached the limit, switch to another.
    - compsciphd 6y ago
      "without somewhat workarounds" sometimes I wonder about people.
- PeterStuer 6y ago
  You'll find out what 'unlimited' means in SaaS speak very, very much sooner than you expect if you tried really to utilize it.
  - ubercow13 6y ago
    Not at 50TB though.
  - sellyme 6y ago
    You'd be surprised. Google has a lot of storage lying around, it takes them a while to notice it being used up. I know of at least three people who on the order of petabytes of data on Google Drive. They took action against one organisation that had multiple 10PB+ users but Google really doesn't seem to care that much about the tertiary institutes giving unlimited GDrive accounts to every data hoarder who pretends to enrol.
bawolff 6y ago
Im more curious how someone pulls down 56 terabytes in a very short period of time without sysadmins at parler noticing. I'm surprised they didn't unintentionally DoS them.
- foepys 6y ago
  Parler used AWS and AWS is always happy to serve any request without problems and notify you about it hours later if you decided to create usage alerts.
  - berkes 6y ago
    > and notify you about it hours later if you decided to create usage alerts and bill you at the end of the month. This is the crucial part: AWS will serve whatever is request. It brings them money.
    - INTPenis 6y ago
      Sorry for OT but what about a simple small time private AWS user like me who uses it for static hosting my blog and stuff. Does that mean someone could request a lot of files, over and over, to increase my bill? Or is that served by some cache? Let's say for example I host Mastodon media on S3, about 30G of unique data. Could someone use that to increase the cost of my bill? I do of course have a budget alert set but I have to react to that alert too.
      - nindalf 6y ago
        This is always a possibility so you take steps to protect yourself. If it’s a few static assets, put a CDN in front of it. You won’t be charged extra by S3 because the CDN would cache it. If you’re hosting Mastodon, then I assume you’d take steps to ensure that only an authenticated user can access any data. And that user would also need to be authorised to access only specific data. And that authenticated and authorised user would be rate limited so they couldn’t scrape everything they have access to easily. If you do all these things, you’ll be fine.
flukus 6y ago
All the content was hosted on s3 and you just needed the URL's, security by obscurity.
mickotron 6y ago
Crowdsourced. The crawling and downloading was able to be coordinated and performed by a bunch of people at the same time.
PeterStuer 6y ago
The storage isn't that much of a deal, but I bet that was not on some cheap consumer Internet subscription as an ISP would have throttled her into oblivion after the first few TB.
azeirah 6y ago
Modern hdd's store up to 18tb. I saw 6tb hdd's for €114 on my local site, 16tb hdd's for €370. It's not exactly cheap, but if you're doing it for a serious project like archiving an entire politically relevant social media website, I'm sure you'll have 1-2 thousand eur lying around for a couple of hard disks
jamesponddotco 6y ago
I got two 90 TB servers that I pay a small amount of peanuts per month at Hetzner to server as backup servers. As long as you stay away from the cloud, storage is dirty cheap.

AWS. If not them because in you're worried they'd also shut you down than probably BackBlaze. Could also just buy a bunch of fairly cheap 100 mbit unmetered boxes off OVH/Kimsufi for a total cost probably of ~$300/m.

ikiris 6y ago

You can get that in 5 drives from best buy these days. Not exactly a huge leap for cloud storage.

j2bax 6y ago

I believe the person that pulled it down is a digital archivist. I’m sure she has plenty of storage laying around for such occasions.

adwww 6y ago
Presumably just another s3 bucket? Do all your transferring from an EC2 instance in the same region and it never needs to waste bandwidth going over the public internet anyway.
- jackweirdy 6y ago
  Or local storage. The DataHoarder subreddit where a lot of similar efforts are coordinated has a lot of info about building dense home storage on the cheap
polar 6y ago
> laying around ITYM lying around, unless this is a quirk of US English. Sorry to be pedantic!
- nkrisc 6y ago
  > unless this is a quirk of US English It is indeed. Very common in colloquial speech around here.

asdff 6y ago

work gave us google drive accounts with "unlimited" capacity

compsciphd 6y ago
except you can only upload 750GB a day without somewhat workarounds
- imvictor 6y ago
  You can. Create multiple service accounts, add them to the Shared drives and connect the service accounts using Rclone, a CLI tool that allows you to perform I/O operations on multiple cloud storage. Once a service account reached the limit, switch to another.
  - compsciphd 6y ago
    "without somewhat workarounds" sometimes I wonder about people.
PeterStuer 6y ago
You'll find out what 'unlimited' means in SaaS speak very, very much sooner than you expect if you tried really to utilize it.
- ubercow13 6y ago
  Not at 50TB though.
- sellyme 6y ago
  You'd be surprised. Google has a lot of storage lying around, it takes them a while to notice it being used up. I know of at least three people who on the order of petabytes of data on Google Drive. They took action against one organisation that had multiple 10PB+ users but Google really doesn't seem to care that much about the tertiary institutes giving unlimited GDrive accounts to every data hoarder who pretends to enrol.

bawolff 6y ago

Im more curious how someone pulls down 56 terabytes in a very short period of time without sysadmins at parler noticing. I'm surprised they didn't unintentionally DoS them.

foepys 6y ago
Parler used AWS and AWS is always happy to serve any request without problems and notify you about it hours later if you decided to create usage alerts.
- berkes 6y ago
  > and notify you about it hours later if you decided to create usage alerts and bill you at the end of the month. This is the crucial part: AWS will serve whatever is request. It brings them money.
  - INTPenis 6y ago
    Sorry for OT but what about a simple small time private AWS user like me who uses it for static hosting my blog and stuff. Does that mean someone could request a lot of files, over and over, to increase my bill? Or is that served by some cache? Let's say for example I host Mastodon media on S3, about 30G of unique data. Could someone use that to increase the cost of my bill? I do of course have a budget alert set but I have to react to that alert too.
    - nindalf 6y ago
      This is always a possibility so you take steps to protect yourself. If it’s a few static assets, put a CDN in front of it. You won’t be charged extra by S3 because the CDN would cache it. If you’re hosting Mastodon, then I assume you’d take steps to ensure that only an authenticated user can access any data. And that user would also need to be authorised to access only specific data. And that authenticated and authorised user would be rate limited so they couldn’t scrape everything they have access to easily. If you do all these things, you’ll be fine.

flukus 6y ago

All the content was hosted on s3 and you just needed the URL's, security by obscurity.

mickotron 6y ago

Crowdsourced. The crawling and downloading was able to be coordinated and performed by a bunch of people at the same time.

PeterStuer 6y ago

The storage isn't that much of a deal, but I bet that was not on some cheap consumer Internet subscription as an ISP would have throttled her into oblivion after the first few TB.

azeirah 6y ago

Modern hdd's store up to 18tb. I saw 6tb hdd's for €114 on my local site, 16tb hdd's for €370. It's not exactly cheap, but if you're doing it for a serious project like archiving an entire politically relevant social media website, I'm sure you'll have 1-2 thousand eur lying around for a couple of hard disks

jamesponddotco 6y ago

I got two 90 TB servers that I pay a small amount of peanuts per month at Hetzner to server as backup servers. As long as you stay away from the cloud, storage is dirty cheap.