3 ms·
Wait, are you trying to get everything from these services, or just some portion? (Oldest, highest-traffic, lowest-traffic, etc.) A quick-and-dirty estimate sa
by Bartweiss 7y ago
Wait, are you trying to get everything from these services, or just some portion? (Oldest, highest-traffic, lowest-traffic, etc.)
A quick-and-dirty estimate says Imgur was adding perhaps 1.5PB of data per year way back in 2012; that much storage is going to run you $30k even on offline tape. Downloading that much data is more doable than I expected (1/3 of a year on a gigabit connection), but just keeping up with Imgur's new uploads sounds like a monumental task.
(It's a good task, and it does seem possible for one dedicated person. I'm just really curious to hear how you're managing it!)
- deleted 7y ago[deleted]
- toomuchtodo 7y agoLTO8 for cold storage, Backblaze styled pods for warm storage. I’ve had some help in acquiring supplies and equipment. All of the racking, colo, networking, and sysadmin is me for my gear (and my wife; she is a cable management fanatic, the right color for the appropriate VLAN!).
- brokensegue 7y agoIs this your job?
- toomuchtodo 7y agoJust one of my hobbies (digital preservationist).
- brokensegue 7y agocrazy. you work with archive team? I've wanted to get involved with that for a while.
- toomuchtodo 7y agohttps://www.archiveteam.org/index.php?title=Archiveteam:IRC https://www.archiveteam.org/index.php?title=Archiveteam:IRC
- ssully 7y agoSome of his favorite porn on Imgur removed and he vowed to never let it happen again.