3 ms·
> archiving all of Imgur and Reddit’s image hosting system that's a huge task? where do you have funding for that?
by brokensegue 7y ago
> archiving all of Imgur and Reddit’s image hosting system
that's a huge task? where do you have funding for that?
- deleted 7y ago[deleted]
- toomuchtodo 7y agoStorage is cheap when host it yourself.
- Bartweiss 7y agoWait, are you trying to get everything from these services, or just some portion? (Oldest, highest-traffic, lowest-traffic, etc.) A quick-and-dirty estimate says Imgur was adding perhaps 1.5PB of data per year way back in 2012; that much storage is going to run you $30k even on offline tape. Downloading that much data is more doable than I expected (1/3 of a year on a gigabit connection), but just keeping up with Imgur's new uploads sounds like a monumental task. (It's a good task, and it does seem possible for one dedicated person. I'm just really curious to hear how you're managing it!)
- deleted 7y ago[deleted]
- toomuchtodo 7y agoLTO8 for cold storage, Backblaze styled pods for warm storage. I’ve had some help in acquiring supplies and equipment. All of the racking, colo, networking, and sysadmin is me for my gear (and my wife; she is a cable management fanatic, the right color for the appropriate VLAN!).
- brokensegue 7y agoIs this your job?
- toomuchtodo 7y agoJust one of my hobbies (digital preservationist).
- brokensegue 7y agocrazy. you work with archive team? I've wanted to get involved with that for a while.
- toomuchtodo 7y agohttps://www.archiveteam.org/index.php?title=Archiveteam:IRC https://www.archiveteam.org/index.php?title=Archiveteam:IRC
- ssully 7y agoSome of his favorite porn on Imgur removed and he vowed to never let it happen again.
- Bartweiss 7y agoArchive Team routs its collected data to the Internet Archive, with tools that I believe any individual could apply independently. As far as storage, apparently the Internet Archive has a $10M budget, which will go quite a long way if you accept slow or even offline storage for the content. But I'm still not sure how an individual would extract the images fast enough. Archive Team's largest project was ~1.5 petabytes of Google+. In 2012, Imgur had 300M images uploaded, which at 5MB per image would be 1.5PB. Simply keeping up with daily throughput would be an enormous undertaking, much less digging out all of the historical content.
- geggam 7y agoGiven the fact 10 years ago Y! Groups had over 4 Petabytes of data this might be a record for them