3 ms·
r/datahoarders will kindly and rightly disagree with you on that regard. For an enthusiast, a 720TB array is pretty reachable. A dedicated enthusiast can get a
by bayindirh 4y ago
r/datahoarders will kindly and rightly disagree with you on that regard.
For an enthusiast, a 720TB array is pretty reachable. A dedicated enthusiast can get a 1PB flash array in 2U.
- TremendousJudge 4y agoI don't think anybody on /r/datahoarders believes it's possible for a private individual to archive the entirety of Youtube. More than 200,000 hours of video are uploaded every day. Generously assuming something like 1 GB/hour for 1080p, that's 200TB per day that you have to add. No home array can handle that.
- orangepurple 4y agoDon't worry, only a very small subset is worth keeping
- jasonladuke0311 4y agoThat’s what you think now.
- autoexec 4y agoEven now, the most garbage youtube video out there is still probably useful as training data for some AI (maybe even to generate horrible youtube videos)
- WalterBright 4y agoSo true. I look at photos I've taken in the past, and discovered that I took pictures of all the wrong things.
- bee_rider 4y agoThis is a weird hackernews phenomenon where two sides of a discussion present the technical aspect of the thing they want to do, and are correct in their description of the technical aspects, without addressing the fact that they are talking about accomplishing totally unrelated objectives. It is probably possible to horde more Youtube videos than you could ever watch, probably including most of the ones that you might ever be interested in. And it is almost certainly impossible for any individual to capture every video which goes through Youtube. Neither of these seem to address the issue of whether there exist videos which will retrospectively have archival value which are not captured.
- autoexec 4y ago> Neither of these seem to address the issue of whether there exist videos which will retrospectively have archival value which are not captured. Unless the entirety of youtube can be archived it's safe to assume that there will be something of value which isn't being preserved. It's an unsolved problem and not one Google wants to see solved.
- MikePlacid 4y ago> probably including most of the ones that you might ever be interested in. That’s a really interesting question: how to determine videos that I might ever be interested in. > whether there exist videos which will retrospectively have archival value which are not captured. And that’s not really a question: there definitely exist videos that have a certain historical value which were deleted from YouTube, and most of them before I archived them cause I am lazy. I would gladly pay for a personal archive.org - a solution that automatically archives each page I visited and video I watched. I guess the required storage amount will be pretty affordable.
- bee_rider 4y ago> > whether there exist videos which will retrospectively have archival value which are not captured. > And that’s not really a question: there definitely exist videos that have a certain historical value which were deleted from YouTube, and most of them before I archived them cause I am lazy. Sure, but you aren't the only one backing up YouTube videos. It seems at least plausible that the aggregate storage capacity of the entire data horder community and their propensity for backing up whatever they come across could result in a situation where if something is interesting, somebody ends up capturing it, right?
- selfhoster11 4y agoThere are some communities with a collaborative video index of who has what backed up.
- munificent 4y agoAccording to this article [1], 500 hours of video are uploaded to YouTube every minute. Depending on the video size and framerate, YouTube recommends up to 240 Mbps for 8k@60FPS [2]. Of course most video isn't that high res. Let's take a conservative guess that it averages somewhere between 2K and 4K and pick a middle bitrate of 24 Mbps. That's: 24 Mbps / 8 bit/byte * 60 seconds/minute * 60 minute/hour = 10800 megabytes per hour of footage = 10.8 gigabytes per hour of footage At 500 hours of footage per minute, that means 5.4 terabytes are uploaded every minute. Your 720 TB array would be completely full a little over two hours' worth of content that is uploaded to YouTube every single day, day after day. At the current upload rate, 2,838.24 petabytes are uploaded every year. I don't think you'll see hobbyist archives of YouTube any time soon. [1]: https://www.tubefilter.com/2019/05/07/number-hours-video-uploaded-to-youtube-per-minute/ https://www.tubefilter.com/2019/05/07/number-hours-video-upl... [2]: https://support.google.com/youtube/answer/1722171?hl=en#zippy=%2Ccontainer-mp%2Cbitrate https://support.google.com/youtube/answer/1722171?hl=en#zipp...
- yig 4y agoI wonder how much of the uploaded content is public versus private or unlisted.
- zacmps 4y agoFor archival 24Mbps is an insane bitrate, you could get away with 1/3th to 1/6th of that. You're also going to limit to public videos (unlisted and private will make up some share of those uploads) and probably to those with non-zero views. I suspect archiving only videos with >100 views would probably cut the amount you archive to 1/10th.
- Godel_unicode 4y agoThis is a distinction without a difference, you’re still talking about O(1PB/day).
- chiph 4y agoSome back-of-the-envelope math shows that YouTube would have to populate and rack a minimum of four 4U storage chassis (60 20TB drives each) per 8 hour shift to store that much. Roughly a little less than half a 42U rack. And that's before allowing for HDD drive parity, redundancy, and distribution across the globe.
- MacsHeadroom 4y agoI have 4PB, but my understanding is that I would fill that mirroring a single day of YouTube at reduced quality. The Internet Archive could surely handle mirroring older YouTube content with a large grant. But the upload rate plus video quality in recent years is definitely cost prohibitive to replicate.
- bayindirh 4y agoThe point I was trying to make is, data hoarders collectively archive a lot of data. They have interests and tend to mirror their interests from the sites. As a result, they can mirror a substantial amount of videos or knowledge from any gigantic site. I'm aware that even a horde of data hoarders can't archive a drop in the YouTube ocean, but by archiving high-impact channels, they can backup both important and vast amounts of information.
- saagarjha 4y agoThe size of YouTube is likely measured in exabytes. I think it would be hard for any entity that was not organized and well-funded to mirror all of it, let alone make it available in a reasonable fashion.