10 ms·
Does YouTube have any incentive to “preserve” anything once it costs them more money than it brings in? If not, maybe it's a little early to say they are actual
by velcrovan 7y ago
Does YouTube have any incentive to “preserve” anything once it costs them more money than it brings in? If not, maybe it's a little early to say they are actually preserving anything. (The article does kind of acknowledge this.) Maybe its real role is as a link in a chain that leads to /r/datahoarders
- cobookman 7y agoYou overestimate the cost of storage at this scale. It's totally worth them storing the data given the marginal cost. A 10TiB HDD is only 150$ retail. These videos can likely be stored in less than 200MiB on average. Storage has gotten cheaper YoY
- deleted 7y ago[deleted]
- kibwen 7y agoI'd like someone to do the math on that 200MiB estimate. YouTube stores every video at multiple resolutions, and has replicas in data centers all around the world.
- zegl 7y agoFor low-traffic videos they likely only keep a copy in a single region.
- 1f60c 7y agoThat, and I’m sure they employ “cold storage” using slower (and cheaper) spinning disks.
- netsharc 7y agoInterestingly they could also just keep the best resolution version of rare videos, and have really powerful encoders if anyone suddenly asked for a lowres version. Then they can deliver as well as cache the lowres version in case someone else wants to look at it an hour later, but maybe drop it after a week of no access.
- jandrese 7y agoThey don't even have to be all that powerful. GPU transcoding is fast as heck. You really just need to decide where the line is between the storage tradeoff and needing a bigger GPU farm to handle more frequent transcoding requests. I'm betting you could get away with having just one copy for 90-95% of Youtube's content. There is an ocean of video on Youtube that is accessed less than once a month or even once a year.
- Ayeceedee 7y agoAh! Small note. There are methods of compression and encoding that allow for scalability. There's some fancy signal processing you can do to encode multiple <framerates / resolutions / compression qualities> into a single bitstream without necessarily storing redundant data. But, I'm not in industry, and the last time I poked my head in I recall things being way uglier and more complicated than I had imagined. There was a good conference talk that was linked here once but I've lost it. Talked about the sort of awful, buggy things (formatting/file wise) that people try to upload that break everything.
- kibwen 7y agoInteresting, thanks! I'd love to read more about ways to encode video that allows streaming at a variety of resolutions with a minimum of redundancy, if anyone has any links.
- Ayeceedee 7y agoFor me, it came up in a course I took last term which used the textbook "Video Processing and Communications" by Y. Wang, J.Ostermann and Y-Q. Zhang. It's a bit out of date (2002), but the core video encoder material was solid. It's a bit heavy in representing concepts using symbolic/equation notation. That, for me, made the jump from 1D signal processing to multidimensional signal processing tougher than it needed to be. There are probably better resources so definitely don't just take my word for it! :) (For scabilility, IIRC the terms "enhancement layer" and "base layer" are particularly important, as are the block diagrams that generate them.)
- chupasaurus 7y ago> Small note. There are methods of compression and encoding that allow for scalability. There's some fancy signal processing you can do to encode multiple <framerates / resolutions / compression qualities> into a single bitstream without necessarily storing redundant data. A big note. All of this wasn't used by YT because they've always used a widely used standard codecs and media containers. All the lower qualities of videos they provide use storage nearly the same as the highest one (so 2x total), all bitrates are the same for all videos. As for download part, the players in auto quality would get first a maximum quality they can for current download speed, if there's a room or connection isn't stable - also 480p version, and if the speed was enough to download current and next chunk in lower quality it would download the best and switch to it over on completion (there were a times when player was downloading the whole video in the best quality after it had finished in an optimal). > the sort of awful, buggy things (formatting/file wise) that people try to upload that break everything. YT has very narrow list of formats allowed to upload with hard breaks in converting process, especially for the audio.
- duiker101 7y agoJust a few days ago there was a discussion about how Google is now indexing less contented from a years ago[1]. If Google can't keep indexes which are relatively inexpensive why should they keep videos that are more expensive and possibly no one will watch? Especially as the length and number of videos is increasing faster than ever? [1] https://news.ycombinator.com/item?id=19762907 https://news.ycombinator.com/item?id=19762907
- Already__Taken 7y agoYou're not scanning through bulk storage like that all the time. Indexes have to be managed to be useful too which is another cost.
- amelius 7y agoOkay, so the videos may become preserved but inaccessible.
- duiker101 7y agoVideos still need to be indexed, why would Google keep the videos without indexing them?
- chillacy 7y agoLet’s not confuse the videos with the index of the videos. Videos can be stored on the slowest and cheapest medium, indexes might be kept in memory or flash. Each video takes hundreds of mb compressed, an entry in the index could be constant per video
- tbirrell 7y agoIndexing and storage are two different things. If something if not accessed regularly, then you don't want to index it. You can still access it, it just takes longer to find.
- the-rc 7y agoThe Google indices are very expensive to create and serve. For one, they use a ton of memory. YouTube has TPMs whose only focus is making storage very cheap.
- RandomInteger4 7y agoMaybe I'm mistaken, but my understanding is that YouTube has been operating at a loss for its entire existence.
- RandomInteger4 7y agoSources: https://www.wsj.com/articles/viewers-dont-add-up-to-profit-for-youtube-1424897967 https://www.wsj.com/articles/viewers-dont-add-up-to-profit-f... http://fortune.com/2016/10/18/youtube-profits-ceo-susan-wojcicki/ http://fortune.com/2016/10/18/youtube-profits-ceo-susan-wojc... https://www.bloomberg.com/news/articles/2019-02-04/alphabet-profit-margins-slide-as-google-costs-march-upwards https://www.bloomberg.com/news/articles/2019-02-04/alphabet-... While there's nothing concrete concerning how much YouTube costs and brings in, it seems to be generally understood that YouTube is an investment that hasn't reached profitability yet and only exists in its current state due to the fact that Google is willing to lose money on it in the mean time.
- adrianN 7y agoThere is an incredibly long tail of videos that are (almost) never watched. All those 200mb videos add up to a lot of money.
- cosarara 7y ago> A 10TiB HDD is only 150$ retail. Where is that? I can't find anything so cheap.
- cobookman 7y agohttps://www.bestbuy.com/site/wd-easystore-10tb-external-usb-3-0-hard-drive-black/6278208.p?skuId=6278208 https://www.bestbuy.com/site/wd-easystore-10tb-external-usb-...
- akhilcacharya 7y agoThe bizarre thing is I can't find bare drives cheaper than that on Amazon - how are they making a profit on these?
- jandrese 7y agoMaybe it's one of those drives that comes with a demo of the backup software and they're hoping you'll buy the full version?
- dogecoinbase 7y agoGenerally speaking the drives in external enclosures are from the bottom slice of the still-viable drive pool. You'll often find that they have shorter/less comprehensive warranties.
- vatueil 7y agoLarger market of average, undemanding home users for external hard drives. Smaller market of more demanding enthusiast users for internal drives. (Larger businesses aren't sourcing drives from Amazon or Newegg, unless the company is called Backblaze.) Latter costs more to support and as niche users generally pay more. (The more price conscious resort to shucking external drives.) At least, that's how it's been explained to me. I don't know if that's the entire story though it sounds reasonable enough.
- thoughtfunction 7y agoThese videos are hours long, and unless your doing low quality 480p, which will make it hard to follow, would definitely not take 200MB.
- chippy 7y agoYou are correct about the cost of storage. However that is not the cost of storing these videos. When you set up a server you just buy a hard disk and some other hardware correct. But Youtube also has to buy staff to maintain, update and services and networks, offices for them, legal stuff, free food, insurance, electricity and every other thing a huge business needs of course. Google after 2007 has been about doing things that make business sense. Just storing these videos is a cost. It's why there's been a glacial pace in new features, APIs and support in youtube for over a decade - it's not a financial priority.
- cobookman 7y agoThat was retail price. Most the operational aspects at google are somewhat fixed costs. Power Cooling are per Server. However given the low load on longtail video content & ability for google to co-mingle high IOPs workloads with low IOPs workload. Its not that simple... TLDR, storage is fairly cheap in scheme of things. Even if having infinite video storage costs 100mil/year, google likely makes more from content monitization.
- montag 7y agoHow many megabytes of storage are subsidized by, say, 100 views per year?
- SkyBelow 7y agoIsn't this true of any entity? Even a Museum will eventually have to pay bills and have to decide to downsize or make changes. Even government funded preservation can have voters decide it isn't worth the costs and preservation can be dropped. I do think private companies will be the ones most fastest to drop preservation and the ones least likely to look for external entity to swap preservation with, thus making YouTube worse than a museum.
- nullandvoid 7y agoI don't think this is a fair comparison as a museum is normally opened with the sole desire of preserving history ( Profit is often desired but not the incentive for opening ) Do you think google bought YouTube originally to preserve history or to create a new line of profit and enhance its brand?
- chillacy 7y agoMuseums don’t own all of their collections though, and they too make decisions on what to display at any moment. Not all objects are worthy of display, some get rotated out back into some rich person’s attic.
- orcdork 7y agoI'd be very interested to hear about any respectable museum that sends off items that are not displayed to be stored into some rich person's attic (or for display at a private residence as I assume you are implying).
- cabalamat 7y ago> Does YouTube have any incentive to “preserve” anything once it costs them more money than it brings in? They clearly have little incentive to do so. If there was a law that prevented websites from using technical or legal means to stop people scraping user-generated content, then I would agree that YouTube has not such obligation. But since they make it hard for people to make archives of their content, then I think they do have a moral obligation to keep it up.
- dpkonofa 7y agoI was going to say something similar. I actually think YouTube (as a hosting/sharing platform) is a net-negative for the internet because there are already millions of videos that were embedded in pages that are no longer working because YouTube removed them, the account was deleted, or any number of things. If the internet worked the way it was intended, these videos would have been hosted on the site rather than embedded and would be still working. If the video wasn't there anymore, that usually means the content isn't there anymore. It's almost worse than dead hyperlinks because it's almost always supplemental content that references a video that can't be accessed anymore. :( That being said, it's a net-positive when it comes to content created. The internet wouldn't be as crazy without YouTube, so thanks for that.
- criley2 7y agoIn nearly all of those cases, the content was removed because either the uploader wanted it removed, or the owner of the intellectual property in the video wanted it removed. In both of those cases, I don't see how this is Youtube's fault or decentralized hosting would be better. In the first case (uploader wants it gone) you have a sticky situation where loosely managed websites refuse (or can't) respond to requests to remove, which can start interfering with various laws in various places. In the second case, where IP has been violated, those same loosely managed sites run significant legal risk of lawsuit by not responding. From the perspective of a content owner or video creator, why would you want your work decentrally stored in places that you can't control it? The other angles here have to do with monetization, too. When you use youtube, you can serve user aware ads and track who is watching your video as it spreads. If it is self-hosted, your content has been stolen and is enriching someone else illegally. From a creators perspective I can't see how decentrally stored videos would be good. Perhaps what we need is a Library of Congress type organization to begin a serious long-term archive of significant portions of this type of media.
- dpkonofa 7y agoI would agree with you if thousands of videos and accounts weren't removed daily due to false "intellectual property" claims and automated reporting. YouTube's system is terrible because they're so large that they've had to try to automate a system that can't really be automated. A decentralized system would require those claims to actually be validated before they're processed because, otherwise, it would be a waste of time and money for the companies making the false claims. I agree with you on the Library of Congress. We need a digital LoC.
- TrainedMonkey 7y agoIt is a ponzi scheme right now. As long as cost of storage is falling exponentially, they can afford to keep the old stuff.
- GuB-42 7y agoThat the cost of storage is falling exponentially certainly helps but it isn't necessary if we want to keep the old stuff. In fact most of the exponential increase in storage space is used to cover the even more exponential increase in video file size. But there is a limit on how big video can get. Our perception is limited and passed a certain point, we won't notice the improvement. Audio has been "perfect" for quite a while now, and with 4k video, we are getting there too. There is also a limit on the amount of content users can produce before people do nothing but film themselves. We may find ourselves running out of stuff to store before we need to delete the old stuff.
- TrainedMonkey 7y agoYou are thinking only about quality of video increases. I agree that is essentially finite if we ignore upcoming technologies such as 360 video and eventually VR ready video (fully 3d, any perspective). My argument was primarily based on the number of videos uploaded (see https://tubularinsights.com/hours-minute-uploaded-youtube/ https://tubularinsights.com/hours-minute-uploaded-youtube/). Looking at the area under the curve, supporting the historic videos is a fraction of what it takes to store the most recent ones. I am arguing as long as the youtube continues to be able to support to store the recent videos, storing the historic ones is not that much more expensive. edit: also, running out of stuff to store is not going to be a thing ever. Amount of cameras in the world is increasing and people will keep shooting more videos.
- buzzert 7y agoI think you’re looking at it the wrong way. YouTube is not preserving video game history by providing a place to host these videos. Instead, they’re offering an incentive for people to create these videos that are currently hosted on YouTube but will eventually be archived somewhere else. In other words, YouTube is somewhat responsible for the creation of this new content, but has no expectation or responsibility to maintain its legacy.
- euske 7y agoAnd the same thing can be said for any type of videos, not just video games videos. YouTube gave an incentive (not necessarily monetary incentive) to preserve something to those who previously wouldn't if such a service didn't exist.