10 ms·

You overestimate the cost of storage at this scale. It's totally worth them storing the data given the marginal cost. A 10TiB HDD is only 150$ retail. These v

You overestimate the cost of storage at this scale. It's totally worth them storing the data given the marginal cost. A 10TiB HDD is only 150$ retail. These videos can likely be stored in less than 200MiB on average. Storage has gotten cheaper YoY

deleted 7y ago
[deleted]
kibwen 7y ago
I'd like someone to do the math on that 200MiB estimate. YouTube stores every video at multiple resolutions, and has replicas in data centers all around the world.
- zegl 7y ago
  For low-traffic videos they likely only keep a copy in a single region.
  - 1f60c 7y ago
    That, and I’m sure they employ “cold storage” using slower (and cheaper) spinning disks.
  - netsharc 7y ago
    Interestingly they could also just keep the best resolution version of rare videos, and have really powerful encoders if anyone suddenly asked for a lowres version. Then they can deliver as well as cache the lowres version in case someone else wants to look at it an hour later, but maybe drop it after a week of no access.
    - jandrese 7y ago
      They don't even have to be all that powerful. GPU transcoding is fast as heck. You really just need to decide where the line is between the storage tradeoff and needing a bigger GPU farm to handle more frequent transcoding requests. I'm betting you could get away with having just one copy for 90-95% of Youtube's content. There is an ocean of video on Youtube that is accessed less than once a month or even once a year.
- Ayeceedee 7y ago
  Ah! Small note. There are methods of compression and encoding that allow for scalability. There's some fancy signal processing you can do to encode multiple <framerates / resolutions / compression qualities> into a single bitstream without necessarily storing redundant data. But, I'm not in industry, and the last time I poked my head in I recall things being way uglier and more complicated than I had imagined. There was a good conference talk that was linked here once but I've lost it. Talked about the sort of awful, buggy things (formatting/file wise) that people try to upload that break everything.
  - kibwen 7y ago
    Interesting, thanks! I'd love to read more about ways to encode video that allows streaming at a variety of resolutions with a minimum of redundancy, if anyone has any links.
    - Ayeceedee 7y ago
      For me, it came up in a course I took last term which used the textbook "Video Processing and Communications" by Y. Wang, J.Ostermann and Y-Q. Zhang. It's a bit out of date (2002), but the core video encoder material was solid. It's a bit heavy in representing concepts using symbolic/equation notation. That, for me, made the jump from 1D signal processing to multidimensional signal processing tougher than it needed to be. There are probably better resources so definitely don't just take my word for it! :) (For scabilility, IIRC the terms "enhancement layer" and "base layer" are particularly important, as are the block diagrams that generate them.)
  - chupasaurus 7y ago
    > Small note. There are methods of compression and encoding that allow for scalability. There's some fancy signal processing you can do to encode multiple <framerates / resolutions / compression qualities> into a single bitstream without necessarily storing redundant data. A big note. All of this wasn't used by YT because they've always used a widely used standard codecs and media containers. All the lower qualities of videos they provide use storage nearly the same as the highest one (so 2x total), all bitrates are the same for all videos. As for download part, the players in auto quality would get first a maximum quality they can for current download speed, if there's a room or connection isn't stable - also 480p version, and if the speed was enough to download current and next chunk in lower quality it would download the best and switch to it over on completion (there were a times when player was downloading the whole video in the best quality after it had finished in an optimal). > the sort of awful, buggy things (formatting/file wise) that people try to upload that break everything. YT has very narrow list of formats allowed to upload with hard breaks in converting process, especially for the audio.
duiker101 7y ago
Just a few days ago there was a discussion about how Google is now indexing less contented from a years ago[1]. If Google can't keep indexes which are relatively inexpensive why should they keep videos that are more expensive and possibly no one will watch? Especially as the length and number of videos is increasing faster than ever? [1] https://news.ycombinator.com/item?id=19762907 https://news.ycombinator.com/item?id=19762907
- Already__Taken 7y ago
  You're not scanning through bulk storage like that all the time. Indexes have to be managed to be useful too which is another cost.
  - amelius 7y ago
    Okay, so the videos may become preserved but inaccessible.
  - duiker101 7y ago
    Videos still need to be indexed, why would Google keep the videos without indexing them?
    - chillacy 7y ago
      Let’s not confuse the videos with the index of the videos. Videos can be stored on the slowest and cheapest medium, indexes might be kept in memory or flash. Each video takes hundreds of mb compressed, an entry in the index could be constant per video
    - tbirrell 7y ago
      Indexing and storage are two different things. If something if not accessed regularly, then you don't want to index it. You can still access it, it just takes longer to find.
- the-rc 7y ago
  The Google indices are very expensive to create and serve. For one, they use a ton of memory. YouTube has TPMs whose only focus is making storage very cheap.
RandomInteger4 7y ago
Maybe I'm mistaken, but my understanding is that YouTube has been operating at a loss for its entire existence.
- RandomInteger4 7y ago
  Sources: https://www.wsj.com/articles/viewers-dont-add-up-to-profit-for-youtube-1424897967 https://www.wsj.com/articles/viewers-dont-add-up-to-profit-f... http://fortune.com/2016/10/18/youtube-profits-ceo-susan-wojcicki/ http://fortune.com/2016/10/18/youtube-profits-ceo-susan-wojc... https://www.bloomberg.com/news/articles/2019-02-04/alphabet-profit-margins-slide-as-google-costs-march-upwards https://www.bloomberg.com/news/articles/2019-02-04/alphabet-... While there's nothing concrete concerning how much YouTube costs and brings in, it seems to be generally understood that YouTube is an investment that hasn't reached profitability yet and only exists in its current state due to the fact that Google is willing to lose money on it in the mean time.
adrianN 7y ago
There is an incredibly long tail of videos that are (almost) never watched. All those 200mb videos add up to a lot of money.
cosarara 7y ago
> A 10TiB HDD is only 150$ retail. Where is that? I can't find anything so cheap.
- cobookman 7y ago
  https://www.bestbuy.com/site/wd-easystore-10tb-external-usb-3-0-hard-drive-black/6278208.p?skuId=6278208 https://www.bestbuy.com/site/wd-easystore-10tb-external-usb-...
  - akhilcacharya 7y ago
    The bizarre thing is I can't find bare drives cheaper than that on Amazon - how are they making a profit on these?
    - jandrese 7y ago
      Maybe it's one of those drives that comes with a demo of the backup software and they're hoping you'll buy the full version?
    - dogecoinbase 7y ago
      Generally speaking the drives in external enclosures are from the bottom slice of the still-viable drive pool. You'll often find that they have shorter/less comprehensive warranties.
    - vatueil 7y ago
      Larger market of average, undemanding home users for external hard drives. Smaller market of more demanding enthusiast users for internal drives. (Larger businesses aren't sourcing drives from Amazon or Newegg, unless the company is called Backblaze.) Latter costs more to support and as niche users generally pay more. (The more price conscious resort to shucking external drives.) At least, that's how it's been explained to me. I don't know if that's the entire story though it sounds reasonable enough.
thoughtfunction 7y ago
These videos are hours long, and unless your doing low quality 480p, which will make it hard to follow, would definitely not take 200MB.
chippy 7y ago
You are correct about the cost of storage. However that is not the cost of storing these videos. When you set up a server you just buy a hard disk and some other hardware correct. But Youtube also has to buy staff to maintain, update and services and networks, offices for them, legal stuff, free food, insurance, electricity and every other thing a huge business needs of course. Google after 2007 has been about doing things that make business sense. Just storing these videos is a cost. It's why there's been a glacial pace in new features, APIs and support in youtube for over a decade - it's not a financial priority.
- cobookman 7y ago
  That was retail price. Most the operational aspects at google are somewhat fixed costs. Power Cooling are per Server. However given the low load on longtail video content & ability for google to co-mingle high IOPs workloads with low IOPs workload. Its not that simple... TLDR, storage is fairly cheap in scheme of things. Even if having infinite video storage costs 100mil/year, google likely makes more from content monitization.
montag 7y ago
How many megabytes of storage are subsidized by, say, 100 views per year?

deleted 7y ago

[deleted]

kibwen 7y ago

I'd like someone to do the math on that 200MiB estimate. YouTube stores every video at multiple resolutions, and has replicas in data centers all around the world.

zegl 7y ago
For low-traffic videos they likely only keep a copy in a single region.
- 1f60c 7y ago
  That, and I’m sure they employ “cold storage” using slower (and cheaper) spinning disks.
- netsharc 7y ago
  Interestingly they could also just keep the best resolution version of rare videos, and have really powerful encoders if anyone suddenly asked for a lowres version. Then they can deliver as well as cache the lowres version in case someone else wants to look at it an hour later, but maybe drop it after a week of no access.
  - jandrese 7y ago
    They don't even have to be all that powerful. GPU transcoding is fast as heck. You really just need to decide where the line is between the storage tradeoff and needing a bigger GPU farm to handle more frequent transcoding requests. I'm betting you could get away with having just one copy for 90-95% of Youtube's content. There is an ocean of video on Youtube that is accessed less than once a month or even once a year.
Ayeceedee 7y ago
Ah! Small note. There are methods of compression and encoding that allow for scalability. There's some fancy signal processing you can do to encode multiple <framerates / resolutions / compression qualities> into a single bitstream without necessarily storing redundant data. But, I'm not in industry, and the last time I poked my head in I recall things being way uglier and more complicated than I had imagined. There was a good conference talk that was linked here once but I've lost it. Talked about the sort of awful, buggy things (formatting/file wise) that people try to upload that break everything.
- kibwen 7y ago
  Interesting, thanks! I'd love to read more about ways to encode video that allows streaming at a variety of resolutions with a minimum of redundancy, if anyone has any links.
  - Ayeceedee 7y ago
    For me, it came up in a course I took last term which used the textbook "Video Processing and Communications" by Y. Wang, J.Ostermann and Y-Q. Zhang. It's a bit out of date (2002), but the core video encoder material was solid. It's a bit heavy in representing concepts using symbolic/equation notation. That, for me, made the jump from 1D signal processing to multidimensional signal processing tougher than it needed to be. There are probably better resources so definitely don't just take my word for it! :) (For scabilility, IIRC the terms "enhancement layer" and "base layer" are particularly important, as are the block diagrams that generate them.)
- chupasaurus 7y ago
  > Small note. There are methods of compression and encoding that allow for scalability. There's some fancy signal processing you can do to encode multiple <framerates / resolutions / compression qualities> into a single bitstream without necessarily storing redundant data. A big note. All of this wasn't used by YT because they've always used a widely used standard codecs and media containers. All the lower qualities of videos they provide use storage nearly the same as the highest one (so 2x total), all bitrates are the same for all videos. As for download part, the players in auto quality would get first a maximum quality they can for current download speed, if there's a room or connection isn't stable - also 480p version, and if the speed was enough to download current and next chunk in lower quality it would download the best and switch to it over on completion (there were a times when player was downloading the whole video in the best quality after it had finished in an optimal). > the sort of awful, buggy things (formatting/file wise) that people try to upload that break everything. YT has very narrow list of formats allowed to upload with hard breaks in converting process, especially for the audio.

duiker101 7y ago

Just a few days ago there was a discussion about how Google is now indexing less contented from a years ago[1]. If Google can't keep indexes which are relatively inexpensive why should they keep videos that are more expensive and possibly no one will watch? Especially as the length and number of videos is increasing faster than ever? [1] https://news.ycombinator.com/item?id=19762907 https://news.ycombinator.com/item?id=19762907

Already__Taken 7y ago
You're not scanning through bulk storage like that all the time. Indexes have to be managed to be useful too which is another cost.
- amelius 7y ago
  Okay, so the videos may become preserved but inaccessible.
- duiker101 7y ago
  Videos still need to be indexed, why would Google keep the videos without indexing them?
  - chillacy 7y ago
    Let’s not confuse the videos with the index of the videos. Videos can be stored on the slowest and cheapest medium, indexes might be kept in memory or flash. Each video takes hundreds of mb compressed, an entry in the index could be constant per video
  - tbirrell 7y ago
    Indexing and storage are two different things. If something if not accessed regularly, then you don't want to index it. You can still access it, it just takes longer to find.
the-rc 7y ago
The Google indices are very expensive to create and serve. For one, they use a ton of memory. YouTube has TPMs whose only focus is making storage very cheap.

RandomInteger4 7y ago

Maybe I'm mistaken, but my understanding is that YouTube has been operating at a loss for its entire existence.

RandomInteger4 7y ago
Sources: https://www.wsj.com/articles/viewers-dont-add-up-to-profit-for-youtube-1424897967 https://www.wsj.com/articles/viewers-dont-add-up-to-profit-f... http://fortune.com/2016/10/18/youtube-profits-ceo-susan-wojcicki/ http://fortune.com/2016/10/18/youtube-profits-ceo-susan-wojc... https://www.bloomberg.com/news/articles/2019-02-04/alphabet-profit-margins-slide-as-google-costs-march-upwards https://www.bloomberg.com/news/articles/2019-02-04/alphabet-... While there's nothing concrete concerning how much YouTube costs and brings in, it seems to be generally understood that YouTube is an investment that hasn't reached profitability yet and only exists in its current state due to the fact that Google is willing to lose money on it in the mean time.

adrianN 7y ago

There is an incredibly long tail of videos that are (almost) never watched. All those 200mb videos add up to a lot of money.

cosarara 7y ago

> A 10TiB HDD is only 150$ retail. Where is that? I can't find anything so cheap.

cobookman 7y ago
https://www.bestbuy.com/site/wd-easystore-10tb-external-usb-3-0-hard-drive-black/6278208.p?skuId=6278208 https://www.bestbuy.com/site/wd-easystore-10tb-external-usb-...
- akhilcacharya 7y ago
  The bizarre thing is I can't find bare drives cheaper than that on Amazon - how are they making a profit on these?
  - jandrese 7y ago
    Maybe it's one of those drives that comes with a demo of the backup software and they're hoping you'll buy the full version?
  - dogecoinbase 7y ago
    Generally speaking the drives in external enclosures are from the bottom slice of the still-viable drive pool. You'll often find that they have shorter/less comprehensive warranties.
  - vatueil 7y ago
    Larger market of average, undemanding home users for external hard drives. Smaller market of more demanding enthusiast users for internal drives. (Larger businesses aren't sourcing drives from Amazon or Newegg, unless the company is called Backblaze.) Latter costs more to support and as niche users generally pay more. (The more price conscious resort to shucking external drives.) At least, that's how it's been explained to me. I don't know if that's the entire story though it sounds reasonable enough.

thoughtfunction 7y ago

These videos are hours long, and unless your doing low quality 480p, which will make it hard to follow, would definitely not take 200MB.

chippy 7y ago

You are correct about the cost of storage. However that is not the cost of storing these videos. When you set up a server you just buy a hard disk and some other hardware correct. But Youtube also has to buy staff to maintain, update and services and networks, offices for them, legal stuff, free food, insurance, electricity and every other thing a huge business needs of course. Google after 2007 has been about doing things that make business sense. Just storing these videos is a cost. It's why there's been a glacial pace in new features, APIs and support in youtube for over a decade - it's not a financial priority.

cobookman 7y ago
That was retail price. Most the operational aspects at google are somewhat fixed costs. Power Cooling are per Server. However given the low load on longtail video content & ability for google to co-mingle high IOPs workloads with low IOPs workload. Its not that simple... TLDR, storage is fairly cheap in scheme of things. Even if having infinite video storage costs 100mil/year, google likely makes more from content monitization.

montag 7y ago

How many megabytes of storage are subsidized by, say, 100 views per year?