16 ms·

Disclaimer: I used to work at a live video streaming company as a financial analyst so quite familiar with this The biggest cost is as you imagine the streamin

Disclaimer: I used to work at a live video streaming company as a financial analyst so quite familiar with this The biggest cost is as you imagine the streaming - getting the video to the viewer. It was a large part of our variable cost and we had a (literal) mad genius dev ops person holed up in his own office cave that managed the whole operation. Ive long forgotten the special optimizations he did but he would keep finding ways to improve margin / efficiency. Encoding is a cost but I don’t recall it being significant Storage isnt generally expensive. Think about how cheap you as a consumer can go get 2 TB of storage, and extrapolate. The other big expense - people! All those engineers to build back and front end systems. That’s what ruined us - too many people were needed and not enough money coming in so we were burning cash.

tithe 2y ago
Curious: was your distribution client-server or peer-to-peer? Or both, similar to Skype's supernode model?
- michaelt 2y ago
  The overwhelming majority of "legitimate" video streaming sites operate on a client-server model, which allows videos to be watched in web browsers, and on mobile devices (which don't generally do well in P2P as they find uploading difficult). And generally torrent-based streamers don't hire financial analysts :)
  - Dylan16807 2y ago
    Thankfully the FCC definition of "broadband" is getting more symmetrical over time. And doesn't webrtc take care of connecting browsers pretty well? The current definition requires 20Mbps of upload, and uploading a youtube-quality video to two other people would not take a big fraction of that. Though it would help if ISPs stop trying to set bandwidth caps at <5% utilization levels.
    - michaelt 2y ago
      It's not only the amount of upload bandwidth and the usage caps (although those are both big issues). It's that you're also probably going to get CGNAT - and maybe even a firewall blocking unusual ports. And you're going to be running the power-hungry data connection at least twice as much, bad for battery life. And mobile connections are less reliable - transitions between towers, going through tunnels, switching between 4G and WiFi. And mobile OSes are eager to suspend things - especially things that are using a lot of data and battery.
      - Dylan16807 2y ago
        That's a problem if all your users are mobile, yeah. I'm thinking of the situation where most of the users are using home connections and have power cables always in or in reach.
foota 2y ago
I'm guessing live video looks a lot different from a more static video site. I think encoding and storage are both quite expensive. You want to encode videos that are likely to be watched in the most efficient ways possible to reduce network bandwidth usage, and every video needs at least some encoding. Based on some power laws etc., I would guess most videos have only a handful of views, so storing them forever and the cost to encode them initially is probably significant.
- deleted 2y ago
  [deleted]
- timr 2y ago
  Encoding and storage aren't significant, relative to the bandwidth costs. Bandwidth is the high order bit. The primary difference between live and static video is the bursts -- get to a certain scale as a static video provider, and you can roughly estimate your bandwidth 95th percentiles. But one big live event can blow you out of the water, and push you over into very expensive tiers that will kill your economics.
  - foota 2y ago
    If they're not significant, then why does youtube build ASICs for doing video encoding? See e.g., https://arstechnica.com/gadgets/2021/04/youtube-is-now-building-its-own-video-transcoding-chips/ https://arstechnica.com/gadgets/2021/04/youtube-is-now-build...
    - jupp0r 2y ago
      Because when you are Youtube, even relatively marginal cost improvements can be huge in absolute. There is also the UX of having to wait X minutes for an uploaded video to be ready that is improved by this.
    - cebert 2y ago
      Doing so wouldn’t hurt and would make a sizable impact at the scale of Google?
contingencies 2y ago
Interesting background. I worked twice in digital video, once ~2000-2001 (ancient history - early IP, ISDN, the dead-end of H.323, bonded GSM channels, etc.) and once ~2009-2010. The second episode was fascinating, we specialised in mobile video at a time when it was just appearing on the consumer market. Most of the global mobile device manufacturers were clients. It got to the point where they would build the hardware and we would get airdropped in to their R&D to make it work - they had no idea how performant the architecture was going to be, because they'd never tried it. We also built the server side, the billing architecture with revenue share, carrier billing support (only possible with device preloaded apps due to Google Play (then "Google Apps"?) store restrictions on third party payment mechanisms), etc. Encoding, scaling and transcoding are relatively cheap for stored content, and relatively expensive if you want real or near-real time. If you want DRM (digital rights management = ~ineffective copy protection) then you need to add a bit more overhead for that, both in terms of processing and latency. If you need multi-DRM (different DRM systems for different devices the consumer owns) and a good cross-device experience (like pause and resume across devices), it gets real hard real fast. It helps to be targeting a standard platform, for example a modern widescreen TV with H.265 support and solid 4K decoding. Otherwise you need a different version for every resolution, a different version for every CODEC, a different version for every bitrate, etc. We had great experience adjusting bitrates and encoding parameters for different device categories, for example if you had a certain phone and you ran it at max spec it might look great but if you were looking to preserve battery and were running on battery save mode the decode would fail and you'd get choppy performance and stuttering audio. This sort of thing was rife then. As a series of specialist video providers emerged, ~all the cloud providers went and added these services, basically 95% of which are frontends to ffmpeg and some internal cloud storage scheme or similar. Finally, billing is hard. Users have an expectation of free content now. No experience with real time stream economics, but saw the inside of LA's stadium video control center one day. Didn't look inexpensive, I'll tell you that much. Probably for events with multiple cameras you're mostly paying site fees, ie. reliable bandwidth, humans, mixing desk if required. For studio broadcast these costs will be reduced. Both will have a slight real time encoding tax vs. stored content. If you want to figure out how to do it cheaply, look at the porn industry.
- gravescale 2y ago
hnthrowaway0328 2y ago
Thanks for sharing. Is it possible to join his team?
bushbaba 2y ago
It’s not that expensive at YouTube scale. We are talking fractions of a penny per GiB transferred.
- superb_dev 2y ago
  Yeah, YouTube is big enough to put their own cache nodes directly in ISP datacenters
  - WookieRushing 2y ago
    Yup! This is the reason why its so cheap for them. Other companies in similar positions have cache nodes in the ISPs and this dramatically lowers the cost
  - mikepurvis 2y ago
    Which would help for the crazy popular meme videos, but I bet the long tail on YouTube is insanely big, even if you did have the “watch next” engine getting in on the game steering you toward content already present in your nearby caches.
    - selcuka 2y ago
      YouTube is estimated to have 1 exabyte [1] of data. Petabyte level storage is not unheard of [2], and a gateway server with 5PB storage would cover ~0.5% of all YouTube videos, which should be sufficient to serve a very high percentage of the most popular videos. They can still afford to serve the occasional obscure video from the origin servers. [1] https://www.qqtube.com/blog/how-much-storage-does-youtube-have https://www.qqtube.com/blog/how-much-storage-does-youtube-ha... [2] https://www.qnap.com/solution/petabyte-storage/en/ https://www.qnap.com/solution/petabyte-storage/en/
  - bobmcnamara 2y ago
    Bingo, and also in Internet Exchanges - every ISP at KCIX the exchange has direct handoff to YouTube, Google, NetFlix, Cloudflare, ...
- 2y ago
shmerl 2y ago
Youtube owns a huge CDN to deliver video quickly.
- NewJazz 2y ago
  https://support.google.com/interconnect/answer/9058809?hl=en https://support.google.com/interconnect/answer/9058809?hl=en
eternityforest 2y ago
I'm actually kind of surprised serving media isn't trivial and solved yet. Routers have ASIC switching, why can't we have dedicated cache appliances with a bunch of RAM and some kind of modified GPU with network access and crypto acceleration in each core?
- saq7 2y ago
  Humorous
- zokier 2y ago
  Have you seen Netflix OCAs? Current off-the-shelf hardware goes really long way https://openconnect.netflix.com/en/appliances/ https://openconnect.netflix.com/en/appliances/ https://news.ycombinator.com/item?id=32519881 https://news.ycombinator.com/item?id=32519881
ezoe 2y ago
I disagree. Storage is expensive. Think of an old video uploaded 15 years ago with total view count of 1k. You can't just put it to a cheap cold storage. Someday, somebody is going to watch it and you have to retrieve it instantly or that somebody will be disappointed.
- Delmololo 2y ago
  You can. Yt for example deletes your 720p after a while and replaces it with a potato. And if you watch a old not relevant yt and it starts after 10 seconds instead of now, no one really cares. You can put that old highly encoded potato at your huge and cheap storage system de located somewhere around the globe were it's just cheap (energy). You can also calculate in the time for a band robot and only store half or the first minute of that potato on your cheap storage and let the robot grab the rest of it. After all if video is your main thing plenty of weird optimizations start to make sense.
  - olivermuty 2y ago
    Tape robot time is NOT measured in minutes but hours. Also you wouldn't pull individual files like this out of tape. Tape is for long term sunk storage, not cold infrequent access like a youtube video. I know aws glacier has an "expedited retrieval time" of 1-5 minutes, but that is not how typical tape setups work. Frankly I would be very interested in what actually hides behind that product.
    - joking 2y ago
      You can use big disk, but not be able to access all the files on the disk with the same frequency, so you have 20% of the disk dedicated to hot storage, and 80% of it to cold storage. Cold storage access is queued , so the 1-5 minutes can come from there.
      - olivermuty 2y ago
        Yeah I looked closer and I think they are basically just packaging up some kind of offering on top of spinning rust for the two first glacier tiers.
solarized 2y ago
Gilfoyle: ":smiling"
graemep 2y ago
> The biggest cost is as you imagine the streaming - getting the video to the viewer. I seem to remember Google own some network infrastructure? That saves some money. On top of that at their size you are going to get things cheaper. > The other big expense - people! All those engineers to build back and front end systems. That’s what ruined us - too many people were needed and not enough money coming in so we were burning cash. There should be economies of scale on that. Its harder to build and maintain bigger systems, but the work required does not scale linearly with size.
- mike_hearn 2y ago
  Google own vast network infrastructure. The day Google acquired YouTube (I was there) they discovered that YT was on the verge of collapse and would run out of bandwidth entirely within months. There was an emergency programme put in place with hundreds of engineers reallocated to an effort to avoid site collapse, with the clever/amusing name of BandAid. BandAid was a success. YouTube's history might look very different if it wasn't. It consisted of a massive crash buildout of a global CDN, something that Google historically hadn't had (CDNs aren't useful if everything you serve is dynamically generated). One reason BandAid could happen was that Google had a large and sophisticated NetOps operation already in place, which was already acquiring long haul unused fibre wavelengths at scale for the purposes of moving the web search index about. So CDN nodes didn't need to be far from a Google backbone POP, and at that point moving bits around on that backbone was to some extent "free" because the bulk of the cost was in the fibre rental and the rackspace+equipment on either end. Also it was being paid for already by the needs of search+ads. Over time Google steadily moved more stuff over to their infrastructure and off YouTube's own, but it was all driven by whatever would break next. Then you have all the costs that YouTube has that basic video sites don't. ContentID alone had costs that would break the back of nearly any other company but was made effectively mandatory by the copyright lawsuits against YouTube early on, which Google won but (according to internal legal analysis at least) that was largely due to the enormous good-faith and successful effort demonstrated by ContentID. And then of course you need a global ad sales team, a ton of work on ad targeting, etc. This story explains why Google can afford to do YouTube and you can't. The reality is that whilst they certainly have some sort of internal number for what YouTube costs, all such figures are inevitably kind of fantastical because so much infrastructure is shared and cross-subsidised by other businesses. You can't just magic up a global team of network engineers and fibre contracts in a few months, which is what YouTube needed in order to survive (and one of the main reasons they were forced to sell). No matter what price you come up with that in internal booking it will always be kinda dubious because such things aren't sold on the market.
game_the0ry 2y ago
> The biggest cost is as you imagine the streaming - getting the video to the viewer. It was a large part of our variable cost and we had a (literal) mad genius dev ops person holed up in his own office cave that managed the whole operation. Sort of make's Cloudflare's R2 look more impressive since they do not charge for egress.
- gowthamgts12 2y ago
  I'm digressing from the topic, but R2 looked good on paper, but have a long way to go in terms of reliability.
  - game_the0ry 2y ago
    Hmm could you elaborate?
emodendroket 2y ago
Moderation seems like another big issue although the solution afaict seems to mostly involve shipping this work off to the Philippines or wherever and making people look at the most horrifying content imaginable for 40 hours a week at very low wages.

tithe 2y ago

Curious: was your distribution client-server or peer-to-peer? Or both, similar to Skype's supernode model?

michaelt 2y ago
The overwhelming majority of "legitimate" video streaming sites operate on a client-server model, which allows videos to be watched in web browsers, and on mobile devices (which don't generally do well in P2P as they find uploading difficult). And generally torrent-based streamers don't hire financial analysts :)
- Dylan16807 2y ago
  Thankfully the FCC definition of "broadband" is getting more symmetrical over time. And doesn't webrtc take care of connecting browsers pretty well? The current definition requires 20Mbps of upload, and uploading a youtube-quality video to two other people would not take a big fraction of that. Though it would help if ISPs stop trying to set bandwidth caps at <5% utilization levels.
  - michaelt 2y ago
    It's not only the amount of upload bandwidth and the usage caps (although those are both big issues). It's that you're also probably going to get CGNAT - and maybe even a firewall blocking unusual ports. And you're going to be running the power-hungry data connection at least twice as much, bad for battery life. And mobile connections are less reliable - transitions between towers, going through tunnels, switching between 4G and WiFi. And mobile OSes are eager to suspend things - especially things that are using a lot of data and battery.
    - Dylan16807 2y ago
      That's a problem if all your users are mobile, yeah. I'm thinking of the situation where most of the users are using home connections and have power cables always in or in reach.

foota 2y ago

I'm guessing live video looks a lot different from a more static video site. I think encoding and storage are both quite expensive. You want to encode videos that are likely to be watched in the most efficient ways possible to reduce network bandwidth usage, and every video needs at least some encoding. Based on some power laws etc., I would guess most videos have only a handful of views, so storing them forever and the cost to encode them initially is probably significant.

deleted 2y ago
[deleted]
timr 2y ago
Encoding and storage aren't significant, relative to the bandwidth costs. Bandwidth is the high order bit. The primary difference between live and static video is the bursts -- get to a certain scale as a static video provider, and you can roughly estimate your bandwidth 95th percentiles. But one big live event can blow you out of the water, and push you over into very expensive tiers that will kill your economics.
- foota 2y ago
  If they're not significant, then why does youtube build ASICs for doing video encoding? See e.g., https://arstechnica.com/gadgets/2021/04/youtube-is-now-building-its-own-video-transcoding-chips/ https://arstechnica.com/gadgets/2021/04/youtube-is-now-build...
  - jupp0r 2y ago
    Because when you are Youtube, even relatively marginal cost improvements can be huge in absolute. There is also the UX of having to wait X minutes for an uploaded video to be ready that is improved by this.
  - cebert 2y ago
    Doing so wouldn’t hurt and would make a sizable impact at the scale of Google?

contingencies 2y ago

Interesting background. I worked twice in digital video, once ~2000-2001 (ancient history - early IP, ISDN, the dead-end of H.323, bonded GSM channels, etc.) and once ~2009-2010. The second episode was fascinating, we specialised in mobile video at a time when it was just appearing on the consumer market. Most of the global mobile device manufacturers were clients. It got to the point where they would build the hardware and we would get airdropped in to their R&D to make it work - they had no idea how performant the architecture was going to be, because they'd never tried it. We also built the server side, the billing architecture with revenue share, carrier billing support (only possible with device preloaded apps due to Google Play (then "Google Apps"?) store restrictions on third party payment mechanisms), etc. Encoding, scaling and transcoding are relatively cheap for stored content, and relatively expensive if you want real or near-real time. If you want DRM (digital rights management = ~ineffective copy protection) then you need to add a bit more overhead for that, both in terms of processing and latency. If you need multi-DRM (different DRM systems for different devices the consumer owns) and a good cross-device experience (like pause and resume across devices), it gets real hard real fast. It helps to be targeting a standard platform, for example a modern widescreen TV with H.265 support and solid 4K decoding. Otherwise you need a different version for every resolution, a different version for every CODEC, a different version for every bitrate, etc. We had great experience adjusting bitrates and encoding parameters for different device categories, for example if you had a certain phone and you ran it at max spec it might look great but if you were looking to preserve battery and were running on battery save mode the decode would fail and you'd get choppy performance and stuttering audio. This sort of thing was rife then. As a series of specialist video providers emerged, ~all the cloud providers went and added these services, basically 95% of which are frontends to ffmpeg and some internal cloud storage scheme or similar. Finally, billing is hard. Users have an expectation of free content now. No experience with real time stream economics, but saw the inside of LA's stadium video control center one day. Didn't look inexpensive, I'll tell you that much. Probably for events with multiple cameras you're mostly paying site fees, ie. reliable bandwidth, humans, mixing desk if required. For studio broadcast these costs will be reduced. Both will have a slight real time encoding tax vs. stored content. If you want to figure out how to do it cheaply, look at the porn industry.

gravescale 2y ago

hnthrowaway0328 2y ago

Thanks for sharing. Is it possible to join his team?

bushbaba 2y ago

It’s not that expensive at YouTube scale. We are talking fractions of a penny per GiB transferred.

superb_dev 2y ago
Yeah, YouTube is big enough to put their own cache nodes directly in ISP datacenters
- WookieRushing 2y ago
  Yup! This is the reason why its so cheap for them. Other companies in similar positions have cache nodes in the ISPs and this dramatically lowers the cost
- mikepurvis 2y ago
  Which would help for the crazy popular meme videos, but I bet the long tail on YouTube is insanely big, even if you did have the “watch next” engine getting in on the game steering you toward content already present in your nearby caches.
  - selcuka 2y ago
    YouTube is estimated to have 1 exabyte [1] of data. Petabyte level storage is not unheard of [2], and a gateway server with 5PB storage would cover ~0.5% of all YouTube videos, which should be sufficient to serve a very high percentage of the most popular videos. They can still afford to serve the occasional obscure video from the origin servers. [1] https://www.qqtube.com/blog/how-much-storage-does-youtube-have https://www.qqtube.com/blog/how-much-storage-does-youtube-ha... [2] https://www.qnap.com/solution/petabyte-storage/en/ https://www.qnap.com/solution/petabyte-storage/en/
- bobmcnamara 2y ago
  Bingo, and also in Internet Exchanges - every ISP at KCIX the exchange has direct handoff to YouTube, Google, NetFlix, Cloudflare, ...
2y ago

shmerl 2y ago

Youtube owns a huge CDN to deliver video quickly.

NewJazz 2y ago
https://support.google.com/interconnect/answer/9058809?hl=en https://support.google.com/interconnect/answer/9058809?hl=en

eternityforest 2y ago

I'm actually kind of surprised serving media isn't trivial and solved yet. Routers have ASIC switching, why can't we have dedicated cache appliances with a bunch of RAM and some kind of modified GPU with network access and crypto acceleration in each core?

saq7 2y ago
Humorous
zokier 2y ago
Have you seen Netflix OCAs? Current off-the-shelf hardware goes really long way https://openconnect.netflix.com/en/appliances/ https://openconnect.netflix.com/en/appliances/ https://news.ycombinator.com/item?id=32519881 https://news.ycombinator.com/item?id=32519881

ezoe 2y ago

I disagree. Storage is expensive. Think of an old video uploaded 15 years ago with total view count of 1k. You can't just put it to a cheap cold storage. Someday, somebody is going to watch it and you have to retrieve it instantly or that somebody will be disappointed.

Delmololo 2y ago
You can. Yt for example deletes your 720p after a while and replaces it with a potato. And if you watch a old not relevant yt and it starts after 10 seconds instead of now, no one really cares. You can put that old highly encoded potato at your huge and cheap storage system de located somewhere around the globe were it's just cheap (energy). You can also calculate in the time for a band robot and only store half or the first minute of that potato on your cheap storage and let the robot grab the rest of it. After all if video is your main thing plenty of weird optimizations start to make sense.
- olivermuty 2y ago
  Tape robot time is NOT measured in minutes but hours. Also you wouldn't pull individual files like this out of tape. Tape is for long term sunk storage, not cold infrequent access like a youtube video. I know aws glacier has an "expedited retrieval time" of 1-5 minutes, but that is not how typical tape setups work. Frankly I would be very interested in what actually hides behind that product.
  - joking 2y ago
    You can use big disk, but not be able to access all the files on the disk with the same frequency, so you have 20% of the disk dedicated to hot storage, and 80% of it to cold storage. Cold storage access is queued , so the 1-5 minutes can come from there.
    - olivermuty 2y ago
      Yeah I looked closer and I think they are basically just packaging up some kind of offering on top of spinning rust for the two first glacier tiers.

solarized 2y ago

Gilfoyle: ":smiling"

graemep 2y ago

> The biggest cost is as you imagine the streaming - getting the video to the viewer. I seem to remember Google own some network infrastructure? That saves some money. On top of that at their size you are going to get things cheaper. > The other big expense - people! All those engineers to build back and front end systems. That’s what ruined us - too many people were needed and not enough money coming in so we were burning cash. There should be economies of scale on that. Its harder to build and maintain bigger systems, but the work required does not scale linearly with size.

mike_hearn 2y ago
Google own vast network infrastructure. The day Google acquired YouTube (I was there) they discovered that YT was on the verge of collapse and would run out of bandwidth entirely within months. There was an emergency programme put in place with hundreds of engineers reallocated to an effort to avoid site collapse, with the clever/amusing name of BandAid. BandAid was a success. YouTube's history might look very different if it wasn't. It consisted of a massive crash buildout of a global CDN, something that Google historically hadn't had (CDNs aren't useful if everything you serve is dynamically generated). One reason BandAid could happen was that Google had a large and sophisticated NetOps operation already in place, which was already acquiring long haul unused fibre wavelengths at scale for the purposes of moving the web search index about. So CDN nodes didn't need to be far from a Google backbone POP, and at that point moving bits around on that backbone was to some extent "free" because the bulk of the cost was in the fibre rental and the rackspace+equipment on either end. Also it was being paid for already by the needs of search+ads. Over time Google steadily moved more stuff over to their infrastructure and off YouTube's own, but it was all driven by whatever would break next. Then you have all the costs that YouTube has that basic video sites don't. ContentID alone had costs that would break the back of nearly any other company but was made effectively mandatory by the copyright lawsuits against YouTube early on, which Google won but (according to internal legal analysis at least) that was largely due to the enormous good-faith and successful effort demonstrated by ContentID. And then of course you need a global ad sales team, a ton of work on ad targeting, etc. This story explains why Google can afford to do YouTube and you can't. The reality is that whilst they certainly have some sort of internal number for what YouTube costs, all such figures are inevitably kind of fantastical because so much infrastructure is shared and cross-subsidised by other businesses. You can't just magic up a global team of network engineers and fibre contracts in a few months, which is what YouTube needed in order to survive (and one of the main reasons they were forced to sell). No matter what price you come up with that in internal booking it will always be kinda dubious because such things aren't sold on the market.

game_the0ry 2y ago

> The biggest cost is as you imagine the streaming - getting the video to the viewer. It was a large part of our variable cost and we had a (literal) mad genius dev ops person holed up in his own office cave that managed the whole operation. Sort of make's Cloudflare's R2 look more impressive since they do not charge for egress.

gowthamgts12 2y ago
I'm digressing from the topic, but R2 looked good on paper, but have a long way to go in terms of reliability.
- game_the0ry 2y ago
  Hmm could you elaborate?

emodendroket 2y ago

Moderation seems like another big issue although the solution afaict seems to mostly involve shipping this work off to the Philippines or wherever and making people look at the most horrifying content imaginable for 40 hours a week at very low wages.