4 ms·

Can someone ELI5 the major challenges of delivering "broadcast quality" video over IP compared to cable? It seems crazy that pushing HD content over a standard

Can someone ELI5 the major challenges of delivering "broadcast quality" video over IP compared to cable? It seems crazy that pushing HD content over a standard aux cable is faster than downloading HD over IP. We have been pushing HD content to our TV's for 10+ years, but many people still have trouble streaming an HD movie from Netflix. Is this simply due to the overhead of IP transport supporting bidirectional communication? That is, a TV broadcast only needs to support a fixed set of N unidirectional flows (channels), but IP needs to support a dynamic set of N bidirectional flows?

corndoge 9y ago
As you said, broadcast is usually done with IP multicast, so it's not really bandwidth intensive. The netflix example is different, there you have a large group of people who all want different streams. At netflix's scale serving that many simultaneous connections from any reasonable amount of datacenters has the capability to saturate DC uplinks at peak hours. Hence why they've resorted to caching appliances in IXs.
- cnorthwood 9y ago
  It'd be great if it could be done using IP multicast! Sadly over the Internet to consumer devices multicast isn't an option, sometimes it is within a particular ISP for their services, but from an arbitrary broadcaster to a receiver it's all unicast (and often pull-based using HLS or DASH)
  - corndoge 9y ago
    I meant at real scale where it makes sense. You're correct in that your upstream ISP needs to support PIM to the customer edge, which is hard to get, but available for certain customers and especially so in datacenter fabrics.
    - cnorthwood 9y ago
      ah yes, I thought you were talking about distribution rather than inside a production facility. At that case the bandwidth of uncompressed 4K (which you want to do so you're not introducing compression artifacts) is in the order of several Gbps, so bandwidth is pretty high, although multicast is used.
  - monocasa 9y ago
    Yeah, I worked on a startup to push IP multicast TV to consumers way back when. Went nowhere for the reasons you state, but we got some cool tech out of it regardless.
- chatmasta 9y ago
  Actually, I didn't realize that broadcast was done with IP multicast. I was thinking it used a separate, older technology. So does that mean that generally speaking, modern ISP's that bundle Internet + Cable push the cable content over the IP transport using multicast in the layer 2 DOCSIS network?
  - wil421 9y ago
    I know that AT&T pushes TV over IP but I no longer use a TV provider. Also when I had slower AT&T services over twisted pair they would only send the signal of the TV channel your we’re watching from the DSLAM to your house. It would take a noticeable second to switch channels.
  - discreditable 9y ago
    I used to have a fiber + TV package. It was definitely multicast. I found that when I swapped their router for my own I had to make sure IGMP was handled in order for TV and on-demand to work.
    - sofaofthedamned 9y ago
      Yes, I did the IGMPProxy dance too! I was amazed that a bit of software that is basically in all home routers is so old, unloved and misunderstood. Multicast is a funny thing. For years people thought it was the future until they realised 1) it requires the WAN end to support it and 2) people to watch stuff at the same time.
  - monocasa 9y ago
    It's not the only way, but yes, there are absolutely cable companies that transmit via IP multicast. There'll still be other filtering going that bundles a couple channels together into the radio/DOCSIS version of a channel so that the end devices don't have to digitally filter all of the firehose of data, but then it's IP multicast on top of that.
- myrandomcomment 9y ago
  One really funny story about this. When AT&T was creating U-verse way back when they picked IP Multicast as the deliver method for the video. In one of their first test rollouts to the engineers on the projects house they found an issue. The engineers kid picked up the remote and started hitting the channel up button rapidly. It takes a few second for the Multicast join to be processed so this really did not work that well. The fix was that when you join say channel 100, the box joins the streams for channel 96-104 so you get the picture ASAP if you push the up/down channel button. This is why you see a delay if you jump to 200 from 100. The Multicast join has to be processed. Anyway that is the story I heard from one of their router vendors. Technically it is all correct.
  - iforgotpassword 9y ago
    I think other (more modern?) systems solve that by requesting the first few seconds after a channel switch via unicast in parallel to the join request, to cover up the time until the join is completed. I discovered this when I swapped my ISP router for a Linux box and didn't forward igmp. When turning on the TV I saw a frozen image. After switching the channel I got about 4 seconds of TV and then freeze again. Some wiresharking hinted me at the igmp and multicast I forgot to properly handle, and after setting it all up I saw a couple of seconds of unicast UDP data after every channel switch.
    - myrandomcomment 9y ago
      The time to start the single stream would be not much faster or at all to the mcast join. Most channels are broadcast to the local head node (main router in an service area). The join just builds the tree down past the head to the end user. Last time I looked at / worked on / designed these systems was maybe 8 years ago, but I do not except it to have changed to much. But you never know. I think the most interesting thing was this use case was what mcast was really designed for and the only real world use case I have seen beyond stuff in the finance market (market data feeds from the exchanges are multicast). Well there is OSPF...
      - 9y ago
cnorthwood 9y ago
It's essentially the difference between circuit switching (dedicating a whole physical link to a particular signal) and packet switching (breaking it down into smaller chunks that can each be dealt with independently throughout a network, and then get reassembled on the other end). The former is easier to do (no overhead of disassembling/reassembling) but less flexible.
- cup-of-tea 9y ago
  Yep. Exactly. Networking 101 but easily forgotten when working in a packet switching world for so long.
jgalar 9y ago
I would guess cable video is multicasted once to many customers while online streaming has to be served one-to-one. From the viewer's standpoint, the bandwidth usage is roughly the same, but the broadcaster and CDNs have a lot more data to deliver.
- ajanuary 9y ago
  There is an awful lot of data that is routed around internally before you hit transmission. That's more what's being talked about here, not delivering the video to viewers.
kvm000 9y ago
Challenges are mostly around keeping lock-step timing across a lot of separate devices in the system at scale (using PTP) for frame-accurate switching, and working through getting the SMPTE-2110 standards completed for interop, but there huge benefits on ability to scale to well beyond what is practical with coax, as well as cost savings on cabling with more density (and weight savings, for mobile production trucks which is important). There are also big architectural advantages of having more dynamic infrastructure and device discovery (AMWA NMOS, etc) rather than hard-wired coax signal paths. There are a lot of scenarios with massive scale on the back-end, with lots of streams managed as part of the production process ahead of delivery to consumers... We have an IP video router (https://evertz.com/products/EXE-VSR https://evertz.com/products/EXE-VSR) that has 2,304 10Gbps ports (moving to 25Gbps per port), each of which can do 6x fully uncompressed SMPTE 2110 video (1080i 29.97fps) flows in each direction, with a 46Tbps non-blocking back-plane so the scale can get a bit crazy. The full scale back-end stuff is pretty invisible from the consumer side. Eventually that internal infrastructure feeds into distribution encoders that produce lower bitrate streams for cable/sat/web distribution (for real-time events, with separate file delivery for VOD platforms like iTunes & Netflix)

corndoge 9y ago

As you said, broadcast is usually done with IP multicast, so it's not really bandwidth intensive. The netflix example is different, there you have a large group of people who all want different streams. At netflix's scale serving that many simultaneous connections from any reasonable amount of datacenters has the capability to saturate DC uplinks at peak hours. Hence why they've resorted to caching appliances in IXs.

cnorthwood 9y ago
It'd be great if it could be done using IP multicast! Sadly over the Internet to consumer devices multicast isn't an option, sometimes it is within a particular ISP for their services, but from an arbitrary broadcaster to a receiver it's all unicast (and often pull-based using HLS or DASH)
- corndoge 9y ago
  I meant at real scale where it makes sense. You're correct in that your upstream ISP needs to support PIM to the customer edge, which is hard to get, but available for certain customers and especially so in datacenter fabrics.
  - cnorthwood 9y ago
    ah yes, I thought you were talking about distribution rather than inside a production facility. At that case the bandwidth of uncompressed 4K (which you want to do so you're not introducing compression artifacts) is in the order of several Gbps, so bandwidth is pretty high, although multicast is used.
- monocasa 9y ago
  Yeah, I worked on a startup to push IP multicast TV to consumers way back when. Went nowhere for the reasons you state, but we got some cool tech out of it regardless.
chatmasta 9y ago
Actually, I didn't realize that broadcast was done with IP multicast. I was thinking it used a separate, older technology. So does that mean that generally speaking, modern ISP's that bundle Internet + Cable push the cable content over the IP transport using multicast in the layer 2 DOCSIS network?
- wil421 9y ago
  I know that AT&T pushes TV over IP but I no longer use a TV provider. Also when I had slower AT&T services over twisted pair they would only send the signal of the TV channel your we’re watching from the DSLAM to your house. It would take a noticeable second to switch channels.
- discreditable 9y ago
  I used to have a fiber + TV package. It was definitely multicast. I found that when I swapped their router for my own I had to make sure IGMP was handled in order for TV and on-demand to work.
  - sofaofthedamned 9y ago
    Yes, I did the IGMPProxy dance too! I was amazed that a bit of software that is basically in all home routers is so old, unloved and misunderstood. Multicast is a funny thing. For years people thought it was the future until they realised 1) it requires the WAN end to support it and 2) people to watch stuff at the same time.
- monocasa 9y ago
  It's not the only way, but yes, there are absolutely cable companies that transmit via IP multicast. There'll still be other filtering going that bundles a couple channels together into the radio/DOCSIS version of a channel so that the end devices don't have to digitally filter all of the firehose of data, but then it's IP multicast on top of that.
myrandomcomment 9y ago
One really funny story about this. When AT&T was creating U-verse way back when they picked IP Multicast as the deliver method for the video. In one of their first test rollouts to the engineers on the projects house they found an issue. The engineers kid picked up the remote and started hitting the channel up button rapidly. It takes a few second for the Multicast join to be processed so this really did not work that well. The fix was that when you join say channel 100, the box joins the streams for channel 96-104 so you get the picture ASAP if you push the up/down channel button. This is why you see a delay if you jump to 200 from 100. The Multicast join has to be processed. Anyway that is the story I heard from one of their router vendors. Technically it is all correct.
- iforgotpassword 9y ago
  I think other (more modern?) systems solve that by requesting the first few seconds after a channel switch via unicast in parallel to the join request, to cover up the time until the join is completed. I discovered this when I swapped my ISP router for a Linux box and didn't forward igmp. When turning on the TV I saw a frozen image. After switching the channel I got about 4 seconds of TV and then freeze again. Some wiresharking hinted me at the igmp and multicast I forgot to properly handle, and after setting it all up I saw a couple of seconds of unicast UDP data after every channel switch.
  - myrandomcomment 9y ago
    The time to start the single stream would be not much faster or at all to the mcast join. Most channels are broadcast to the local head node (main router in an service area). The join just builds the tree down past the head to the end user. Last time I looked at / worked on / designed these systems was maybe 8 years ago, but I do not except it to have changed to much. But you never know. I think the most interesting thing was this use case was what mcast was really designed for and the only real world use case I have seen beyond stuff in the finance market (market data feeds from the exchanges are multicast). Well there is OSPF...
    - 9y ago

cnorthwood 9y ago

It's essentially the difference between circuit switching (dedicating a whole physical link to a particular signal) and packet switching (breaking it down into smaller chunks that can each be dealt with independently throughout a network, and then get reassembled on the other end). The former is easier to do (no overhead of disassembling/reassembling) but less flexible.

cup-of-tea 9y ago
Yep. Exactly. Networking 101 but easily forgotten when working in a packet switching world for so long.

jgalar 9y ago

I would guess cable video is multicasted once to many customers while online streaming has to be served one-to-one. From the viewer's standpoint, the bandwidth usage is roughly the same, but the broadcaster and CDNs have a lot more data to deliver.

ajanuary 9y ago
There is an awful lot of data that is routed around internally before you hit transmission. That's more what's being talked about here, not delivering the video to viewers.

kvm000 9y ago

Challenges are mostly around keeping lock-step timing across a lot of separate devices in the system at scale (using PTP) for frame-accurate switching, and working through getting the SMPTE-2110 standards completed for interop, but there huge benefits on ability to scale to well beyond what is practical with coax, as well as cost savings on cabling with more density (and weight savings, for mobile production trucks which is important). There are also big architectural advantages of having more dynamic infrastructure and device discovery (AMWA NMOS, etc) rather than hard-wired coax signal paths. There are a lot of scenarios with massive scale on the back-end, with lots of streams managed as part of the production process ahead of delivery to consumers... We have an IP video router (https://evertz.com/products/EXE-VSR https://evertz.com/products/EXE-VSR) that has 2,304 10Gbps ports (moving to 25Gbps per port), each of which can do 6x fully uncompressed SMPTE 2110 video (1080i 29.97fps) flows in each direction, with a 46Tbps non-blocking back-plane so the scale can get a bit crazy. The full scale back-end stuff is pretty invisible from the consumer side. Eventually that internal infrastructure feeds into distribution encoders that produce lower bitrate streams for cable/sat/web distribution (for real-time events, with separate file delivery for VOD platforms like iTunes & Netflix)