5 ms·
People just do not appreciate how many gotchas can pop up doing anything live. Sure, Netflix might have a great CDN that works great for their canned content an
by dylan604 2y ago
People just do not appreciate how many gotchas can pop up doing anything live. Sure, Netflix might have a great CDN that works great for their canned content and I could see how they might have assumed that's the hardest part.
Live has changed over the years from large satellite dishes beaming to a geosat and back down to the broadcast center($$$$$), to microwave to a more local broadcast center($$$$), to running dedicated fiber long haul back to a broadcast center($$$), to having a kit with multiple cell providers pushing a signal back to a broadcast center($$), to having a direct internet connection to a server accepting a live http stream($).
I'd be curious to know what their live plan was and what their redundant plan was.
- diggan 2y ago> People just do not appreciate how many gotchas can pop up doing anything live. Sure thing, but also, how much resources do you think Netflix threw on this event? If organizations like FOSSDEM and CCC can do live events (although with way smaller viewership) across the globe without major hiccups on (relatively) tiny budgets and smaller infrastructure overall, how could Netflix not?
- phyrex 2y agoScale changes everything, I don't think it's fair to shrug this off
- tiluha 2y agoThis is true, but scale comes after production. Once you have the video encoded on a server with a stable connection the hard part is over. What netflix failed to do is spread the files to enough servers around the globe to handle the load. I'm surprised they were unable(?) to use their network of edge servers to handle the live stream. Just run the stream with a 10 second delay and in that time push the stream segments to the edge server
- dylan604 2y agoThis right here is where I'd expect the failure to occur. This isn't Joey Beercan running OBS using their home internet connectivity. This is a major broadcast. I'd expect a full on broadcast truck/trailer. If they were attempting to broadcast this with the ($) option directly to a server from onsite, then I would demand my money back. Broadcasting a live IP signal just falls on its face so many times it's only the cheap bastard option. Get the video signal as a video signal away from the live location to a facility with stable redundant networking. This is the kind of thinking someone only familiar with computers/software/networking would think of rather than someone in broadcasting. It's nice to think about disrupting, but this is the kind of failure that disruptors never think about. Broadcasters have been there done that with ensuring live broadcasts don't go down because an internet connection wasn't able to keep up.
- shrubble 2y agoLumen has their Vyvvyx product/service which uses fiber for broadcast television.
- chgs 2y agoI’ve been using vyvx since it was called global crossing/genesis, it was fairly unique when it started, but point to point ip distributon of programs has been the norm for at least 15 years. Still have backup paths on major events on a different technology, you’d be surprised how common a dual failure on two paths can be. For example output from the euro football this summer my mai paths were on a couple of leased lines with -7, but still had a backup on some local internet into a different city just incase there was a meltdown of the main providers network (it’s happened before with ipath, automation is great until it isn’t)
- mmooss 2y ago> Once you have the video encoded on a server with a stable connection the hard part is over. The hard part is over, and people new to the problem think they are almost done, but then the next part turns out to be 100x harder. Lots of people can encode a video.
- diggan 2y agoYeah, I agree with this, especially the everything part. Netflix isn't exactly a scrappy FOSS/hackers organization or similar.
- BlueTemplar 2y agoLast I checked, p2p solves a lot of the scaling issues. Haven't Asian live sports been using p2p already two decades ago ? (What is the biggest Peertube livestream so far ?)
- dylan604 2y ago> how much resources do you think Netflix threw on this event? Based on the results, I hope it was a small team working 20% time on the idea. If you tell me they threw everything they had at it to this result, then that's even more embarrassing for them.
- throw0101b 2y ago> If organizations like FOSSDEM and CCC can do live events (although with way smaller viewership) […] Or, for that matter, Youtube (Live) and Twitch.
- bostik 2y agoThe CCC video crew has its fair share of geeks from broadcasting corporations and studio houses. Their combined institutional knowledge about live events and streaming distribution is probably in the same ballpark as that of giant global TV networks. They also have the benefit of having practiced their craft at the CCC events for more than a decade. Twice a year. (Their summer event is smaller but still fairly well known. Links to talks show up on HN every now and then.) Funky anecdote: the video crew at Assembly have more broadcasting and live AV gear for their annual event than most medium-sized studios.
- akira2501 2y ago> Their combined institutional knowledge about live events and streaming distribution Now if they could just get audio levels and compression figured out.
- bena 2y agoIt is weird because this was a solved problem. Every major network can broadcast the Super Bowl without issue. And while Netflix claims it streamed to 280 million, that’s if every single subscriber viewed it. Actual numbers put it in the 120 million range. Which is in line with the Super Bowl. Maybe Netflix needs to ask CBS or ABC how to broadcast
- ironhaven 2y agoDo you live stream the superbowl? Me and everyone I know watch it over antenna broadcast tv. I think it is easier to have millions of tvs catch airwaves vs millions of point to point https video streams.
- dylan604 2y agoIf you watch it over cable, you're live streaming it. Let's face it, that's where the vast majority of viewers see it. Few people view OTA even if the quality is better. Live sports do not broadcast the event directly to a streamer. They push it to their broadcast centers. It then gets distributed from there to whatever avenues it needs to go. Trying to push a live IP stream directly from the remote live venue rarely works as expected. That's precisely why the broadcasters/networks do not do it that way
- throw0101b 2y ago> If you watch it over cable, you're live streaming it. Which is probably done over the cableco's private network (not the public Internet) with a special VLAN used for television (as opposed to general web access). They're probably using multicast.
- toast0 2y agoIs cable video over IP now? Last time I looked (which was forever ago), even switched video was atsc with a bit of messaging for the cable box to ask what channel to tune to, and to keep the stream alive. TV over teleco systems seems to be highly multicast, so kind of similar, headend only has to send the content once, in a single bitrate. Not really the same as an IP service live stream where the distribution point is sending out one copy per viewer and participating in bitrate adaptation. AFAIK, Netflix hasn't publicly described how they do live events, but I think it's safe to assume they have some amount of onsite production that outputs the master feed for archiving and live transcoding for the different bitrate targets (that part may be onsite, or at a broadcast center or something cloudy), and then goes to a distribution network. I'd imagine their broadcast center/or onsite processing feeds to a limited number of highly connected nodes that feed to most of their CDN nodes; maybe more layers. And then clients stream from the CDN nodes. Nobody would stream an event like this direct from the event; you've got to have something to increase capacity.
- colesantiago 2y agoThis is the whole point of chaos engineering that was invented at Netflix, which tests the resiliency of these systems. I guess we now know the limits of what "at scale" is for Netflix's live-streaming solution. They shouldn't be failing at scale on a huge stage like this. I look forward to reading the post mortem about this.
- dylan604 2y agoEveryone keeps mentioning at scale. I seriously doubt this was an "at scale" problem. I have strong suspicion this was a failure at the origination point being able to push a stable signal. That is not an "at scale" issue, but a hubris of we can do better/cheaper than broadcasting standard practices
- zinodaur 2y agoIf it was a problem at origin, why did it get better/worse as viewership fell/rose?
- kristjansson 2y agoAs counterpoint, I observed 2-3 drops in bitrate, but an otherwise fine experience. So the problem seems to have been in dissemination, not at the origin.
- collinrapp 2y agoYeah, I was switching between my phone and desktop to watch the stream and I had a seamless experience on both devices the entire time. I’m not sure why so many people are assuming this was a universal experience.
- woobar 2y agoI've tried to watch an old Seinfeld episode during this event. It was freezing every few minutes even at downgraded bitrate. A video that should be on my local CDN node.
- kortilla 2y ago
- xyst 2y agoYou are making excuses for a multibillion dollar company that has been in this game for many years. Maybe the first to market in streaming. This isn’t NFLX’s first rodeo in live streaming. Have seen a handful of events pop up in their apps. There is no excuse. All of the resources and talent at their disposal, and they looked absolutely amateurish. Poor optics. I would be amazed if they are able to secure another exclusive contract like this in the future.
- DrillShopper 2y agoA company that readily admits it burns out SWRs and SREs in exchange for the big bucks. Just what the fuck are these people doing? If I were a major investor in them I'd be pissed.
- throwaway2037 2y agoYou probably are a major investor, incidentally, through your pension fund(s) and retirement savings. It is hard to avoid Netflix if you are doing any kind of broad-based index investing.
- Xenoamorphous 2y agoSorry for the off topic but what’s this thing that I only come across in Hacker News about referring to a company by their stock exchange name (APPL, MSFT, etc) outside of a stock context? It seems really weird to me.
- rashabd 2y agomerriam-webster.com/dictionary/brevity
- Xenoamorphous 2y agoWriting APPL instead of Apple doesn’t get you any fewer keystrokes.
- chgs 2y agoYou’re talking about the contribution from the venue to the boardcast centre, increasingly not a full program but being mixed remotely. That’s a very different area to transmission of live to end users.
- dylan604 2y agoWhat are you talking about? The signal coming from a live event is the full package. The output of “the truck” has multiple outs including the full mix of all grafix, some only have the mix minus any branding, etc. While the isos get recorded in the truck, they are not pushed out to the broadcast center. All of the “mixing” as you call it is done in the truck. If you’ve never seen it, it is quite impressive. In one part of the truck is the director and the technical director. The director is the one calling things like “ready camera 1”, “take 1”, etc. the TD is the one on the switcher pushing the actual buttons on the console to make it happen. Next to them is the graphics team prepping all of the stats made available to the TD to key in. In another area is the team of slomo/replay that are taking the feeds from all of the cameras to recorders that allow the operators to pull out the selects and make available for the director/TD to cut to. Typically in the back of the truck is the audio mixer that mixes all of the mics around the event in real time. All of that creates the signal you see on your screen. It leaves the back of the truck and heads out to wherever the broadcaster has better control
- chgs 2y agoNot nowadays, more and more remote production for larger and larger events, and it’s coming on rapidly. Directors are increasing sitting in centralised control rooms rather than in a scanner. BT sport are interesting, spin up graphics, replay, etc in an AWS environment a couple of hours before. I was impressed by their uefa youth league coverage a couple of years ago, and they aren’t slowing down https://www.limitlessbroadcast.tv/portfolio/uefa-youth-league-match-coverage/ https://www.limitlessbroadcast.tv/portfolio/uefa-youth-leagu... https://www.svgeurope.org/blog/headlines/stratospheric-revolution-bt-sport-reaches-for-cloud-in-the-next-phase-of-its-decentalised-remote-production-evolution/ https://www.svgeurope.org/blog/headlines/stratospheric-revol... Obviously not every broadcast, or even most, are remote now, but it’s an ever increasing number. I don’t know how the US industry works, I suspect the heavy union presence I’ve seen at places like NAB will slow it, but in Europe remote production is increasingly the future.
- avidphantasm 2y agoIs multicast a thing on the commercial internet? Seems like that could help.
- dilyevsky 2y agoIf commercial = public, then no - you can not use multicast for this. It is heavily used within some enterprise networks though like if you go to a gym with lots of TVs they are all likely on multicast