3 ms·

It's a bit off topic, but I've never understood while people hold Netflix as some engineering Holy Grail. Their product is straightforward feature-wise, and pu

It's a bit off topic, but I've never understood while people hold Netflix as some engineering Holy Grail. Their product is straightforward feature-wise, and pushing a low-teen megabits per second of static video data on the internet in a somewhat timely manner is not a huge technical challenge nowadays (or 10 years ago). Doing stuff like real-time video streaming, where you have to encode and push video to users with very low latency requirements (like Google Stadia) or with moderately relaxed latency but broadcast to a lot of users, like Twitch, or having a mind-bogglingly huge library like Youtube, is probably orders of magnitude harder. I do like their shows, and probably a lot of technical wizardry VFX goes into making them, but getting the bytes to the end user is not it. I'm sure there's a lot of adversarial smarts there, where brilliant engineers come up with incredibly complex solutions to simple problems, and it requires even more brilliance to make things run smoothly, but I'm sure their problems could be solved with simple pragmatic engineering.

j45 2y ago
It’s not straight forward at scale
- sieabahlpark 2y ago
  Care to elaborate? Most things boil down to client side features like streaming the video with HLS or DASH, even if it's not live just to get the encoding benefits. DRM is its own thing. I think it's important to have the distinction between delivering video content and the in-app experience. I do think streaming VOD is way easier than live content.
  - YZF 2y ago
    You also need a CDN. NetFlix deploys boxes all over the place to cache the content. I agree there's nothing in there that's not understood from an engineering/technology perspective. It's not fusion or curing cancer. But scale and complexity compound simple engineering problems. There monitoring, there's billing, there's machine learning, there's software lifecycle, multitude of different devices types and networks, all of that at extreme scales that need to work pretty reliably.
    - sieabahlpark 2y ago
      [dead]
hluska 2y ago
Take those low teen megabits and scale it through $x users in $y locations streaming $z content items. Consider that 15% of those $x users will take to the internet to complain about problems. Also consider that once you’re a big enough name, mainstream media will cover those complaints on slow news days. Finally consider that a large portion of budget goes into constantly maximizing x, y and z. Thats where the engineering problem goes from trivial to extremely complicated. Lots and lots of people demanding a similar quality of service across a wide pool of content on a wide pool of devices. And that doesn’t even fully cover last mile issues.
crdrost 2y ago
Slightly bigger challenges Netflix faces: • You have deals with N big media companies who each have their own restrictions on who can stream what from where. The list is constantly changing, so permissions to view media are locale-specific and revokable; you need a way to say “okay this person is not allowed any more to do that.” • Multiple-screen detection emphatically needs to be rock solid. Someone is going to unplug their Roku player when their laptop says “you're watching from too many screens,” and by the time they get back to their laptop you need to be detecting them as streaming from 0 screens. At the same time a hiccup in this process shouldn't cause like 3% of your users to get a big streaming interruption as they don't seem to be online. • You have to recommend stuff based on what this person has watched. An acquisition team needs to do cluster analysis on this to get new stuff to fill all of the different clusters of interests that emerge in your user base. • People will search for shows you don't have. (Because of point 1, the big media companies only permit access to a fraction of their backlog.) You have to know this media that you don't have access to, well enough to recommend something related that might keep the user on Netflix instead of hopping to another service. • All of this has to happen on pretty low latencies when someone starts up Netflix. That is, anybody who jumps into Netflix should see a personalized view of what they were watching, what they can watch, filtered by your allow lists and not cached on their device, within just a few seconds. • All of this has to be portable to all of the different platforms Netflix supports.
- torginus 2y ago
  > You have deals with N big media companies who each have their own restrictions on who can stream what from where. The list is constantly changing, so permissions to view media are locale-specific and revokable; you need a way to say “okay this person is not allowed any more to do that.” Yeah, international media content licensing is very likely Netflix's moat. I'd wager starting a streaming service, and getting the big studios to host their content is pretty much impossible. Recommender systems are pretty well studied, and anyways, I'm sure it's significance for Netflix is overrated. People (including me) tend to watch the show everyone's talking about. It's a much more hairy issue for a site like Youtube, where users upload decades worth of content every day, and the site has to figure out what to show to which viewers, where the content's shelf-life might be measured in days, so building up collaborative filtering data might not be feasible. All the other technical issues are probably non-trivial, but I don't think any of them requires world-changing engineering prowess. Most engineering is non-trivial. If I attempted to design a washing machine, I'd probably fail miserably, and figuring how to do it well probably took collectively thousands of engineer-years. Doesn't mean it requires exclusively unicorn engineers.
deleted 2y ago
[deleted]
phonon 2y ago
Bingo! Watch and be afraid.... https://youtu.be/s-vJcOfrvi0 https://youtu.be/s-vJcOfrvi0
- computerdork 2y ago
  Both the video clip and the guy's personal experience at Netflix were Hilarious. And have gone through this too (as most of us have). Yeah, typical corporate development, where often the smallest of changes can be a nightmare:)
- jarsin 2y ago
  This made reading this whole thread worth it. Right when i was feeling bad about coasting in some small company..
- TremendousJudge 2y ago
  All that effort to make a weird "feature" that no user actually wanted. The reason it took them all that work to do is that the assumption that you wouldn't reorder seasons was baked into the architecture. For the engineers that designed it was just unthinkable, and for good reason.
kapad 2y ago
A bit "off" the off-topic point. - I'm not really sure Stadia can be called a "technical achievement at scale". Mainly because it didn't achieve all that much scale to begin with. Definitely a technical achievement in cloud gaming. But the implementation at scale is something that I now see Sony and Xbox working toward (along with their console partners - hardware and software). - Youtube, definitely, given the content base, the number of active users at literally anytime, is a monumental achievement. But so are a bunch of porn websites. And I know the porn industry doesn't have a good rep. Even in engineering. Still, porn sees more active users than Youtube and netflix combined, albeit, on a bunch of different websites, but a few large enough websites that are comparable to Youtube in terms of active users, and catalogue size. Also, these companies manage to host all of this content and delivery it all over the world on a much smaller budget than Youtube. Not to take away any of Youtube's achievements, but from a technical point of view, I do think the porn industry wins. To begin with, porn streaming is older than youtube, but quite a few years actually. Additionally, from what I've heard, some of the popular live streams on OnlyFans (and similar websites) see more concurrent viewers than most YouTube live streams.
hnfong 2y ago
I've always thought the myth came from the fact/rumor that Netflix paid above FAA(N?)G rates for engineers...

j45 2y ago

It’s not straight forward at scale

sieabahlpark 2y ago
Care to elaborate? Most things boil down to client side features like streaming the video with HLS or DASH, even if it's not live just to get the encoding benefits. DRM is its own thing. I think it's important to have the distinction between delivering video content and the in-app experience. I do think streaming VOD is way easier than live content.
- YZF 2y ago
  You also need a CDN. NetFlix deploys boxes all over the place to cache the content. I agree there's nothing in there that's not understood from an engineering/technology perspective. It's not fusion or curing cancer. But scale and complexity compound simple engineering problems. There monitoring, there's billing, there's machine learning, there's software lifecycle, multitude of different devices types and networks, all of that at extreme scales that need to work pretty reliably.
  - sieabahlpark 2y ago
    [dead]

hluska 2y ago

Take those low teen megabits and scale it through $x users in $y locations streaming $z content items. Consider that 15% of those $x users will take to the internet to complain about problems. Also consider that once you’re a big enough name, mainstream media will cover those complaints on slow news days. Finally consider that a large portion of budget goes into constantly maximizing x, y and z. Thats where the engineering problem goes from trivial to extremely complicated. Lots and lots of people demanding a similar quality of service across a wide pool of content on a wide pool of devices. And that doesn’t even fully cover last mile issues.

crdrost 2y ago

Slightly bigger challenges Netflix faces: • You have deals with N big media companies who each have their own restrictions on who can stream what from where. The list is constantly changing, so permissions to view media are locale-specific and revokable; you need a way to say “okay this person is not allowed any more to do that.” • Multiple-screen detection emphatically needs to be rock solid. Someone is going to unplug their Roku player when their laptop says “you're watching from too many screens,” and by the time they get back to their laptop you need to be detecting them as streaming from 0 screens. At the same time a hiccup in this process shouldn't cause like 3% of your users to get a big streaming interruption as they don't seem to be online. • You have to recommend stuff based on what this person has watched. An acquisition team needs to do cluster analysis on this to get new stuff to fill all of the different clusters of interests that emerge in your user base. • People will search for shows you don't have. (Because of point 1, the big media companies only permit access to a fraction of their backlog.) You have to know this media that you don't have access to, well enough to recommend something related that might keep the user on Netflix instead of hopping to another service. • All of this has to happen on pretty low latencies when someone starts up Netflix. That is, anybody who jumps into Netflix should see a personalized view of what they were watching, what they can watch, filtered by your allow lists and not cached on their device, within just a few seconds. • All of this has to be portable to all of the different platforms Netflix supports.

torginus 2y ago
> You have deals with N big media companies who each have their own restrictions on who can stream what from where. The list is constantly changing, so permissions to view media are locale-specific and revokable; you need a way to say “okay this person is not allowed any more to do that.” Yeah, international media content licensing is very likely Netflix's moat. I'd wager starting a streaming service, and getting the big studios to host their content is pretty much impossible. Recommender systems are pretty well studied, and anyways, I'm sure it's significance for Netflix is overrated. People (including me) tend to watch the show everyone's talking about. It's a much more hairy issue for a site like Youtube, where users upload decades worth of content every day, and the site has to figure out what to show to which viewers, where the content's shelf-life might be measured in days, so building up collaborative filtering data might not be feasible. All the other technical issues are probably non-trivial, but I don't think any of them requires world-changing engineering prowess. Most engineering is non-trivial. If I attempted to design a washing machine, I'd probably fail miserably, and figuring how to do it well probably took collectively thousands of engineer-years. Doesn't mean it requires exclusively unicorn engineers.

deleted 2y ago

[deleted]

phonon 2y ago

Bingo! Watch and be afraid.... https://youtu.be/s-vJcOfrvi0 https://youtu.be/s-vJcOfrvi0

computerdork 2y ago
Both the video clip and the guy's personal experience at Netflix were Hilarious. And have gone through this too (as most of us have). Yeah, typical corporate development, where often the smallest of changes can be a nightmare:)
jarsin 2y ago
This made reading this whole thread worth it. Right when i was feeling bad about coasting in some small company..
TremendousJudge 2y ago
All that effort to make a weird "feature" that no user actually wanted. The reason it took them all that work to do is that the assumption that you wouldn't reorder seasons was baked into the architecture. For the engineers that designed it was just unthinkable, and for good reason.

kapad 2y ago

A bit "off" the off-topic point. - I'm not really sure Stadia can be called a "technical achievement at scale". Mainly because it didn't achieve all that much scale to begin with. Definitely a technical achievement in cloud gaming. But the implementation at scale is something that I now see Sony and Xbox working toward (along with their console partners - hardware and software). - Youtube, definitely, given the content base, the number of active users at literally anytime, is a monumental achievement. But so are a bunch of porn websites. And I know the porn industry doesn't have a good rep. Even in engineering. Still, porn sees more active users than Youtube and netflix combined, albeit, on a bunch of different websites, but a few large enough websites that are comparable to Youtube in terms of active users, and catalogue size. Also, these companies manage to host all of this content and delivery it all over the world on a much smaller budget than Youtube. Not to take away any of Youtube's achievements, but from a technical point of view, I do think the porn industry wins. To begin with, porn streaming is older than youtube, but quite a few years actually. Additionally, from what I've heard, some of the popular live streams on OnlyFans (and similar websites) see more concurrent viewers than most YouTube live streams.

hnfong 2y ago

I've always thought the myth came from the fact/rumor that Netflix paid above FAA(N?)G rates for engineers...