3 ms·

Netflix is supposed to be the bastion of microservices and the trailblazer of all-aws infrastructure. But as time goes by I just ask, all this work and costs a

Netflix is supposed to be the bastion of microservices and the trailblazer of all-aws infrastructure. But as time goes by I just ask, all this work and costs and complexity, to serve files? Yeah don't get me wrong, the size of the files are really big, AND they are streamed, noted. But it's not the programming complexity challenge that one would expect, almost all of the complexity seems to stem from metadata like when users stop watching, and how to recommend them titles to keep them hooked, and when to cut the titles and autoplay the next video to make them addicted to binge watching. Case in point, the blogpost speaks of a CPU concurrency bug and clients being servers? But never once refers to an actual business domain purpose. Like are these servers even loading video content? My bet is they are more on the optimizing engagement side of things. And I make this bet knowing that these are servers with high video-like load, but I'm confident that these guys are juggling 10TB/s of mouse metadata into some ML system more than I'm confident that they have some problem with the core of their technology which has worked since launch. As I say this, I know I'm probably wrong, surely the production issues are cause by high peak loads like a new chapter of the latest series or whatever. I'm all over the place, I just don't like netflix is what I'm saying

romanobro56 2y ago
[flagged]
- TZubiri 2y ago
  Rizz skibidi bro
PittleyDunkin 2y ago
> But as time goes by I just ask, all this work and costs and complexity, to serve files? You could say the same thing about the entire web.
- nicce 2y ago
  Not really. People are not posting data into Netlix. Netflix is mostly read-only. That is huge complexity reducer.
  - TZubiri 2y ago
    I thought about the complexity in terms of compute, but I guess if there's no input then there's no compute possible, as all functions are idempotent and static. At the very least their results are cacheable, or the input is centralized (admins/show producers)
  - jacksontheel 2y ago
    Every time you like/dislike/watchlist a movie you're posting data. When you're watching a movie your progress is constantly updated, posting data. Simple stuff but there's possibly hundreds of thousands of concurrent users doing that at any given moment.
    - nicce 2y ago
      Yes, but it is still counts only a fraction of the purpose of their infrastructure. There are no hard global real-time sync requirements. > When you're watching a movie your progress is constantly updated, posting data This can be implemented on server side and with read requests only. A proper comparison would be YouTube where people upload videos and comment stuff in real-time.
      - PittleyDunkin 2y ago
        > A proper comparison would be YouTube where people upload videos and comment stuff in real-time. Even in this one sentence you're conflating two types of interaction. Surely downloading videos is yet a third, and possibly the rest of the assets on the site a fourth. Why not just say the exact problem you think is worth of discussion with your full chest if you so clearly have one in mind?
        TZubiri 2y ago
toast0 2y ago
> But as time goes by I just ask, all this work and costs and complexity, to serve files? IMHO, a large amount of the complexity is all the other stuff. Account information, browsing movies, recommendations, viewed/not/how much seen, steering to local CDN nodes, DRM stuff, etc. The file servers have a lot less complexity; copy content to CDN nodes, send the client to the right node for the content, serve 400Gbps+ per node. Probably some really interesting stuff for their real time streams (but I haven't seen a blog/presentation on those) Transcoding is probably interesting too. Managing job queues isn't new, but there's probably some fun stuff around cost effectiveness.
bobdvb 2y ago
Netflix has done massive amounts of work on BSD to improve it's network throughput, that's part of them enabling their file delivery from their CDN appliances. https://people.freebsd.org/~gallatin/talks/euro2022.pdf https://people.freebsd.org/~gallatin/talks/euro2022.pdf They've also contributed significantly to open source tools for video processing, one of the biggest things that stands out is probably their VMAF tool for quantifying perceptual quality in video. It's probably the best open source tool for measuring video quality out there right now. It's also absolutely true that in any streaming service, the orchestration, account management, billing and catalogue components are waaaay more complex than actually delivering video on-demand. To counter one thing you've said: mouse movement... most viewing of premium content isn't done on web or even mobile devices. Most viewing time of paid content is done on a TV, where you're not measuring focus. But that's just a piece of trivia. As you said, you just don't like them, but they've done a lot for the open source community and that should be understood.
- TZubiri 2y ago
  Yeah I stand corrected. Video being one of the highest entropy types of data probably means they face state of the art throughput challenges. Which are inherently tied to cost and monetization. That said, free apps like tiktok and youtube probably face higher throughput, so the user-pays model probably means netflix is at the state of the art at high volume quality (both app experience and content) rather than sheer volume low quality or premium quality low volume markets. I mean serving millions of customers at 8 bucks per month. Which is not quite like serving billions.

romanobro56 2y ago

[flagged]

TZubiri 2y ago
Rizz skibidi bro

PittleyDunkin 2y ago

> But as time goes by I just ask, all this work and costs and complexity, to serve files? You could say the same thing about the entire web.

nicce 2y ago
Not really. People are not posting data into Netlix. Netflix is mostly read-only. That is huge complexity reducer.
- TZubiri 2y ago
  I thought about the complexity in terms of compute, but I guess if there's no input then there's no compute possible, as all functions are idempotent and static. At the very least their results are cacheable, or the input is centralized (admins/show producers)
- jacksontheel 2y ago
  Every time you like/dislike/watchlist a movie you're posting data. When you're watching a movie your progress is constantly updated, posting data. Simple stuff but there's possibly hundreds of thousands of concurrent users doing that at any given moment.
  - nicce 2y ago
    Yes, but it is still counts only a fraction of the purpose of their infrastructure. There are no hard global real-time sync requirements. > When you're watching a movie your progress is constantly updated, posting data This can be implemented on server side and with read requests only. A proper comparison would be YouTube where people upload videos and comment stuff in real-time.
    - PittleyDunkin 2y ago
      > A proper comparison would be YouTube where people upload videos and comment stuff in real-time. Even in this one sentence you're conflating two types of interaction. Surely downloading videos is yet a third, and possibly the rest of the assets on the site a fourth. Why not just say the exact problem you think is worth of discussion with your full chest if you so clearly have one in mind?
      - TZubiri 2y ago

toast0 2y ago

> But as time goes by I just ask, all this work and costs and complexity, to serve files? IMHO, a large amount of the complexity is all the other stuff. Account information, browsing movies, recommendations, viewed/not/how much seen, steering to local CDN nodes, DRM stuff, etc. The file servers have a lot less complexity; copy content to CDN nodes, send the client to the right node for the content, serve 400Gbps+ per node. Probably some really interesting stuff for their real time streams (but I haven't seen a blog/presentation on those) Transcoding is probably interesting too. Managing job queues isn't new, but there's probably some fun stuff around cost effectiveness.

bobdvb 2y ago

Netflix has done massive amounts of work on BSD to improve it's network throughput, that's part of them enabling their file delivery from their CDN appliances. https://people.freebsd.org/~gallatin/talks/euro2022.pdf https://people.freebsd.org/~gallatin/talks/euro2022.pdf They've also contributed significantly to open source tools for video processing, one of the biggest things that stands out is probably their VMAF tool for quantifying perceptual quality in video. It's probably the best open source tool for measuring video quality out there right now. It's also absolutely true that in any streaming service, the orchestration, account management, billing and catalogue components are waaaay more complex than actually delivering video on-demand. To counter one thing you've said: mouse movement... most viewing of premium content isn't done on web or even mobile devices. Most viewing time of paid content is done on a TV, where you're not measuring focus. But that's just a piece of trivia. As you said, you just don't like them, but they've done a lot for the open source community and that should be understood.

TZubiri 2y ago
Yeah I stand corrected. Video being one of the highest entropy types of data probably means they face state of the art throughput challenges. Which are inherently tied to cost and monetization. That said, free apps like tiktok and youtube probably face higher throughput, so the user-pays model probably means netflix is at the state of the art at high volume quality (both app experience and content) rather than sheer volume low quality or premium quality low volume markets. I mean serving millions of customers at 8 bucks per month. Which is not quite like serving billions.