4 ms·

I'm still surprised they did not get away from Nginx at that point and did something like Cloudflare.

by Thaxll 2y ago

alberth 2y ago
When you’re pushing the amount of data Netflix is, you need to work directly with the ISP & Exchanges. At least that’s my guess.
- yencabulator 2y ago
  Parent is talking purely about software. https://blog.cloudflare.com/how-we-built-pingora-the-proxy-that-connects-cloudflare-to-the-internet/ https://blog.cloudflare.com/how-we-built-pingora-the-proxy-t...
toast0 2y ago
Do you mean moving away from using Nginx, like Cloudflare moved to a custom replacement? [1] I don't think that's as needed for Netflix. My understanding is their CDN nodes are using Nginx to serve static files --- their content management system directs off peak transfers, and routes clients to nodes that it knows has the files. They don't run a traditional caching (reverse) proxy, and they most likely don't hit a lot of the things Cloudflare was hitting, because their use case is so different. (I haven't seen a discussion of how Netflix handles live events, that's clearly a different process than their traditional prerecorded media) [1] https://blog.cloudflare.com/how-we-built-pingora-the-proxy-that-connects-cloudflare-to-the-internet/ https://blog.cloudflare.com/how-we-built-pingora-the-proxy-t...
- drewg123 2y ago
  Yes, exactly.
- Thaxll 2y ago
  Yes I'm talking about Nginx running on those bsd boxes, they have such a custom design that writing your own static file process would have made sense.
  - toast0 2y ago
    Getting an HTTP server to work with the diversity of HTTP clients in the world that Netflix supports is not going to be fun, and NGINX is right there. As I understand it, they've made some changes to NGINX, but I don't think they've made a lot, and I don't think anything where the structure of NGINX was not conducive to it or limiting. I'm not one to shy away from building a custom design, but it's a lot easier when you control the clients, and Netflix has to work with everything from browsers to weird SDKs in TVs. Netflix OCA performance seems mostly bottlenecked on I/O/memory bandwidth (and cpu overhead for pacing?) and any sensible HTTP server for static files isn't going to use a lot of memory bandwidth processing inbound requests and calling sendfile on static files. So why spend the limited time of a small team to build something that's not going to make a big difference.

I'm still surprised they did not get away from Nginx at that point and did something like Cloudflare.

by Thaxll 2y ago

I'm still surprised they did not get away from Nginx at that point and did something like Cloudflare.

When you’re pushing the amount of data Netflix is, you need to work directly with the ISP & Exchanges. At least that’s my guess.

yencabulator 2y ago
Parent is talking purely about software. https://blog.cloudflare.com/how-we-built-pingora-the-proxy-that-connects-cloudflare-to-the-internet/ https://blog.cloudflare.com/how-we-built-pingora-the-proxy-t...

toast0 2y ago

Do you mean moving away from using Nginx, like Cloudflare moved to a custom replacement? [1] I don't think that's as needed for Netflix. My understanding is their CDN nodes are using Nginx to serve static files --- their content management system directs off peak transfers, and routes clients to nodes that it knows has the files. They don't run a traditional caching (reverse) proxy, and they most likely don't hit a lot of the things Cloudflare was hitting, because their use case is so different. (I haven't seen a discussion of how Netflix handles live events, that's clearly a different process than their traditional prerecorded media) [1] https://blog.cloudflare.com/how-we-built-pingora-the-proxy-that-connects-cloudflare-to-the-internet/ https://blog.cloudflare.com/how-we-built-pingora-the-proxy-t...

drewg123 2y ago
Yes, exactly.
Thaxll 2y ago
Yes I'm talking about Nginx running on those bsd boxes, they have such a custom design that writing your own static file process would have made sense.
- toast0 2y ago
  Getting an HTTP server to work with the diversity of HTTP clients in the world that Netflix supports is not going to be fun, and NGINX is right there. As I understand it, they've made some changes to NGINX, but I don't think they've made a lot, and I don't think anything where the structure of NGINX was not conducive to it or limiting. I'm not one to shy away from building a custom design, but it's a lot easier when you control the clients, and Netflix has to work with everything from browsers to weird SDKs in TVs. Netflix OCA performance seems mostly bottlenecked on I/O/memory bandwidth (and cpu overhead for pacing?) and any sensible HTTP server for static files isn't going to use a lot of memory bandwidth processing inbound requests and calling sendfile on static files. So why spend the limited time of a small team to build something that's not going to make a big difference.