6 ms·

So this is perhaps a lame post but I'm super tired, been up since 4am. I'll try and follow up with more detail in the morning. This article, in my opinion, is

So this is perhaps a lame post but I'm super tired, been up since 4am. I'll try and follow up with more detail in the morning. This article, in my opinion, is way off. I base that on the fact that I've been dancing with Netflix for a month, I might end up working with them to try and make NUMA machines serve up content faster. As in I'd be working on exactly what happens when you hit play. My take on how Netflix serves you movies is nothing like what this article says. They have servers in every ISP, the servers send a heartbeat to a conductor in AWS, the heartbeat says "I've got this content and I am this over worked", when you hit play the app reaches out the conductor and says "I want this", the conductor looks around and finds a server close to you that is not overloaded and off you go. That might look easy. It's not. Take a look at this post about how they fill a 100Gbit pipe: https://news.ycombinator.com/item?id=15367421 https://news.ycombinator.com/item?id=15367421 I'm a kernel guy, I'm old school, I get what they are doing there, that is impressive. I wish hacker news got excited about the filling the pipe post and less excited about this thread.

side_up_down 9y ago
> I wish hacker news got excited about the filling the pipe post and less excited about this thread. That's disingenuous considering the number of votes and comments each thread have received. HN IS more excited by the pipe post as reflected by user participation. Would love to hear more detail insofar as this article is concerned.
- luckydude 9y ago
  The 100Gb pipe article was off the front page and not really commented on that much. I see post after post about containers this and vagrant that, all sorts of posts about stuff that I find pretty uninteresting. Something as remarkable as the 100Gbit thing gets a little attention but it is clear that it's not that interesting to the majority of the people here. I'm just depressed that real low level systems stuff seems to be not sexy any more. When I was at Sun the kernel group was the unchallenged top of the heap, it was the place to be. When I was fixing source management at Sun (as a hobby, it wasn't my official job) they asked me to go work in the tools group. Are you frigging kidding me? Noone in their right mind would leave the kernel group for the tools group. Sorry to be snooty but that just wasn't a thing. I just got back from a FreeBSD conference at Netflix and was asking about the state of the world and it's depressing. People don't seem to write solid papers like they used to. Sun produced papers on vnodes, the VM system architecture and implementation, Sparc, I wrote one on making UFS perform like an extent based file system. Who is doing that now? I looked in the usual conference proceedings and it was full of academic stuff, not very interesting stuff, but no industry stuff.
  - sliken 9y ago
    I think that must have been common. Nobody good worked in tools. I managed a couple sunos and later solaris boxes. Userspace was a disaster. I ended up replacing it all with GNU and every piece I installed had a warning. Don't use Solaris make. Don't use solaris tar. Don't use the solaris compiler, etc. Ended up having to install gcc from binaries, then build zip, make, etc. Even tail (max 10k lines) and awk (max 16 columns or something) had issues. While debugging even xterm would die if you made it too wide. Once you replaced the userspace with GNU + X11 the kernel was solid, although SunOS and early solaris didn't multitask well. For some workloads I'd end up disabling all but one CPU.
    - luckydude 9y ago
      So I believe you on the Solaris userspace, it was full of System 5 garbage. GNU was a much better answer. But on SunOS 4.x, you had userspace problems? That was a pretty stock BSD userspace with a lot of bugfixes. Every open source makefile in the world just worked when you typed make on SunOS. Seemed pretty solid to me, I think the only thing I would do is add perl, it took Sun a long time to decide to include perl (if they ever did).
arca_vorago 9y ago
Is there a GNU/Linux project equivalent of the netflix/freebsd sendfile? Besides this use case, what is your assessment of the linux (netfilter) vs bsd netstacks? What kernel is going to be at the center of faster and faster switch fabrics?
- Thaxll 9y ago
  Senfile is a syscall in the Linux kernel since 2003~~
- luckydude 9y ago
  I wrote up a hand wavy way of doing a generalized sendfile here: http://mcvoy.com/lm/papers/splice.pdf http://mcvoy.com/lm/papers/splice.pdf and linux took some of that. As I recall, they don't use it much because the user land apps (like web servers) end up touching the data for TLS (https). Netflix/FreeBSD are working with the NIC folks to get the byte by byte TLS pushed down in the card along with the TCP offload. I think they'll get there first because BSD is so heavily used by CDN people (not just Netflix, Limelight is very active in BSD as well).
Slartie 9y ago
Hehe, I can feel your pain. Had a discussion just a few days ago with some guys demonstrating the typical enthusiasm about cloud stuff, especially AWS. Because after all, Netflix is using it, and they create 1/3rd of all Internet traffic, right? There's this very widespread misconception about Netflix being a monolithic service running on Amazons' cloud infrastructure, even though the truth is that just the "rather boring" routine stuff like billing, view history tracking, suggestions, everything necessary to show the UI is running there. Netflix does a good job at this, but after all it's not that impressive when you've architected some distributed systems yourself. Not even their extreme take on the microservices concept - that is after all just a nice way of letting their devs do "their thing" in the way they want with as little restrictions from the environment as possible (which basically only works because they clearly have above-average-competence devs who can deal with these degrees of freedom). What's really crazy is the way they squeeze unimaginable amounts of bytes per second out of modern hardware and into the internet infrastructure. They saturate 100Gbit links that usually serve an aggregation of many boxes in a datacenter with just ONE box! This is way beyond what even most above-average devs are capable of doing - you NEED old-school guys which still managed to stay on top of the crazy stack created by the evolution of hardware and low-level software in the last decade. There's not many of those out there, and Netflix apparently managed to catch a good bunch of them. These guys do the magic, and the magic they do never touches that damn Amazon Cloud. It just floats way above it.
- SomeStupidPoint 9y ago
  Netflix is my goto example of "do your core competency in-house, outsource the rest". They do one thing themselves in their stack: their distribution network for content, and they do an incredible job of it. Every post I see about Netflix's CDN for video is insightful and a learning experience. Then they throw a CRUD app in the cloud on top of it and call it a day. Okay, a little simplified -- there's still some neat tech in the DRM, in load-balancing the CDN, and in keeping all of their tech highly available. But conceivably, Netflix could retain much of their value by simply offering all the content to other websites that displayed it to customers -- the hard part of what they do is the CDN (and contracts with content owners), and opening their platform to other interfaces doesn't change that. (Heck, Netflix might be worth more if they opened their content to other interfaces, since they're not actually very good at the front end experience.) But when I was doing consulting (about cloud stuff), that was my advice: do the core of your business yourself (eg, CDN) then offload as much of the rest as you can.
nickjj 9y ago
Is it every ISP, or only certain ISPs who opt in? I remember OpenConnect being mentioned back when the ISP I use came on board. The service has been really good. I haven't gotten a single buffer in like 3 years. If anyone is curious, glance through this. It's a pretty cool initiative: https://openconnect.netflix.com/en/ https://openconnect.netflix.com/en/ If you get crappy Netflix service you should call your ISP and point them to this page.
anonu 9y ago
Can you recommend a more technical version of OP's article or a video-taped talk?

side_up_down 9y ago

> I wish hacker news got excited about the filling the pipe post and less excited about this thread. That's disingenuous considering the number of votes and comments each thread have received. HN IS more excited by the pipe post as reflected by user participation. Would love to hear more detail insofar as this article is concerned.

luckydude 9y ago
The 100Gb pipe article was off the front page and not really commented on that much. I see post after post about containers this and vagrant that, all sorts of posts about stuff that I find pretty uninteresting. Something as remarkable as the 100Gbit thing gets a little attention but it is clear that it's not that interesting to the majority of the people here. I'm just depressed that real low level systems stuff seems to be not sexy any more. When I was at Sun the kernel group was the unchallenged top of the heap, it was the place to be. When I was fixing source management at Sun (as a hobby, it wasn't my official job) they asked me to go work in the tools group. Are you frigging kidding me? Noone in their right mind would leave the kernel group for the tools group. Sorry to be snooty but that just wasn't a thing. I just got back from a FreeBSD conference at Netflix and was asking about the state of the world and it's depressing. People don't seem to write solid papers like they used to. Sun produced papers on vnodes, the VM system architecture and implementation, Sparc, I wrote one on making UFS perform like an extent based file system. Who is doing that now? I looked in the usual conference proceedings and it was full of academic stuff, not very interesting stuff, but no industry stuff.
- sliken 9y ago
  I think that must have been common. Nobody good worked in tools. I managed a couple sunos and later solaris boxes. Userspace was a disaster. I ended up replacing it all with GNU and every piece I installed had a warning. Don't use Solaris make. Don't use solaris tar. Don't use the solaris compiler, etc. Ended up having to install gcc from binaries, then build zip, make, etc. Even tail (max 10k lines) and awk (max 16 columns or something) had issues. While debugging even xterm would die if you made it too wide. Once you replaced the userspace with GNU + X11 the kernel was solid, although SunOS and early solaris didn't multitask well. For some workloads I'd end up disabling all but one CPU.
  - luckydude 9y ago
    So I believe you on the Solaris userspace, it was full of System 5 garbage. GNU was a much better answer. But on SunOS 4.x, you had userspace problems? That was a pretty stock BSD userspace with a lot of bugfixes. Every open source makefile in the world just worked when you typed make on SunOS. Seemed pretty solid to me, I think the only thing I would do is add perl, it took Sun a long time to decide to include perl (if they ever did).

arca_vorago 9y ago

Is there a GNU/Linux project equivalent of the netflix/freebsd sendfile? Besides this use case, what is your assessment of the linux (netfilter) vs bsd netstacks? What kernel is going to be at the center of faster and faster switch fabrics?

Thaxll 9y ago
Senfile is a syscall in the Linux kernel since 2003~~
luckydude 9y ago
I wrote up a hand wavy way of doing a generalized sendfile here: http://mcvoy.com/lm/papers/splice.pdf http://mcvoy.com/lm/papers/splice.pdf and linux took some of that. As I recall, they don't use it much because the user land apps (like web servers) end up touching the data for TLS (https). Netflix/FreeBSD are working with the NIC folks to get the byte by byte TLS pushed down in the card along with the TCP offload. I think they'll get there first because BSD is so heavily used by CDN people (not just Netflix, Limelight is very active in BSD as well).

Slartie 9y ago

Hehe, I can feel your pain. Had a discussion just a few days ago with some guys demonstrating the typical enthusiasm about cloud stuff, especially AWS. Because after all, Netflix is using it, and they create 1/3rd of all Internet traffic, right? There's this very widespread misconception about Netflix being a monolithic service running on Amazons' cloud infrastructure, even though the truth is that just the "rather boring" routine stuff like billing, view history tracking, suggestions, everything necessary to show the UI is running there. Netflix does a good job at this, but after all it's not that impressive when you've architected some distributed systems yourself. Not even their extreme take on the microservices concept - that is after all just a nice way of letting their devs do "their thing" in the way they want with as little restrictions from the environment as possible (which basically only works because they clearly have above-average-competence devs who can deal with these degrees of freedom). What's really crazy is the way they squeeze unimaginable amounts of bytes per second out of modern hardware and into the internet infrastructure. They saturate 100Gbit links that usually serve an aggregation of many boxes in a datacenter with just ONE box! This is way beyond what even most above-average devs are capable of doing - you NEED old-school guys which still managed to stay on top of the crazy stack created by the evolution of hardware and low-level software in the last decade. There's not many of those out there, and Netflix apparently managed to catch a good bunch of them. These guys do the magic, and the magic they do never touches that damn Amazon Cloud. It just floats way above it.

SomeStupidPoint 9y ago
Netflix is my goto example of "do your core competency in-house, outsource the rest". They do one thing themselves in their stack: their distribution network for content, and they do an incredible job of it. Every post I see about Netflix's CDN for video is insightful and a learning experience. Then they throw a CRUD app in the cloud on top of it and call it a day. Okay, a little simplified -- there's still some neat tech in the DRM, in load-balancing the CDN, and in keeping all of their tech highly available. But conceivably, Netflix could retain much of their value by simply offering all the content to other websites that displayed it to customers -- the hard part of what they do is the CDN (and contracts with content owners), and opening their platform to other interfaces doesn't change that. (Heck, Netflix might be worth more if they opened their content to other interfaces, since they're not actually very good at the front end experience.) But when I was doing consulting (about cloud stuff), that was my advice: do the core of your business yourself (eg, CDN) then offload as much of the rest as you can.

nickjj 9y ago

Is it every ISP, or only certain ISPs who opt in? I remember OpenConnect being mentioned back when the ISP I use came on board. The service has been really good. I haven't gotten a single buffer in like 3 years. If anyone is curious, glance through this. It's a pretty cool initiative: https://openconnect.netflix.com/en/ https://openconnect.netflix.com/en/ If you get crappy Netflix service you should call your ISP and point them to this page.

anonu 9y ago

Can you recommend a more technical version of OP's article or a video-taped talk?