9 ms·

How Netflix works: the stuff that happens when you hit Play

195 points by sds111 9y ago

johnwheeler 9y ago
i don’t understand how microservices solves the problem of broken interfaces. if an api changes or disappears, how is that different from the locations.txt file changing or disappearing? i think it’s just another instance of humans making things more complicated than they need be. Same line of reasoning Linus went with a monolith vs a micro kernel.
- reificator 9y ago
  Microservices set out to do the same thing as Object Oriented programming set out to do. You define an {object,microservice} by exposing methods, and other services and clients interact by calling those methods. Theoretically, if your API remains stable, your internal implementation can change drastically and the system will continue to function as it should. There's no reason you can't draw these boundaries inside a monolith, but with microservices you'd have to go out of your way to not have those boundaries. IMO by using HTTP to communicate, they ended up being significantly closer to the original concepts behind OOP and message passing. http://lists.squeakfoundation.org/pipermail/squeak-dev/1998-October/017019.html http://lists.squeakfoundation.org/pipermail/squeak-dev/1998-... Now, whether microservices are successful in dealing with the problems they set out to solve, and are worth the tradeoffs they entail is still up for debate.
  - deleted 9y ago
    [deleted]
- solipsism 9y ago
  Absolutely right. locations.txt is an API. And there's nothing monolithic about the example. It involves two completely separate apps with an API in between!
- Godel_unicode 9y ago
  I think you have missed the point. I think the point of that section actually has nothing to do with the locations.txt interface between the maps app and locolist. It's actually about the fact that, with microservices, any piece of the system only depends on it's dependency graph, not on every piece of the system working. "... on a huge service like Netflix the entire application going down because a change was made to one part of it..." To extend your OS example, just because there was a kernel panic in the Bluetooth stack is no reason to stop servicing requests over the Ethernet NIC which is already bound by the web server. There's also no need to edit or recompile the Ethernet driver because of a Bluetooth problem; see others posts about encapsulation.
- ImSkeptical 9y ago
  You might change the location of locations.text or store the locations in a database, or on the cloud, etc. You shouldn't change the commitment your service has to vend data about locations etc. You make a contract saying you will do X when given Y but you don't commit to anything about how you will do it, and that can change as necessary.
- fuckuservices9 9y ago
  They don’t, they just let the trendy nerds beat off over unnecessarily complex software. C.f. The entirety of the JavaScript ecosystem.
deleted 9y ago
[deleted]
reificator 9y ago
Ouch. Post an article, then two weeks later your headline image makes you look a bit tasteless. Bad timing... EDIT: And then I started reading the first paragraph... EDIT 2: I'm not saying they did anything wrong, the article was posted on the 17th.
- CaveTech 9y ago
  Kevin Spacey did some terrible things, and now we must remove all mention of any associated works from the world - so far as to call it insensitive to mention the name of the show he was in? Dismissing not only the hundreds to thousands of unrelated people who had a role in creating it, but also turning an unrelated act into it's coverstory. My god the US has some serious problems these days.
  - deleted 9y ago
    [deleted]
  - reificator 9y ago
    Is that what I said? > now we must remove all mention of any associated works from the world Yep. I was advocating this. > so far as to call it insensitive to mention the name of the show he was in? Absolutely, I'm sorry I called it insensitive. > Dismissing the hundreds to thousands of unrelated people who had a role in creating the show. I know, it was wrong of me to fire those people. I'm sorry. > My god the US has some serious problems these days. I know, it's ridiculous. I mean, look at the backlash to someone saying it would be "a bit tasteless" to post that today, and that it's unfortunate that the article is being shared two weeks after it was published such that some will assume it was published today.
    - benchaney 9y ago
      > I know, it's ridiculous. I mean, look at the backlash to someone saying it would be There was no backlash.
      - deleted 9y ago
X86BSD 9y ago
Their open conect FreeBSD boxes are nuts. The amount of data they spew is crazy. 1/3 of all internet traffic. One service. Mind blowing.
- toomuchtodo 9y ago
  It gives me hope that someone could still go out and challenge YouTube using a stable of these at various internet exchanges and a central object store/web front end.
  - X86BSD 9y ago
    I think YouTube simply has first to market mindshare. I personally use and prefer Vimeo. It's a better experience and quality. I don't think anything technical or special keeps YouTube number one other than first to market mindshare.
    - reificator 9y ago
      Part of that mindshare is that I only go to Vimeo when I can't find something on YouTube, and then when I find it on Vimeo it's incomplete and in approximately 240p. It's not fair to Vimeo, because they actually had a copy and YouTube didn't, but because of multiple experiences like that I was surprised to see a positive remark about their quality.
    - analogic 9y ago
      just hope there's at least enough competition to keep them halfway honest
    - blocked_again 9y ago
      > I don't think anything technical or special keeps YouTube number one other than first to market mindshare It's the youtube content creators like PewDiPie, Casey Niestat, Kurzgesagt etc that makes Youtube special just like that community that makes Hacker news special.
- StillBored 9y ago
luckydude 9y ago
So this is perhaps a lame post but I'm super tired, been up since 4am. I'll try and follow up with more detail in the morning. This article, in my opinion, is way off. I base that on the fact that I've been dancing with Netflix for a month, I might end up working with them to try and make NUMA machines serve up content faster. As in I'd be working on exactly what happens when you hit play. My take on how Netflix serves you movies is nothing like what this article says. They have servers in every ISP, the servers send a heartbeat to a conductor in AWS, the heartbeat says "I've got this content and I am this over worked", when you hit play the app reaches out the conductor and says "I want this", the conductor looks around and finds a server close to you that is not overloaded and off you go. That might look easy. It's not. Take a look at this post about how they fill a 100Gbit pipe: https://news.ycombinator.com/item?id=15367421 https://news.ycombinator.com/item?id=15367421 I'm a kernel guy, I'm old school, I get what they are doing there, that is impressive. I wish hacker news got excited about the filling the pipe post and less excited about this thread.
- side_up_down 9y ago
  > I wish hacker news got excited about the filling the pipe post and less excited about this thread. That's disingenuous considering the number of votes and comments each thread have received. HN IS more excited by the pipe post as reflected by user participation. Would love to hear more detail insofar as this article is concerned.
  - luckydude 9y ago
    The 100Gb pipe article was off the front page and not really commented on that much. I see post after post about containers this and vagrant that, all sorts of posts about stuff that I find pretty uninteresting. Something as remarkable as the 100Gbit thing gets a little attention but it is clear that it's not that interesting to the majority of the people here. I'm just depressed that real low level systems stuff seems to be not sexy any more. When I was at Sun the kernel group was the unchallenged top of the heap, it was the place to be. When I was fixing source management at Sun (as a hobby, it wasn't my official job) they asked me to go work in the tools group. Are you frigging kidding me? Noone in their right mind would leave the kernel group for the tools group. Sorry to be snooty but that just wasn't a thing. I just got back from a FreeBSD conference at Netflix and was asking about the state of the world and it's depressing. People don't seem to write solid papers like they used to. Sun produced papers on vnodes, the VM system architecture and implementation, Sparc, I wrote one on making UFS perform like an extent based file system. Who is doing that now? I looked in the usual conference proceedings and it was full of academic stuff, not very interesting stuff, but no industry stuff.
alexnewman 9y ago
Wish they would have mentioned the widevine drm Shit show
fasouto 9y ago
Correct me if I'm wrong but the data center picture looks very similar to the CERN computing center inside CERN installations in Meyrin.
- saagarjha 9y ago
  The caption says that it is: > An Amazon Web Services data center in Frankfurt, Germany, specially dedicated to CERN.
  - ephimetheus 9y ago
    But CERN is not in Frankfurt and there is a data center on site here.
  - hellofunk 9y ago
    Another comment here claims that that photo is mislabeled and not an Amazon property.
rb666 9y ago
"A special piece of code is also added to these files to lock them with what is called digital rights management or DRM — a technological measure which prevents piracy of films." They should probably update this to say "tries to prevent", as NF DRM has been long cracked.
- nitrogen 9y ago
  Also nice would be a note that DRM prevents fair use.
- colde 9y ago
  Do you have a source for it being "cracked"? Circumvented sure, but the actual cryptography components haven't been cracked as far as I am aware.
  - wolco 9y ago
    I think lower resolutions have been cracked. I don't think 4K has been cracked yet but the comment might be related to this bug. https://www.reddit.com/r/Piracy/comments/6pkypj/direct_stream_copy_has_netflix_4k_streaming_drm/ https://www.reddit.com/r/Piracy/comments/6pkypj/direct_strea...
  - jccalhoun 9y ago
    I don't have a source that explicitly says it has been cracked but I saw a reddit post stating that all episodes of stranger things 2 was available on torrent sites less than 10 minutes after it went live on Netflix and people commenting on how it was possible because the drm was cracked
qualitytime 9y ago
You know what, I'm going to use this post to tell you what happened last night when I hit play in firefox. Nothing happened. I saw The spinning loading wheel and the firefox/netflix header saying "blah blah audio video software being installed try again blah restart blah"... I waited, I reloaded the page, I googled, I checked the DRM settings, I did blah blah blah. Wasted time and frustrated I then opened microsoft edge and everything worked. You know that firefox share is at 8% according to W3Counter? You know this kind of crap will only reduce this? And then we'll only have the big daddy corporate sponsored browsers. And all because users want to watch netflix. What a bag of mediocre horse shit.
- crypt1d 9y ago
  FWIW, Firefox has been behaving quite weird for me for the last 2-4 weeks (Linux version). It randomly freezes and slows down during page loading, even crashed a few times. Seems to happen mostly when there is some video content on one of the tabs, so I'm blaiming the plugins for now. Kinda sucks because I switched from Chrome few months ago only. I really hope I didn't make a mistake.
  - bababooey 9y ago
    Same thing happened to me last year. I got very hyped up on the switch. But then it turned into hot garbage. Tried to get support on the firefox subreddit and they told me I shouldn't be using Firefox Beta if I'm not "technically inclined". Okay, so I switched to the normal release and it was too slow for me to even bother. Back to Google... I'm wondering if their new quantum engine fixes issues. Are you on that newest version?
  - JBlue42 9y ago
    Have also had issues for a month or so now. Weird stuff like trying to do a search in incognito mode, which should do a Google search, and it spinning forever.
- kilburn 9y ago
  I'm not upvoting because of the snarkiness, but I found myself in the exact same situation yesterday. Today it magically works fine again. If there is any netflixer around here: please, please push for better testing to prevent that from happening again (i.e: firefox should be one of the test platforms!)
kaplun 9y ago
An Amazon Web Services data center in Frankfurt, Germany, specially dedicated to CERN. I believe this is actually the CERN computer center itself and totally unrelated with Amazon. http://cds.cern.ch/record/1103476 http://cds.cern.ch/record/1103476
pidybi 9y ago
nice to know ;)
wscott 9y ago
BTW. Netflix is in the process of transitioning to TLS (https) transport. Not because they need it for any reason, but because shady advertisers keep snooping connections and using that to profile users. Netflix is tired of being accused of selling user data. My understanding is that the movie was already encrypted, but that stream could be fingerprinted to identify which movie it was. Using TLS is a lot more expensive so that costs them money, so I have to respect them for that. Grandma's TV is still unencrypted, but anyone who updates their client is protected. They do have to deal with a whole ton of legacy clients.
- arianvanp 9y ago
  Alas. The https protected videos can be fingerprinted as well. https://news.ycombinator.com/item?id=14070130 https://news.ycombinator.com/item?id=14070130
Thaxll 9y ago
I still believe that Netflix stack is way too complicated for what Netflix needs. It's a CDN with a recommendation engine that is completely garbage ( the 95% stuff I'm interested ). Also comparing Youtube and Netflix speed, Youtube is like 2/3x faster to load any content.

195 points by sds111 9y ago

i don’t understand how microservices solves the problem of broken interfaces. if an api changes or disappears, how is that different from the locations.txt file changing or disappearing? i think it’s just another instance of humans making things more complicated than they need be. Same line of reasoning Linus went with a monolith vs a micro kernel.

reificator 9y ago
Microservices set out to do the same thing as Object Oriented programming set out to do. You define an {object,microservice} by exposing methods, and other services and clients interact by calling those methods. Theoretically, if your API remains stable, your internal implementation can change drastically and the system will continue to function as it should. There's no reason you can't draw these boundaries inside a monolith, but with microservices you'd have to go out of your way to not have those boundaries. IMO by using HTTP to communicate, they ended up being significantly closer to the original concepts behind OOP and message passing. http://lists.squeakfoundation.org/pipermail/squeak-dev/1998-October/017019.html http://lists.squeakfoundation.org/pipermail/squeak-dev/1998-... Now, whether microservices are successful in dealing with the problems they set out to solve, and are worth the tradeoffs they entail is still up for debate.
- deleted 9y ago
  [deleted]
solipsism 9y ago
Absolutely right. locations.txt is an API. And there's nothing monolithic about the example. It involves two completely separate apps with an API in between!
Godel_unicode 9y ago
I think you have missed the point. I think the point of that section actually has nothing to do with the locations.txt interface between the maps app and locolist. It's actually about the fact that, with microservices, any piece of the system only depends on it's dependency graph, not on every piece of the system working. "... on a huge service like Netflix the entire application going down because a change was made to one part of it..." To extend your OS example, just because there was a kernel panic in the Bluetooth stack is no reason to stop servicing requests over the Ethernet NIC which is already bound by the web server. There's also no need to edit or recompile the Ethernet driver because of a Bluetooth problem; see others posts about encapsulation.
ImSkeptical 9y ago
You might change the location of locations.text or store the locations in a database, or on the cloud, etc. You shouldn't change the commitment your service has to vend data about locations etc. You make a contract saying you will do X when given Y but you don't commit to anything about how you will do it, and that can change as necessary.
fuckuservices9 9y ago
They don’t, they just let the trendy nerds beat off over unnecessarily complex software. C.f. The entirety of the JavaScript ecosystem.

deleted 9y ago

[deleted]

reificator 9y ago

Ouch. Post an article, then two weeks later your headline image makes you look a bit tasteless. Bad timing... EDIT: And then I started reading the first paragraph... EDIT 2: I'm not saying they did anything wrong, the article was posted on the 17th.

CaveTech 9y ago
Kevin Spacey did some terrible things, and now we must remove all mention of any associated works from the world - so far as to call it insensitive to mention the name of the show he was in? Dismissing not only the hundreds to thousands of unrelated people who had a role in creating it, but also turning an unrelated act into it's coverstory. My god the US has some serious problems these days.
- deleted 9y ago
  [deleted]
- reificator 9y ago
  Is that what I said? > now we must remove all mention of any associated works from the world Yep. I was advocating this. > so far as to call it insensitive to mention the name of the show he was in? Absolutely, I'm sorry I called it insensitive. > Dismissing the hundreds to thousands of unrelated people who had a role in creating the show. I know, it was wrong of me to fire those people. I'm sorry. > My god the US has some serious problems these days. I know, it's ridiculous. I mean, look at the backlash to someone saying it would be "a bit tasteless" to post that today, and that it's unfortunate that the article is being shared two weeks after it was published such that some will assume it was published today.
  - benchaney 9y ago
    > I know, it's ridiculous. I mean, look at the backlash to someone saying it would be There was no backlash.
    - deleted 9y ago

X86BSD 9y ago

Their open conect FreeBSD boxes are nuts. The amount of data they spew is crazy. 1/3 of all internet traffic. One service. Mind blowing.

toomuchtodo 9y ago
It gives me hope that someone could still go out and challenge YouTube using a stable of these at various internet exchanges and a central object store/web front end.
- X86BSD 9y ago
  I think YouTube simply has first to market mindshare. I personally use and prefer Vimeo. It's a better experience and quality. I don't think anything technical or special keeps YouTube number one other than first to market mindshare.
  - reificator 9y ago
    Part of that mindshare is that I only go to Vimeo when I can't find something on YouTube, and then when I find it on Vimeo it's incomplete and in approximately 240p. It's not fair to Vimeo, because they actually had a copy and YouTube didn't, but because of multiple experiences like that I was surprised to see a positive remark about their quality.
  - analogic 9y ago
    just hope there's at least enough competition to keep them halfway honest
  - blocked_again 9y ago
    > I don't think anything technical or special keeps YouTube number one other than first to market mindshare It's the youtube content creators like PewDiPie, Casey Niestat, Kurzgesagt etc that makes Youtube special just like that community that makes Hacker news special.
StillBored 9y ago

luckydude 9y ago

So this is perhaps a lame post but I'm super tired, been up since 4am. I'll try and follow up with more detail in the morning. This article, in my opinion, is way off. I base that on the fact that I've been dancing with Netflix for a month, I might end up working with them to try and make NUMA machines serve up content faster. As in I'd be working on exactly what happens when you hit play. My take on how Netflix serves you movies is nothing like what this article says. They have servers in every ISP, the servers send a heartbeat to a conductor in AWS, the heartbeat says "I've got this content and I am this over worked", when you hit play the app reaches out the conductor and says "I want this", the conductor looks around and finds a server close to you that is not overloaded and off you go. That might look easy. It's not. Take a look at this post about how they fill a 100Gbit pipe: https://news.ycombinator.com/item?id=15367421 https://news.ycombinator.com/item?id=15367421 I'm a kernel guy, I'm old school, I get what they are doing there, that is impressive. I wish hacker news got excited about the filling the pipe post and less excited about this thread.

side_up_down 9y ago
> I wish hacker news got excited about the filling the pipe post and less excited about this thread. That's disingenuous considering the number of votes and comments each thread have received. HN IS more excited by the pipe post as reflected by user participation. Would love to hear more detail insofar as this article is concerned.
- luckydude 9y ago
  The 100Gb pipe article was off the front page and not really commented on that much. I see post after post about containers this and vagrant that, all sorts of posts about stuff that I find pretty uninteresting. Something as remarkable as the 100Gbit thing gets a little attention but it is clear that it's not that interesting to the majority of the people here. I'm just depressed that real low level systems stuff seems to be not sexy any more. When I was at Sun the kernel group was the unchallenged top of the heap, it was the place to be. When I was fixing source management at Sun (as a hobby, it wasn't my official job) they asked me to go work in the tools group. Are you frigging kidding me? Noone in their right mind would leave the kernel group for the tools group. Sorry to be snooty but that just wasn't a thing. I just got back from a FreeBSD conference at Netflix and was asking about the state of the world and it's depressing. People don't seem to write solid papers like they used to. Sun produced papers on vnodes, the VM system architecture and implementation, Sparc, I wrote one on making UFS perform like an extent based file system. Who is doing that now? I looked in the usual conference proceedings and it was full of academic stuff, not very interesting stuff, but no industry stuff.

alexnewman 9y ago

Wish they would have mentioned the widevine drm Shit show

fasouto 9y ago

Correct me if I'm wrong but the data center picture looks very similar to the CERN computing center inside CERN installations in Meyrin.

saagarjha 9y ago
The caption says that it is: > An Amazon Web Services data center in Frankfurt, Germany, specially dedicated to CERN.
- ephimetheus 9y ago
  But CERN is not in Frankfurt and there is a data center on site here.
- hellofunk 9y ago
  Another comment here claims that that photo is mislabeled and not an Amazon property.

rb666 9y ago

"A special piece of code is also added to these files to lock them with what is called digital rights management or DRM — a technological measure which prevents piracy of films." They should probably update this to say "tries to prevent", as NF DRM has been long cracked.

nitrogen 9y ago
Also nice would be a note that DRM prevents fair use.
colde 9y ago
Do you have a source for it being "cracked"? Circumvented sure, but the actual cryptography components haven't been cracked as far as I am aware.
- wolco 9y ago
  I think lower resolutions have been cracked. I don't think 4K has been cracked yet but the comment might be related to this bug. https://www.reddit.com/r/Piracy/comments/6pkypj/direct_stream_copy_has_netflix_4k_streaming_drm/ https://www.reddit.com/r/Piracy/comments/6pkypj/direct_strea...
- jccalhoun 9y ago
  I don't have a source that explicitly says it has been cracked but I saw a reddit post stating that all episodes of stranger things 2 was available on torrent sites less than 10 minutes after it went live on Netflix and people commenting on how it was possible because the drm was cracked

qualitytime 9y ago

You know what, I'm going to use this post to tell you what happened last night when I hit play in firefox. Nothing happened. I saw The spinning loading wheel and the firefox/netflix header saying "blah blah audio video software being installed try again blah restart blah"... I waited, I reloaded the page, I googled, I checked the DRM settings, I did blah blah blah. Wasted time and frustrated I then opened microsoft edge and everything worked. You know that firefox share is at 8% according to W3Counter? You know this kind of crap will only reduce this? And then we'll only have the big daddy corporate sponsored browsers. And all because users want to watch netflix. What a bag of mediocre horse shit.

crypt1d 9y ago
FWIW, Firefox has been behaving quite weird for me for the last 2-4 weeks (Linux version). It randomly freezes and slows down during page loading, even crashed a few times. Seems to happen mostly when there is some video content on one of the tabs, so I'm blaiming the plugins for now. Kinda sucks because I switched from Chrome few months ago only. I really hope I didn't make a mistake.
- bababooey 9y ago
  Same thing happened to me last year. I got very hyped up on the switch. But then it turned into hot garbage. Tried to get support on the firefox subreddit and they told me I shouldn't be using Firefox Beta if I'm not "technically inclined". Okay, so I switched to the normal release and it was too slow for me to even bother. Back to Google... I'm wondering if their new quantum engine fixes issues. Are you on that newest version?
- JBlue42 9y ago
  Have also had issues for a month or so now. Weird stuff like trying to do a search in incognito mode, which should do a Google search, and it spinning forever.
kilburn 9y ago
I'm not upvoting because of the snarkiness, but I found myself in the exact same situation yesterday. Today it magically works fine again. If there is any netflixer around here: please, please push for better testing to prevent that from happening again (i.e: firefox should be one of the test platforms!)

kaplun 9y ago

An Amazon Web Services data center in Frankfurt, Germany, specially dedicated to CERN. I believe this is actually the CERN computer center itself and totally unrelated with Amazon. http://cds.cern.ch/record/1103476 http://cds.cern.ch/record/1103476

pidybi 9y ago

nice to know ;)

wscott 9y ago

BTW. Netflix is in the process of transitioning to TLS (https) transport. Not because they need it for any reason, but because shady advertisers keep snooping connections and using that to profile users. Netflix is tired of being accused of selling user data. My understanding is that the movie was already encrypted, but that stream could be fingerprinted to identify which movie it was. Using TLS is a lot more expensive so that costs them money, so I have to respect them for that. Grandma's TV is still unencrypted, but anyone who updates their client is protected. They do have to deal with a whole ton of legacy clients.

arianvanp 9y ago
Alas. The https protected videos can be fingerprinted as well. https://news.ycombinator.com/item?id=14070130 https://news.ycombinator.com/item?id=14070130

Thaxll 9y ago

I still believe that Netflix stack is way too complicated for what Netflix needs. It's a CDN with a recommendation engine that is completely garbage ( the 95% stuff I'm interested ). Also comparing Youtube and Netflix speed, Youtube is like 2/3x faster to load any content.