12 ms·

Netflix: What Happens When You Press Play?

517 points by yarapavan 9y ago

jedberg 9y ago
> I pulled this chapter together from dozens of sources that were at times somewhat contradictory. Facts on the ground change over time and depend who is telling the story and what audience they're addressing. You know old time journalists had a solution to this. Interviewing subjects with first hand knowledge. I know I rag on highscalability a lot, but I feel like they deserve it. They've written multiple articles about topics in which I would be a first person resource (Reddit and Netflix), and not once have they ever contacted me or anyone I worked with for an interview. They've quoted me from presentations, but the quotes were out of context and wrong they way they quoted it. All of this could have been cleared up with a single interview of anyone who is a primary source expert.
- radicalbyte 9y ago
  A really good way for you to get your point across would be to write a reactionary blog post detailing the actual information. Netflix seem to be pretty cool & open about things so it should be possible?
  - jedberg 9y ago
    A good idea, except I haven't worked there in almost three years, so a lot of my knowledge it out of date. I have to start by interviewing my old co-workers! :)
    - kfriede 9y ago
      "Netflix: What Happens When You Press Play (2014)" - 2017/12/25
- frik 9y ago
  I am all for video-on-demand, as long it's free. So YouTube, Twitch, etc are great value. In case I need to spend money, I prefer to own the thing I pay for. So Blu-ray/DVD/CDs discs for me. The gathered video/music can be played on all my devices from car to phone, even in flight-mode offline. I trade a bit of less convince upfront to really own it, but then I can play it as often and exactly when and where I want. Anyway every few years one could get a higher resolution version, unfortunately the films aren't sold in source versions, but downscaled, but the video-on-demand at always lagging behind and stream worse bit rates.
- tybit 9y ago
  Which just goes to show how much of a market there is for easily digestible information in this space. Netflix in particular seems to have some really interesting ideas hidden behind unclear blogs(at least for me personally) and long presentations I rarely have time to sit through, even though I often appreciate them when I do.
- frozenport 9y ago
  Whats wrong with the information?
- marcoperaza 9y ago
  Modern journalism in a nutshell. Now realize that most journalism is this sloppy, not just the pieces about things you happen to know about.
  - oculusthrift 9y ago
    not true. many print news sources call their sources and sometimes verify each quote twice. i’ve heard stories from people who were simply quoted in a new yorker article about movies and they were called twice to verify their quote when it was just a small part of the article
- sandworm101 9y ago
  Try writing tech articles for money. You have to produce them quickly and to point. Interviewing people takes time. By the time the subject recieves the request and gets permission to talk to the reporter, the dealine has passed. An interview might also overturn the theme of the article, a theme dictated by a manager/purchaser based on thier brief reading of a few other articles/blogs. If the reporter wants to be paid they better generate the story as requested and on time. It isnt good jounalism but it is how you make rent.
- katastic 9y ago
  I've actually heard a lot of people say this. At least a couple Ph.Ds, though I can't recall the links. How many failures in journalism are attributed to "lazy journalism" where nobody will even leave the building these days to even ask someone involved in the story or topic. How the internet made it "easier" but they've become so over-reliant on it that many journalists fail to include information that isn't already online. Then again, Netflix is probably a phone call away, so that may be a whole 'nother level of lack-of-followup.
- ufmace 9y ago
  I get your point, but how many of these tech companies will actually authorize their employees who know what they're talking about to speak freely with tech reporters about how their system works? I would figure most would redirect you to a PR flunky to feed meaningless corporate-speak to any reporters, and then it's all "Netflix uses the latest internet technology to ensure the best possible viewing experience for all viewers around the world. Unfortunately, the specifics of our system are proprietary and we are unable to discuss that."
  - gtaylor 9y ago
    Reddit is constantly out presenting, talking, and even writing about architecture, culture, and our unique challenges. Netflix seems to be similar. Their open source record speaks for itself, and their technical blog and presence at conferences and meetups is outstanding. Source: though these opinions are my own, I work on Infrastructure at Reddit.
  - drdrey 9y ago
    Netflix is particularly open about how it works. There are some confidential bits, but most of the infrastructure projects are not considered proprietary. (I work at Netflix)
- toddh 9y ago
  Hi Jeremy, I'm very sorry if I've misquoted you or quoted you out of context. I'd certainly correct them instantly if contacted. I've searched my email and I don't see any corrections, so I'm not sure how I can make amends. I pull a lot of quotes from a lot of sources. Many thousands a year. My goal is always to highlight ideas that I think might help readers do their job better. Highlight is they key word here. That you say those sort of things often is not surprising. My hope is always readers will be made curious enough to follow the link and learn more on their own. And unfortunately there is no "they" at HighScalability.com. There's only me and there's only ever been me. So all mistakes are mine and only mine. There's never been much money or time to be as comprehensive as I'd like. I do the best I can, but I often fall short, no doubt. I'm not a journalist. I never said I was. I'm a programmer. Have been all my life. I started this blog a long time ago. It was just stuff I found interesting. For whatever reason it got popular for awhile so I tried to make something out of it. That, like most of the industry is petering out now, so we'll see what the future holds. The point about interviews is a good one and one I've wrestled with. Over the years I've conducted a lot of interviews. Much less so lately, and that's for a reason. I've learned that presentations at major conferences are usually the best resources. Presenters put a huge amount of effort into making those talks. And everything they say is open and approved for public consumption by definition. Interviews with someone like me are often a bother, a time sync, so I don't usually get as much out of them. Interviews have a high overhead, taking a lot of time to setup and there's often a lot of red tape and legal wrangling that must go on to get access and content approval. Which is again, why, presentations are such awesome resources. This article was derived mostly from people working on a project making a major presentation or from primary sources writing an article. I couldn't do better than that. And regardless, I would still make mistakes. Believe me. It's surprisingly hard to make sense of an interview and turn something from thread mode into document mode. That process is an error prone one because I have to fill in the gaps and there are always gaps. It's rare that I've ever just published an interview. I don't think they are that useful in general. I want people to read a highly condensed document they can get something concrete out of with minimal effort or just ignore easily if it's not relevant to them. The strange style I've developed reflects that goal. This article was of course different because it has a different audience. So again, I apologize. But I guarantee my mistakes are never intentional, never for money, and never for lack of caring or effort.
hetfeld 9y ago
What happens when i press play? It checks DRM settings in my browser and gets me lowest quality because my browser is open source. That is what happens.
- synotna 9y ago
  You have the freedom to watch something else
  - Walf 9y ago
    And content creators have the freedom to choose where they distribute content, many of whom choose Netflix.
    - synotna 9y ago
      Correct, so passively aggressively complaining about DRM is a moot point
      - izacus 9y ago
        It's what customers do in free market to get better service.
        beefield 9y ago
        Or turn from government regulated monopolistic market to actually free market i.e. torrents.
pavs 9y ago
Many of the things Netflix does here google does too and has been doing for a long time like GGC (Google Global Cache) for free at ISP NOC or datacenters - their GGC nodes are many orders of magnitude time is distribution. 90% of the Google traffic through these devices are youtube videos. They mention it in the article like Netflix pioneered this idea with open-connect. If anything it's harder to get OC devices from Netflix then it is from Google. Even Akamai does have a similar program where they let you put their CDN on ISP NOC.
- amelius 9y ago
  That sounds a bit like a violation of net neutrality avant la lettre.
  - pavs 9y ago
    There is a whole world outside the United States.
  - virtuallynathan 9y ago
    It's good for everyone -- if you are an ISP with 35%+ of your traffic coming from Netflix, you might be begging for these boxes in your network.
    - pavs 9y ago
      Yup, as an ISP owner, I can tell you 70-75% of my traffic come through GGC, for every 1gbps traffic from cache nodes 100mbps on average is pulled from the internet. Even with GGC nodes, it's still very expensive, BW-wise.
      - PuffinBlue 9y ago
        I saw another comment on here in the distant past that said an ISP owner decided to pull out the GGC boxes because they would sync vast amounts of data (largely YouTube IIRC) and it wasn't worth it bandwdith wise for them. It sounds like 10:1 is a decent saving but still objectively a lot of data.
deleted 9y ago
[deleted]
inciampati 9y ago
Nothing. So you go to your nearest torrent aggregator and are streaming it down in 3 minutes.
- HenryBemis 9y ago
  I read the word "nothing" and I thought to mysel: now THIS will turn interesting!!! (thinking that this will point to removal of Net Neutrality and the risks and conflicts it will bring to online services once ISPs turn aggro to both content producers/distributors and end-customers. Buuuuuuuuut no.. it was a silly short-lived comment about piracy (not even an educating one)
  - inciampati 9y ago
    Actually it was a comment questioning why they don't use torrents to provide their services. And also pointing out that p2p based solutions somehow outperform theirs in many cases (for me, not US based so the net neutrality stuff is maybe not relevant). Anyway sorry I was too silly and didn't clarify these points.
    - Klathmon 9y ago
      P2P has traditionally not worked out well for companies. Even recently Microsoft tried it for updates, and people were quite angry. I honestly don't see a situation where someone would prefer to use their own bandwidth for a paid service rather than having a server do the heavy lifting. Either as the provider, or the consumer.
      - inciampati 9y ago
        This is true, and I wonder if someone has done an analysis of the many failures. It is impressive that something which works so well in the wild provides so little benefit when tamed. Maybe Netflix or a streaming service could pay for bytes uploaded. Abuse would be difficult to mitigate but in principle this would encourage peers to provide the company's service to the network.
      - 9y ago
verletx64 9y ago
There's something about this article, it kind of feels like a sequence of staccato thoughts. It feels like it kind of jumps around.
- toddh 9y ago
  I did try to tell a story, but I don't think I succeeded as well as I would have liked. It definitely does jump around more than it should. It's hard to cover the important bits and link them seamlessly together. I'm just not that good a writer.
  - verletx64 9y ago
    Practice makes perfect! It was frustrating because I think I saw what you were going for and it's a good idea. Don't be afraid to drop things, even if they're important. You've got a title, right? "What happens when you press play". Answer that question. There's a whole load there that is interesting but doesn't answer the question of "Given Netflix have a bunch of video ready to go, how does that reach me?". Don't get me wrong, the rest is interesting, and important when thinking about Netflix as a whole, but ultimately I'm unsure they fit in the article. You might be worried about missing important bits, I personally worry about losing readers. It doesn't matter how important something is, if my target audience quits 15% of the way through the article. Anyway, no sweat really, blogging is hard enough. It's been a long time since I've written regularly, and I just started a personal blog, and yeah, a few weeks later im sitting on a mound of drafts I need to take a 'minimum viable post' eye to. :) Merry Christmas!
wz1000 9y ago
> Netflix has more than 110 million subscribers. > Netflix operates in more than 200 countries The five largest public torrent trackers track 30-50 million peers at any given time, serving over 10 million torrents. Torrents are estimated to have over a quarter of a billion active users. Torrents can be streamed, and in the end you have a DRM free copy of the file that you can rewatch at your pleasure, on any device(adding custom subtitles if you need them). It amazes me how much effort has been expended on pushing malware to almost everybody(binary, unauditable blobs in every browser) solely so that massive cartels can make money on something that has been solved since 2001. Subscribing to Netflix, or any other proprietary streaming (dis)service is not only foolish(in the sense of surrendering your own freedom, not to mention your hard earned money to nefarious corporate interests), but deeply unethical.
- madeofpalk 9y ago
  > not to mention your hard earned money to nefarious corporate interests Yes, this is the basis of our society. I work for money to buy things, mostly from corporations. There's nothing nefarious about that. I find it really hard to believe that paying for content is unethical.
  - wz1000 9y ago
    > I find it really hard to believe that paying for content is unethical. Your money isn't going to the people who created the content. It is going to cartels that have no interests in common with you, and have massive lobbying power in order to force you to download malware and restrict the access of information from you.
    - madeofpalk 9y ago
      Of course money goes to the creators. I have an actor friend who was in a 'Netflix Original'. He got paid for it. Sure, others are making profits on it but, again, that's not surprising. That's how our society works. I wasn't 'lobbied' or forced to install 'malware'. I'm aware of what's going on (I too have heard of R.M.S.!) and I made a conscious decision to subscribe to such services.
bendbro 9y ago
It ticks to 25% loaded even though your internet is down.
Walf 9y ago
I know what it does when I don't press play: fills my memory with a boatload of bloated interface that has the shittiest navigation of any media player I've ever used. And I still can't just view a list of all series/movies by date.
- nabeards 9y ago
  Which media player has an interface you like? I'm usually an outcast for it, but I actually like the iTunes Movies interface myself.
  - juanmirocks 9y ago
    Popcorn has a better interface than Netflix IMO
  - Nala_Alan 9y ago
    I like mpv the most. It's simple, but still has everything you need (and you can do rest with the keyboard shortcuts), and it's pretty. In my opinion, however, the problem with Netflix player, is not its design. It's that it's pretty laggy, it is not nearly as smooth as for example YouTube player.
- JustSomeNobody 9y ago
  Amazon's UI on the fire stick is so much worse.
  - PuffinBlue 9y ago
    I'm not sure about that really. It's like bashing your shin against a coffee table vs walking into a lamppost. Both painful experiences in their own unique ways.
- ashark 9y ago
  I’ve hated it since pressing play stopped being necessary to start video playing. Now I have to keep the focus bouncing around or else whatever it’s currently on will start yelling at me. Can’t just stop and look around at what’s on screen. It’s stressful. [EDIT] in fact, now that I think about it, I almost never just browse the catalogue anymore, while I used to a lot. I open it and go to search ASAP, then close it if I didn't find what I want or see something else interesting in the related results. The auto-playing is annoying enough that it's almost totally driven me away from that part of the interface. I knew I didn't like it but hadn't thought about how it'd affected my behavior, but there it is. Time browsing catalogue from "lots" to "nearly none". [EDIT EDIT] I might add that full-screen ads you have to scroll past (with no indication that's what you need to do) and other general mucking with the plain grid-layout interface make it wicked confusing for my parents. They just got a Roku solely for Netflix and if it keeps doing that I bet they won't use it very much.
temp 9y ago
I'd be interested in finding out what happens when I press the thumbs down/up icon on Netflix, because the match ratings that are displayed to me seem to have nothing to do with either my ratings or with my watching history.
- juanmirocks 9y ago
  It records your like/dislike, but it doesn’t distribute the change to its database immediately. Same for YouTube or so many other services.
  - temp 9y ago
    The thing is I don't feel like any of the match ratings in the last year and a half since I've been using Netflix have had anything to do with my interests, the ratings I've given out or with my viewing history. Unless propagation to their database takes two years, I'm really interested in how they actually arrive at match ratings.
    - syshum 9y ago
      I believe largely because a single "like / dislike" is massively inferior to the old traditional Rating system where I am able to Rank content I like...
- saryant 9y ago
  AFAIK from some conversations I had at a conference last year with a few Netflix engineers who work on this system, it's sort of realtime. Recommendations are build up using Apache Flink, which provides stateful stream compute. So your click turns into an event on a Kafka topic which is eventually consumed by Flink, joined with whatever other data sources they need and then written out to a store somewhere. IIRC the various machine learning models exist inside their Flink architecture. https://ci.apache.org/projects/flink/flink-docs-release-1.4/dev/libs/ml/index.html https://ci.apache.org/projects/flink/flink-docs-release-1.4/...
- dexterdog 9y ago
nvr219 9y ago
All this tells me is that Netflix is way too powerful a force in my life. Yikes.
plaidfuji 9y ago
ITT: people complaining about Netflix
Abishek_Muthian 9y ago
I feel bit sorry for Netflix, for the lengths they have gone to get their high performance/reliable architecture but the clients are still at the mercy of the OS, browsers, CPU (decoding codecs) of-course assuming high speed Internet. From their website - https://help.netflix.com/en/node/23742 https://help.netflix.com/en/node/23742 Resolution: Google Chrome Up to 720p on Windows, Mac, and Linux. Up to 1080p on Chrome OS. Internet Explorer up to 1080p. Microsoft Edge up to 4K. Mozilla Firefox up to 720p. Opera up to 720p. Safari up to 1080p on Mac OS X 10.10.3 or later. Streaming in 4K requires an HDCP 2.2 compliant connection to a 4K capable display, Intel's 7th generation Core CPU, and the latest Windows updates. P.S Interesting to note that Chrome OS plays 1080p, I assume because it has HW acceleration enabled by default (in browser alone, not in android apps though).
- jmiserez 9y ago
  Is it browser limitations, or just DRM limitations? Youtube seems to do 4K just fine.
  - Abishek_Muthian 9y ago
    I think it has to do with HEVC h.265 codec & HW acceleration support available to the browser in the respective platforms. https://caniuse.com/#feat=hevc https://caniuse.com/#feat=hevc Not sure why Chrome doesn't support HW for it though albeit Chrome OS. My Celeron N3060 Chromebook can decode it. I read Netflix has started using VP9 for streaming, newer CPU's have VP9 decoders in them.
    - izacus 9y ago
      No it's actually a DRM limitation. You're not allowed to watch high definition content of your platform doesn't support latest Widewine DRM even if you have the HW or SW decoding capability.
  - Nala_Alan 9y ago
    Yeah, IE and Edge are the only browsers with PlayReady DRM. Rest of them (besides Apple devices) are using Widevine DRM.
tlholaday 9y ago
Author uses acronym CDN without first defining it. Explain Like I Do Not Already Know What You Are Explaining.
- jordache 9y ago
  that article is pretty technical, beyond the level of technicality of simply knowing what a CDN is.
  - JustSomeNobody 9y ago
    But even technical people sometimes don't know simple acronyms. It doesn't hurt to define them anyway.
    - jordache 9y ago
      mentioning cdn would be akin to a car review magazine mentioning ABS in an article and then dedicated a couple sentences explaining what ABS is
      - yesenadam 9y ago
        Hmm I dont know what CDN or ABS are.
        BoorishBears 9y ago
        Content Delivery Network Anti-lock Braking System Both are exactly what they sound like
        yesenadam 9y ago
        OK thanks :-)
  - tlholaday 9y ago
    I red-circle the style-guide violating unexplained CDN acronym because the author titled the work from which this is excerpted “Explain Like I’m 10.” I have no objection to using terms-of-art in articles written for experts.
dominotw 9y ago
> Netflix Saves Money In AWS How much of it due to special discount that netflix gets from AWS. Would a company thats not highly visible as netflix save the same amount of money ? > Recommendations. I find netflix recommendations pretty bad. Does it really work for other people?
- juanmirocks 9y ago
  The recommendations don’t work for me. The media selection by Netflix is not for me yet. Ultimately, I want a service that recommends me only three possibilities, and at least one is a total hit.
- donarb 9y ago
  It's not visibility, it's how much you spend/use. I'll bet there are large companies out there you've never heard of that get just as much of a discount from Amazon as Netflix does.
  - qaq 9y ago
    I bet very few do I remember watching someone high up from Netflix presenting at a conference he quoted that they get 70% discount.
- syshum 9y ago
  >>I find netflix recommendations pretty bad. Does it really work for other people? They used to, the old system I found the recommendation to be pretty good and discovered some new things as a result The new, like/dislike system is garbage and most of the stuff is recommends are not something I would enjoy... I wonder how much of that is even based on my personal rating anymore or it is just stuff netflix wants to promote...
tw1010 9y ago
I guess the discrepancy between what I expected the article to be, based on the title, and what it actually was is a good reflection on the fact that development of these types of services is no longer built from first principles, low level to high level, but is rather constructed like a building from a set of abstracted lego pieces.
deleted 9y ago
[deleted]
erstap 9y ago
I feel like this article could be cut in half if it had been given a sharper focus. I don't care why I am seeing a specific image for a film and the reasoning for image personalization. Save that for a different article. I just want to know what happens after I click play.
curun1r 9y ago
> The advantage of having three regions is that any one region can fail, and the other regions will step in handle all the members in the failed region. At a previous employer, we met with AWS engineers who told us that this sort of thinking wasn't correct. Availability zones are for failure tolerance. Regions are conceptually different and for things like moving compute closer to clients, mitigating political instability or ensuring that data is held in a certain jurisdiction. To back up that claim, they told us that amazon.com ran out of a single region, us-west-2. As an aside, another interesting point from that meeting was that autoscaling wasn't a good strategy for scaling up in the event of the loss of an availability zone. The reason why is that in those scenarios, a lot of other customers will be trying to fail over as well and autoscaling can get very slow. Instead of autoscaling, they recommended that we run 150% of our per-AZ capacity in 3 AZs so that if an AZ went offline, the remaining two would already have the necessary capacity. We still used autoscaling, but it was for responding to spikes and troughs in our own load, not failure scenarios. I'm curious whether these two observations are correct since it seems like a lot of people view regions and autoscaling through a failure-recovery lens.
- aaronblohowiak 9y ago
  Sometimes there have been region-wide issues. Running cold has its own set of problems. We do not rely exclusively on auto scaling when we do failover. Disclaimer: while I work at Netflix, these views are not necessarily those of my employer
- mimerme 9y ago
  I think this is another case of "your not really supposed to...but it works ¯\_(ツ)_/¯" For your first point, I think it depends on your usage scenario. With a big company that requires good performance to satisfy their users, like Netflix, having multiple regions is a given. But what the article probably missed is that they also most likely use AZs within the same region to mitigate failures, but since they have multiple regions at hand why not use them to mitigate failures as well? And their monthly tests are a simulation of a worst case scenario if an entire region goes down. As for autoscaling I also think that people think of it as a magic solution. More users? Autoscale. Zone down? Autoscale. But the problem arises when they think that it's instant because it isn't. Things have to happen behind the scenes, but for many people, they only care about what happens on stage, so for them configuring autoscale is just better because it's easier than load balancing on 3 different AZs.
forgotpassagan 9y ago
I love how Netflix is so open about their architecture, but I have one strong criticism. After trying to get Spinnaker running, which is like 10 services for a fairly straightforward application, and reading about how Netflix has Cron jobs running to 'clean up' data inconsistencies... I started to think Google has a better approach to scaling the code. Netflix is strong on eventual consistency, and making services as small as possible. Having tried to build something in that image, holy shit the issue of distributed transactions becomes a nightmare. We had to build error handling in everything to handle the failure of everything else. Maybe half our code was 'just in case' to deal with exploding failure scenario complexity during service call fanout. If your service calls out to two or more services that also do data updates, God help you. There's no call ordering or way to make sure both succeed or fail, so if one does you could easily end up in an invalid state. Contrast this to Google's approach. Maps is one service, Docs is one service. Their service boundaries are much larger so they can shove the complexity of consistency and rollbacks back into the DB where it belongs. And they avoid eventual consistency as much as possible. If Google can make these big 'monoliths' work at huge scale I don't see any advantage to 'microservices', it's just a bunch of pointless overhead. I think Netflix has had a little too much industry koolaid ...
- saryant 9y ago
  > Netflix is strong on eventual consistency, and making services as small as possible. Having tried to build something in that image, holy shit the issue of distributed transactions becomes a nightmare. Distributed transactions are a sure sign that your service boundaries have been incorrectly drawn.
  - forgotpassagan 9y ago
    Not according to Google :). They actually support and encourage distributed transactions across boundaries. Imagine all the data Google stores with your user profile across their hundreds of application boundaries. To keep things consistent you would either need to store all profile data in one massive service or support distributed transactions. The Netflix model is fault tolerance, with every service supporting various failed profile update scenarios. Google just decided to add distributed transactions support to spare all that overhead.
macpete 9y ago
Feels more like an ad for AWS than a technical post
twic 9y ago
This article is a chapter from my new book Explain the Cloud Like I'm 10. The normal expression is "explain like I'm 5", but in the cloud, you need to double everything, in case an AZ goes down.
kzzzznot 9y ago
> The video comes in a high definition format that’s many terabytes in size. A terabyte is big. Imagine 60 stacks of paper as tall as the Eiffel Tower. That’s a terabyte. What... No. An HD ProRes will be 150-200GB for a 1.5-2 hour film. That is a 1920x1080 Apple ProRes 422 HQ @ ~190Mbps which is the 'gold standard' accepted Mezz file for Netflix, Google Play, iTunes, AIV (Microsoft, Sony, TalkTalk etc). There are different codecs for broadcasters but this is the standard for VoD. If you deliver uncompressed (uncommon, few relative gains) then it might run into the hundreds of GB but never a multi-terabyte file for a feature. Although most of this article has been informative, I am dismayed by the outright BS. It basically ruins any trust I would have of this otherwise well written piece. Can your exaggeration be explained or justified please?
thinkMOAR 9y ago
If they could please make it so that you can search on <any> subtitle languages, it would really help when the in-laws visit.

517 points by yarapavan 9y ago

> I pulled this chapter together from dozens of sources that were at times somewhat contradictory. Facts on the ground change over time and depend who is telling the story and what audience they're addressing. You know old time journalists had a solution to this. Interviewing subjects with first hand knowledge. I know I rag on highscalability a lot, but I feel like they deserve it. They've written multiple articles about topics in which I would be a first person resource (Reddit and Netflix), and not once have they ever contacted me or anyone I worked with for an interview. They've quoted me from presentations, but the quotes were out of context and wrong they way they quoted it. All of this could have been cleared up with a single interview of anyone who is a primary source expert.

radicalbyte 9y ago
A really good way for you to get your point across would be to write a reactionary blog post detailing the actual information. Netflix seem to be pretty cool & open about things so it should be possible?
- jedberg 9y ago
  A good idea, except I haven't worked there in almost three years, so a lot of my knowledge it out of date. I have to start by interviewing my old co-workers! :)
  - kfriede 9y ago
    "Netflix: What Happens When You Press Play (2014)" - 2017/12/25
frik 9y ago
I am all for video-on-demand, as long it's free. So YouTube, Twitch, etc are great value. In case I need to spend money, I prefer to own the thing I pay for. So Blu-ray/DVD/CDs discs for me. The gathered video/music can be played on all my devices from car to phone, even in flight-mode offline. I trade a bit of less convince upfront to really own it, but then I can play it as often and exactly when and where I want. Anyway every few years one could get a higher resolution version, unfortunately the films aren't sold in source versions, but downscaled, but the video-on-demand at always lagging behind and stream worse bit rates.
tybit 9y ago
Which just goes to show how much of a market there is for easily digestible information in this space. Netflix in particular seems to have some really interesting ideas hidden behind unclear blogs(at least for me personally) and long presentations I rarely have time to sit through, even though I often appreciate them when I do.
frozenport 9y ago
Whats wrong with the information?
marcoperaza 9y ago
Modern journalism in a nutshell. Now realize that most journalism is this sloppy, not just the pieces about things you happen to know about.
- oculusthrift 9y ago
  not true. many print news sources call their sources and sometimes verify each quote twice. i’ve heard stories from people who were simply quoted in a new yorker article about movies and they were called twice to verify their quote when it was just a small part of the article
sandworm101 9y ago
Try writing tech articles for money. You have to produce them quickly and to point. Interviewing people takes time. By the time the subject recieves the request and gets permission to talk to the reporter, the dealine has passed. An interview might also overturn the theme of the article, a theme dictated by a manager/purchaser based on thier brief reading of a few other articles/blogs. If the reporter wants to be paid they better generate the story as requested and on time. It isnt good jounalism but it is how you make rent.
katastic 9y ago
I've actually heard a lot of people say this. At least a couple Ph.Ds, though I can't recall the links. How many failures in journalism are attributed to "lazy journalism" where nobody will even leave the building these days to even ask someone involved in the story or topic. How the internet made it "easier" but they've become so over-reliant on it that many journalists fail to include information that isn't already online. Then again, Netflix is probably a phone call away, so that may be a whole 'nother level of lack-of-followup.
ufmace 9y ago
I get your point, but how many of these tech companies will actually authorize their employees who know what they're talking about to speak freely with tech reporters about how their system works? I would figure most would redirect you to a PR flunky to feed meaningless corporate-speak to any reporters, and then it's all "Netflix uses the latest internet technology to ensure the best possible viewing experience for all viewers around the world. Unfortunately, the specifics of our system are proprietary and we are unable to discuss that."
- gtaylor 9y ago
  Reddit is constantly out presenting, talking, and even writing about architecture, culture, and our unique challenges. Netflix seems to be similar. Their open source record speaks for itself, and their technical blog and presence at conferences and meetups is outstanding. Source: though these opinions are my own, I work on Infrastructure at Reddit.
- drdrey 9y ago
  Netflix is particularly open about how it works. There are some confidential bits, but most of the infrastructure projects are not considered proprietary. (I work at Netflix)
toddh 9y ago
Hi Jeremy, I'm very sorry if I've misquoted you or quoted you out of context. I'd certainly correct them instantly if contacted. I've searched my email and I don't see any corrections, so I'm not sure how I can make amends. I pull a lot of quotes from a lot of sources. Many thousands a year. My goal is always to highlight ideas that I think might help readers do their job better. Highlight is they key word here. That you say those sort of things often is not surprising. My hope is always readers will be made curious enough to follow the link and learn more on their own. And unfortunately there is no "they" at HighScalability.com. There's only me and there's only ever been me. So all mistakes are mine and only mine. There's never been much money or time to be as comprehensive as I'd like. I do the best I can, but I often fall short, no doubt. I'm not a journalist. I never said I was. I'm a programmer. Have been all my life. I started this blog a long time ago. It was just stuff I found interesting. For whatever reason it got popular for awhile so I tried to make something out of it. That, like most of the industry is petering out now, so we'll see what the future holds. The point about interviews is a good one and one I've wrestled with. Over the years I've conducted a lot of interviews. Much less so lately, and that's for a reason. I've learned that presentations at major conferences are usually the best resources. Presenters put a huge amount of effort into making those talks. And everything they say is open and approved for public consumption by definition. Interviews with someone like me are often a bother, a time sync, so I don't usually get as much out of them. Interviews have a high overhead, taking a lot of time to setup and there's often a lot of red tape and legal wrangling that must go on to get access and content approval. Which is again, why, presentations are such awesome resources. This article was derived mostly from people working on a project making a major presentation or from primary sources writing an article. I couldn't do better than that. And regardless, I would still make mistakes. Believe me. It's surprisingly hard to make sense of an interview and turn something from thread mode into document mode. That process is an error prone one because I have to fill in the gaps and there are always gaps. It's rare that I've ever just published an interview. I don't think they are that useful in general. I want people to read a highly condensed document they can get something concrete out of with minimal effort or just ignore easily if it's not relevant to them. The strange style I've developed reflects that goal. This article was of course different because it has a different audience. So again, I apologize. But I guarantee my mistakes are never intentional, never for money, and never for lack of caring or effort.

hetfeld 9y ago

What happens when i press play? It checks DRM settings in my browser and gets me lowest quality because my browser is open source. That is what happens.

synotna 9y ago
You have the freedom to watch something else
- Walf 9y ago
  And content creators have the freedom to choose where they distribute content, many of whom choose Netflix.
  - synotna 9y ago
    Correct, so passively aggressively complaining about DRM is a moot point
    - izacus 9y ago
      It's what customers do in free market to get better service.
      - beefield 9y ago
        Or turn from government regulated monopolistic market to actually free market i.e. torrents.

pavs 9y ago

Many of the things Netflix does here google does too and has been doing for a long time like GGC (Google Global Cache) for free at ISP NOC or datacenters - their GGC nodes are many orders of magnitude time is distribution. 90% of the Google traffic through these devices are youtube videos. They mention it in the article like Netflix pioneered this idea with open-connect. If anything it's harder to get OC devices from Netflix then it is from Google. Even Akamai does have a similar program where they let you put their CDN on ISP NOC.

amelius 9y ago
That sounds a bit like a violation of net neutrality avant la lettre.
- pavs 9y ago
  There is a whole world outside the United States.
- virtuallynathan 9y ago
  It's good for everyone -- if you are an ISP with 35%+ of your traffic coming from Netflix, you might be begging for these boxes in your network.
  - pavs 9y ago
    Yup, as an ISP owner, I can tell you 70-75% of my traffic come through GGC, for every 1gbps traffic from cache nodes 100mbps on average is pulled from the internet. Even with GGC nodes, it's still very expensive, BW-wise.
    - PuffinBlue 9y ago
      I saw another comment on here in the distant past that said an ISP owner decided to pull out the GGC boxes because they would sync vast amounts of data (largely YouTube IIRC) and it wasn't worth it bandwdith wise for them. It sounds like 10:1 is a decent saving but still objectively a lot of data.

deleted 9y ago

[deleted]

inciampati 9y ago

Nothing. So you go to your nearest torrent aggregator and are streaming it down in 3 minutes.

HenryBemis 9y ago
I read the word "nothing" and I thought to mysel: now THIS will turn interesting!!! (thinking that this will point to removal of Net Neutrality and the risks and conflicts it will bring to online services once ISPs turn aggro to both content producers/distributors and end-customers. Buuuuuuuuut no.. it was a silly short-lived comment about piracy (not even an educating one)
- inciampati 9y ago
  Actually it was a comment questioning why they don't use torrents to provide their services. And also pointing out that p2p based solutions somehow outperform theirs in many cases (for me, not US based so the net neutrality stuff is maybe not relevant). Anyway sorry I was too silly and didn't clarify these points.
  - Klathmon 9y ago
    P2P has traditionally not worked out well for companies. Even recently Microsoft tried it for updates, and people were quite angry. I honestly don't see a situation where someone would prefer to use their own bandwidth for a paid service rather than having a server do the heavy lifting. Either as the provider, or the consumer.
    - inciampati 9y ago
      This is true, and I wonder if someone has done an analysis of the many failures. It is impressive that something which works so well in the wild provides so little benefit when tamed. Maybe Netflix or a streaming service could pay for bytes uploaded. Abuse would be difficult to mitigate but in principle this would encourage peers to provide the company's service to the network.
    - 9y ago

verletx64 9y ago

There's something about this article, it kind of feels like a sequence of staccato thoughts. It feels like it kind of jumps around.

toddh 9y ago
I did try to tell a story, but I don't think I succeeded as well as I would have liked. It definitely does jump around more than it should. It's hard to cover the important bits and link them seamlessly together. I'm just not that good a writer.
- verletx64 9y ago
  Practice makes perfect! It was frustrating because I think I saw what you were going for and it's a good idea. Don't be afraid to drop things, even if they're important. You've got a title, right? "What happens when you press play". Answer that question. There's a whole load there that is interesting but doesn't answer the question of "Given Netflix have a bunch of video ready to go, how does that reach me?". Don't get me wrong, the rest is interesting, and important when thinking about Netflix as a whole, but ultimately I'm unsure they fit in the article. You might be worried about missing important bits, I personally worry about losing readers. It doesn't matter how important something is, if my target audience quits 15% of the way through the article. Anyway, no sweat really, blogging is hard enough. It's been a long time since I've written regularly, and I just started a personal blog, and yeah, a few weeks later im sitting on a mound of drafts I need to take a 'minimum viable post' eye to. :) Merry Christmas!

wz1000 9y ago

> Netflix has more than 110 million subscribers. > Netflix operates in more than 200 countries The five largest public torrent trackers track 30-50 million peers at any given time, serving over 10 million torrents. Torrents are estimated to have over a quarter of a billion active users. Torrents can be streamed, and in the end you have a DRM free copy of the file that you can rewatch at your pleasure, on any device(adding custom subtitles if you need them). It amazes me how much effort has been expended on pushing malware to almost everybody(binary, unauditable blobs in every browser) solely so that massive cartels can make money on something that has been solved since 2001. Subscribing to Netflix, or any other proprietary streaming (dis)service is not only foolish(in the sense of surrendering your own freedom, not to mention your hard earned money to nefarious corporate interests), but deeply unethical.

madeofpalk 9y ago
> not to mention your hard earned money to nefarious corporate interests Yes, this is the basis of our society. I work for money to buy things, mostly from corporations. There's nothing nefarious about that. I find it really hard to believe that paying for content is unethical.
- wz1000 9y ago
  > I find it really hard to believe that paying for content is unethical. Your money isn't going to the people who created the content. It is going to cartels that have no interests in common with you, and have massive lobbying power in order to force you to download malware and restrict the access of information from you.
  - madeofpalk 9y ago
    Of course money goes to the creators. I have an actor friend who was in a 'Netflix Original'. He got paid for it. Sure, others are making profits on it but, again, that's not surprising. That's how our society works. I wasn't 'lobbied' or forced to install 'malware'. I'm aware of what's going on (I too have heard of R.M.S.!) and I made a conscious decision to subscribe to such services.

bendbro 9y ago

It ticks to 25% loaded even though your internet is down.

Walf 9y ago

I know what it does when I don't press play: fills my memory with a boatload of bloated interface that has the shittiest navigation of any media player I've ever used. And I still can't just view a list of all series/movies by date.

nabeards 9y ago
Which media player has an interface you like? I'm usually an outcast for it, but I actually like the iTunes Movies interface myself.
- juanmirocks 9y ago
  Popcorn has a better interface than Netflix IMO
- Nala_Alan 9y ago
  I like mpv the most. It's simple, but still has everything you need (and you can do rest with the keyboard shortcuts), and it's pretty. In my opinion, however, the problem with Netflix player, is not its design. It's that it's pretty laggy, it is not nearly as smooth as for example YouTube player.
JustSomeNobody 9y ago
Amazon's UI on the fire stick is so much worse.
- PuffinBlue 9y ago
  I'm not sure about that really. It's like bashing your shin against a coffee table vs walking into a lamppost. Both painful experiences in their own unique ways.
ashark 9y ago
I’ve hated it since pressing play stopped being necessary to start video playing. Now I have to keep the focus bouncing around or else whatever it’s currently on will start yelling at me. Can’t just stop and look around at what’s on screen. It’s stressful. [EDIT] in fact, now that I think about it, I almost never just browse the catalogue anymore, while I used to a lot. I open it and go to search ASAP, then close it if I didn't find what I want or see something else interesting in the related results. The auto-playing is annoying enough that it's almost totally driven me away from that part of the interface. I knew I didn't like it but hadn't thought about how it'd affected my behavior, but there it is. Time browsing catalogue from "lots" to "nearly none". [EDIT EDIT] I might add that full-screen ads you have to scroll past (with no indication that's what you need to do) and other general mucking with the plain grid-layout interface make it wicked confusing for my parents. They just got a Roku solely for Netflix and if it keeps doing that I bet they won't use it very much.

temp 9y ago

I'd be interested in finding out what happens when I press the thumbs down/up icon on Netflix, because the match ratings that are displayed to me seem to have nothing to do with either my ratings or with my watching history.

juanmirocks 9y ago
It records your like/dislike, but it doesn’t distribute the change to its database immediately. Same for YouTube or so many other services.
- temp 9y ago
  The thing is I don't feel like any of the match ratings in the last year and a half since I've been using Netflix have had anything to do with my interests, the ratings I've given out or with my viewing history. Unless propagation to their database takes two years, I'm really interested in how they actually arrive at match ratings.
  - syshum 9y ago
    I believe largely because a single "like / dislike" is massively inferior to the old traditional Rating system where I am able to Rank content I like...
saryant 9y ago
AFAIK from some conversations I had at a conference last year with a few Netflix engineers who work on this system, it's sort of realtime. Recommendations are build up using Apache Flink, which provides stateful stream compute. So your click turns into an event on a Kafka topic which is eventually consumed by Flink, joined with whatever other data sources they need and then written out to a store somewhere. IIRC the various machine learning models exist inside their Flink architecture. https://ci.apache.org/projects/flink/flink-docs-release-1.4/dev/libs/ml/index.html https://ci.apache.org/projects/flink/flink-docs-release-1.4/...
dexterdog 9y ago

nvr219 9y ago

All this tells me is that Netflix is way too powerful a force in my life. Yikes.

plaidfuji 9y ago

ITT: people complaining about Netflix

Abishek_Muthian 9y ago

I feel bit sorry for Netflix, for the lengths they have gone to get their high performance/reliable architecture but the clients are still at the mercy of the OS, browsers, CPU (decoding codecs) of-course assuming high speed Internet. From their website - https://help.netflix.com/en/node/23742 https://help.netflix.com/en/node/23742 Resolution: Google Chrome Up to 720p on Windows, Mac, and Linux. Up to 1080p on Chrome OS. Internet Explorer up to 1080p. Microsoft Edge up to 4K. Mozilla Firefox up to 720p. Opera up to 720p. Safari up to 1080p on Mac OS X 10.10.3 or later. Streaming in 4K requires an HDCP 2.2 compliant connection to a 4K capable display, Intel's 7th generation Core CPU, and the latest Windows updates. P.S Interesting to note that Chrome OS plays 1080p, I assume because it has HW acceleration enabled by default (in browser alone, not in android apps though).

jmiserez 9y ago
Is it browser limitations, or just DRM limitations? Youtube seems to do 4K just fine.
- Abishek_Muthian 9y ago
  I think it has to do with HEVC h.265 codec & HW acceleration support available to the browser in the respective platforms. https://caniuse.com/#feat=hevc https://caniuse.com/#feat=hevc Not sure why Chrome doesn't support HW for it though albeit Chrome OS. My Celeron N3060 Chromebook can decode it. I read Netflix has started using VP9 for streaming, newer CPU's have VP9 decoders in them.
  - izacus 9y ago
    No it's actually a DRM limitation. You're not allowed to watch high definition content of your platform doesn't support latest Widewine DRM even if you have the HW or SW decoding capability.
- Nala_Alan 9y ago
  Yeah, IE and Edge are the only browsers with PlayReady DRM. Rest of them (besides Apple devices) are using Widevine DRM.

tlholaday 9y ago

Author uses acronym CDN without first defining it. Explain Like I Do Not Already Know What You Are Explaining.

jordache 9y ago
that article is pretty technical, beyond the level of technicality of simply knowing what a CDN is.
- JustSomeNobody 9y ago
  But even technical people sometimes don't know simple acronyms. It doesn't hurt to define them anyway.
  - jordache 9y ago
    mentioning cdn would be akin to a car review magazine mentioning ABS in an article and then dedicated a couple sentences explaining what ABS is
    - yesenadam 9y ago
      Hmm I dont know what CDN or ABS are.
      - BoorishBears 9y ago
        Content Delivery Network Anti-lock Braking System Both are exactly what they sound like
        yesenadam 9y ago
        OK thanks :-)
- tlholaday 9y ago
  I red-circle the style-guide violating unexplained CDN acronym because the author titled the work from which this is excerpted “Explain Like I’m 10.” I have no objection to using terms-of-art in articles written for experts.

dominotw 9y ago

> Netflix Saves Money In AWS How much of it due to special discount that netflix gets from AWS. Would a company thats not highly visible as netflix save the same amount of money ? > Recommendations. I find netflix recommendations pretty bad. Does it really work for other people?

juanmirocks 9y ago
The recommendations don’t work for me. The media selection by Netflix is not for me yet. Ultimately, I want a service that recommends me only three possibilities, and at least one is a total hit.
donarb 9y ago
It's not visibility, it's how much you spend/use. I'll bet there are large companies out there you've never heard of that get just as much of a discount from Amazon as Netflix does.
- qaq 9y ago
  I bet very few do I remember watching someone high up from Netflix presenting at a conference he quoted that they get 70% discount.
syshum 9y ago
>>I find netflix recommendations pretty bad. Does it really work for other people? They used to, the old system I found the recommendation to be pretty good and discovered some new things as a result The new, like/dislike system is garbage and most of the stuff is recommends are not something I would enjoy... I wonder how much of that is even based on my personal rating anymore or it is just stuff netflix wants to promote...

tw1010 9y ago

I guess the discrepancy between what I expected the article to be, based on the title, and what it actually was is a good reflection on the fact that development of these types of services is no longer built from first principles, low level to high level, but is rather constructed like a building from a set of abstracted lego pieces.

deleted 9y ago

[deleted]

erstap 9y ago

I feel like this article could be cut in half if it had been given a sharper focus. I don't care why I am seeing a specific image for a film and the reasoning for image personalization. Save that for a different article. I just want to know what happens after I click play.

curun1r 9y ago

> The advantage of having three regions is that any one region can fail, and the other regions will step in handle all the members in the failed region. At a previous employer, we met with AWS engineers who told us that this sort of thinking wasn't correct. Availability zones are for failure tolerance. Regions are conceptually different and for things like moving compute closer to clients, mitigating political instability or ensuring that data is held in a certain jurisdiction. To back up that claim, they told us that amazon.com ran out of a single region, us-west-2. As an aside, another interesting point from that meeting was that autoscaling wasn't a good strategy for scaling up in the event of the loss of an availability zone. The reason why is that in those scenarios, a lot of other customers will be trying to fail over as well and autoscaling can get very slow. Instead of autoscaling, they recommended that we run 150% of our per-AZ capacity in 3 AZs so that if an AZ went offline, the remaining two would already have the necessary capacity. We still used autoscaling, but it was for responding to spikes and troughs in our own load, not failure scenarios. I'm curious whether these two observations are correct since it seems like a lot of people view regions and autoscaling through a failure-recovery lens.

aaronblohowiak 9y ago
Sometimes there have been region-wide issues. Running cold has its own set of problems. We do not rely exclusively on auto scaling when we do failover. Disclaimer: while I work at Netflix, these views are not necessarily those of my employer
mimerme 9y ago
I think this is another case of "your not really supposed to...but it works ¯\_(ツ)_/¯" For your first point, I think it depends on your usage scenario. With a big company that requires good performance to satisfy their users, like Netflix, having multiple regions is a given. But what the article probably missed is that they also most likely use AZs within the same region to mitigate failures, but since they have multiple regions at hand why not use them to mitigate failures as well? And their monthly tests are a simulation of a worst case scenario if an entire region goes down. As for autoscaling I also think that people think of it as a magic solution. More users? Autoscale. Zone down? Autoscale. But the problem arises when they think that it's instant because it isn't. Things have to happen behind the scenes, but for many people, they only care about what happens on stage, so for them configuring autoscale is just better because it's easier than load balancing on 3 different AZs.

forgotpassagan 9y ago

I love how Netflix is so open about their architecture, but I have one strong criticism. After trying to get Spinnaker running, which is like 10 services for a fairly straightforward application, and reading about how Netflix has Cron jobs running to 'clean up' data inconsistencies... I started to think Google has a better approach to scaling the code. Netflix is strong on eventual consistency, and making services as small as possible. Having tried to build something in that image, holy shit the issue of distributed transactions becomes a nightmare. We had to build error handling in everything to handle the failure of everything else. Maybe half our code was 'just in case' to deal with exploding failure scenario complexity during service call fanout. If your service calls out to two or more services that also do data updates, God help you. There's no call ordering or way to make sure both succeed or fail, so if one does you could easily end up in an invalid state. Contrast this to Google's approach. Maps is one service, Docs is one service. Their service boundaries are much larger so they can shove the complexity of consistency and rollbacks back into the DB where it belongs. And they avoid eventual consistency as much as possible. If Google can make these big 'monoliths' work at huge scale I don't see any advantage to 'microservices', it's just a bunch of pointless overhead. I think Netflix has had a little too much industry koolaid ...

saryant 9y ago
> Netflix is strong on eventual consistency, and making services as small as possible. Having tried to build something in that image, holy shit the issue of distributed transactions becomes a nightmare. Distributed transactions are a sure sign that your service boundaries have been incorrectly drawn.
- forgotpassagan 9y ago
  Not according to Google :). They actually support and encourage distributed transactions across boundaries. Imagine all the data Google stores with your user profile across their hundreds of application boundaries. To keep things consistent you would either need to store all profile data in one massive service or support distributed transactions. The Netflix model is fault tolerance, with every service supporting various failed profile update scenarios. Google just decided to add distributed transactions support to spare all that overhead.

macpete 9y ago

Feels more like an ad for AWS than a technical post

twic 9y ago

This article is a chapter from my new book Explain the Cloud Like I'm 10. The normal expression is "explain like I'm 5", but in the cloud, you need to double everything, in case an AZ goes down.

kzzzznot 9y ago

> The video comes in a high definition format that’s many terabytes in size. A terabyte is big. Imagine 60 stacks of paper as tall as the Eiffel Tower. That’s a terabyte. What... No. An HD ProRes will be 150-200GB for a 1.5-2 hour film. That is a 1920x1080 Apple ProRes 422 HQ @ ~190Mbps which is the 'gold standard' accepted Mezz file for Netflix, Google Play, iTunes, AIV (Microsoft, Sony, TalkTalk etc). There are different codecs for broadcasters but this is the standard for VoD. If you deliver uncompressed (uncommon, few relative gains) then it might run into the hundreds of GB but never a multi-terabyte file for a feature. Although most of this article has been informative, I am dismayed by the outright BS. It basically ruins any trust I would have of this otherwise well written piece. Can your exaggeration be explained or justified please?

thinkMOAR 9y ago

If they could please make it so that you can search on <any> subtitle languages, it would really help when the in-laws visit.