6 ms·
Netflix seems so over engineered to me. They basically have a catalog of a few thousand movies that are negotiated months in advance of actual use. Basically, t
by RcouF1uZ4gsC 10y ago
Netflix seems so over engineered to me. They basically have a catalog of a few thousand movies that are negotiated months in advance of actual use. Basically, they just need to encode them and put them on a box and ship them to edge caches. Caching immutable data scales incredibly well.I would also bet that 99.99% of the movies people actually watch on Netflix, would fit on a single box.
In regard to the analytics, they have 100 million subscribers. Let's say each subscriber watches an average of 100 episodes/movies a day. For each watch you record subscriber ID, movie ID, start time, stop time and get 32 bytes * 100 * 100,000,000 = 320 gigabytes of data per day total. I am pretty sure that you could get a commercial database and and business intelligence package that could support the type of analytics you need (mainly clustering analysis) at that scale. A national grocery chain, probably has a similar amount of data ingestion and a similar analytics need. In addition, I have subscribed to multiple Netflix type services and I have never weighed the quality of suggestions very high, giving much more weight to the functionality of the client, lack of ads, and large catalog of good movies.
As evidence that this is a solved engineering problem, just look at the number of similar movie services: Amazon, Google play, Hulu, PlayStation Vue, Pureflix, Crackle, etc.
Google, Facebook, Baidu, Amazon, the self driving car companies are doing cutting edge stuff in terms of scalability and analysis, but not Netflix. The complexity of their operations, seems mainly to be one of their own doing and not intrinsic to the service they provide.
So I look at stuff like the article here and see a bunch of very smart engineers who are bored with the (solved) core problem and spend their time making cool stuff which is actually a pretty good thing.
- mdekkers 10y agoDon't they also have some kind of recommendation engine? I bet that takes a bit more data and processing (who watched what, did they finish, maybe even more like what kind of scenes did they stop watching, or watched again, etc)
- pkaye 10y agoI can see this kind of over engineering in all the companies that hit the VC/IPO lottery. They all hire like crazy the best engineers and have to keep them busy doing challenging engineering stuff. And managers want to protect their turf so they join in. So you have ever more complex designs in anticipation of growth, side projects, detailed analysis, blogs with pretty graphs, etc. A combination of Parkinson's Law and Second System Effect.
- lmkg 10y agoAt one point, Netflix video streaming accounted for something absurd like 20% of all Internet traffic in the country. Regardless of whether it "scaled incredibly well," I would imagine there are still novel issues with that much scaling. Reliability also matters differently for video than it does for normal web traffic. It's one thing to shuttle 5GB (or whatever) of data over the course of an hour. It's another thing to shuttle 5GB of data with no hiccups for an hour. Detecting and routing around machine or network issues fast enough that real-time video playback is not impeded sounds to me like a difficult challenge.
- yomly 10y agoBeat me to it. I am sure a large part Netflix's success is how seamless its products are: there's minimal to no loading time and the video stream is virtually always smooth - including during times of action/high frame rate. Creating a video streaming platform is one thing, but creating a content delivery platform that is effectively as reliable as TV that is also cost efficient is no small task if you ask me...
- deleted 10y ago[deleted]
- johansch 10y agoBullcrap. The tech is good enough (but not that much better beyond that level). >90% of the success comes from the content creation/licensing strategy/execution and marketing. I.e. non-tech. At this point it's a bit like arguing that Disney's great success in Bluray sales comes from their encoding/Bluray mastering expertise.
- sheeshkebab 10y agoTech part was "good enough" for last 20+ years - it's why most companies call their tech teams "IT department". There is however something about that last 5-10% where moving a few bits around the right way gives you business advantage, over places like Disney. That said, it does seem like some of this Netflix stuff is overbuilt, although I don't need to run tens of 1000's of ec2, so maybe it's worth it.
- sethev 10y agoHow does the fact that other companies have streaming video services lead to Netflix not having to solve that problem? This seems a bit like saying that Walmart doesn't need to worry about managing their supply chain because Cosco has already solved that problem. Also if you've ever tried a TV channel's streaming service you'll realize that recording the start and end time won't tell you much about the quality of the experience.
- aquilaFiera 10y agoThis sounds like one of those developer estimates that takes 10x longer than stated.
- always_good 10y ago"Netflix is over-engineered, it's just videos lol" Classic HN hubris. For example, Netflix dynamically adjusts video quality on demand. Pretty easy feature. The top HN comment will explain how they would implement that in five minutes -- hypothetically, of course!
- orf 10y ago"Responsible 20% of the entire us internet traffic? Solved problem mate, i could do it on a single machine with mongodb. Overengineered!"
- rhizome 10y agoJust think how much more they could do if they stopped using MySQL. /s
- user5994461 10y agoForget about the joke for a minute and let's be realistic. It is entirely possible and reasonable to prototype a streaming service serving 20% of the us traffic with a single machine and mongodb. Well, a single machine, mongodb AND akamai.
- orf 10y agoWell of course, mongodb is web scale don't ya know
- user5994461 10y agoMongoDB is shit. No need to use it. What's important is really the distribution and caching layer, and Akamai has been offering that for a very long time for all the big internet companies.
- jefe_ 10y agoI get the impression they're running analytics not just on the users but also the content (which would take more horsepower). Guessing they use the analysis to model scripts and generate ideas for their Original Series. Thinking of what is needed to do this, you need to analyze themes, characters, story lines, visual effects, music, many components, and then tie them to user types and viewing patterns. Then they create series with either wide appeal or strong appeal to a very small niche. On top of that, these shows load almost instantly while you're still determining whether to watch or not.
- spydum 10y agoFor sure there is a bit of self-flagellation. However, what it doesn't tell you is how cost effective or as they elude to: how improves developer velocity (a stand in for productivity) is. Those other guys may also be offering streaming, but how quickly can they adapt?
- philsnow 10y agohttp://www.smbc-comics.com/comic/2011-12-13 http://www.smbc-comics.com/comic/2011-12-13 If we make a scale from 1 to 4 out of those four panels, I'd say Google is around a 2.5 and Netflix is somewhere above 3. It's tough to assign numbers much past 3 because the technology becomes indistinguishable from magic pretty quickly.
- kayoone 10y agoIn the end Netflix is one of the places people would like to work because of stuff like this. The sheer scale of operations at netflix combined with modern technology is very interesting to many engineers, so it also kind of work in favour of their HR.
- zaidos 10y agoThat is the core of their product. However, there are a lot of by-products and back office software needed to run the company. As a consumer, you most likely are only perceiving a small percentage of the technology running a company. - Applications for all devices Netflix is on (TVs, Phones, Displays, Roku, Apple TV, etc.) and all the support around it. - Recommendation and analytical software within Netflix. Not even accounting for other areas of the company where ML can be applied. - Billing and financial software for Netflix users and partners - Proprietary/internal customer support software - Content management system, along with all the international and legal challenges around that. - Any/all software used by their marketing teams
- inopinatus 10y ago* Operations and control plane for all of the above that is robust enough for SREs to deal with million-to-one-chance occurrences happening every few minutes. * Complexity multiplier (variations of contracts, catalogues, infrastructure, language, currency, payment methods, taxation, government regulation, corporate governance, market structure, customer preferences) of offering service in >100 countries. The suggestion that building a service like Netflix is a "solved problem" is naive to the point of idiocy.
- emptythought 10y agoAnother thing worth noting is that they still support basically every piece of hardware they ever did. Think built in apps on old smart TVs or early blu-ray decks that haven't been patched in YEARS, devices from before 2009~ even. You can still watch netflix on an original Wii. That hasn't gotten any OS or software patches since like 2011. That's a long tail of legacy clients to have to deal with.
- toast0 10y agoGiven that streaming is basically competing with discs, it makes sense not to drop any devices ever; a DVD produced today is expected to work on a DVD player from however long ago, provided it's still operational.
- 10y ago
- alpha_squared 10y ago> Google, Facebook, Baidu, Amazon, the self driving car companies are doing cutting edge stuff in terms of scalability and analysis, but not Netflix. I think you're missing a quantifier of some sort. Perhaps "consumer-facing"? Of any AWS customer, Netflix is far and away the leader in understanding and making use of AWS. It pushes the limits more often than not. Their scalability, failure/recovery handling and (assumed) cost-effective deployments are _the_ standard in AWS and cloud computing. They even have an open-source toolset[0] for testing your infrastructure against failures and recovery that's quickly being picked up by several other companies to harden their infrastructure. Then you realize that regularly, throughout the day, they're intentionally bringing their own infrastructure down and they can still operate with nary a hiccup. I wouldn't call that over-engineered. [0] https://github.com/Netflix/SimianArmy https://github.com/Netflix/SimianArmy
- jperras 10y ago> A national grocery chain, probably has a similar amount of data ingestion and a similar analytics need. It's off-topic, but having interacted with national-level grocery chains before, I think you grossly over-estimate their technological capacity and savvy. They're probably one of the slowest industries to adopt new technological advancements, for better or worse.
- Gelob 10y agoWhile the contracts may be negotiated months in advance the videos aren't delivered until very close to the release date, the same for Google and amazon. In some cases it may be a few days early but production schedules are tight and even if the post house has it done early contracts and controls forbid them from sending it to Netflix, etc too early.
- manigandham 10y ago> Basically, they just need to encode them and put them on a box and ship them to edge caches. Caching immutable data scales incredibly well.I would also bet that 99.99% of the movies people actually watch on Netflix, would fit on a single box. This is exactly what they do. The video serving part isn't that complicated, it's the rest of the site and APIs. Although those are a tad over engineered considering the final output is often a slow and unwieldy UX.