By not collecting that much data. Netflix is a streaming platform. So... stream the movie. That's it. Collect just enough data for performance analysis and for keeping track of where the user stopped watching.
That's mere millions of data points per day, not trillions.
Like... why do they need real time machine learning at that rate? No, really: Why!? Would it be catastrophic to their business if they ran ML as batch jobs? Do their recommendations need to change second-to-second?
Another crazy quote: "At that time, Netflix had ~500 microservices, generating more than 10PB data every day in the ecosystem."
Wat?
That's a 170 MB of logs pre customer per day! Most of those customers might watch an episode of a TV show per day, or watch one movie per day. Call it 2 hours of usage per day? These guys were blasting out 1.4MB of logs per minute per active user while essentially doing nothing much more than streaming big binary blocks of data from a CDN!
In my mind that's the crux of the issue. The architecture astronauts have gone amok, solving problems that shouldn't exist...
The argument / excuse is: "Many product features, such as personalized recommendation, search, etc., can benefit from fresher data to improve user experiences, leading to higher user retention, engagement, etc."
My experience with NetFlix is that their recommendations are garbage and getting worse over time. Exabytes of data won't solve this.
I feel like the majority of celebrity tech companies are like this. They attract a certain kind of person who gets easily bored and needs interesting things to work on to stick around. There is also internal pressure to justify promotions by building things with "large impact". And CV padding to move onto the next big thing.
So you end up with lots of large, complex, engineering projects with virtually 0 or most likely negative actual business impact, to keep all the smart kids happy. And these beget more projects to deal with the complexity. It's pretty much Parkinson's Law applied to tech.
> and the recommendation system seems far worse than the old days
That's partially also because they lost or let expire the licenses to so much content.
Maybe governments should, in the interest of all people, mandate that all owners of IP/content should fairly license them to everyone who asks to operate a streaming or purchase service... that way we'd get competition on service quality and features between streaming providers instead of having to pay more for all the "content islands" than we did for cable TV.
500 microservices is less than I expected, assuming they're actually "micro". Stuff that comes up from the top of my mind: bandwidth estimation, best cache finding, rating, recommendations, kids experience, various aspects of user management, subscription management, analytics, ... - each of these could be multiple microservices. And that's just the user facing part. They also have corporate stuff, ingestion, ...
Reminds me of the Nutri-Matic machine:
> When the Drink button was pressed it made an instant but highly detailed examination of the subject's taste buds, a spectroscopic analysis of the subject's metabolism and then sent tiny experimental signals down the neural pathways to the taste centers of the subject's brain to see what was likely to go down well. However, no one knew quite why it did this because it invariably delivered a cupful of liquid that was almost, but not quite, entirely unlike tea.
Couldn't agree more. While it's true that Netflix does have some unique engineering problems to solve, this smells like "Paycheck Driven Development". When you employ 10,000 software engineers, if you don't have enough problems for them to solve, they'll start inventing problems.
20 trillion events per day? Are you kidding me? This sounds like some sort of Montessori day care for software engineers.
I recently had a discussion with a former Netflixer about keeping track of user views. It’s important to pick up where you left off and is likely important to pay the owners of the movies. It’s a very hard problem to solve. The data is highly skewed by skips and afk viewers.
This is mere kilobytes of data over the length of the entire movie, and can tolerate hours of latency for most aspects (e.g.: analysis, charge-back, ML, etc...).
A lot of the time I look at "modern cloud architectures" and I see a firehose of raw data being spewed out in hideously inefficient formats. Think 1 kilobyte of JSON to represent a metric, often a single number or data point. It's not just NetFlix, this is the default for most of Azure Monitor and its agents as an example that would affect a lot of people.
The write amplification is just insanity. Orders of magnitude of overhead, not data.
It's as if the entire world forgot that binary formats exist. Google in no small part benefits from remembering this lost art -- they default to gRPC and similar packed binary formats. One reason Kubernetes (written mostly by ex-Googlers) is so fast is because internally it's all compiled code and gRPC. Compared to, say, Azure itself it is ludicrously faster. Literally hundreds of times lower latencies for common operations.
deleted 5y ago
[deleted]
I'm with you on this one. I was an architect of a quite large SVOD streaming service and this sounds like a runaway engineering division that's lost.
Sure, you can log and trace every tiny event but they've forgotten to ask whether or not they should and what are you getting out of it. It's creating a problem that doesn't exist, then solving it (by creating 2 more complex systems). 500 microservices sounds excessive but not out of the realm of belief.
I do wonder if it might relate to the way they're measuring engineering performance.
I think you're misunderstanding Netflix's scale and their culture. I am sure the SVOD service you architected was great but it probably wasn't Netflix scale.
Netflix's culture is heavily biased towards collecting metrics and experimenting with them. Their entire ethos is data oriented therefore they tend to capture a lot of metrics. You are terming it as a 'runaway engineering division that's lost' is unfair. You probably need to work at their scale for a while to appreciate the complexity of their business and technology needs. If it was easy to build Netflix, then everyone would.
From my BA experience it could also be a runaway project manager issue or runaway data analytics issue as people just say hey let's get that piece of data.
Is it useful? Rarely asked. But more importantly, they rarely got pulled off when proved to be useless.
It’s clown eng run amok. But like any whale, no one tells them because they pay well.
Lol, reminded of of wat: https://www.destroyallsoftware.com/talks/wat https://www.destroyallsoftware.com/talks/wat
And I completely agree. What the hell are they doing, and is it worth the candle? Copyright aside, torrents and a shitty imdb wannabe site would have smaller CO² footprint and better content.
It sure is convenient to just say "don't do that", but it's a lazy answer divorced from reality.
Netflix does way more than "streaming big binary blocks of data from a CDN". Think of all the backend stuff behind the scenes that might go into this - encoding/re-encoding all these "big binary blocks", request tracing, audit logs, BI, etc.
Part of Netflix's success is powered by collecting this detailed information. Hiring 6 smart people to build something that can handle that volume of events is a complete no-brainer compared to the business value it generates.
> Netflix does way more than "streaming big binary blocks of data from a CDN".
Yes, but our point (or at least mine) is that it should not. Transcoding takes place once and for all when you first upload the video, then you can call it a day and let caching proxies distribute bits to people who want to download them.
Do users benefit from the machine learning algorithms and statistics collected by Netflix? No. Managers and shareholders benefit from it, and they're gonna let such big corporations destroy our entire planet if that's what it takes for their fame and profits. Just think about the ecological impact of what you're building, for humanity's sake.
You’re focusing on the “ecological impact” of an event bus over the impact of massive transcoding jobs and the infrastructure required to serve it fast to users?
> Transcoding takes place once and for all when you first upload the video, then you can call it a day and let caching proxies distribute bits
Damn! You should let Netflix know that, I’m sure they would be relieved to know it’s as simple as that, as anyone who’s ever worked with serving video to many devices would attest!
> Do users benefit from the machine learning algorithms and statistics collected by Netflix?
Yes. Those metrics allow them to take targeted bets on new types of content that other studios might pass over, across the entire world.
Your whole argument seems to be that “Netflix could be run with just python -m http.server /movies/, all these kids better get off my lawn because they are messing it up with all this complexity that I don’t appreciate because I’ve got no clue what’s required to build and operate something like Netflix at scale”. I don’t buy it.