6 ms·

I am one of the ones who complain about their microservices architecture quite a lot. This comes from both first-hand experience of talking to several of their

I am one of the ones who complain about their microservices architecture quite a lot. This comes from both first-hand experience of talking to several of their directors when consulted upon on how to make certain systems of theirs better. It's not just a matter of guarantees, it's a matter of complexity. Like right now Google search is dying and there's nothing that they can do to fix it because they have given up control. The same thing happened with Netflix where they wanted to push too hard to be a tech company and have their tech blogs filled with interesting things. On the back end they went too deep on the microservices complexity. And on the front end for a long time they suffered with their whole RxJS problem. So it's not an objective matter of what's better. It's more cultural problem at Netflix. Plus the fact that they want to be associated with "Faang" and yet their product is not really technology based.

kjellsbells 2y ago
> they want to be associated with "Faang" and yet their product is not really technology based. You lost me. Netflix built a massive CDN, a recommendation engine, did dynamic transcoding of video, and a bunch of other things, at scale, quite some years before everyone else. They may have enshittified in the last five years, but I dont see any reason why they dont have a genuinely legitimate claim to being a founder member of the FAANG club. I have a much harder time believing that companies with AI in their name or domain are doing any kind of AI, by contrast.
- ilrwbwrkhv 2y ago
  Pornhub has better, more reliable technology than Netflix, yet you don't see their tech blog very often do you?
  - always_imposter 2y ago
    fair. I have always wondered how do they deliver their content and what goes on behind the scenes and nobody on tech twitter or even youtubers talk about pornhub's infra for some reason. A lot of the innovation in tech has roots in people wanting to see high quality tiddies on the internet.
    - E39M5S62 2y ago
      I worked on PH infra for a few years - * The frontend itself runs on bare metal * A lot of the backend was built out as microservices, running on top of Mesos and then later K8s * CDN was in-house. The long tail is surprisingly long, but the front page videos were cached globally The unifying theme behind PH is that they host everything on their own equipment, no AWS/GCP involved.
  - bdangubic 2y ago
    PH has to stream for 20 seconds on average while boxing match is a lot longer :-)
    - johnnyanmac 2y ago
      Sounds like a consumer issue ;)
- badpun 2y ago
  Same thing was done a little later by HBO, Disney and a plethora of others, which points to the task not really being uber-difficult.
  - moomin 2y ago
    Speaking as a consumer, Netflix’s solution is objectively better than it’s competitors. It handles network blips better, it’s more responsive to use, it has far fewer random bugs you need to work around. You can argue whether or not that edge translates into more revenue, but the edge is objectively there.
    - johnnyanmac 2y ago
      Hard to speak "objectively" as a consumer who has their own regional biases and knows none of the sausage underneath. Maybe you're in a rural area and Netflix scaled gracefully. Maybe you're deep in SF and Netflix simply outspent to give minimal disruption to a population hub. These could both be true but don't speak to what performs better overall.
    - IIsi50MHz 2y ago
      Agree. Hulu, HBO/Max, and Disney Plus each do most of these: - frequently decide that episodes I've watched are either completely unwatched (with random fully watched eps of the show mixed in). - seemingly every time I leave at the start of the end-credits, I surely must have intended to come back and watch them. - rebuild the entire interface (progressively, slowly) when I've left the tab unfocussed for too long. Instead of letting continue where I was, they show it for less than a second, the rebuild the world. - keep resetting the closed-caption setting to "none", regardless of choosing "always" or "on instant replay"; worse, they sometimes still have the correct setting in the interface, but have disabled captions anyway. Netflix has only once since they started streaming forgotten playback position or episode completion. They politely suggest when to reload the page (via a tiny footer banner), but even that might not appear for months. They usually know where end-credits really start, and count that as completion. They don't seem to mess with captions.
  - Agingcoder 2y ago
jiggawatts 2y ago
You can explain these problems with simple business metrics that technologists like to ignore. Right before the recent Twitter acquisition, the various bits of info that came to the limelight included the "minor detail" that they had more than doubled their headcount and associated expenses, but had not doubled either their revenue or profits. Technology complexity went up, the business went backwards. Thousands of programmers doesn't always translate to more value! Netflix regularly puts out blog articles proudly proclaiming that they process exabytes of logs per microsecond or whatever it is that their microservices Rube Goldberg machine spits out these days, patting themselves on the back for a heroic job well done. Meanwhile, I've been able to go on the same rant year after year that they're still unable to publish more than five subtitle languages per region. These are 40 KB files! They had an employee argue with me about this in another forum, saying that the distribution of these files is "harder than I thought". It's not hard! They're solving the wrong problems. The problems they're solving are fun for engineers, but pointless for the business or their customers. From a customer perspective Netflix is either treading water or noticeably getting worse. Their catalog is smaller than it was. They've lost licensing deals for movies and series that I want to watch. The series they're producing themselves are not things I want to watch any more. They removed content ratings, so I can't even pick something that is good without using my phone to look up each title manually! Microservices solve none of these issues (or make it worse), yet this is all we hear about when Netflix comes up in technology discussions. I've only ever read one article that is actually relevant to their core business of streaming video, which was a blog about using kTLS in BSD to stream directly from the SSD to the NIC and bypassing the CPU. Even that is questionable! They do this to enable HTTPS... which they don't need! They could have just used a cryptographic signature on their static content, which the clients can verify with the same level of assurance as HTTPS. Many other large content distribution networks do this. It's 100% certain that someone could pretend to be Elon, fire 200-500 staff from the various Netflix microservices teams and then hire just one junior tech to figure out how to distribute subtitles... and that would materially improve customer retention while cutting costs with no downside whatsoever.
- aetimmes 2y ago
  > Right before the recent Twitter acquisition, the various bits of info that came to the limelight included the "minor detail" that they had more than doubled their headcount and associated expenses, but had not doubled either their revenue or profits. Every tech company massively inflated their headcount during the leadup to the Twitter acquisition because money was free. I interviewed at Meta in 2021 and asked an EM what he would do if given a magic wand to fix one problem at the company. His response: "I would instantly hire 10,000 more engineers." Elon famously did the opposite and now revenue is down 80%.
inquirerGeneral 2y ago
[dead]
aetimmes 2y ago
Google search is dying because of business reasons, not technical ones. The ads branch is actively cannibalizing search quality to make people perform more searches and view more ads. "Microservices" have nothing to do with it.
wlonkly 2y ago
They want to be associated with FAANG? What does the N stand for?
- ilrwbwrkhv 2y ago
  Forced assoc.

kjellsbells 2y ago

> they want to be associated with "Faang" and yet their product is not really technology based. You lost me. Netflix built a massive CDN, a recommendation engine, did dynamic transcoding of video, and a bunch of other things, at scale, quite some years before everyone else. They may have enshittified in the last five years, but I dont see any reason why they dont have a genuinely legitimate claim to being a founder member of the FAANG club. I have a much harder time believing that companies with AI in their name or domain are doing any kind of AI, by contrast.

ilrwbwrkhv 2y ago
Pornhub has better, more reliable technology than Netflix, yet you don't see their tech blog very often do you?
- always_imposter 2y ago
  fair. I have always wondered how do they deliver their content and what goes on behind the scenes and nobody on tech twitter or even youtubers talk about pornhub's infra for some reason. A lot of the innovation in tech has roots in people wanting to see high quality tiddies on the internet.
  - E39M5S62 2y ago
    I worked on PH infra for a few years - * The frontend itself runs on bare metal * A lot of the backend was built out as microservices, running on top of Mesos and then later K8s * CDN was in-house. The long tail is surprisingly long, but the front page videos were cached globally The unifying theme behind PH is that they host everything on their own equipment, no AWS/GCP involved.
- bdangubic 2y ago
  PH has to stream for 20 seconds on average while boxing match is a lot longer :-)
  - johnnyanmac 2y ago
    Sounds like a consumer issue ;)
badpun 2y ago
Same thing was done a little later by HBO, Disney and a plethora of others, which points to the task not really being uber-difficult.
- moomin 2y ago
  Speaking as a consumer, Netflix’s solution is objectively better than it’s competitors. It handles network blips better, it’s more responsive to use, it has far fewer random bugs you need to work around. You can argue whether or not that edge translates into more revenue, but the edge is objectively there.
  - johnnyanmac 2y ago
    Hard to speak "objectively" as a consumer who has their own regional biases and knows none of the sausage underneath. Maybe you're in a rural area and Netflix scaled gracefully. Maybe you're deep in SF and Netflix simply outspent to give minimal disruption to a population hub. These could both be true but don't speak to what performs better overall.
  - IIsi50MHz 2y ago
    Agree. Hulu, HBO/Max, and Disney Plus each do most of these: - frequently decide that episodes I've watched are either completely unwatched (with random fully watched eps of the show mixed in). - seemingly every time I leave at the start of the end-credits, I surely must have intended to come back and watch them. - rebuild the entire interface (progressively, slowly) when I've left the tab unfocussed for too long. Instead of letting continue where I was, they show it for less than a second, the rebuild the world. - keep resetting the closed-caption setting to "none", regardless of choosing "always" or "on instant replay"; worse, they sometimes still have the correct setting in the interface, but have disabled captions anyway. Netflix has only once since they started streaming forgotten playback position or episode completion. They politely suggest when to reload the page (via a tiny footer banner), but even that might not appear for months. They usually know where end-credits really start, and count that as completion. They don't seem to mess with captions.
- Agingcoder 2y ago

jiggawatts 2y ago

You can explain these problems with simple business metrics that technologists like to ignore. Right before the recent Twitter acquisition, the various bits of info that came to the limelight included the "minor detail" that they had more than doubled their headcount and associated expenses, but had not doubled either their revenue or profits. Technology complexity went up, the business went backwards. Thousands of programmers doesn't always translate to more value! Netflix regularly puts out blog articles proudly proclaiming that they process exabytes of logs per microsecond or whatever it is that their microservices Rube Goldberg machine spits out these days, patting themselves on the back for a heroic job well done. Meanwhile, I've been able to go on the same rant year after year that they're still unable to publish more than five subtitle languages per region. These are 40 KB files! They had an employee argue with me about this in another forum, saying that the distribution of these files is "harder than I thought". It's not hard! They're solving the wrong problems. The problems they're solving are fun for engineers, but pointless for the business or their customers. From a customer perspective Netflix is either treading water or noticeably getting worse. Their catalog is smaller than it was. They've lost licensing deals for movies and series that I want to watch. The series they're producing themselves are not things I want to watch any more. They removed content ratings, so I can't even pick something that is good without using my phone to look up each title manually! Microservices solve none of these issues (or make it worse), yet this is all we hear about when Netflix comes up in technology discussions. I've only ever read one article that is actually relevant to their core business of streaming video, which was a blog about using kTLS in BSD to stream directly from the SSD to the NIC and bypassing the CPU. Even that is questionable! They do this to enable HTTPS... which they don't need! They could have just used a cryptographic signature on their static content, which the clients can verify with the same level of assurance as HTTPS. Many other large content distribution networks do this. It's 100% certain that someone could pretend to be Elon, fire 200-500 staff from the various Netflix microservices teams and then hire just one junior tech to figure out how to distribute subtitles... and that would materially improve customer retention while cutting costs with no downside whatsoever.

aetimmes 2y ago
> Right before the recent Twitter acquisition, the various bits of info that came to the limelight included the "minor detail" that they had more than doubled their headcount and associated expenses, but had not doubled either their revenue or profits. Every tech company massively inflated their headcount during the leadup to the Twitter acquisition because money was free. I interviewed at Meta in 2021 and asked an EM what he would do if given a magic wand to fix one problem at the company. His response: "I would instantly hire 10,000 more engineers." Elon famously did the opposite and now revenue is down 80%.

inquirerGeneral 2y ago

[dead]

aetimmes 2y ago

Google search is dying because of business reasons, not technical ones. The ads branch is actively cannibalizing search quality to make people perform more searches and view more ads. "Microservices" have nothing to do with it.

wlonkly 2y ago

They want to be associated with FAANG? What does the N stand for?

ilrwbwrkhv 2y ago
Forced assoc.