3 ms·

Netflix is good only on streaming ready made content, not live streaming, but; 1. Netflix is a 300B company, this isn't a resources issue. 2. This isn't the f

by ksec 2y ago

Netflix is good only on streaming ready made content, not live streaming, but; 1. Netflix is a 300B company, this isn't a resources issue. 2. This isn't the first time they have done live streaming at this scale either. They already have prior failure experience, you expect the 2nd time to be better, if not perfect. 3. There were plenty of time between first massive live streaming to second. Meaning plenty of time to learn and iterate.

crowcroft 2y ago
Yea, the issue here isn't just that they're having issues, it's that they're having the same issues they've had before.
ilrwbwrkhv 2y ago
You can't solve your way out of a complex problem that you have created and which wasn't needed in the first place. The entire microservices thing was overly complex with zero benefits I spoke to multiple Netflix senior technicians about this. They said that's the whole shtick.
- iLoveOncall 2y ago
  That's a ridiculous statement. PrimeVideo is the leader in terms of sports events streaming over internet and it is composed of hundreds of microservices. Live streaming is just much harder than streaming, and it takes a years of work and a huge headcount to get something good.
  - sgarland 2y ago
    Prime famously undid some amount of their microservices recently because it couldn’t keep up, and was hideously expensive.
    - iLoveOncall 2y ago
      It was a single team for a very specific use case. To be clear when I said that PrimeVideo is composed of hundreds of microservices, I actually meant that it's composed of hundreds of services, themselves composed, more often than not, of multiple microservices. Depending on your definition of a microservice, my team alone owns dozens.
    - crop_rotation 2y ago
      This comment shows how a very random blog about a very small part of a product can dominate all conversation about it. Prime video famously did not undo anything. Out of 100+ teams one team undid one service. But somehow similar comments are common on HN. I am making no judgement or microservice or not just on this particular comment.
that_guy_iain 2y ago
They have the NFL next month on Christmas day. So that'll be a big streaming session but I think it'll be nothing compared to this. Even Twitter was having problems handling the live pirate streams there.
- jonlucc 2y ago
  > Even Twitter was having problems Is that a surprise? They're not who I would think of first as a gold standard for high viewership live streams.
  - that_guy_iain 2y ago
    Well, considering it was multiple small streams I would expect them to keep them up. No have their entire streaming service have issues.
hughesjj 2y ago
Yeah didn't they crash on love is blind or one of their reality shows recently-ish?
deleted 2y ago
[deleted]
freefaler 2y ago
The problem is that provisioning vast capacity for peak viewership is expensive and requires long-term commitment. Some providers won't give you more connectivity to their network unless you sign a 12 month deal where you prepay that. Peak traffic is very expensive to run, because you're building capacity that will be empty/unsused when the event ends. Who'd pay for that? That's why it's tricky and that's why Akamai charges these insane prices for live streaming. A "public" secret in that network layer is usually not redundant in your datacenter even if it's promised. To have redundant network you'd need to double your investment and it'll seat idle of at 50% max capacity. For 2hr downtime per year when you restart the high-capacity routers it's not cost efficient for most clients.
- treflop 2y ago
  Then sign a contract with Akamai, who has been in business for 25 years? You outsource if you aren’t planning to do something very often. There is no middle ground where you commit a mediocre amount of resources, end up with downtime and a mediocre experience, and then go “but we saved money.”
  - freefaler 2y ago
    Well, they didn't want to spend the money or more likely their own technical team/boss promised that they can do it themselves. They indeed have a great CDN network, but it's not very good for this particular type of traffic. May be they will know/fix/buy next time...
    - ksec 2y ago
      When Apple moved off Akamai for their Keynote live streaming, ( I remember they also used Limestone or EdgeCast ) they had some percentage of audience using Akamai and some on their own CDN. I think it took them three years before they completely moved off Akamai. Not sure if that is still case as that was more than 10 years ago. But like you stated, they dont want to spend money and their technical people couldn't deliver on time. This isn't a technical issue a lot of people on HN and Twitter wants to discuss about. It is a management issue.
rajnathani 2y ago
Apple was clearly larger than Google when they came out with Apple Maps, and it was issue-laden for a long time. It is not a resource-issue, but a tech development maturity issue.
brokenmachine 2y ago
>They already have prior failure experience What was the previous fail?

by ksec 2y ago

crowcroft 2y ago

Yea, the issue here isn't just that they're having issues, it's that they're having the same issues they've had before.

ilrwbwrkhv 2y ago

You can't solve your way out of a complex problem that you have created and which wasn't needed in the first place. The entire microservices thing was overly complex with zero benefits I spoke to multiple Netflix senior technicians about this. They said that's the whole shtick.

iLoveOncall 2y ago
That's a ridiculous statement. PrimeVideo is the leader in terms of sports events streaming over internet and it is composed of hundreds of microservices. Live streaming is just much harder than streaming, and it takes a years of work and a huge headcount to get something good.
- sgarland 2y ago
  Prime famously undid some amount of their microservices recently because it couldn’t keep up, and was hideously expensive.
  - iLoveOncall 2y ago
    It was a single team for a very specific use case. To be clear when I said that PrimeVideo is composed of hundreds of microservices, I actually meant that it's composed of hundreds of services, themselves composed, more often than not, of multiple microservices. Depending on your definition of a microservice, my team alone owns dozens.
  - crop_rotation 2y ago
    This comment shows how a very random blog about a very small part of a product can dominate all conversation about it. Prime video famously did not undo anything. Out of 100+ teams one team undid one service. But somehow similar comments are common on HN. I am making no judgement or microservice or not just on this particular comment.

that_guy_iain 2y ago

They have the NFL next month on Christmas day. So that'll be a big streaming session but I think it'll be nothing compared to this. Even Twitter was having problems handling the live pirate streams there.

jonlucc 2y ago
> Even Twitter was having problems Is that a surprise? They're not who I would think of first as a gold standard for high viewership live streams.
- that_guy_iain 2y ago
  Well, considering it was multiple small streams I would expect them to keep them up. No have their entire streaming service have issues.

hughesjj 2y ago

Yeah didn't they crash on love is blind or one of their reality shows recently-ish?

deleted 2y ago

[deleted]

freefaler 2y ago

The problem is that provisioning vast capacity for peak viewership is expensive and requires long-term commitment. Some providers won't give you more connectivity to their network unless you sign a 12 month deal where you prepay that. Peak traffic is very expensive to run, because you're building capacity that will be empty/unsused when the event ends. Who'd pay for that? That's why it's tricky and that's why Akamai charges these insane prices for live streaming. A "public" secret in that network layer is usually not redundant in your datacenter even if it's promised. To have redundant network you'd need to double your investment and it'll seat idle of at 50% max capacity. For 2hr downtime per year when you restart the high-capacity routers it's not cost efficient for most clients.

treflop 2y ago
Then sign a contract with Akamai, who has been in business for 25 years? You outsource if you aren’t planning to do something very often. There is no middle ground where you commit a mediocre amount of resources, end up with downtime and a mediocre experience, and then go “but we saved money.”
- freefaler 2y ago
  Well, they didn't want to spend the money or more likely their own technical team/boss promised that they can do it themselves. They indeed have a great CDN network, but it's not very good for this particular type of traffic. May be they will know/fix/buy next time...
  - ksec 2y ago
    When Apple moved off Akamai for their Keynote live streaming, ( I remember they also used Limestone or EdgeCast ) they had some percentage of audience using Akamai and some on their own CDN. I think it took them three years before they completely moved off Akamai. Not sure if that is still case as that was more than 10 years ago. But like you stated, they dont want to spend money and their technical people couldn't deliver on time. This isn't a technical issue a lot of people on HN and Twitter wants to discuss about. It is a management issue.

rajnathani 2y ago

Apple was clearly larger than Google when they came out with Apple Maps, and it was issue-laden for a long time. It is not a resource-issue, but a tech development maturity issue.

brokenmachine 2y ago

>They already have prior failure experience What was the previous fail?