4 ms·

and Netflix?

by rizjoj 9y ago

and Netflix?

eikenberry 9y ago
They have a completely different use case that benefits from being able to scale up and down quickly. I don't see them moving away from AWS anytime soon.
kyledrake 9y ago
They run BGP and do an enormous amount of IX peering. https://www.peeringdb.com/asn/2906 https://www.peeringdb.com/asn/2906 AWS doesn't even let you do BGP. If you want to use them for a CDN, you're monopolized into their network and their blend of ridiculously overpriced bandwidth. They might be using AWS for the canonical store of data, but the sanity of using "the cloud" goes out the window the second you need to ship a lot of traffic to the public internet.
- pravinva 9y ago
  What are you on about? AWS practically requires BGP. Static routing is rarely the best choice when you set up DX. Netflix rolls its own CDNs because Cloudfront can't meet all their demands, yet.
- jimktrains2 9y ago
  > heir blend of ridiculously overpriced bandwidth. For someone not at netflix's level, can you recommend somewhere with significantly cheaper bandwidth.
  - lsjdfkljdfwkwdf 9y ago
    At that traffic volume Netflix has more negotiation power with ISPs
  - toomuchtodo 9y ago
    Any colo or dedicated server provider.
    - jimktrains2 9y ago
      At a CoLo I have the rack, managment, and hardware overhead too, though? Also fewer PoP. EDIT: I guess I was talking about CloudFront specifically, not EC2. EC2 is pretty pricey.
      - toomuchtodo 9y ago
        Price compare various CDNs. It's a commodity business. Fastly comes to mind, but we had quite a few outages with them at my last org.
  - thezilch 9y ago
    Pretty much anywhere.
  - corford 9y ago
    Pretty much anyone... Hetzner, Linode, Choopa
    - jimktrains2 9y ago
      For those I need to run a VPS, which also costs money and has fewer PoP and more mangment overhead. EDIT: I guess I was talking about CloudFront specifically, not EC2. EC2 is pretty pricey.
      - corford 9y ago
        Sorry misunderstood. You could try here https://www.cdnperf.com/ https://www.cdnperf.com/ and compare egress prices/limits.
  - NightlyDev 9y ago
    While I don't know what pricing AWS has for high bandwidth customers since it's not public, I do know that the public prices are really high. The thing is, even for smaller companies it's easy to get way better deals on bandwidth. I'm currently using a provider that runs openstack and I'm paying less than $0.009/GB, from the first byte(no commitment). I'd say that is very cheap compared to AWS pricing at $0.05/GB above 350 TB. With colocation or renting dedicated servers and IP-transit you can get way lower prices. A dedicated 1 Gbps is about 300 USD/month with my provider, and the prices drops hard for dedicated 10, 40 and 100 Gbps uplinks. Never used OVH myself, but they are providing 2 Gbps at £340/month for their highest quality bandwidth, and £64/month for bandwidth they think would be good for downloads. I think using cloud providers for somewhat high bandwidth services are basically throwing money out the window. Until prices really drops(if they ever will) I'll continue thinking that clouds are for temporary workloads or prototyping with the ability to easily scale at a high cost.
    - CharlesW 9y ago
      > I'm currently using a provider that runs openstack and I'm paying less than $0.009/GB, from the first byte(no commitment). Would you mind sharing who that is?
      - NightlyDev 9y ago
        I'm not allowed to say, but I think OVH also runs openstack with the "public cloud" offering, and I think bandwidth are even cheaper with them.
    - user5994461 9y ago
      It's interesting that what you consider a great provider is only doing 6 times cheaper than AWS. That's not much of a discount.
  - dasil003 9y ago
    I built a global video CDN using SoftLayer a couple years ago and ended up paying around $0.01/GB.
    - jimktrains2 9y ago
      How? https://www.ibm.com/cloud-computing/bluemix/content-delivery-network https://www.ibm.com/cloud-computing/bluemix/content-delivery... (which is linked to from http://www.softlayer.com/ http://www.softlayer.com/) says $0.12/GB. https://aws.amazon.com/cloudfront/pricing/ https://aws.amazon.com/cloudfront/pricing/ says $0.085/GB.
      - dasil003 9y ago
        At the time they included 20TB of traffic per month per server, and it went into regional pools. I think they discontinued that because of us :)
  - shaklee3 9y ago
    Packet.net
  - notyourday 9y ago
    Below $10k/mo bandwidth plays are irrelevant - the cost of the support infrastructure is just too high. At $10k/mo what you do is plop a rack in one of the well connected buildings in NYC, VA, SF, SJC, CHI and get either flat rate 10Gs or 1G commits over 10Gs. You should be paying between 0.55 and 0.75 per Mbit/sec on 95th percentile on 1G commit. You put into the same cabinet your edge nodes that actually would be pushing the traffic out and use AWS or GCP for your compute workloads.
- lsjdfkljdfwkwdf 9y ago
  Name a CDN (excluding DDoS mitigation) that offers useful BGP service to customers? Even some CDNs are not quite good at running BGP
- _pmf_ 9y ago
  > you're monopolized into their network and their blend of ridiculously overpriced bandwidth. Wouldn't a customer the size of Netflix be granted more liberty in this regard?
andyfleming 9y ago
To be fair, Netflix only runs some of their infrastructure on AWS. Basically, if I understand correctly, their applications run on AWS and their delivery is through their own CDN built on top of various ISPs[1]. [1] https://media.netflix.com/en/company-blog/how-netflix-works-with-isps-around-the-globe-to-deliver-a-great-viewing-experience https://media.netflix.com/en/company-blog/how-netflix-works-...
- yeukhon 9y ago
  Correct, but some is probably an understatment. They are pretty much run everyting on AWS except what you cited (e.g video processing and content delivery). Netflix is the biggest AWS user.
  - scurvy 9y ago
    Everything on AWS except their largest, most critical piece of infrastructure that needs to be the highest performing. That says it all right there.
    - erikbye 9y ago
      https://www.networkworld.com/article/3037428/cloud-computing/netflix-is-not-really-all-in-on-amazon-s-cloud.html https://www.networkworld.com/article/3037428/cloud-computing... And, > We rely on the cloud for all of our scalable computing and storage needs — our business logic, distributed databases and big data processing/analytics, recommendations, transcoding, and hundreds of other functions that make up the Netflix application. Video is delivered through Netflix Open Connect, our content delivery network that is distributed globally to efficiently deliver our bits to members’ devices. Source: https://media.netflix.com/en/company-blog/completing-the-netflix-cloud-migration https://media.netflix.com/en/company-blog/completing-the-net... Netflix still operates its own data centers and CDN to deliver video. This is not done on AWS.
    - yeukhon 9y ago
      Isn't that what I said above?

and Netflix?

by rizjoj 9y ago

and Netflix?

They have a completely different use case that benefits from being able to scale up and down quickly. I don't see them moving away from AWS anytime soon.

kyledrake 9y ago

They run BGP and do an enormous amount of IX peering. https://www.peeringdb.com/asn/2906 https://www.peeringdb.com/asn/2906 AWS doesn't even let you do BGP. If you want to use them for a CDN, you're monopolized into their network and their blend of ridiculously overpriced bandwidth. They might be using AWS for the canonical store of data, but the sanity of using "the cloud" goes out the window the second you need to ship a lot of traffic to the public internet.

pravinva 9y ago
What are you on about? AWS practically requires BGP. Static routing is rarely the best choice when you set up DX. Netflix rolls its own CDNs because Cloudfront can't meet all their demands, yet.
jimktrains2 9y ago
> heir blend of ridiculously overpriced bandwidth. For someone not at netflix's level, can you recommend somewhere with significantly cheaper bandwidth.
- lsjdfkljdfwkwdf 9y ago
  At that traffic volume Netflix has more negotiation power with ISPs
- toomuchtodo 9y ago
  Any colo or dedicated server provider.
  - jimktrains2 9y ago
    At a CoLo I have the rack, managment, and hardware overhead too, though? Also fewer PoP. EDIT: I guess I was talking about CloudFront specifically, not EC2. EC2 is pretty pricey.
    - toomuchtodo 9y ago
      Price compare various CDNs. It's a commodity business. Fastly comes to mind, but we had quite a few outages with them at my last org.
- thezilch 9y ago
  Pretty much anywhere.
- corford 9y ago
  Pretty much anyone... Hetzner, Linode, Choopa
  - jimktrains2 9y ago
    For those I need to run a VPS, which also costs money and has fewer PoP and more mangment overhead. EDIT: I guess I was talking about CloudFront specifically, not EC2. EC2 is pretty pricey.
    - corford 9y ago
      Sorry misunderstood. You could try here https://www.cdnperf.com/ https://www.cdnperf.com/ and compare egress prices/limits.
- NightlyDev 9y ago
  While I don't know what pricing AWS has for high bandwidth customers since it's not public, I do know that the public prices are really high. The thing is, even for smaller companies it's easy to get way better deals on bandwidth. I'm currently using a provider that runs openstack and I'm paying less than $0.009/GB, from the first byte(no commitment). I'd say that is very cheap compared to AWS pricing at $0.05/GB above 350 TB. With colocation or renting dedicated servers and IP-transit you can get way lower prices. A dedicated 1 Gbps is about 300 USD/month with my provider, and the prices drops hard for dedicated 10, 40 and 100 Gbps uplinks. Never used OVH myself, but they are providing 2 Gbps at £340/month for their highest quality bandwidth, and £64/month for bandwidth they think would be good for downloads. I think using cloud providers for somewhat high bandwidth services are basically throwing money out the window. Until prices really drops(if they ever will) I'll continue thinking that clouds are for temporary workloads or prototyping with the ability to easily scale at a high cost.
  - CharlesW 9y ago
    > I'm currently using a provider that runs openstack and I'm paying less than $0.009/GB, from the first byte(no commitment). Would you mind sharing who that is?
    - NightlyDev 9y ago
      I'm not allowed to say, but I think OVH also runs openstack with the "public cloud" offering, and I think bandwidth are even cheaper with them.
  - user5994461 9y ago
    It's interesting that what you consider a great provider is only doing 6 times cheaper than AWS. That's not much of a discount.
- dasil003 9y ago
  I built a global video CDN using SoftLayer a couple years ago and ended up paying around $0.01/GB.
  - jimktrains2 9y ago
    How? https://www.ibm.com/cloud-computing/bluemix/content-delivery-network https://www.ibm.com/cloud-computing/bluemix/content-delivery... (which is linked to from http://www.softlayer.com/ http://www.softlayer.com/) says $0.12/GB. https://aws.amazon.com/cloudfront/pricing/ https://aws.amazon.com/cloudfront/pricing/ says $0.085/GB.
    - dasil003 9y ago
      At the time they included 20TB of traffic per month per server, and it went into regional pools. I think they discontinued that because of us :)
- shaklee3 9y ago
  Packet.net
- notyourday 9y ago
  Below $10k/mo bandwidth plays are irrelevant - the cost of the support infrastructure is just too high. At $10k/mo what you do is plop a rack in one of the well connected buildings in NYC, VA, SF, SJC, CHI and get either flat rate 10Gs or 1G commits over 10Gs. You should be paying between 0.55 and 0.75 per Mbit/sec on 95th percentile on 1G commit. You put into the same cabinet your edge nodes that actually would be pushing the traffic out and use AWS or GCP for your compute workloads.
lsjdfkljdfwkwdf 9y ago
Name a CDN (excluding DDoS mitigation) that offers useful BGP service to customers? Even some CDNs are not quite good at running BGP
_pmf_ 9y ago
> you're monopolized into their network and their blend of ridiculously overpriced bandwidth. Wouldn't a customer the size of Netflix be granted more liberty in this regard?

andyfleming 9y ago

To be fair, Netflix only runs some of their infrastructure on AWS. Basically, if I understand correctly, their applications run on AWS and their delivery is through their own CDN built on top of various ISPs[1]. [1] https://media.netflix.com/en/company-blog/how-netflix-works-with-isps-around-the-globe-to-deliver-a-great-viewing-experience https://media.netflix.com/en/company-blog/how-netflix-works-...

yeukhon 9y ago
Correct, but some is probably an understatment. They are pretty much run everyting on AWS except what you cited (e.g video processing and content delivery). Netflix is the biggest AWS user.
- scurvy 9y ago
  Everything on AWS except their largest, most critical piece of infrastructure that needs to be the highest performing. That says it all right there.
  - erikbye 9y ago
    https://www.networkworld.com/article/3037428/cloud-computing/netflix-is-not-really-all-in-on-amazon-s-cloud.html https://www.networkworld.com/article/3037428/cloud-computing... And, > We rely on the cloud for all of our scalable computing and storage needs — our business logic, distributed databases and big data processing/analytics, recommendations, transcoding, and hundreds of other functions that make up the Netflix application. Video is delivered through Netflix Open Connect, our content delivery network that is distributed globally to efficiently deliver our bits to members’ devices. Source: https://media.netflix.com/en/company-blog/completing-the-netflix-cloud-migration https://media.netflix.com/en/company-blog/completing-the-net... Netflix still operates its own data centers and CDN to deliver video. This is not done on AWS.
  - yeukhon 9y ago
    Isn't that what I said above?