4 ms·

And yet a lot of folks think Netflix is on AWS. I would think AWS gives a huge discount to Netflix and ask 'just run at least something on aws'. Those promotion

by betaby 5y ago

And yet a lot of folks think Netflix is on AWS. I would think AWS gives a huge discount to Netflix and ask 'just run at least something on aws'. Those promotional videos have nothing useful about netflix on aws https://aws.amazon.com/solutions/case-studies/netflix-case-study/ https://aws.amazon.com/solutions/case-studies/netflix-case-s... and targeted to managers and unaware of the scale engineers.

dangrossman 5y ago
Netflix is on AWS -- the website and app backends, not the video streams. * Netflix.com resolves to ec2-52-3-144-142.compute-1.amazonaws.com from my computer at this moment. Some more domains their apps reach out to include: * nflxvideo.net: ec2-46-137-171-215.eu-west-1.compute.amazonaws.com. * appboot.netflix.com: ec2-34-199-156-113.compute-1.amazonaws.com. * uiboot.netflix.com: ec2-52-203-145-205.compute-1.amazonaws.com.
- betaby 5y ago
  That exactly what I mean. AWS would represent single digit of the infrastructure and traffic.
  - dangrossman 5y ago
    Maybe of their traffic, but not their infrastructure. Netflix rents 300,000 CPUs on AWS just for video encoding. Encoding one TV series produces over 1,000 files that sit on S3, ready for those PoPs to pull them in and cache them. They also do more data processing, analytics and ML than most people realize and that's all happening on AWS. For example, not only are each user's recommendations personalized, but all of the artwork is personalized too. If you like comedy, the cover art for Good Will Hunting may be a photo of Robin Williams. If you like romance, the cover art for the same movie may be a photo of Matt Damon and Minnie Driver going in for a kiss. They're also doing predictive analytics on what each person will want to watch tomorrow in particular. Every evening, all those non-AWS servers go ask AWS what video files they should download from S3 to proactively cache the next day of streaming.
    - betaby 5y ago
      ~300000 CPUs is about 2K racks worth of servers. I would think if you have such constant load it would be more economical to move that on-premise.
      - jeffbee 5y ago
        300k cpu cores is “only” about 1200 nodes these days, definitely not 2000 racks.
        betaby 5y ago
        If CPU == core thread, then yes.
      - kristjansson 5y ago
        The load is emphatically not constant though? Encoding has to be done once, as fast as possible, and updated rarely. That warehouse (or just aisle) of racks would be sitting idle much of the time…
      - killingtime74 5y ago
        You’re thinking like an engineer. A business person would just ask for a discount (which I’m sure they do) and constantly renegotiate. Don’t forget the financing of on premise and data centre workers and how that looks on your balance sheet.
    - jeffbee 5y ago
      I wonder if there is any discussion of whether they could make it cheaper with Google’s compression ASICs, which I assume will be a cloud service at some point.
      - whoknowswhat11 5y ago
        Amazon offers something not has fancy (but maybe more flexible?) VT1 instances can support streams up to 4K UHD resolution at 60 frames per second (fps) and can transcode up to 64 simultaneous 1080p60 streams in real time. One thing is that a place like netflix has got to have a LOT of target formats (think of range of devices netflix delivers to). ASICS sometimes struggle when faced with this needed flexibility.
- Xorlev 5y ago
  And their batch processing. I recall from a talk at Re:Invent forever ago, they had everything on AWS except: - Their CDN, which has to be deployed wherever there's demand. - An old Oracle setup with some ancient software that runs their DVD business. The latter may not be true anymore. In any case, it's smart to break from cloud providers when it makes sense for your problem. Video streaming is network-heavy, so pushing the actual streaming to where consumers is really smart.
- isclever 5y ago
  Go to fast.com, open webdev tools on the network tab. Run a speedtest, the hosts are the local caches (OpenConnect) that you would use to stream movies. Note: I don't know for sure, but its the most likely.
  - virtuallynathan 5y ago
    Yep, that is generally the case. The steering is slightly different, but close enough.
bsaul 5y ago
wait, aren't they running anything at all ??? I was 100% sure netflix had the most advanced use of aws ( inventing things like chaos monkey, etc)... What are they running on ?
- johneth 5y ago
  They run their backend on AWS (i.e. accounts, catalogue, watch history, etc.) They serve their video data using their own custom hardware outside of AWS.
- elzbardico 5y ago
  They run EVERYTHING but last mile file serving on aws. Encoding, storage, catalog, recommendations, accounts, etc. all of it in AWS.
  - dralley 5y ago
    They have resident cache boxes hooked directly into many ISPs. Those aren't on AWS.
    - lvh 5y ago
      That sounds like the last-mile the GP is talking about?
Xorlev 5y ago
Netflix runs ~everything but video streaming from AWS. That's more than just lip service.
teleforce 5y ago
I have a friend who used to work with iflix, an Asian version of Netflix [1]. According to him all of their contents are on AWS. [1]https://www.iflix.com/ https://www.iflix.com/
robjan 5y ago
Netflix use AWS for their commodity workloads and rolled their own CDN because delivering high quality streams without using too much bandwidth on the public internet is one of their differentiators.
Hikikomori 5y ago
Laughably wrong.

by betaby 5y ago

dangrossman 5y ago

Netflix is on AWS -- the website and app backends, not the video streams. * Netflix.com resolves to ec2-52-3-144-142.compute-1.amazonaws.com from my computer at this moment. Some more domains their apps reach out to include: * nflxvideo.net: ec2-46-137-171-215.eu-west-1.compute.amazonaws.com. * appboot.netflix.com: ec2-34-199-156-113.compute-1.amazonaws.com. * uiboot.netflix.com: ec2-52-203-145-205.compute-1.amazonaws.com.

betaby 5y ago
That exactly what I mean. AWS would represent single digit of the infrastructure and traffic.
- dangrossman 5y ago
  Maybe of their traffic, but not their infrastructure. Netflix rents 300,000 CPUs on AWS just for video encoding. Encoding one TV series produces over 1,000 files that sit on S3, ready for those PoPs to pull them in and cache them. They also do more data processing, analytics and ML than most people realize and that's all happening on AWS. For example, not only are each user's recommendations personalized, but all of the artwork is personalized too. If you like comedy, the cover art for Good Will Hunting may be a photo of Robin Williams. If you like romance, the cover art for the same movie may be a photo of Matt Damon and Minnie Driver going in for a kiss. They're also doing predictive analytics on what each person will want to watch tomorrow in particular. Every evening, all those non-AWS servers go ask AWS what video files they should download from S3 to proactively cache the next day of streaming.
  - betaby 5y ago
    ~300000 CPUs is about 2K racks worth of servers. I would think if you have such constant load it would be more economical to move that on-premise.
    - jeffbee 5y ago
      300k cpu cores is “only” about 1200 nodes these days, definitely not 2000 racks.
      - betaby 5y ago
        If CPU == core thread, then yes.
    - kristjansson 5y ago
      The load is emphatically not constant though? Encoding has to be done once, as fast as possible, and updated rarely. That warehouse (or just aisle) of racks would be sitting idle much of the time…
    - killingtime74 5y ago
      You’re thinking like an engineer. A business person would just ask for a discount (which I’m sure they do) and constantly renegotiate. Don’t forget the financing of on premise and data centre workers and how that looks on your balance sheet.
  - jeffbee 5y ago
    I wonder if there is any discussion of whether they could make it cheaper with Google’s compression ASICs, which I assume will be a cloud service at some point.
    - whoknowswhat11 5y ago
      Amazon offers something not has fancy (but maybe more flexible?) VT1 instances can support streams up to 4K UHD resolution at 60 frames per second (fps) and can transcode up to 64 simultaneous 1080p60 streams in real time. One thing is that a place like netflix has got to have a LOT of target formats (think of range of devices netflix delivers to). ASICS sometimes struggle when faced with this needed flexibility.
Xorlev 5y ago
And their batch processing. I recall from a talk at Re:Invent forever ago, they had everything on AWS except: - Their CDN, which has to be deployed wherever there's demand. - An old Oracle setup with some ancient software that runs their DVD business. The latter may not be true anymore. In any case, it's smart to break from cloud providers when it makes sense for your problem. Video streaming is network-heavy, so pushing the actual streaming to where consumers is really smart.
isclever 5y ago
Go to fast.com, open webdev tools on the network tab. Run a speedtest, the hosts are the local caches (OpenConnect) that you would use to stream movies. Note: I don't know for sure, but its the most likely.
- virtuallynathan 5y ago
  Yep, that is generally the case. The steering is slightly different, but close enough.

bsaul 5y ago

wait, aren't they running anything at all ??? I was 100% sure netflix had the most advanced use of aws ( inventing things like chaos monkey, etc)... What are they running on ?

johneth 5y ago
They run their backend on AWS (i.e. accounts, catalogue, watch history, etc.) They serve their video data using their own custom hardware outside of AWS.
elzbardico 5y ago
They run EVERYTHING but last mile file serving on aws. Encoding, storage, catalog, recommendations, accounts, etc. all of it in AWS.
- dralley 5y ago
  They have resident cache boxes hooked directly into many ISPs. Those aren't on AWS.
  - lvh 5y ago
    That sounds like the last-mile the GP is talking about?

Xorlev 5y ago

Netflix runs ~everything but video streaming from AWS. That's more than just lip service.

teleforce 5y ago

I have a friend who used to work with iflix, an Asian version of Netflix [1]. According to him all of their contents are on AWS. [1]https://www.iflix.com/ https://www.iflix.com/

robjan 5y ago

Netflix use AWS for their commodity workloads and rolled their own CDN because delivering high quality streams without using too much bandwidth on the public internet is one of their differentiators.

Hikikomori 5y ago

Laughably wrong.