4 ms·

Does this save much money compared to using Amazon's spot capacity? In the early days I remember the spot price closely tracked the "per operating hour" RI pri

Does this save much money compared to using Amazon's spot capacity? In the early days I remember the spot price closely tracked the "per operating hour" RI prices. It would be a very nice feature if EC2 could incorporate each user's pool of reserved instances into their view of the spot market, but maybe that would create too much complexity on Amazon's side.

samstave 11y ago
These are diff models. The spot pool is amazons unused capacity vs their allotment of RIs. RIs always get priority capacity over spot, and spot instances always have highest bidder priority. If there are a ton of spot capacity one hour, then Netflix scales up in that zone to launch jobs on their RIs - the RIs get launched and spot instances get whacked with the message "capacity over subscribed". The RI is a guarantee that when you choose to launch that instance, capacity exists. Period. (Unless your instance limit for that type/zone/region is hit) So, no Amazon could never do that. However, AFAIK, Netflix can do what it wants with the RIs... Think of Heroku: millions of sub-leases on their RIs to user apps.... But Amazon cannot meddle with the availability of the RIs that Heroku has purchased
- cperciva 11y ago
  I know RIs are different from spot instances. But it should be possible to have a spot instance API front-end which pulls from unused RIs first -- just like on-demand instances do, in fact.
  - samstave 11y ago
    But that is how it already works... The diff is that AWS spot instance pull from capacity that is unused, irrespective of RIs that you don't own... But when i come to launch my RI and capacity is full, your spot instance gets whacked. What netflix is doing is an internal meta-spot-market on THEIR idle RIs. This compute capacity will never be available to anyone but them unless they create a service to sublet the capacity via the same tools they do to their internal teams... From amazons persective that capacity is already paid for regardless of consumption... Forgive me if I am articulating this poorly... In the end, I just find the numbers disclosed in here to be awesome and I suspect they are just but a fraction of total capacity Netflix requires. Given that they disclose 12k idle trough RIs is very telling in efficiencies - but since I don't have the whole picture of their footprint, or instance demographic, I can't tell how much of this completes the picture of their landscape. I'll assume 12k represents perhaps 80% of the machines they use... They have control-plan, orchestration, DR, private peering nodes (can't recall the name) but like backblaze pods, and any number of varied stage test debut envs... So from a pure "stream or encode my shit" maybe 80% works, hopefully it's better - but could be worse... Netflix's original mandate to push all DC ops and what not out of the building is not only super awesome, but is one of the linchpins in AWS business model. There should be whole career paths in college designed around this model...
    - cperciva 11y ago
      You're not understanding me. If I have a Reserved Instance already paid for and tell Amazon that I want a Spot Instance, I'll be charged the Spot Instance price for it. What I'm saying is that launching a spot instance should use my idle RI capacity and bill me as if I had launched those instances as on-demand.
      - samstave 11y ago
        I understand you, but that is an incorrect assumption. Aws has resources. They are priced at X. If you buy on demand, you pay X If you buy an RI, you pay X-[Ri discount] If you launch an instance, and you have an RI that applies, you get the discount. Spot instances are a diff game. Spot and RI are mutually exclusive when costing. Spot is only affected by capacity when it comes to RIs. If you have RIs, you can't swap a running instance between spot or RI or OD pricing... Aws calls this "lifecycle" - spot or normal. RI discount "coupons" only apply to an instance in the same zone on normal lifecycle that can receive the available RI pricing discount. They do not overlap...
        cperciva 11y ago
        I know they don't overlap! I'm saying they should overlap.
        samstave 11y ago
        Had this discussion with aws multiple times. Never going to happen

samstave 11y ago

These are diff models. The spot pool is amazons unused capacity vs their allotment of RIs. RIs always get priority capacity over spot, and spot instances always have highest bidder priority. If there are a ton of spot capacity one hour, then Netflix scales up in that zone to launch jobs on their RIs - the RIs get launched and spot instances get whacked with the message "capacity over subscribed". The RI is a guarantee that when you choose to launch that instance, capacity exists. Period. (Unless your instance limit for that type/zone/region is hit) So, no Amazon could never do that. However, AFAIK, Netflix can do what it wants with the RIs... Think of Heroku: millions of sub-leases on their RIs to user apps.... But Amazon cannot meddle with the availability of the RIs that Heroku has purchased

cperciva 11y ago
I know RIs are different from spot instances. But it should be possible to have a spot instance API front-end which pulls from unused RIs first -- just like on-demand instances do, in fact.
- samstave 11y ago
  But that is how it already works... The diff is that AWS spot instance pull from capacity that is unused, irrespective of RIs that you don't own... But when i come to launch my RI and capacity is full, your spot instance gets whacked. What netflix is doing is an internal meta-spot-market on THEIR idle RIs. This compute capacity will never be available to anyone but them unless they create a service to sublet the capacity via the same tools they do to their internal teams... From amazons persective that capacity is already paid for regardless of consumption... Forgive me if I am articulating this poorly... In the end, I just find the numbers disclosed in here to be awesome and I suspect they are just but a fraction of total capacity Netflix requires. Given that they disclose 12k idle trough RIs is very telling in efficiencies - but since I don't have the whole picture of their footprint, or instance demographic, I can't tell how much of this completes the picture of their landscape. I'll assume 12k represents perhaps 80% of the machines they use... They have control-plan, orchestration, DR, private peering nodes (can't recall the name) but like backblaze pods, and any number of varied stage test debut envs... So from a pure "stream or encode my shit" maybe 80% works, hopefully it's better - but could be worse... Netflix's original mandate to push all DC ops and what not out of the building is not only super awesome, but is one of the linchpins in AWS business model. There should be whole career paths in college designed around this model...
  - cperciva 11y ago
    You're not understanding me. If I have a Reserved Instance already paid for and tell Amazon that I want a Spot Instance, I'll be charged the Spot Instance price for it. What I'm saying is that launching a spot instance should use my idle RI capacity and bill me as if I had launched those instances as on-demand.
    - samstave 11y ago
      I understand you, but that is an incorrect assumption. Aws has resources. They are priced at X. If you buy on demand, you pay X If you buy an RI, you pay X-[Ri discount] If you launch an instance, and you have an RI that applies, you get the discount. Spot instances are a diff game. Spot and RI are mutually exclusive when costing. Spot is only affected by capacity when it comes to RIs. If you have RIs, you can't swap a running instance between spot or RI or OD pricing... Aws calls this "lifecycle" - spot or normal. RI discount "coupons" only apply to an instance in the same zone on normal lifecycle that can receive the available RI pricing discount. They do not overlap...
      - cperciva 11y ago
        I know they don't overlap! I'm saying they should overlap.
        samstave 11y ago
        Had this discussion with aws multiple times. Never going to happen