4 ms·

It is completely incorrect to characterize these observations as "mispricing" - this is a quirk of automatically-determined prices across very different product

It is completely incorrect to characterize these observations as "mispricing" - this is a quirk of automatically-determined prices across very different products. If the author actually tried to use these instances in any significant volume they would understand the driver - capacity pools are nowhere near equal, and not as interchangeable for AWS as the article implies they would be for a user. Prices reflect demand munged with available capacity - uncommon instance types are uncommon precisely because they aren't used as much, so there aren't the same signals to drive the price up and down automatically. Instances with attached NVMe are available in much lower volumes than others, as are AMD instances. Obviously these pools cannot be used as a drop-in replacement for non-"d" instances or Intel families.

ericpauley 4y ago
Author here. The key here is that customers can leverage these pools in addition to their existing pools, improving capacity and price. AWS actually supports this out of the box (including substituting instances with drives) by specifying core and memory requirements directly instead of instance types.
- Moissanite 4y ago
  Totally agree with that; it is a pretty common approach. The only part I don't agree with is calling out the price differences as some kind of "gotcha" that AWS somehow missed, particularly given the speculative "lost revenue" data which have no basis in reality.
  - ericpauley 4y ago
    See the emphasis on transparent substitutes in the article. This analysis is limited strictly to sets of instances that are fully hardware compatible, meaning AWS could resell one instance as another. There are way more savings to be had as a customer by leveraging instances that aren't transparent substitutes.
    - Moissanite 4y ago
      I read it all, and don't agree with your interpretation of "transparent substitutes" in several of the cases.
      - ericpauley 4y ago
        Which instances are not transparent substitutes, in your opinion? Keep in mind the defintion here is that Amazon could substitute the image transparently, e.g., by ignoring the additional resources in hypervisor, not that the instances are by default indistinguishable. That being said, the substitute instances considered could be trivially accepted by any task running on the original instance, so long as it doesn't misbehave when given too many resources. In the case of vCPU, you can even hide extra vCPU cores, so a c6g.xlarge can be made effectively indistinguishable from a m6g.2xlarge by disabling the vCPUs at the hypervisor level.
- snake_doc 4y ago
  > Across all AWS availability zones instances are mispriced by roughly $400/hr at any given time. This means that, with just a single instance of each type, Amazon is missing out on $200/hr or roughy $1.7 million each year. This is over roughly 15,000 pools of instances. Given Amazon controls roughly 100 million IPs, we can guess that each instance pool probably has on the order of 1000 instances (more for smaller instances, less for larger instances). Given this, the average mispriced pool might have hundreds of instances, meaning hundreds of millions each year in missed revenue due to mispriced spot instances. Because amazon keeps their number of instances a secret, it’s difficult to make a precise estimate from the outside, but the missed revenue probably falls somewhere in this range. You are hypothesizing that the price differences produce "lost" revenues. An alternative hypothesis can be that the price differences produce similar or higher level of revenues for AWS through price segmentation, with Amazon recognizing the lack of adoption of certain spot instance bidding features and auction markets reacting appropriately. Unless you have the capacity and quantity demanded for each instance types, you can't prove your hypothesis. You are assuming scenario 3 (below) with no insights into price elasticity of the underlying customers. Example: Baseline: Instance types A and B are equivalent. A is priced at $3, with capacity of 1000, quantity demanded of 800. B is priced at $2, with capacity of 1000, quantity demanded of 200. Total quantity demanded = 1,000. Revenues from instance type A = $3 x 800 = $2,400 Revenues from instance type B = $2 x 200 = $ 400 Total revenues = $2,800 Scenario 1: All customers purchase instance B instead due to better price discovery. Revenues from instance type A = $3 x 0 = $0 Revenues from instance type B = $2 x 1,000 = $ $2,000 Total quantity demanded = 1,000. Total revenues = $2,000 Amazon loses $800 in revenues, there are no "lost" revenues" recovered. Scenario 2: Amazon changes instance type B price to $3. Total quantity demand decreases to 900 due to price elasticity of instance type B customers. Revenues from instance type A = $3 x 800 = $2,400 Revenues from instance type B = $3 x 100 = $300 Total revenues = $2,700 Amazon loses $100 in revenues, there are no "lost" revenues recovered. Scenario 3: Amazon changes instance type B price to $3. Total quantity demand remains at 1,000. Revenues from instance type A = $3 x 800 = $2,400 Revenues from instance type B = $3 x 200 = $600 Total revenues = $3,000 Amazon recovers $200 in "lost" revenues.
  - pclmulqdq 4y ago
    The mispricing is likely good for Amazon. It indicates that most people aren't doing this arbitrage, so Amazon can milk them for extra money.
pclmulqdq 4y ago
In financial markets, this quirk of automatically-determined prices across different products is frequently called "mispricing" when those products logically should have a relationship with each other. Straightforwardly: All hosts with space for a c6gd spot instance have space for a c6g instance. If Amazon is willing to host a c6gd instance in that slot for $X, they should be willing to also host a c6g instance there for $X. In financial markets, the way this gets handled is through arbitrage: someone will buy the equivalent of the c6gd instance, and sell the c6g part for the higher price (they may also sell the "d" part for even more money). This has the effect of "correcting" the price. The AWS spot market does not allow you to do arbitrage, and AWS doesn't appear to do the arbitrage for you. AWS probably likes this inefficiency in their market: some instance types are more popular than others, and some customers make assumptions that require them to use a very specific instance type (ie a c6gd would not work as a substitute for their c6g instance). However, the vast majority of users probably could work just fine if their c6g instance were a c6gd, and don't look for the arbitrage opportunity. That means Amazon gets paid extra.
- Moissanite 4y ago
  > If Amazon is willing to host a c6gd instance in that slot for $X, they should be willing to also host a c6g instance there for $X. The reality is that direct c6gd demand might be an order of magnitude lower than c6g direct demand - if AWS can get some more flexible people to adopt c6gd by offering a lower price, c6g capacity is slightly stabilized for on-demand usage by people who don't value the flexibility. Also note that c6g to c6gd has a non-zero switching cost - extra NVMe on the instance adds a new source of potential hardware failure, increasing the probability of termination very slightly. There might be other software-related costs depending on whether your application makes any ill-advised assumptions about attached storage during setup. So overall, I would just be happier to read this article if it was framed as "PSA: having more features in an ec2 instance is sometimes cheaper! Don't rule yourself out of extra savings by making overly-constrained fleet requests." The extra commentary about foregone revenue makes too many assumptions and detracts from the core point.
  - pclmulqdq 4y ago

ericpauley 4y ago

Author here. The key here is that customers can leverage these pools in addition to their existing pools, improving capacity and price. AWS actually supports this out of the box (including substituting instances with drives) by specifying core and memory requirements directly instead of instance types.

Moissanite 4y ago
Totally agree with that; it is a pretty common approach. The only part I don't agree with is calling out the price differences as some kind of "gotcha" that AWS somehow missed, particularly given the speculative "lost revenue" data which have no basis in reality.
- ericpauley 4y ago
  See the emphasis on transparent substitutes in the article. This analysis is limited strictly to sets of instances that are fully hardware compatible, meaning AWS could resell one instance as another. There are way more savings to be had as a customer by leveraging instances that aren't transparent substitutes.
  - Moissanite 4y ago
    I read it all, and don't agree with your interpretation of "transparent substitutes" in several of the cases.
    - ericpauley 4y ago
      Which instances are not transparent substitutes, in your opinion? Keep in mind the defintion here is that Amazon could substitute the image transparently, e.g., by ignoring the additional resources in hypervisor, not that the instances are by default indistinguishable. That being said, the substitute instances considered could be trivially accepted by any task running on the original instance, so long as it doesn't misbehave when given too many resources. In the case of vCPU, you can even hide extra vCPU cores, so a c6g.xlarge can be made effectively indistinguishable from a m6g.2xlarge by disabling the vCPUs at the hypervisor level.
snake_doc 4y ago
> Across all AWS availability zones instances are mispriced by roughly $400/hr at any given time. This means that, with just a single instance of each type, Amazon is missing out on $200/hr or roughy $1.7 million each year. This is over roughly 15,000 pools of instances. Given Amazon controls roughly 100 million IPs, we can guess that each instance pool probably has on the order of 1000 instances (more for smaller instances, less for larger instances). Given this, the average mispriced pool might have hundreds of instances, meaning hundreds of millions each year in missed revenue due to mispriced spot instances. Because amazon keeps their number of instances a secret, it’s difficult to make a precise estimate from the outside, but the missed revenue probably falls somewhere in this range. You are hypothesizing that the price differences produce "lost" revenues. An alternative hypothesis can be that the price differences produce similar or higher level of revenues for AWS through price segmentation, with Amazon recognizing the lack of adoption of certain spot instance bidding features and auction markets reacting appropriately. Unless you have the capacity and quantity demanded for each instance types, you can't prove your hypothesis. You are assuming scenario 3 (below) with no insights into price elasticity of the underlying customers. Example: Baseline: Instance types A and B are equivalent. A is priced at $3, with capacity of 1000, quantity demanded of 800. B is priced at $2, with capacity of 1000, quantity demanded of 200. Total quantity demanded = 1,000. Revenues from instance type A = $3 x 800 = $2,400 Revenues from instance type B = $2 x 200 = $ 400 Total revenues = $2,800 Scenario 1: All customers purchase instance B instead due to better price discovery. Revenues from instance type A = $3 x 0 = $0 Revenues from instance type B = $2 x 1,000 = $ $2,000 Total quantity demanded = 1,000. Total revenues = $2,000 Amazon loses $800 in revenues, there are no "lost" revenues" recovered. Scenario 2: Amazon changes instance type B price to $3. Total quantity demand decreases to 900 due to price elasticity of instance type B customers. Revenues from instance type A = $3 x 800 = $2,400 Revenues from instance type B = $3 x 100 = $300 Total revenues = $2,700 Amazon loses $100 in revenues, there are no "lost" revenues recovered. Scenario 3: Amazon changes instance type B price to $3. Total quantity demand remains at 1,000. Revenues from instance type A = $3 x 800 = $2,400 Revenues from instance type B = $3 x 200 = $600 Total revenues = $3,000 Amazon recovers $200 in "lost" revenues.
- pclmulqdq 4y ago
  The mispricing is likely good for Amazon. It indicates that most people aren't doing this arbitrage, so Amazon can milk them for extra money.

pclmulqdq 4y ago

In financial markets, this quirk of automatically-determined prices across different products is frequently called "mispricing" when those products logically should have a relationship with each other. Straightforwardly: All hosts with space for a c6gd spot instance have space for a c6g instance. If Amazon is willing to host a c6gd instance in that slot for $X, they should be willing to also host a c6g instance there for $X. In financial markets, the way this gets handled is through arbitrage: someone will buy the equivalent of the c6gd instance, and sell the c6g part for the higher price (they may also sell the "d" part for even more money). This has the effect of "correcting" the price. The AWS spot market does not allow you to do arbitrage, and AWS doesn't appear to do the arbitrage for you. AWS probably likes this inefficiency in their market: some instance types are more popular than others, and some customers make assumptions that require them to use a very specific instance type (ie a c6gd would not work as a substitute for their c6g instance). However, the vast majority of users probably could work just fine if their c6g instance were a c6gd, and don't look for the arbitrage opportunity. That means Amazon gets paid extra.

Moissanite 4y ago
> If Amazon is willing to host a c6gd instance in that slot for $X, they should be willing to also host a c6g instance there for $X. The reality is that direct c6gd demand might be an order of magnitude lower than c6g direct demand - if AWS can get some more flexible people to adopt c6gd by offering a lower price, c6g capacity is slightly stabilized for on-demand usage by people who don't value the flexibility. Also note that c6g to c6gd has a non-zero switching cost - extra NVMe on the instance adds a new source of potential hardware failure, increasing the probability of termination very slightly. There might be other software-related costs depending on whether your application makes any ill-advised assumptions about attached storage during setup. So overall, I would just be happier to read this article if it was framed as "PSA: having more features in an ec2 instance is sometimes cheaper! Don't rule yourself out of extra savings by making overly-constrained fleet requests." The extra commentary about foregone revenue makes too many assumptions and detracts from the core point.
- pclmulqdq 4y ago