5 ms·

> We reached out to some contacts at AWS to find out why the Aurora team built this. Did I/O Optimized do some clever engineering with sharding and storing data

> We reached out to some contacts at AWS to find out why the Aurora team built this. Did I/O Optimized do some clever engineering with sharding and storing data in S3? Were they just feeling generous? No surprises here. Come what may, Amazon has always strived to lower costs (the low costs, more customers, more volume fly-wheel?). This is but one example. AWS adopted cost-follow pricing for S3 (very different from value-based pricing) after apparently a lengthy debate: As they get more efficient, they want to pass down those savings to customers (as price reductions): S3 would be a tiered monthly subscription service based on average storage use, with a free tier. Customers would choose a monthly subscription rate based on how much data they typically needed to store. Simple ... The engineering team was ready to move on to the next question. Except that day we never got to the next question. We kept discussing this question. We really did not know how developers would use S3 when it launched. Would they store mostly large objects with low retrieval rates? Small objects with high retrieval rates? How often would updates happen versus reads? ... All those factors were unknown yet could meaningfully impact our costs ... was there a way to structure our pricing [to] ensure that it would be affordable to our customers and to Amazon? ... the discussion moved away from a tiered subscription pricing strategy and toward a cost-following strategy. "Cost following" means that your pricing model is driven primarily by your costs, which are then passed on to your customer. This is what construction companies use, because building your customer's gazebo out of redwood will cost you a lot more than building it out of pine. If we were to use a cost-following strategy, we'd be sacrificing the simplicity of subscription pricing, but both our customers and Amazon would benefit. With cost following, whatever the developer did with S3, they would use it in a way that would meet their requirements, and they would strive to minimise their cost and, therefore, our cost too. There would be no gaming of the system, and we wouldn't have to estimate how the mythical average customer would use S3 to set our prices. From: https://archive.is/lT5zT https://archive.is/lT5zT I wonder what explains AWS' high egress costs, though.

Nextgrid 3y ago
> I wonder what explains AWS' high egress costs, though. Vendor lock-in. It prevents people from otherwise picking the best provider for the task at hand - for example using some managed AWS services, but keeping the bulk of your compute on-prem or at a (much cheaper) bare-metal host. It makes sense, but I wish there was an option to opt-out, as to allow high-bandwidth applications that are fully on AWS (at the moment AWS is a non-starter for many of those even if you have no intention of using AWS competitors like in the scenario above). Maybe they should just price end-user egress vs competitor egress differently (datacenter and business provider IPs are priced like now, but consumer-grade provider IPs are much cheaper/free)? That would discourage provider-hopping, while making AWS a viable provider even for high-bandwidth applications such as serving or proxying media.
- coredog64 3y ago
  Unpopular opinion: There are a lot of free networking products, and it probably makes sense to hide all that in the margins of a few products (egress, NAT gateways) rather than penny pinch every API.
  - Dylan16807 3y ago
    Even one cent per gigabyte would pay for so much of that stuff. If subsidizing the networking products would require a 10x or higher markup on egress, then that is not the correct way to organize prices.
- Szpadel 3y ago
  Explanation makes sense, except cross availability zone traffic is also expensive and by this logic it should not be.
  - twoodfin 3y ago
    It’s presumably expensive because there’s a lot less inter-AZ bandwidth than there is intra-AZ bandwidth for the obvious reasons.
    - Szpadel 3y ago
      But if that's the case, this could be the reason of egress cost. I mean egress cannot be cheaper for aws than cross az. So if we assume that this is just expensive then we should assume the same for egress
  - hughesjj 3y ago
    IIRC some cross az traffic uses vendors' fiber (albeit definitely encrypted) but I could be completely wrong.
- shrubble 3y ago
  DirectConnect pricing might be cheaper, if a company is really trying to get stuff out, but I haven't run the numbers to determine if the connection charge plus DTO (data transfer out) would save money over regular S3 egress charges. see https://aws.amazon.com/directconnect/pricing/ https://aws.amazon.com/directconnect/pricing/
pahkah 3y ago
> No surprises here. Come what may, Amazon has always strived to lower costs Maybe it's just a goal to reduce costs, but it seems likely that this is a response to Google's introduction of AlloyDB, a Postgres-compatible database competing with Aurora that is advertised as having "no...opaque I/O charges". I doubt Amazon was feeling generous.
victor106 3y ago
> Come what may, Amazon has always strived to lower costs Agreed. Azure on the other hand is notoriously expensive and do everything possible to raise prices.
bilalq 3y ago
More than anything else, I'm bothered by Cloudwatch's pricing. $3/dashboard/month when it's essentially just a JSON config? $0.30/metric/month, with the risk of a junior engineer easily being able to add one line of code that 1000X or more multiplies your AWS bill? And then tie in how some AWS services only a binary choice between logging ridiculous volumes of data or nothing at all (looking at you, AppSync) to make the log ingestion fees high too.
BiteCode_dev 3y ago
> No surprises here. Come what may, Amazon has always strived to lower costs I have a hard time reading "amazon" and "lower costs" in the same sentence. The cloud is consistently an order of magnitude more expensive than self hosting, even when including people cost, unless you need every single bit of load balancing/high availability/normalization they provide. The vast majority of projects don't.

Nextgrid 3y ago

> I wonder what explains AWS' high egress costs, though. Vendor lock-in. It prevents people from otherwise picking the best provider for the task at hand - for example using some managed AWS services, but keeping the bulk of your compute on-prem or at a (much cheaper) bare-metal host. It makes sense, but I wish there was an option to opt-out, as to allow high-bandwidth applications that are fully on AWS (at the moment AWS is a non-starter for many of those even if you have no intention of using AWS competitors like in the scenario above). Maybe they should just price end-user egress vs competitor egress differently (datacenter and business provider IPs are priced like now, but consumer-grade provider IPs are much cheaper/free)? That would discourage provider-hopping, while making AWS a viable provider even for high-bandwidth applications such as serving or proxying media.

coredog64 3y ago
Unpopular opinion: There are a lot of free networking products, and it probably makes sense to hide all that in the margins of a few products (egress, NAT gateways) rather than penny pinch every API.
- Dylan16807 3y ago
  Even one cent per gigabyte would pay for so much of that stuff. If subsidizing the networking products would require a 10x or higher markup on egress, then that is not the correct way to organize prices.
Szpadel 3y ago
Explanation makes sense, except cross availability zone traffic is also expensive and by this logic it should not be.
- twoodfin 3y ago
  It’s presumably expensive because there’s a lot less inter-AZ bandwidth than there is intra-AZ bandwidth for the obvious reasons.
  - Szpadel 3y ago
    But if that's the case, this could be the reason of egress cost. I mean egress cannot be cheaper for aws than cross az. So if we assume that this is just expensive then we should assume the same for egress
- hughesjj 3y ago
  IIRC some cross az traffic uses vendors' fiber (albeit definitely encrypted) but I could be completely wrong.
shrubble 3y ago
DirectConnect pricing might be cheaper, if a company is really trying to get stuff out, but I haven't run the numbers to determine if the connection charge plus DTO (data transfer out) would save money over regular S3 egress charges. see https://aws.amazon.com/directconnect/pricing/ https://aws.amazon.com/directconnect/pricing/

pahkah 3y ago

> No surprises here. Come what may, Amazon has always strived to lower costs Maybe it's just a goal to reduce costs, but it seems likely that this is a response to Google's introduction of AlloyDB, a Postgres-compatible database competing with Aurora that is advertised as having "no...opaque I/O charges". I doubt Amazon was feeling generous.

victor106 3y ago

> Come what may, Amazon has always strived to lower costs Agreed. Azure on the other hand is notoriously expensive and do everything possible to raise prices.

bilalq 3y ago

More than anything else, I'm bothered by Cloudwatch's pricing. $3/dashboard/month when it's essentially just a JSON config? $0.30/metric/month, with the risk of a junior engineer easily being able to add one line of code that 1000X or more multiplies your AWS bill? And then tie in how some AWS services only a binary choice between logging ridiculous volumes of data or nothing at all (looking at you, AppSync) to make the log ingestion fees high too.

BiteCode_dev 3y ago

> No surprises here. Come what may, Amazon has always strived to lower costs I have a hard time reading "amazon" and "lower costs" in the same sentence. The cloud is consistently an order of magnitude more expensive than self hosting, even when including people cost, unless you need every single bit of load balancing/high availability/normalization they provide. The vast majority of projects don't.