5 ms·

I'm not very familiar with AWS or The Cloud, but I'm having trouble understanding what you said about Amazon leaving money on the table by not directing custome

I'm not very familiar with AWS or The Cloud, but I'm having trouble understanding what you said about Amazon leaving money on the table by not directing customers toward specific-purpose services as opposed to EC2? Wouldn't (for AWS to make a profit anyway) whatever managed service have to be cheaper than some equivalent service running on an EC2 VM? I get the concerns re: pricing and predictability, but it still seems like more $$$ for AWS.

mumblemumble 6y ago
It's not just about a straight cost comparison. It's about how organizational decision-making works. The people shopping for products are not spending their own money, but they are spending their own time and energy. The people approving budgets are not considering all possible alternatives, they are only considering the ones that have been presented to them by the people doing the shopping. If the shoppers decide that an option will cost them too much in time and irritation, then it may be that the people holding the purse-strings are never even made aware that it exists. Even if it is the cheapest option.
- gregmac 6y ago
  This is a really good summary of the situation, and I'd add a bit about risk: It's relativity easy to estimate EC2 costs for running some random service, because it's literally just a per-hour fee times number of instances. If you're wrong, the bigger instance size or more instances isn't that much more expensive. For almost every other service, you have to estimate some other much more detailed metric: number of http requests, bytes per message, etc. When you haven't yet written the software, those details can be very fuzzy, making the whole estimation process extremely risky - it could be cheaper than EC2, it could be 10x more, and we won't really know until we've spent at least a coulple months writing code. And let's hope we don't pivot or have our customers do anything in a way we're not expecting..
nucleardog 6y ago
No, usually the managed services are a premium over the bare hardware. When you use RDS for example, you’re paying for the compute resources but also paying for the extra functionality they provide and their management and maintenance they’re doing for you. You can run your own Postgres database, or you can pay the premium for Aurora on RDS and get a multi-region setup with point in time restore and one-click scaling and automatically managed storage size and automatic patching and monitoring integrated into AWS monitoring tools and... They’re leaving money in the table because instead of using “Amazon Managed $X” potentially at a premium or paying a similar amount but in a way where AWS can provide the service with fewer compute resources than you or I would need because of their scale and thus more profitably, people look and see they’ll be paying $0.10/1000 requests and $0.05/1gb of data processed in a query and $0.10/gb for bandwidth for any transfer that leaves the region and... people just give up and go “I have no idea what that will cost or whether I can afford it, but this EC2 instance is $150/mo, I can afford that.”
bane 6y ago
Yeah good question. Sibling comments to this one explain it well, but basically AWS managed services come at a premium price over some equivalent running in just EC2. (Some services in fact do charge you for EC2 time + the service + storage etc.) "Managed" usually means "pay us more in exchange for less work on your part". This is usually pitched as a way to reduce admin/infrastructure/devops type staff and the overhead that goes along with having those people on the payroll.
glogla 6y ago
For example: Managed Airflow Scheduler on AWS with "large" size costs $0.99/hour, or $8,672/year per instance. That's ~ $17,500 considering Airflow for at least non-prod and prod instances. Building it on your own on same size EC2 instance would cost $3,363/year for the EC2. Times two for two environments, let's say $6,700. $4,000 if you prepay the instance. That looks way cheaper, but then you have to do the engineering and the operational support yourself. If you consider just the engineering and assume engineer costs $50/hour and estimate this to initial three weeks of work and then 2.5 days / month for support (upgrades, tuning, ...) that's extra $4,000 upfront and $1,000/month. So on AWS you're at $17,500/year and on-prem you're at best $20,000 first year and $16,000 next years. So the AWS only comes a bit more expensive - but the math is tricky on several parts: - maybe you need 4 environments deployed instead of 2, which is more for AWS but not much more for engineering? - maybe there's less sustaining cost because you're ok with upgrading Airflow only once a quarter? - you probably already pay the engineers, so it's not an extra money cost, it's extra cost of them not working on other stuff - different boxes and budgets - maybe you're in part of a world where good devops engineer doesn't cost $50/hour but $15 hour - I'm ignoring cost of operational support, which can be a lot for on-prem if you need 24/7 - maybe you need 12+ Airflow instances thanks to your fragmented / federated IT and can share the engineering cost - etc, etc. So I think what OP was saying is that if AWS priced Managed Airflow at $0.5 per hour, it would be no brainer to use instead of build your own. The way it is, some customers will surely for their own Airflow instead, because the math favors it. Does that make sense?
- bird_monster 6y ago
  > That looks way cheaper, but then you have to do the engineering and the operational support yourself. In my experience, this is the piece that engineers rarely realize and that is actually one of the biggest factors in evaluating cloud providers vs. home-rolled. Especially if you're a small company, engineering time (really any employee time) is _insanely valuable_. Valuable such that even if Airflow is cash-expensive, if using it allows your engineers to focus on building whatever makes _your business successful_, it is usually a much better idea to just use Airflow and keep moving. Clients usually will not care about whether you implemented your own version of an AWS product (unless that's your company's specific business). Clients will care about the features you ship. If you spent a ton of time re-inventing Airflow to save some cost, but then go bankrupt before you ever ship, rolling your own Airflow implementation clearly didn't save you anything.

mumblemumble 6y ago

It's not just about a straight cost comparison. It's about how organizational decision-making works. The people shopping for products are not spending their own money, but they are spending their own time and energy. The people approving budgets are not considering all possible alternatives, they are only considering the ones that have been presented to them by the people doing the shopping. If the shoppers decide that an option will cost them too much in time and irritation, then it may be that the people holding the purse-strings are never even made aware that it exists. Even if it is the cheapest option.

gregmac 6y ago
This is a really good summary of the situation, and I'd add a bit about risk: It's relativity easy to estimate EC2 costs for running some random service, because it's literally just a per-hour fee times number of instances. If you're wrong, the bigger instance size or more instances isn't that much more expensive. For almost every other service, you have to estimate some other much more detailed metric: number of http requests, bytes per message, etc. When you haven't yet written the software, those details can be very fuzzy, making the whole estimation process extremely risky - it could be cheaper than EC2, it could be 10x more, and we won't really know until we've spent at least a coulple months writing code. And let's hope we don't pivot or have our customers do anything in a way we're not expecting..

nucleardog 6y ago

No, usually the managed services are a premium over the bare hardware. When you use RDS for example, you’re paying for the compute resources but also paying for the extra functionality they provide and their management and maintenance they’re doing for you. You can run your own Postgres database, or you can pay the premium for Aurora on RDS and get a multi-region setup with point in time restore and one-click scaling and automatically managed storage size and automatic patching and monitoring integrated into AWS monitoring tools and... They’re leaving money in the table because instead of using “Amazon Managed $X” potentially at a premium or paying a similar amount but in a way where AWS can provide the service with fewer compute resources than you or I would need because of their scale and thus more profitably, people look and see they’ll be paying $0.10/1000 requests and $0.05/1gb of data processed in a query and $0.10/gb for bandwidth for any transfer that leaves the region and... people just give up and go “I have no idea what that will cost or whether I can afford it, but this EC2 instance is $150/mo, I can afford that.”

bane 6y ago

Yeah good question. Sibling comments to this one explain it well, but basically AWS managed services come at a premium price over some equivalent running in just EC2. (Some services in fact do charge you for EC2 time + the service + storage etc.) "Managed" usually means "pay us more in exchange for less work on your part". This is usually pitched as a way to reduce admin/infrastructure/devops type staff and the overhead that goes along with having those people on the payroll.

glogla 6y ago

For example: Managed Airflow Scheduler on AWS with "large" size costs $0.99/hour, or $8,672/year per instance. That's ~ $17,500 considering Airflow for at least non-prod and prod instances. Building it on your own on same size EC2 instance would cost $3,363/year for the EC2. Times two for two environments, let's say $6,700. $4,000 if you prepay the instance. That looks way cheaper, but then you have to do the engineering and the operational support yourself. If you consider just the engineering and assume engineer costs $50/hour and estimate this to initial three weeks of work and then 2.5 days / month for support (upgrades, tuning, ...) that's extra $4,000 upfront and $1,000/month. So on AWS you're at $17,500/year and on-prem you're at best $20,000 first year and $16,000 next years. So the AWS only comes a bit more expensive - but the math is tricky on several parts: - maybe you need 4 environments deployed instead of 2, which is more for AWS but not much more for engineering? - maybe there's less sustaining cost because you're ok with upgrading Airflow only once a quarter? - you probably already pay the engineers, so it's not an extra money cost, it's extra cost of them not working on other stuff - different boxes and budgets - maybe you're in part of a world where good devops engineer doesn't cost $50/hour but $15 hour - I'm ignoring cost of operational support, which can be a lot for on-prem if you need 24/7 - maybe you need 12+ Airflow instances thanks to your fragmented / federated IT and can share the engineering cost - etc, etc. So I think what OP was saying is that if AWS priced Managed Airflow at $0.5 per hour, it would be no brainer to use instead of build your own. The way it is, some customers will surely for their own Airflow instead, because the math favors it. Does that make sense?

bird_monster 6y ago
> That looks way cheaper, but then you have to do the engineering and the operational support yourself. In my experience, this is the piece that engineers rarely realize and that is actually one of the biggest factors in evaluating cloud providers vs. home-rolled. Especially if you're a small company, engineering time (really any employee time) is _insanely valuable_. Valuable such that even if Airflow is cash-expensive, if using it allows your engineers to focus on building whatever makes _your business successful_, it is usually a much better idea to just use Airflow and keep moving. Clients usually will not care about whether you implemented your own version of an AWS product (unless that's your company's specific business). Clients will care about the features you ship. If you spent a ton of time re-inventing Airflow to save some cost, but then go bankrupt before you ever ship, rolling your own Airflow implementation clearly didn't save you anything.