4 ms·

Be it VC money, or just general apathy, it seems cloud costs are rarely a priority. My biggest gripe is when see that justification that goes like, "developer's

Be it VC money, or just general apathy, it seems cloud costs are rarely a priority. My biggest gripe is when see that justification that goes like, "developer's time is $100/hr and better spent elsewhere." Okay, I agree, doing something like self-hosting everything to reduce costs would be a terrible idea, but there's a fine balance. In 2019, Lyft spent $300M on AWS [1]. Sure, it's a big service, but are you seriously telling my that it's necessary to spend $300M in a single year? I think the meme that developer's time spent on cost control is useless needs to die. Engineers should be encouraged to control costs so that they can take better decisions early on. They should be nudged to pick more cost-efficient choices. This won't only reduce costs, the chances are companies will also end up having dramatically simpler architecture. [1]: https://www.cnbc.com/2019/03/01/lyft-plans-to-spend-300-million-on-aws-through-2021.html https://www.cnbc.com/2019/03/01/lyft-plans-to-spend-300-mill...

IanCal 4y ago
It's absolutely a balance, but I've seen it the other way too. Devs spending weeks to make something more efficient when compute time costs cents. That has a minimum cost of their hourly rate. The maximum cost is the cost of not doing what's most valuable to the company. > In 2019, Lyft spent $300M on AWS [1]. Sure, it's a big service, but are you seriously telling my that it's necessary to spend $300M in a single year? They didn't spend that in a single year. It also depends what they were doing, I was recently very shocked by the data costs of a major hardware store in the US but after breaking it down it made more sense. It all depends what they get for that $300M over 3 years.
boiler_up800 4y ago
> Engineers should be encouraged to control costs For better or worse a lot of folks are allergic to this. A framing that I’ve seen is to just understand the cost drivers from a technical perspective, for example large materialized views in BigQuery consume a lot of CPU. That’s technically interesting and it’s the reason they’re expensive.
paxys 4y ago
Literally from the first line of the article you linked: > Lyft has signed up to pay cloud market leader Amazon Web Services at least $80 million per year for the next three years, totaling at least $300 million. You can launch a massive project, hire dozens of people, buy server and colo space across the globe, rebuild their infra from scratch, and get costs down to – what – $60 million a year? Lyft is closing in on $4 billion in annual revenue. A few million in savings is meaningless, especially when balanced with the massive risks of such a project.
- disgruntledphd2 4y ago
  If Lyft consistently made money, I'd completely agree with you. That's not the world we live in, though.
  - _5uxp 4y ago
    I think that an uber are more like a public service now, lol. So whoever invests in them long term doing the public service and not expecting real good returns.
jayzalowitz 4y ago
I've managed down a cloud budget in that range. Trust me you can get there surprisingly easily if you want resiliency, security, and uptime on thousands of services.
NohatCoder 4y ago
The weird thing is, the nature of their business does not imply any significant computing needs. As a rough estimate they have served on the order of 1 billion customers, so that suggest they need 1 billion, multiplied by some factor of cruft, database rows. Depending on the cruft factor that should be somewhere between easily fitting on a single disk, up to a few of them. Add in redundancy and backup and we have still not filled a rack cabinet. Maybe all those computers are doing something that someone thought was important, but it is not serving the core business.
novok 4y ago
When you have big contracts like that, you get significantly discounted rates. And when you need large amounts of compute like that, you do need a whole bunch of staff to manage it and make it efficient and start reproducing / maintaining similar software created by the cloud providers. Let's say AWS makes a %50 profit margin over the raw cost of the hardware than you doing it yourself in DC, and each engineer is $400k/yr. You need to keep under 100 engineers to just start beating AWS, and your not in the business of running cloud infra. Also when you run the DCs yourself, usually it's much slower for other engineers using your DC infra than using AWS or similar. It gets complicated!
outworlder 4y ago
> it seems cloud costs are rarely a priority. Because they aren't a priority until they are. Execs will either turn a blind eye or simply won't understand that it's a problem, until a massive purchase order lands in accounting and gets escalated to the highest levels. Now it's suddenly a problem and it is up to the operations/devops/SRE/name of the month/SecFinDeVOps to figure out how to reduce costs. Which is completely backwards as they are probably not the group who requested the resources in the first place.

IanCal 4y ago

It's absolutely a balance, but I've seen it the other way too. Devs spending weeks to make something more efficient when compute time costs cents. That has a minimum cost of their hourly rate. The maximum cost is the cost of not doing what's most valuable to the company. > In 2019, Lyft spent $300M on AWS [1]. Sure, it's a big service, but are you seriously telling my that it's necessary to spend $300M in a single year? They didn't spend that in a single year. It also depends what they were doing, I was recently very shocked by the data costs of a major hardware store in the US but after breaking it down it made more sense. It all depends what they get for that $300M over 3 years.

boiler_up800 4y ago

> Engineers should be encouraged to control costs For better or worse a lot of folks are allergic to this. A framing that I’ve seen is to just understand the cost drivers from a technical perspective, for example large materialized views in BigQuery consume a lot of CPU. That’s technically interesting and it’s the reason they’re expensive.

paxys 4y ago

Literally from the first line of the article you linked: > Lyft has signed up to pay cloud market leader Amazon Web Services at least $80 million per year for the next three years, totaling at least $300 million. You can launch a massive project, hire dozens of people, buy server and colo space across the globe, rebuild their infra from scratch, and get costs down to – what – $60 million a year? Lyft is closing in on $4 billion in annual revenue. A few million in savings is meaningless, especially when balanced with the massive risks of such a project.

disgruntledphd2 4y ago
If Lyft consistently made money, I'd completely agree with you. That's not the world we live in, though.
- _5uxp 4y ago
  I think that an uber are more like a public service now, lol. So whoever invests in them long term doing the public service and not expecting real good returns.

jayzalowitz 4y ago

I've managed down a cloud budget in that range. Trust me you can get there surprisingly easily if you want resiliency, security, and uptime on thousands of services.

NohatCoder 4y ago

The weird thing is, the nature of their business does not imply any significant computing needs. As a rough estimate they have served on the order of 1 billion customers, so that suggest they need 1 billion, multiplied by some factor of cruft, database rows. Depending on the cruft factor that should be somewhere between easily fitting on a single disk, up to a few of them. Add in redundancy and backup and we have still not filled a rack cabinet. Maybe all those computers are doing something that someone thought was important, but it is not serving the core business.

novok 4y ago

When you have big contracts like that, you get significantly discounted rates. And when you need large amounts of compute like that, you do need a whole bunch of staff to manage it and make it efficient and start reproducing / maintaining similar software created by the cloud providers. Let's say AWS makes a %50 profit margin over the raw cost of the hardware than you doing it yourself in DC, and each engineer is $400k/yr. You need to keep under 100 engineers to just start beating AWS, and your not in the business of running cloud infra. Also when you run the DCs yourself, usually it's much slower for other engineers using your DC infra than using AWS or similar. It gets complicated!

outworlder 4y ago

> it seems cloud costs are rarely a priority. Because they aren't a priority until they are. Execs will either turn a blind eye or simply won't understand that it's a problem, until a massive purchase order lands in accounting and gets escalated to the highest levels. Now it's suddenly a problem and it is up to the operations/devops/SRE/name of the month/SecFinDeVOps to figure out how to reduce costs. Which is completely backwards as they are probably not the group who requested the resources in the first place.