6 ms·

Ahrefs Saved US$400M in 3 Years by Not Going to the Cloud

118 points by dgudkov 4y ago

djfobbz 4y ago
Way to go, I approve! Been running bare metal for my SaaS since 2007.
bhahn 4y ago
I've only worked at companies running in the cloud. Is there a console (open-source or otherwise) and other tooling similar to AWS' for managing your resources in your own data center?
- ocdtrekkie 4y ago
  Plenty of them, yes. I probably would decline to move to a job with a cloud-based setup. It's short-term thinking at it's finest.
- mdaniel 4y ago
  The OpenStack console is pretty good, imho If you have an 8GB vm you can see it in action via https://docs.openstack.org/devstack/zed/guides/single-vm.html https://docs.openstack.org/devstack/zed/guides/single-vm.htm... (takes about 45 minutes) or if you want the "full" experience fire up an education cluster from https://openmetal.io/use-cases/on-demand-openstack-cloud/ https://openmetal.io/use-cases/on-demand-openstack-cloud/ (it takes about 4 hours or so but it's also a "real" setup unlike the vm which for sure allows running instances via qemu but isn't going to allow exercising the complete suite as easily)
Nextgrid 4y ago
Duplicate: https://news.ycombinator.com/item?id=35094407 https://news.ycombinator.com/item?id=35094407
karmicthreat 4y ago
I can definitely see advantages for very specific use cases in favor of bare metal colo. But AWS does provide a good amount of value in flexibility and reliability. If you have a whole lot of bandwidth or storage needs you should probably look at a hybrid system. But you will probably pay for it with increased engineering and maintenance costs.
Xeoncross 4y ago
Another interesting thing is that the farther you go from the CPU cache, the more copies of data and the more extra processing you have to do. Think of how many copies of the data you need (and how much processing it takes) to use: - a L3 CPU cache. - a 16GB RAM cache. - an embedded C/Rust/Go SQLite database. - a distributed, fault tolerant Cassandra/elastic cache cluster. You can literally solve the same typeahead/autocomplete/user lookup/session management/etc.. features using the last three of these - but the CPU cycle, memory usage and latency costs (plus complexity) explodes. I'm always impressed by people that handle 1 million connections on a single box while I'm watching teams trying to handle a thousand connections with a whole fleet of infrastructure and caching layers. Then again, part of the issue is the wrong language and wrong designs patterns. Highly optimized Rust and Go someone obsessed on all weekend is going to shred your team's unoptimized vanilla Ruby or Java install you didn't even have time to write tests for and don't honestly care about because you're too worried about being laid off.
- hot_gril 4y ago
  For use cases with simple business logic and high throughput, yeah the optimized Rust, maybe on your own hardware, makes sense. Tons of cloud customers running NodeJS or Java stuff don't fit this. Maybe they have a few optimized pieces with high throughput and majority unoptimized where resource usage is low either way. Or they might be ok spending 10X compute resources even on something high-throughput because landing new features/products matters way more than optimizing a small part of their budget. Like, there are businesses renting big 8-socket servers with TBs of RAM to support some monstrous corporate DB that can only run on a single machine.
  - Xeoncross 4y ago
    Yeah its kind of sad that we still have to use node and java with better options out there. Don't get me wrong, Typescript, Deno, JVM, etc.. are all doing their best to right those ships, but it's would still be better to just scrap that stuff for a better foundation. That is what everyone knows, so that is what's taught, so that is what everyone knows, so that is what's taught. I'd gladly onboard projects to Erlang, Rust, Zig, Go, or anything modern. Would save our planet so much wasted energy.
mdaniel 4y ago
Damn, how many times is this article going to be submitted? https://news.ycombinator.com/from?site=ahrefs.com https://news.ycombinator.com/from?site=ahrefs.com https://news.ycombinator.com/item?id=35108429 https://news.ycombinator.com/item?id=35108429 https://news.ycombinator.com/item?id=35100866 https://news.ycombinator.com/item?id=35100866 https://news.ycombinator.com/item?id=35095604 https://news.ycombinator.com/item?id=35095604 https://news.ycombinator.com/item?id=35094407 https://news.ycombinator.com/item?id=35094407
- Atlas22 4y ago
  Just gotta wait for the comment claiming admin/maintenance costs makes non-cloud more expensive than cloud then we can have yet another full repeat discussion.
- elijaht 4y ago
  Yea, I thought that was weird. Was wondering why there weren’t many comments when it had been on the front page for so long
ramesh31 4y ago
This fits Ahrefs because computation and data is their secret sauce, not software. For the average SaaS this isn't the case.
- twelve40 4y ago
  great point, i didn't even realize how much compute power they need (and how much money they make)
- PaywallBuster 4y ago
  right! Most services are probably well served by AWS/Heroku and others and simply saving the engineering time on developing the product, or optimizing when it becomes a problem AWS can be 2x as expensive as DO for a simple VM, which you can easily work around by acquiring Reserve Instances or using "spot" instances but, the "AWS premium" becomes a problem: - 10x when dealing with bandwidth which you cannot do much except a very big enterprise package and getting some discount that even then it will be multiple times higher than the alternatives The way I see it, AWS is great but you need to consider your 1) key infra requirements 2) consider cost effectiveness of different solutions/services 3) consider alternatives i.e. - Cloudfront is fine, but if you doing 100 tb/month probably Cloudflare would be best! - Lambda is fine but if you doing 100M requests/month EC2/ECS would probably work better (or more cost effective) Consider 1) what you need to get started and ship 2) what you need to scale up 3) what's the most cost effective solution long term The requirements will change over time, be ready to adapt!
marcopicentini 4y ago
Me too! Running a B2B SaaS using Digital Ocean and Cloud66. I am free to switch vendor every 12 months without pain.
- intev 4y ago
  How has the experience been with Cloud66? Why them over using something like caprover?
  - marcopicentini 4y ago
    I don’t know about caprover. I like Cloud66 because: - Pricing is fair. I spend 100$/month for my stack - They back up db and host it to a different cloud provider (AWS) - It’s easy to import Db - UI is clean and easy to use - It’s easy to switch vendor. Just clone the app and it will install all in a new server. I have switched from Digital Ocean to Azure in a couple of hours with 30 min downtime.
codingbot3000 4y ago
It looks clear to me that given the hardware they need, and the scale they operate at, buying their own metal makes sense. Also them sticking to similar servers is a good idea imho. What I am missing though in the monthly cost is the personnel cost. I would assume that a decently qualified system and network administrator in Singapore, capable of operating and servicing this setup, will cost upward of $10,000 per month. And only having one is probably not enough, due to SPOF and other considerations. To be honest, even for the cloud you probably need someone to administer the setup, but chances are high that there is much less to do. Paying for, and finding qualified staff is from my experience the main driver for small and mid-size companies to go to the cloud. And for many this actually makes sense financially. Even though many create unnecessary complex setups or get trapped in vendor lock-ins.
- pphysch 4y ago
  A small-medium DC can be a lot of work to install, specifically in terms of backup power/cooling, but not much to maintain on a daily basis. Occasionally you need to replace expired components, and install new systems, but that's about it. There will be weeklong periods where no action is required. Finding qualified staff is definitely hard. Especially nowadays, there are a lot of DevOps astronauts that have never swapped a bad DIMM.
  - indymike 4y ago
    I've had great luck just leasing what I need from a company who already has the DC and is willing to manage the hardware as part of the lease.
  - kkielhofner 4y ago
    Building a DC is yet another level. Co-location facilities drop redundant power in your cabinet and manage cooling. All you need to do is make sure you plug your redundant power supplies into the right PDU. Also, any decent co-lo provider will rack and provide remote access to HW. For geo-redundancy (unless you’re at massive scale beyond an HN comment) a couple of cabinets per facility is an incredible amount of compute. For hardware issues get support from your hardware vendor. When equipment fails (drive in a RAID array, power supply, entire chassis - whatever) they’ll dispatch a tech directly to the facility and take care of it. Depending on what you’re willing to pay it’s either same day or next (and warranty support is usually the first thing vendors will discount if you negotiate). I’ve “deployed” to multiple data centers that ran for years without ever stepping foot in them. That said I’m with you - I’m a little concerned Big Cloud is here to stay after 20 years of incredibly effective marketing and mindshare dominance. We now have an entire generation who’s sum total understanding of hardware is a (reversible!) USB-C connector.
aldarisbm 4y ago
> So we decided to consolidate all the costs spent for this data center, divide them by the number of servers, and compare them to the cost of a similar installation in the Amazon Web Services (AWS) cloud (we use this as the leading IaaS example). so compare them to buying AWS VMs you don’t need? one of the main attractions of the cloud is the fact that you don’t have to pay for what you don’t use. Elasticity. Every time I see one of these is: I bought 100 servers (which i use 40 of), I would’ve saved money if I was renting 100 servers of AWS. Yeah … no shit. If you were renting only 40, and had the ability to go down as usage decreases, there would be savings, mind you there are such things as, RIs, savings plans, etc, etc. meh.
margorczynski 4y ago
Once your operation is stable and you know (almost) exactly what you need then sticking to the Cloud makes less and less sense. The premium they charge is enormous. Not sure why e.g. Netflix doesn't exit AWS considering their needs flatlined. Or maybe they have some special agreement with much lower prices than market.
- yegle 4y ago
  A 24/7 oncall rotation team needs 5 people to keep it <1 rotation per month. This alone costs 1M/y. I think many companies simply get away by having unhealthy oncall rotations. E.g. ask people to oncall outside business hours constantly.
  - cbrozefsky 4y ago
    Or their stuff doesn't break such that being oncall is a formality.
    - pgwhalen 4y ago
      This is a great way to put it, and I don’t see this reality acknowledged on HN very often.
- marcinzm 4y ago
  What the cloud gives you is the removal of the inter-department conflict between IT and Product Engineering. This is especially bad since IT is a cost-center rather than a revenue generator. Having a single provider who in turn has a single client is a horrible dynamic. It can work for the short term but long term you get really odd dynamics. Sure the cloud is expensive but so is the aggregate opportunity cost of waiting 3 months and spending 20 hours in meetings for a single dev server.
- trollied 4y ago
  This quite often comes down to CapEx vs OpEx.
andrewstuart 4y ago
There a fantasy that says you don’t need experts to run your cloud infrastructure. The same fantasy says that if you own your own hardware you’ll need lots of experts that are so expensive and so hard to find that it makes no sense to self host. Cloud is crazy expensive and delivers much slower computing power per dollar. Consider if you need kubernetes plus cloud scaling, or you just need one modern ripping fast 16 core machine with 128GB RAM.
- AtlasBarfed 4y ago
  It's bottom lines like the article's where I wonder if all the companies who had HORRID penny-wise pound-foolish (6 months of paperwork to get a server, and only then would they order the servers) and went cloud simply did what these guys appear to be doing: buy enough to stay ahead of demand. If you have unused servers, who cares, at least you don't have allocated by unused AWS servers that pay for themselves in hardware costs every... four months? I don't remember how long it is.
  - lamontcg 4y ago
    I sold all my AMZN stock by 2006 when I quit thinking that everyone would figure out the secret to doing your own hardware cheap. I then went to try to convert other businesses over to doing it cheaply, but as an engineer I just wound up getting into fights with my managers who loved spending money. It is all about Empire building and managers wanted those expensive VMWare and EMC vendor relationships to manage in order to fluff up their resumes. They want more people under their umbrella. If you start hiring Directors of Network Engineering you'll find that you start hiring lots of expensive CCIEs and spending a lot of money on Cisco. It is almost a problem of being too small, since you can't quite easily point out how having single SATA drives in all the servers will save millions over having hardware RAID cards, and having simple networking in your rack design will also save millions over what Cisco would like to push on you. So you overspend by $250k here and there and everywhere, and pretty quickly you're back up to cloud costs. At a smaller level it isn't quite as obvious how obscenely expensive to your capex these kinds of managers are, and they know hot to talk in ways that make Managment believe that they're buying all kinds of Enterprise-class resiliancy and leveragable vendor relationships and all that kind of nonsense. Hiring smart engineers instead is something that most Managers don't know how to do either, which is what you need in order to get a decent level of automation in place in order to scale the ability to manage it all. And don't get me started on high priced consultants that can be brought in to overdesign the most complicated solutions possible by never saying 'no' to any kind of requirement that IT management can dream up. I've seen VLAN diagrams that you wouldn't believe... But anyway, if you can solve that human problem, through hiring the right people and having a corporate culture of "Frugality" (as much as I hate it, that focus Amazon has is what let them do IT cheaply and then resell it). But it isn't a technical problem, or a financial problem, or even really a staffing problem, it is fundamentally a culture problem. And then the trick in my mind would be to do Frugality right without being as much of a dick about it as Amazon is.
not_me_ever 4y ago
How can you save >$150M/year on a service that should cost <$2M/year to run?

118 points by dgudkov 4y ago

Way to go, I approve! Been running bare metal for my SaaS since 2007.

I've only worked at companies running in the cloud. Is there a console (open-source or otherwise) and other tooling similar to AWS' for managing your resources in your own data center?

ocdtrekkie 4y ago
Plenty of them, yes. I probably would decline to move to a job with a cloud-based setup. It's short-term thinking at it's finest.
mdaniel 4y ago
The OpenStack console is pretty good, imho If you have an 8GB vm you can see it in action via https://docs.openstack.org/devstack/zed/guides/single-vm.html https://docs.openstack.org/devstack/zed/guides/single-vm.htm... (takes about 45 minutes) or if you want the "full" experience fire up an education cluster from https://openmetal.io/use-cases/on-demand-openstack-cloud/ https://openmetal.io/use-cases/on-demand-openstack-cloud/ (it takes about 4 hours or so but it's also a "real" setup unlike the vm which for sure allows running instances via qemu but isn't going to allow exercising the complete suite as easily)

Nextgrid 4y ago

Duplicate: https://news.ycombinator.com/item?id=35094407 https://news.ycombinator.com/item?id=35094407

karmicthreat 4y ago

I can definitely see advantages for very specific use cases in favor of bare metal colo. But AWS does provide a good amount of value in flexibility and reliability. If you have a whole lot of bandwidth or storage needs you should probably look at a hybrid system. But you will probably pay for it with increased engineering and maintenance costs.

Xeoncross 4y ago

Another interesting thing is that the farther you go from the CPU cache, the more copies of data and the more extra processing you have to do. Think of how many copies of the data you need (and how much processing it takes) to use: - a L3 CPU cache. - a 16GB RAM cache. - an embedded C/Rust/Go SQLite database. - a distributed, fault tolerant Cassandra/elastic cache cluster. You can literally solve the same typeahead/autocomplete/user lookup/session management/etc.. features using the last three of these - but the CPU cycle, memory usage and latency costs (plus complexity) explodes. I'm always impressed by people that handle 1 million connections on a single box while I'm watching teams trying to handle a thousand connections with a whole fleet of infrastructure and caching layers. Then again, part of the issue is the wrong language and wrong designs patterns. Highly optimized Rust and Go someone obsessed on all weekend is going to shred your team's unoptimized vanilla Ruby or Java install you didn't even have time to write tests for and don't honestly care about because you're too worried about being laid off.

hot_gril 4y ago
For use cases with simple business logic and high throughput, yeah the optimized Rust, maybe on your own hardware, makes sense. Tons of cloud customers running NodeJS or Java stuff don't fit this. Maybe they have a few optimized pieces with high throughput and majority unoptimized where resource usage is low either way. Or they might be ok spending 10X compute resources even on something high-throughput because landing new features/products matters way more than optimizing a small part of their budget. Like, there are businesses renting big 8-socket servers with TBs of RAM to support some monstrous corporate DB that can only run on a single machine.
- Xeoncross 4y ago
  Yeah its kind of sad that we still have to use node and java with better options out there. Don't get me wrong, Typescript, Deno, JVM, etc.. are all doing their best to right those ships, but it's would still be better to just scrap that stuff for a better foundation. That is what everyone knows, so that is what's taught, so that is what everyone knows, so that is what's taught. I'd gladly onboard projects to Erlang, Rust, Zig, Go, or anything modern. Would save our planet so much wasted energy.

mdaniel 4y ago

Damn, how many times is this article going to be submitted? https://news.ycombinator.com/from?site=ahrefs.com https://news.ycombinator.com/from?site=ahrefs.com https://news.ycombinator.com/item?id=35108429 https://news.ycombinator.com/item?id=35108429 https://news.ycombinator.com/item?id=35100866 https://news.ycombinator.com/item?id=35100866 https://news.ycombinator.com/item?id=35095604 https://news.ycombinator.com/item?id=35095604 https://news.ycombinator.com/item?id=35094407 https://news.ycombinator.com/item?id=35094407

Atlas22 4y ago
Just gotta wait for the comment claiming admin/maintenance costs makes non-cloud more expensive than cloud then we can have yet another full repeat discussion.
elijaht 4y ago
Yea, I thought that was weird. Was wondering why there weren’t many comments when it had been on the front page for so long

ramesh31 4y ago

This fits Ahrefs because computation and data is their secret sauce, not software. For the average SaaS this isn't the case.

twelve40 4y ago
great point, i didn't even realize how much compute power they need (and how much money they make)
PaywallBuster 4y ago
right! Most services are probably well served by AWS/Heroku and others and simply saving the engineering time on developing the product, or optimizing when it becomes a problem AWS can be 2x as expensive as DO for a simple VM, which you can easily work around by acquiring Reserve Instances or using "spot" instances but, the "AWS premium" becomes a problem: - 10x when dealing with bandwidth which you cannot do much except a very big enterprise package and getting some discount that even then it will be multiple times higher than the alternatives The way I see it, AWS is great but you need to consider your 1) key infra requirements 2) consider cost effectiveness of different solutions/services 3) consider alternatives i.e. - Cloudfront is fine, but if you doing 100 tb/month probably Cloudflare would be best! - Lambda is fine but if you doing 100M requests/month EC2/ECS would probably work better (or more cost effective) Consider 1) what you need to get started and ship 2) what you need to scale up 3) what's the most cost effective solution long term The requirements will change over time, be ready to adapt!

marcopicentini 4y ago

Me too! Running a B2B SaaS using Digital Ocean and Cloud66. I am free to switch vendor every 12 months without pain.

intev 4y ago
How has the experience been with Cloud66? Why them over using something like caprover?
- marcopicentini 4y ago
  I don’t know about caprover. I like Cloud66 because: - Pricing is fair. I spend 100$/month for my stack - They back up db and host it to a different cloud provider (AWS) - It’s easy to import Db - UI is clean and easy to use - It’s easy to switch vendor. Just clone the app and it will install all in a new server. I have switched from Digital Ocean to Azure in a couple of hours with 30 min downtime.

codingbot3000 4y ago

It looks clear to me that given the hardware they need, and the scale they operate at, buying their own metal makes sense. Also them sticking to similar servers is a good idea imho. What I am missing though in the monthly cost is the personnel cost. I would assume that a decently qualified system and network administrator in Singapore, capable of operating and servicing this setup, will cost upward of $10,000 per month. And only having one is probably not enough, due to SPOF and other considerations. To be honest, even for the cloud you probably need someone to administer the setup, but chances are high that there is much less to do. Paying for, and finding qualified staff is from my experience the main driver for small and mid-size companies to go to the cloud. And for many this actually makes sense financially. Even though many create unnecessary complex setups or get trapped in vendor lock-ins.

pphysch 4y ago
A small-medium DC can be a lot of work to install, specifically in terms of backup power/cooling, but not much to maintain on a daily basis. Occasionally you need to replace expired components, and install new systems, but that's about it. There will be weeklong periods where no action is required. Finding qualified staff is definitely hard. Especially nowadays, there are a lot of DevOps astronauts that have never swapped a bad DIMM.
- indymike 4y ago
  I've had great luck just leasing what I need from a company who already has the DC and is willing to manage the hardware as part of the lease.
- kkielhofner 4y ago
  Building a DC is yet another level. Co-location facilities drop redundant power in your cabinet and manage cooling. All you need to do is make sure you plug your redundant power supplies into the right PDU. Also, any decent co-lo provider will rack and provide remote access to HW. For geo-redundancy (unless you’re at massive scale beyond an HN comment) a couple of cabinets per facility is an incredible amount of compute. For hardware issues get support from your hardware vendor. When equipment fails (drive in a RAID array, power supply, entire chassis - whatever) they’ll dispatch a tech directly to the facility and take care of it. Depending on what you’re willing to pay it’s either same day or next (and warranty support is usually the first thing vendors will discount if you negotiate). I’ve “deployed” to multiple data centers that ran for years without ever stepping foot in them. That said I’m with you - I’m a little concerned Big Cloud is here to stay after 20 years of incredibly effective marketing and mindshare dominance. We now have an entire generation who’s sum total understanding of hardware is a (reversible!) USB-C connector.

aldarisbm 4y ago

> So we decided to consolidate all the costs spent for this data center, divide them by the number of servers, and compare them to the cost of a similar installation in the Amazon Web Services (AWS) cloud (we use this as the leading IaaS example). so compare them to buying AWS VMs you don’t need? one of the main attractions of the cloud is the fact that you don’t have to pay for what you don’t use. Elasticity. Every time I see one of these is: I bought 100 servers (which i use 40 of), I would’ve saved money if I was renting 100 servers of AWS. Yeah … no shit. If you were renting only 40, and had the ability to go down as usage decreases, there would be savings, mind you there are such things as, RIs, savings plans, etc, etc. meh.

margorczynski 4y ago

Once your operation is stable and you know (almost) exactly what you need then sticking to the Cloud makes less and less sense. The premium they charge is enormous. Not sure why e.g. Netflix doesn't exit AWS considering their needs flatlined. Or maybe they have some special agreement with much lower prices than market.

yegle 4y ago
A 24/7 oncall rotation team needs 5 people to keep it <1 rotation per month. This alone costs 1M/y. I think many companies simply get away by having unhealthy oncall rotations. E.g. ask people to oncall outside business hours constantly.
- cbrozefsky 4y ago
  Or their stuff doesn't break such that being oncall is a formality.
  - pgwhalen 4y ago
    This is a great way to put it, and I don’t see this reality acknowledged on HN very often.
marcinzm 4y ago
What the cloud gives you is the removal of the inter-department conflict between IT and Product Engineering. This is especially bad since IT is a cost-center rather than a revenue generator. Having a single provider who in turn has a single client is a horrible dynamic. It can work for the short term but long term you get really odd dynamics. Sure the cloud is expensive but so is the aggregate opportunity cost of waiting 3 months and spending 20 hours in meetings for a single dev server.
trollied 4y ago
This quite often comes down to CapEx vs OpEx.

andrewstuart 4y ago

There a fantasy that says you don’t need experts to run your cloud infrastructure. The same fantasy says that if you own your own hardware you’ll need lots of experts that are so expensive and so hard to find that it makes no sense to self host. Cloud is crazy expensive and delivers much slower computing power per dollar. Consider if you need kubernetes plus cloud scaling, or you just need one modern ripping fast 16 core machine with 128GB RAM.

AtlasBarfed 4y ago
It's bottom lines like the article's where I wonder if all the companies who had HORRID penny-wise pound-foolish (6 months of paperwork to get a server, and only then would they order the servers) and went cloud simply did what these guys appear to be doing: buy enough to stay ahead of demand. If you have unused servers, who cares, at least you don't have allocated by unused AWS servers that pay for themselves in hardware costs every... four months? I don't remember how long it is.
- lamontcg 4y ago
  I sold all my AMZN stock by 2006 when I quit thinking that everyone would figure out the secret to doing your own hardware cheap. I then went to try to convert other businesses over to doing it cheaply, but as an engineer I just wound up getting into fights with my managers who loved spending money. It is all about Empire building and managers wanted those expensive VMWare and EMC vendor relationships to manage in order to fluff up their resumes. They want more people under their umbrella. If you start hiring Directors of Network Engineering you'll find that you start hiring lots of expensive CCIEs and spending a lot of money on Cisco. It is almost a problem of being too small, since you can't quite easily point out how having single SATA drives in all the servers will save millions over having hardware RAID cards, and having simple networking in your rack design will also save millions over what Cisco would like to push on you. So you overspend by $250k here and there and everywhere, and pretty quickly you're back up to cloud costs. At a smaller level it isn't quite as obvious how obscenely expensive to your capex these kinds of managers are, and they know hot to talk in ways that make Managment believe that they're buying all kinds of Enterprise-class resiliancy and leveragable vendor relationships and all that kind of nonsense. Hiring smart engineers instead is something that most Managers don't know how to do either, which is what you need in order to get a decent level of automation in place in order to scale the ability to manage it all. And don't get me started on high priced consultants that can be brought in to overdesign the most complicated solutions possible by never saying 'no' to any kind of requirement that IT management can dream up. I've seen VLAN diagrams that you wouldn't believe... But anyway, if you can solve that human problem, through hiring the right people and having a corporate culture of "Frugality" (as much as I hate it, that focus Amazon has is what let them do IT cheaply and then resell it). But it isn't a technical problem, or a financial problem, or even really a staffing problem, it is fundamentally a culture problem. And then the trick in my mind would be to do Frugality right without being as much of a dick about it as Amazon is.

not_me_ever 4y ago

How can you save >$150M/year on a service that should cost <$2M/year to run?