22 ms·

The cost of cloud, a trillion dollar paradox (2021)

161 points by lnwlebjel 3y ago

notamy 3y ago
(2021)
- ninju 3y ago
  Yeah...that explains why the chart at the beginning of the article only went to 2020
SoftTalker 3y ago
OT: Something on that page is crashing Firefox tabs.
- Glyptodon 3y ago
  Was fine for me, but I do have various blockers.
- wizzard0 3y ago
  Makes the iPhone uncomfortably hot as well
- cosmic_quanta 3y ago
  Scrolling in Edge, i'm noticing terrible lagging. I imagine these things are related
very_good_man 3y ago
Worth having a look at DHH's leadership on this issue: https://world.hey.com/dhh/why-we-re-leaving-the-cloud-654b47e0 https://world.hey.com/dhh/why-we-re-leaving-the-cloud-654b47...
Glyptodon 3y ago
I've always thought the mysterious bit was that cloud doesn't seem like it ever gets cheaper while the cost difference between cloud and own infra and profit associated with AWS suggest it could be. Is there something structurally wrong with competition in cloud?
- sgt101 3y ago
  Yes, massive cross subsidies from core business to cloud business to exclude and destroy competition - aws=books, azure=windows, gcp=search. There are three companies doing monopolistic things, they have got away with it because of that plurality.
  - senderista 3y ago
    AWS is vastly more profitable than Amazon's retail business.
    - sgt101 3y ago
      It is now. It wasn't in the past. It wouldn't be if they had to compete with providers that also wanted to compete.
      - calimoro78 3y ago
        AWS was first to market for many years, and their margins were always higher than ecommerce retail (and that's easy to believe, since retail often runs on 1% margins)
      - nine_k 3y ago
        AWS was basically built as an internal product to support the retail business, and then they realized that they can sell it.
        bcrosby95 3y ago
        There's been several articles posted here over the years from people that worked at Amazon at the time saying this is just a myth.
        senderista 3y ago
        Yes, hence the "MAWS" (Move to AWS) initiative to actually make retail services run on AWS (took many years).
latchkey 3y ago
Previously: The cost of cloud, a trillion dollar paradox https://news.ycombinator.com/item?id=27401081 https://news.ycombinator.com/item?id=27401081 (June 4, 2021 — 7 points, 2 comments)
- dang 3y ago
  Thanks! Macroexpanded: The Cost of Cloud, a Trillion Dollar Paradox - https://news.ycombinator.com/item?id=32832433 https://news.ycombinator.com/item?id=32832433 - Sept 2022 (15 comments) The Cost of Cloud, a Trillion Dollar Paradox - https://news.ycombinator.com/item?id=32672553 https://news.ycombinator.com/item?id=32672553 - Sept 2022 (1 comment) The cost of cloud, a trillion dollar paradox - https://news.ycombinator.com/item?id=27401081 https://news.ycombinator.com/item?id=27401081 - June 2021 (2 comments) The cost of cloud, a trillion dollar paradox - https://news.ycombinator.com/item?id=27306742 https://news.ycombinator.com/item?id=27306742 - May 2021 (207 comments)
AIorNot 3y ago
So now the trend is away from cloud ? What about a good way to to judge the cost of the cloud - pricing has always been overly complex and arcane and full of gaps in understanding the overhead involved
- Macha 3y ago
  Honestly the relative clarity of what cloud costs has been one of the advantages that companies want to pay a premium for, compared to the quagmire of capex, depreciation, attribution of datacenter costs, and interdepartmental chargebacks that on prem usually means. I suspect fewer cloud migrations would occur if companies could map their onprem costs to specific products much more easily rather than going "Oh it might be a lot for this team but who knows?", which opens the room to someone to sell the pricing transparency use case to an exec.
  - nine_k 3y ago
    Now you can operate your own local cloud (Dokku, Rancher, Proxmox, VMWare, MinIO, k8s, you name it). This gives you all the simplicity of deployment and accountability, and some elasticity. It adds a bit of ops complexity at the start though. But running your own rack starts to make sense when your AWS bill is at least the same order of magnitude as an SRE salary; at that point, you already have needs that are varied enough.
bamazizi 3y ago
Interesting to see this article again since it was the trigger of starting a homelab for me. After realizing cloud services are putting a major dent in my pocket to get a lousy startup idea off the ground, I started to wonder if there's "The Cheapest" way? (I'm not cheap but I'm very frugal) Nowadays internet speed is great to do self hosting. I have a business line internet at home with ~1gb up&down! Bought couple of 6-7 year old enterprise Dell servers (2x12core xeon, 128gb ram each) and no longer pay any cloud provider ... i'm also hosting 2 backend solutions for mobile apps with decent traffic for friends' startups! The learning experience has been tremendous! It has actually gotten a lot better and easier with new solutions coming out for homelabs. Get started with Proxmox clusters and go from there...
- ilyt 3y ago
  Renting dedicated server off company like OVH, or even co-locating your own is also far cheaper than cloud, few times over, without fuss of turning your house into datacenter.
  - graemep 3y ago
    There is a time cost, but most of it is a one off, and not that difficult, and there are tools to make it easier.
- p1necone 3y ago
  Yeah the delta in cost/CPU/memory/storage between self hosting on second hand stuff and paying cloud services is insane. It's a no brainer if your use case needs beefier hardware than the typical $5/month VPS host and you don't have enterprise level liability/uptime expectations. With stuff like proxmox you get a pretty similar level of ease of use to managed VM services too.
  - Karrot_Kream 3y ago
    I generally find these discussions unproductive on HN because of how binary they become, but definitely once you go off the beaten path it seems a lot cheaper (sans power though) to run your own infra. I've been thinking of doing some data ETL for something that generates 1 GB / day and has retention for 30 days and it's much cheaper to host a Postgres myself at that scale than run it in the cloud. I could store in something like S3, but then I need to deal with querying S3. I'd like to combine some of this with cloud infra but I suspect cloud bandwidth costs would kill me. Colo-ing 2-3 servers is probably the best bet here.
sgt101 3y ago
Look - if you can optimise your workload over a capital cycle cloud is not for you. Especially if you are a technology company and have the bandwidth to deal with servers and ISV's properly. If you are a changing, growing, non-technology company then cloud is the place you should be. You will get screwed by Oracle, you will get screwed by Accenture. You will not be able to optimise your workload and you may as well get screwed by aws/azure/gcp and not think about infrastructure that much instead.
- SoftTalker 3y ago
  Good luck getting the time of day from Oracle or Accenture if you are not already quite a large established company. Not that you want it anyway though.
  - renewiltord 3y ago
    Oracle was willing to talk to us and the max we would have paid them would have been $10 m / year.
    - 9dev 3y ago
      Being able to pay 10 million per year to a software vendor is absolutely a sign of being a well established company.
  - quickthrower2 3y ago
    Time of day? Sign this $10M contract.
    - deleted 3y ago
      [deleted]
epberry 3y ago
This post, along with this Simon Wardley thread, https://twitter.com/swardley/status/1088780650860158978?s=20 https://twitter.com/swardley/status/1088780650860158978?s=20, attracted me to finops as an engineering problem. Consumption based pricing is really tough for a lot of orgs to handle at the $1M+ range, so much so that it requires dedicated software to line things up. I started helping maintain https://ec2instances.info https://ec2instances.info in my spare time and the code base is literally full of IF statements to paper over AWS billing quirks. Later, I joined Vantage, one of the companies linked in the article. It's a little bit undecided whether finops will have the growth that devops did but the problem is seemingly felt very acutely.
- bigtones 3y ago
  I love using https://ec2instances.info https://ec2instances.info as a reference - thank you for maintaining it !
- avn2109 3y ago
  Thank you for ec2instances! I use it a lot and it's really great :)
- red-iron-pine 3y ago
  > attracted me to finops as an engineering problem. Consumption based pricing is really tough for a lot of orgs to handle at the $1M+ range, so much so that it requires dedicated software to line things up. absolutely. massive struggle at the orgs I've been / am at. processes for provisioning new resources haven't changed, and a lot of teams kind of push their own builds, or get reservations for X amount of space and dollars and then they can play with their space. the result is OpEx shifts wildly from one quarter to the next, and control has been hit or miss. it continues mostly because projects need to deliver -- and do -- but that "do what you gotta do to make it work" approach has turned basically into the wild west and no one plays by the rules.
  - epberry 3y ago
    Yeah. In an infrastructure as code setup there's some more rigor that can be applied, at least in terms of tagging what's going out. But I've generally seen the finops team come in and optimize ex post facto.
michaelt 3y ago
> A billion dollar private software company told us that their public cloud spend amounted to 81% of COR, and that “cloud spend ranging from 75 to 80% of cost of revenue was common among software companies” I find that quite hard to believe. I mean, I can see that if you were selling cloud backup services and stored customer data on S3, I can understand S3 costs being a big part of your budget. But for the vast majority of businesses - I'd expect a supermarket selling a $50 basket of groceries to spend maybe $0.01 on database storage and CPU and whatnot. I don't see that a company like Slack, which has revenue of about $8.75/user/month would need to spend anything like that much on cloud costs.
- Ographer 3y ago
  A chart on the article said that Slack spends an estimated 41% of revenue on cloud costs, calculated using their financials by this author who claims they were being conservative but I don't know how to estimate such a thing myself.
  - mr_toad 3y ago
    So Slack are basically on-selling cloud at a 59% markup.
- paulddraper 3y ago
  You should. Best in class SaaS costs are like 10% of revenue
- quickthrower2 3y ago
  This is high, unless you are a platform as a service, effectively selling the compute on to customers. Add on the free tiers subsidy (for growth!) and it make sense then. If you are selling just software and 80% of that is on the cloud you are undercharging or wasting crazy resources.
- icedchai 3y ago
  I've worked at a few SaaS startups that did not price their service correctly and did not want to pay up front for reserved instances. The cloud costs were very high. 80% of revenue was about right.
asah 3y ago
this makes no sense: can individual companies operate computing infrastructure more efficiently than cloud providers? if not, then instead of migrating just negotiate.
- p1necone 3y ago
  I think the biggest thing is that the minimum extra spend if you want in house managed infra is basically the salary of a full time sysadmin + hardware costs. You can pay for a lot of cloud resources for that much, even when the cloud resources are massively overpriced compared to what you could manage if you had more scale. The only other alternative is finding someone who's good enough at it to make that full time sysadmin job just a small part of their "actual" job responsibilities, but that person is pretty hard to find.
  - acdha 3y ago
    The other thing is remembering it’s not one full time sysadmin but many and also things like security staffing. One dude running some Linux boxes isn’t going to give you the kinds of audit logging, monitoring, encryption, etc. which is table stakes in the cloud world. You can hire those people, buy HSMs, etc. but then you’re talking a much bigger gap before you break even. Then you get a step function cost increase building and operating all of the services your developers aren’t getting out of the box.
  - jodersky 3y ago
    > I think the biggest thing is that the minimum extra spend if you want in house managed infra is basically the salary of a full time sysadmin + hardware costs. The cloud promises to make admin tasks easier, however I have never seen it eliminate the role of a sysadmin in practice. In my experience, most organizations that run their infra on a cloud still have dedicated admin roles (often called "DevOps" or infra teams). Hence, I think that the claim that a sysadmin's salary is an extra expense for self-managed infrastructure is exaggerated. You may need more sysadmins for achieving the same features in a self-managed setup compared to the cloud, but it is not a linear scale.
- tommek4077 3y ago
simonebrunozzi 3y ago
Martin Casado is one of the smartest people I've met in my career in IT (AWS, VMware, etc). This phrase is key: "You’re crazy if you don’t start in the cloud; you’re crazy if you stay on it" Perhaps I'd say that "you're crazy if you keep all your IT on it, as you scale". There's still a case to be made for a small % of your workloads in the cloud. The flexibility is priceless. But for everything else, there's a well managed on-prem. Side note: interesting how, after Mark Andreessen's disastrous post on techno-optimism a few days ago, I approached the reading of this article with rolled eyes and low expectations. Seeing Marting Casado as the author immediately changed it, however.
- nonrandomstring 3y ago
  > "You’re crazy if you don't start in the cloud; you're crazy if you stay on it" Spot on. Gonna have to steal that for my next meeting. Although, given the potential Cavium-Marvell TPM issue viz ECC and that we're not really there with homomorphic compute, I might make a caveat for developers working on very sensitive new ideas; Keep it on-prem until you have a good enough security segmentation to push out what you're happy with on untrusted cloud infra.
- tight-ship 3y ago
  > Perhaps I'd say that "you're crazy if you keep all your IT on it, as you scale". And by scale, I think companies should be at least in the double digit millions spent in cloud infrastructure before they start building out their own. Everybody is used to the cloud APIs and going back to the old days of managing things in a static datacenter are not coming back for these companies.
- deleted 3y ago
  [deleted]
- acdha 3y ago
  > But for everything else, there's a well managed on-prem. That’s begging a number of questions, however - the number of places who can approach a cloud service portfolio at all, much less with a cost savings or security parity, has to be at least an order of magnitude smaller than the number of places who incorrectly thought they could.
codemac 3y ago
It's my hardcore held belief that the cloud industry could wildly be disrupted if it was easy enough to order 5 year capacity contracts. Then AWS / GCP etc would be left with just the elastic demand, rather than the inelastic profit center. However - this would take a huge amount of upfront investment, have high risks for black swan events against 5y contracts without crypto-style escrow, and I have been unsuccessful pitching this direction in elevators to folks, which makes me think I don't have a visceral enough idea.
- bob1029 3y ago
  5 years is a really long time on most types of B2B contracts. Especially ones with very thorough operational considerations bound to technology changing over time. Consider how much premiums would be on $TSLA put options that expire in 2028.
  - codemac 3y ago
    Somehow real estate gets away with it, and they have even more fungible contents!
- latchkey 3y ago
  Lambda went from bragging about short contracts [0] to 3+ years and if you look closely, the only thing with actual pricing is 3 years, otherwise it is contact sales and waitlist. [0] https://lambdalabs.com/blog/cutting-the-cost-of-high-performance-deep-learning https://lambdalabs.com/blog/cutting-the-cost-of-high-perform... [1] https://lambdalabs.com/service/gpu-cloud/reserved https://lambdalabs.com/service/gpu-cloud/reserved
mlsu 3y ago
It cuts the other way, too. A cost for one is revenue for another: In this case, AWS or Microsoft. Your cost of revenue is their revenue! There will be big growth in the "make it easier to manage on prem deployments" sector. Obviously you need capable sysadmin if you are going to move workloads away from cloud but improving the tech (I'm thinking of Oxide Computer here) will make it palatable for smaller and smaller orgs to consider repatriation, as they can do so with less effort and expertise. This makes me hopeful as well that internet infrastructure will become less centralized. It's closer to the spirit of the internet that not every bit of information you see online passes through us-east-1.
hnthrowaway0315 3y ago
I'd say for a lot of companies the Fed wake up call has happened that they have been doing a lot on reducing cloud costs. They did not necessarily move away from cloud but instead sign long term contracts and remove dead or undead services.
VirusNewbie 3y ago
Dropbox has the pay, clout, and pipeline to have zero problems with recruiting talent. Can the same be said of all F500 companies that may want to basically engineer their own cloud? Because the dev team is almost certainly used to all the convenience of the cloud, so you'll need a good infra team.
deleted 3y ago
[deleted]
jdoss 3y ago
Back in 2007ish the CEO of Parallels/SWSoft (the company behind Virtuozzo a very popular hypervisor used back in the day) stood on the stage of the keynote of HostingCon and said that the Cloud (AWS) was going to eat the WebHosting Industry's lunch. His entire keynote was very doom and gloom. Fast forward to 2015ish it totally it did. The days of VPS and Dedicated hosting being the only options for deploying your stuff for the Internet to consume were killed by the big three cloud providers. In one hand I am very thankful that this caused a boom of innovation and tooling to make engineering operations easier and more accessible to our industry as a whole. If I had the tools I have now, managing 50+ racks in my old datacenter would have been so much easier. But in the other hand, the promises the Cloud providers have shoved down our throats have been used against our Industry with raised prices, reduce freedoms on what providers you can choose, and consolidation of duties on Engineering staff. The average developer has to do more work, across many different domains to get paid well. The war waged to kill off Systems Administration as a core job responsibility and using the DevOps movement to kill FTE count for specialized work on bare metal or datacenter infra keep us locked into using Cloud APIs to do our jobs. I think this really sucks. I do think this article hits home on some points I have been seeing over the past few years. The costs are too high for the goods provided in comparison to getting a couple of racks in few datacenters and DIY which can for sure work for the right kinds of companies at the right stage of growth. I also think Hybrid deployments of Cloud + Bare Metal in a datacenter is so much more of a viable option for less mature companies. The pendulum is starting to slowly swing back to doing things on Bare Metal. I am excited for the next 10 years of progress and I hope the Cloud still remains a powerful tool in our toolbox, but it isn't used for everything. The cost savings are too real as this article points out when you don't use the cloud and you'd be surprised how much easier it is these days to get some bare metal deployments going in your eng org.
- sokoloff 3y ago
  > keep us locked into using Cloud APIs to do our jobs I love that we can use Cloud APIs to do our jobs. That's piles better than negotiating with (and waiting for) an Ops team to provision you hardware or networking that you don't have much visibility into and then relying on them to troubleshoot. I don't mind paying more for that (which you absolutely do on a per-unit of compute, network, or storage).
nologic01 3y ago
solving a paradox sometimes requires thinking out of the box repatriation is not the global solution, it applies to mature and large companies the dimension to explore is co-ownership of infrastructure by multiple independent companies, both small and large a captive entity - wholly owned by member companies - manages a vast pool of resources achieving both the economies of scale and the flexibility needed by growing companies. you don't "rent" somebody elses linux server, you invest and own a share. R&D for evolving the platform is something outsourced to third parties (maybe even current cloud providers)
alex_young 3y ago
Sub-linear scaling is powerful. Think about how costs scale in relation to use of a service. If you are paying someone else for the service as a package, the costs scale linearly, or some close approximation of that. AWS gives you a small discount if you have larger volumes for some services, but these are essentially a rounding error. If instead you launch a similar service in house, your costs are often dramatically lower per unit the more units you use. You start with a small team supporting a system with some large capacity you're not using yet because you need room to grow, and you realize large steps along the way where you can handle tens or hundreds of times the volume you see at each staffing level. Buying hardware or developing custom software gets cheaper at volume. This is sub-linear scaling.
andrewprock 3y ago
Any discussion of cloud vs. data center vs. on premises that does not discuss employee costs, documentation, and training is not complete. One of the main reasons for using cloud infrastructure is that many people know how to use it, internal documentation requirements are dramatically lessened, and you don't need to higher infrastructure IT employees to maintain and monitor it. If cloud only costs double internal IT, it's almost certainly cheaper until you reach a billion dollar valuation - or more depending on what your infra needs are.
hn_throwaway_99 3y ago
I think it's a bit nutty that folks always use Dropbox as the example of "cost savings by doing it yourself". I mean, Dropbox itself is essentially a provider of cloud storage. It seems like owning their own data centers was obvious from the beginning: Use AWS initially to scale, but given the size of Dropbox it obviously doesn't make sense to have another middleman in the mix. I just point this out because Dropbox is so unlike your average enterprise user of the cloud. I'd like to see some better examples than Dropbox of enterprises saving more (or innovating as quickly) by going on their own.
- zinodaur 3y ago
  I work at a company that recently tried to shift our (on prem, sharded, 2-metro, 6 replica) mysql deployment to AWS. Any way we sliced it, even after steep discounts, we were seeing massive TCO increases, often up to 10x - the math just doesn't work out for large deployments. It's really a shame that all these companies are forced to write their own (crappy) version of AWS - but price is a feature when it comes to this stuff.
locococo 3y ago
the benefit of the cloud is the elasticity. People neglect sometimes that businesses had to scale their infrastructure to peak demand before the cloud. If your workloads don't have peaks it's an easy problem to solve.
- jmathai 3y ago
  We had to purchase/lease HP storage servers with 10x the storage we needed for our photo startup in 2006. The lead time to get a server was weeks so there was very little elasticity to respond to non linear growth. Would have been a lot easier if we had S3 (launched around this time, no one really knew about it or what it was).
AndrewKemendo 3y ago
I’m leading engineering at a new start up and I’m doing absolutely everything I can to avoid touching the cloud at pretty much any cost. We’re going to be running all of our own dev/test GPU servers with only inference hosted outside our cluster.
- jjn2009 3y ago
  The need for GPUs is a solid reason to stay away.
  - AndrewKemendo 3y ago
    Yeah it’s the main reason because the data transfer costs alone would be insane
efxhoy 3y ago
If moving off the cloud was easy we’d all be doing it. For our org the risk is (unfortunately imo) bigger than the savings. I would love to pay hetzner dedicated boxes rates for compute but the amount of work to migrate everything would be huge. Not to mention the difficulty in hiring people who are actually comfortable with doing the infra work of setting up all our services, with the required multi-AZ config, on bare metal. Its hard enough finding people who can reliably setup services in aws with terraform and keep them alive.
smartbit 3y ago
Previous May 27, 2021 https://news.ycombinator.com/item?id=27306742 https://news.ycombinator.com/item?id=27306742 207 comments Sept 14, 2022 https://news.ycombinator.com/item?id=32832433 https://news.ycombinator.com/item?id=32832433 16 comments

161 points by lnwlebjel 3y ago

notamy 3y ago

(2021)

ninju 3y ago
Yeah...that explains why the chart at the beginning of the article only went to 2020

SoftTalker 3y ago

OT: Something on that page is crashing Firefox tabs.

Glyptodon 3y ago
Was fine for me, but I do have various blockers.
wizzard0 3y ago
Makes the iPhone uncomfortably hot as well
cosmic_quanta 3y ago
Scrolling in Edge, i'm noticing terrible lagging. I imagine these things are related

very_good_man 3y ago

Worth having a look at DHH's leadership on this issue: https://world.hey.com/dhh/why-we-re-leaving-the-cloud-654b47e0 https://world.hey.com/dhh/why-we-re-leaving-the-cloud-654b47...

Glyptodon 3y ago

I've always thought the mysterious bit was that cloud doesn't seem like it ever gets cheaper while the cost difference between cloud and own infra and profit associated with AWS suggest it could be. Is there something structurally wrong with competition in cloud?

sgt101 3y ago
Yes, massive cross subsidies from core business to cloud business to exclude and destroy competition - aws=books, azure=windows, gcp=search. There are three companies doing monopolistic things, they have got away with it because of that plurality.
- senderista 3y ago
  AWS is vastly more profitable than Amazon's retail business.
  - sgt101 3y ago
    It is now. It wasn't in the past. It wouldn't be if they had to compete with providers that also wanted to compete.
    - calimoro78 3y ago
      AWS was first to market for many years, and their margins were always higher than ecommerce retail (and that's easy to believe, since retail often runs on 1% margins)
    - nine_k 3y ago
      AWS was basically built as an internal product to support the retail business, and then they realized that they can sell it.
      - bcrosby95 3y ago
        There's been several articles posted here over the years from people that worked at Amazon at the time saying this is just a myth.
        senderista 3y ago
        Yes, hence the "MAWS" (Move to AWS) initiative to actually make retail services run on AWS (took many years).

latchkey 3y ago

Previously: The cost of cloud, a trillion dollar paradox https://news.ycombinator.com/item?id=27401081 https://news.ycombinator.com/item?id=27401081 (June 4, 2021 — 7 points, 2 comments)

dang 3y ago
Thanks! Macroexpanded: The Cost of Cloud, a Trillion Dollar Paradox - https://news.ycombinator.com/item?id=32832433 https://news.ycombinator.com/item?id=32832433 - Sept 2022 (15 comments) The Cost of Cloud, a Trillion Dollar Paradox - https://news.ycombinator.com/item?id=32672553 https://news.ycombinator.com/item?id=32672553 - Sept 2022 (1 comment) The cost of cloud, a trillion dollar paradox - https://news.ycombinator.com/item?id=27401081 https://news.ycombinator.com/item?id=27401081 - June 2021 (2 comments) The cost of cloud, a trillion dollar paradox - https://news.ycombinator.com/item?id=27306742 https://news.ycombinator.com/item?id=27306742 - May 2021 (207 comments)

AIorNot 3y ago

So now the trend is away from cloud ? What about a good way to to judge the cost of the cloud - pricing has always been overly complex and arcane and full of gaps in understanding the overhead involved

Macha 3y ago
Honestly the relative clarity of what cloud costs has been one of the advantages that companies want to pay a premium for, compared to the quagmire of capex, depreciation, attribution of datacenter costs, and interdepartmental chargebacks that on prem usually means. I suspect fewer cloud migrations would occur if companies could map their onprem costs to specific products much more easily rather than going "Oh it might be a lot for this team but who knows?", which opens the room to someone to sell the pricing transparency use case to an exec.
- nine_k 3y ago
  Now you can operate your own local cloud (Dokku, Rancher, Proxmox, VMWare, MinIO, k8s, you name it). This gives you all the simplicity of deployment and accountability, and some elasticity. It adds a bit of ops complexity at the start though. But running your own rack starts to make sense when your AWS bill is at least the same order of magnitude as an SRE salary; at that point, you already have needs that are varied enough.

bamazizi 3y ago

Interesting to see this article again since it was the trigger of starting a homelab for me. After realizing cloud services are putting a major dent in my pocket to get a lousy startup idea off the ground, I started to wonder if there's "The Cheapest" way? (I'm not cheap but I'm very frugal) Nowadays internet speed is great to do self hosting. I have a business line internet at home with ~1gb up&down! Bought couple of 6-7 year old enterprise Dell servers (2x12core xeon, 128gb ram each) and no longer pay any cloud provider ... i'm also hosting 2 backend solutions for mobile apps with decent traffic for friends' startups! The learning experience has been tremendous! It has actually gotten a lot better and easier with new solutions coming out for homelabs. Get started with Proxmox clusters and go from there...

ilyt 3y ago
Renting dedicated server off company like OVH, or even co-locating your own is also far cheaper than cloud, few times over, without fuss of turning your house into datacenter.
- graemep 3y ago
  There is a time cost, but most of it is a one off, and not that difficult, and there are tools to make it easier.
p1necone 3y ago
Yeah the delta in cost/CPU/memory/storage between self hosting on second hand stuff and paying cloud services is insane. It's a no brainer if your use case needs beefier hardware than the typical $5/month VPS host and you don't have enterprise level liability/uptime expectations. With stuff like proxmox you get a pretty similar level of ease of use to managed VM services too.
- Karrot_Kream 3y ago
  I generally find these discussions unproductive on HN because of how binary they become, but definitely once you go off the beaten path it seems a lot cheaper (sans power though) to run your own infra. I've been thinking of doing some data ETL for something that generates 1 GB / day and has retention for 30 days and it's much cheaper to host a Postgres myself at that scale than run it in the cloud. I could store in something like S3, but then I need to deal with querying S3. I'd like to combine some of this with cloud infra but I suspect cloud bandwidth costs would kill me. Colo-ing 2-3 servers is probably the best bet here.

sgt101 3y ago

Look - if you can optimise your workload over a capital cycle cloud is not for you. Especially if you are a technology company and have the bandwidth to deal with servers and ISV's properly. If you are a changing, growing, non-technology company then cloud is the place you should be. You will get screwed by Oracle, you will get screwed by Accenture. You will not be able to optimise your workload and you may as well get screwed by aws/azure/gcp and not think about infrastructure that much instead.

SoftTalker 3y ago
Good luck getting the time of day from Oracle or Accenture if you are not already quite a large established company. Not that you want it anyway though.
- renewiltord 3y ago
  Oracle was willing to talk to us and the max we would have paid them would have been $10 m / year.
  - 9dev 3y ago
    Being able to pay 10 million per year to a software vendor is absolutely a sign of being a well established company.
- quickthrower2 3y ago
  Time of day? Sign this $10M contract.
  - deleted 3y ago
    [deleted]

epberry 3y ago

This post, along with this Simon Wardley thread, https://twitter.com/swardley/status/1088780650860158978?s=20 https://twitter.com/swardley/status/1088780650860158978?s=20, attracted me to finops as an engineering problem. Consumption based pricing is really tough for a lot of orgs to handle at the $1M+ range, so much so that it requires dedicated software to line things up. I started helping maintain https://ec2instances.info https://ec2instances.info in my spare time and the code base is literally full of IF statements to paper over AWS billing quirks. Later, I joined Vantage, one of the companies linked in the article. It's a little bit undecided whether finops will have the growth that devops did but the problem is seemingly felt very acutely.

bigtones 3y ago
I love using https://ec2instances.info https://ec2instances.info as a reference - thank you for maintaining it !
avn2109 3y ago
Thank you for ec2instances! I use it a lot and it's really great :)
red-iron-pine 3y ago
> attracted me to finops as an engineering problem. Consumption based pricing is really tough for a lot of orgs to handle at the $1M+ range, so much so that it requires dedicated software to line things up. absolutely. massive struggle at the orgs I've been / am at. processes for provisioning new resources haven't changed, and a lot of teams kind of push their own builds, or get reservations for X amount of space and dollars and then they can play with their space. the result is OpEx shifts wildly from one quarter to the next, and control has been hit or miss. it continues mostly because projects need to deliver -- and do -- but that "do what you gotta do to make it work" approach has turned basically into the wild west and no one plays by the rules.
- epberry 3y ago
  Yeah. In an infrastructure as code setup there's some more rigor that can be applied, at least in terms of tagging what's going out. But I've generally seen the finops team come in and optimize ex post facto.

michaelt 3y ago

> A billion dollar private software company told us that their public cloud spend amounted to 81% of COR, and that “cloud spend ranging from 75 to 80% of cost of revenue was common among software companies” I find that quite hard to believe. I mean, I can see that if you were selling cloud backup services and stored customer data on S3, I can understand S3 costs being a big part of your budget. But for the vast majority of businesses - I'd expect a supermarket selling a $50 basket of groceries to spend maybe $0.01 on database storage and CPU and whatnot. I don't see that a company like Slack, which has revenue of about $8.75/user/month would need to spend anything like that much on cloud costs.

Ographer 3y ago
A chart on the article said that Slack spends an estimated 41% of revenue on cloud costs, calculated using their financials by this author who claims they were being conservative but I don't know how to estimate such a thing myself.
- mr_toad 3y ago
  So Slack are basically on-selling cloud at a 59% markup.
paulddraper 3y ago
You should. Best in class SaaS costs are like 10% of revenue
quickthrower2 3y ago
This is high, unless you are a platform as a service, effectively selling the compute on to customers. Add on the free tiers subsidy (for growth!) and it make sense then. If you are selling just software and 80% of that is on the cloud you are undercharging or wasting crazy resources.
icedchai 3y ago
I've worked at a few SaaS startups that did not price their service correctly and did not want to pay up front for reserved instances. The cloud costs were very high. 80% of revenue was about right.

asah 3y ago

this makes no sense: can individual companies operate computing infrastructure more efficiently than cloud providers? if not, then instead of migrating just negotiate.

p1necone 3y ago
I think the biggest thing is that the minimum extra spend if you want in house managed infra is basically the salary of a full time sysadmin + hardware costs. You can pay for a lot of cloud resources for that much, even when the cloud resources are massively overpriced compared to what you could manage if you had more scale. The only other alternative is finding someone who's good enough at it to make that full time sysadmin job just a small part of their "actual" job responsibilities, but that person is pretty hard to find.
- acdha 3y ago
  The other thing is remembering it’s not one full time sysadmin but many and also things like security staffing. One dude running some Linux boxes isn’t going to give you the kinds of audit logging, monitoring, encryption, etc. which is table stakes in the cloud world. You can hire those people, buy HSMs, etc. but then you’re talking a much bigger gap before you break even. Then you get a step function cost increase building and operating all of the services your developers aren’t getting out of the box.
- jodersky 3y ago
  > I think the biggest thing is that the minimum extra spend if you want in house managed infra is basically the salary of a full time sysadmin + hardware costs. The cloud promises to make admin tasks easier, however I have never seen it eliminate the role of a sysadmin in practice. In my experience, most organizations that run their infra on a cloud still have dedicated admin roles (often called "DevOps" or infra teams). Hence, I think that the claim that a sysadmin's salary is an extra expense for self-managed infrastructure is exaggerated. You may need more sysadmins for achieving the same features in a self-managed setup compared to the cloud, but it is not a linear scale.
tommek4077 3y ago

simonebrunozzi 3y ago

Martin Casado is one of the smartest people I've met in my career in IT (AWS, VMware, etc). This phrase is key: "You’re crazy if you don’t start in the cloud; you’re crazy if you stay on it" Perhaps I'd say that "you're crazy if you keep all your IT on it, as you scale". There's still a case to be made for a small % of your workloads in the cloud. The flexibility is priceless. But for everything else, there's a well managed on-prem. Side note: interesting how, after Mark Andreessen's disastrous post on techno-optimism a few days ago, I approached the reading of this article with rolled eyes and low expectations. Seeing Marting Casado as the author immediately changed it, however.

nonrandomstring 3y ago
> "You’re crazy if you don't start in the cloud; you're crazy if you stay on it" Spot on. Gonna have to steal that for my next meeting. Although, given the potential Cavium-Marvell TPM issue viz ECC and that we're not really there with homomorphic compute, I might make a caveat for developers working on very sensitive new ideas; Keep it on-prem until you have a good enough security segmentation to push out what you're happy with on untrusted cloud infra.
tight-ship 3y ago
> Perhaps I'd say that "you're crazy if you keep all your IT on it, as you scale". And by scale, I think companies should be at least in the double digit millions spent in cloud infrastructure before they start building out their own. Everybody is used to the cloud APIs and going back to the old days of managing things in a static datacenter are not coming back for these companies.
deleted 3y ago
[deleted]
acdha 3y ago
> But for everything else, there's a well managed on-prem. That’s begging a number of questions, however - the number of places who can approach a cloud service portfolio at all, much less with a cost savings or security parity, has to be at least an order of magnitude smaller than the number of places who incorrectly thought they could.

codemac 3y ago

It's my hardcore held belief that the cloud industry could wildly be disrupted if it was easy enough to order 5 year capacity contracts. Then AWS / GCP etc would be left with just the elastic demand, rather than the inelastic profit center. However - this would take a huge amount of upfront investment, have high risks for black swan events against 5y contracts without crypto-style escrow, and I have been unsuccessful pitching this direction in elevators to folks, which makes me think I don't have a visceral enough idea.

bob1029 3y ago
5 years is a really long time on most types of B2B contracts. Especially ones with very thorough operational considerations bound to technology changing over time. Consider how much premiums would be on $TSLA put options that expire in 2028.
- codemac 3y ago
  Somehow real estate gets away with it, and they have even more fungible contents!
latchkey 3y ago
Lambda went from bragging about short contracts [0] to 3+ years and if you look closely, the only thing with actual pricing is 3 years, otherwise it is contact sales and waitlist. [0] https://lambdalabs.com/blog/cutting-the-cost-of-high-performance-deep-learning https://lambdalabs.com/blog/cutting-the-cost-of-high-perform... [1] https://lambdalabs.com/service/gpu-cloud/reserved https://lambdalabs.com/service/gpu-cloud/reserved

mlsu 3y ago

It cuts the other way, too. A cost for one is revenue for another: In this case, AWS or Microsoft. Your cost of revenue is their revenue! There will be big growth in the "make it easier to manage on prem deployments" sector. Obviously you need capable sysadmin if you are going to move workloads away from cloud but improving the tech (I'm thinking of Oxide Computer here) will make it palatable for smaller and smaller orgs to consider repatriation, as they can do so with less effort and expertise. This makes me hopeful as well that internet infrastructure will become less centralized. It's closer to the spirit of the internet that not every bit of information you see online passes through us-east-1.

hnthrowaway0315 3y ago

I'd say for a lot of companies the Fed wake up call has happened that they have been doing a lot on reducing cloud costs. They did not necessarily move away from cloud but instead sign long term contracts and remove dead or undead services.

VirusNewbie 3y ago

Dropbox has the pay, clout, and pipeline to have zero problems with recruiting talent. Can the same be said of all F500 companies that may want to basically engineer their own cloud? Because the dev team is almost certainly used to all the convenience of the cloud, so you'll need a good infra team.

deleted 3y ago

[deleted]

jdoss 3y ago

Back in 2007ish the CEO of Parallels/SWSoft (the company behind Virtuozzo a very popular hypervisor used back in the day) stood on the stage of the keynote of HostingCon and said that the Cloud (AWS) was going to eat the WebHosting Industry's lunch. His entire keynote was very doom and gloom. Fast forward to 2015ish it totally it did. The days of VPS and Dedicated hosting being the only options for deploying your stuff for the Internet to consume were killed by the big three cloud providers. In one hand I am very thankful that this caused a boom of innovation and tooling to make engineering operations easier and more accessible to our industry as a whole. If I had the tools I have now, managing 50+ racks in my old datacenter would have been so much easier. But in the other hand, the promises the Cloud providers have shoved down our throats have been used against our Industry with raised prices, reduce freedoms on what providers you can choose, and consolidation of duties on Engineering staff. The average developer has to do more work, across many different domains to get paid well. The war waged to kill off Systems Administration as a core job responsibility and using the DevOps movement to kill FTE count for specialized work on bare metal or datacenter infra keep us locked into using Cloud APIs to do our jobs. I think this really sucks. I do think this article hits home on some points I have been seeing over the past few years. The costs are too high for the goods provided in comparison to getting a couple of racks in few datacenters and DIY which can for sure work for the right kinds of companies at the right stage of growth. I also think Hybrid deployments of Cloud + Bare Metal in a datacenter is so much more of a viable option for less mature companies. The pendulum is starting to slowly swing back to doing things on Bare Metal. I am excited for the next 10 years of progress and I hope the Cloud still remains a powerful tool in our toolbox, but it isn't used for everything. The cost savings are too real as this article points out when you don't use the cloud and you'd be surprised how much easier it is these days to get some bare metal deployments going in your eng org.

sokoloff 3y ago
> keep us locked into using Cloud APIs to do our jobs I love that we can use Cloud APIs to do our jobs. That's piles better than negotiating with (and waiting for) an Ops team to provision you hardware or networking that you don't have much visibility into and then relying on them to troubleshoot. I don't mind paying more for that (which you absolutely do on a per-unit of compute, network, or storage).

nologic01 3y ago

solving a paradox sometimes requires thinking out of the box repatriation is not the global solution, it applies to mature and large companies the dimension to explore is co-ownership of infrastructure by multiple independent companies, both small and large a captive entity - wholly owned by member companies - manages a vast pool of resources achieving both the economies of scale and the flexibility needed by growing companies. you don't "rent" somebody elses linux server, you invest and own a share. R&D for evolving the platform is something outsourced to third parties (maybe even current cloud providers)

alex_young 3y ago

Sub-linear scaling is powerful. Think about how costs scale in relation to use of a service. If you are paying someone else for the service as a package, the costs scale linearly, or some close approximation of that. AWS gives you a small discount if you have larger volumes for some services, but these are essentially a rounding error. If instead you launch a similar service in house, your costs are often dramatically lower per unit the more units you use. You start with a small team supporting a system with some large capacity you're not using yet because you need room to grow, and you realize large steps along the way where you can handle tens or hundreds of times the volume you see at each staffing level. Buying hardware or developing custom software gets cheaper at volume. This is sub-linear scaling.

andrewprock 3y ago

Any discussion of cloud vs. data center vs. on premises that does not discuss employee costs, documentation, and training is not complete. One of the main reasons for using cloud infrastructure is that many people know how to use it, internal documentation requirements are dramatically lessened, and you don't need to higher infrastructure IT employees to maintain and monitor it. If cloud only costs double internal IT, it's almost certainly cheaper until you reach a billion dollar valuation - or more depending on what your infra needs are.

hn_throwaway_99 3y ago

I think it's a bit nutty that folks always use Dropbox as the example of "cost savings by doing it yourself". I mean, Dropbox itself is essentially a provider of cloud storage. It seems like owning their own data centers was obvious from the beginning: Use AWS initially to scale, but given the size of Dropbox it obviously doesn't make sense to have another middleman in the mix. I just point this out because Dropbox is so unlike your average enterprise user of the cloud. I'd like to see some better examples than Dropbox of enterprises saving more (or innovating as quickly) by going on their own.

zinodaur 3y ago
I work at a company that recently tried to shift our (on prem, sharded, 2-metro, 6 replica) mysql deployment to AWS. Any way we sliced it, even after steep discounts, we were seeing massive TCO increases, often up to 10x - the math just doesn't work out for large deployments. It's really a shame that all these companies are forced to write their own (crappy) version of AWS - but price is a feature when it comes to this stuff.

locococo 3y ago

the benefit of the cloud is the elasticity. People neglect sometimes that businesses had to scale their infrastructure to peak demand before the cloud. If your workloads don't have peaks it's an easy problem to solve.

jmathai 3y ago
We had to purchase/lease HP storage servers with 10x the storage we needed for our photo startup in 2006. The lead time to get a server was weeks so there was very little elasticity to respond to non linear growth. Would have been a lot easier if we had S3 (launched around this time, no one really knew about it or what it was).

AndrewKemendo 3y ago

I’m leading engineering at a new start up and I’m doing absolutely everything I can to avoid touching the cloud at pretty much any cost. We’re going to be running all of our own dev/test GPU servers with only inference hosted outside our cluster.

jjn2009 3y ago
The need for GPUs is a solid reason to stay away.
- AndrewKemendo 3y ago
  Yeah it’s the main reason because the data transfer costs alone would be insane

efxhoy 3y ago

If moving off the cloud was easy we’d all be doing it. For our org the risk is (unfortunately imo) bigger than the savings. I would love to pay hetzner dedicated boxes rates for compute but the amount of work to migrate everything would be huge. Not to mention the difficulty in hiring people who are actually comfortable with doing the infra work of setting up all our services, with the required multi-AZ config, on bare metal. Its hard enough finding people who can reliably setup services in aws with terraform and keep them alive.

smartbit 3y ago

Previous May 27, 2021 https://news.ycombinator.com/item?id=27306742 https://news.ycombinator.com/item?id=27306742 207 comments Sept 14, 2022 https://news.ycombinator.com/item?id=32832433 https://news.ycombinator.com/item?id=32832433 16 comments