14 ms·

The cost of cloud, a trillion dollar paradox

198 points by StratusBen 5y ago

api 5y ago
This isn't surprising at all. This industry is incredibly fad driven. Cloud became the fad, and the buzz was that cloud saves money, so everyone goes cloud, and then cloud eventually costs more than the original stuff did. The same is happening with SaaS. SaaS eliminates the need for in-house IT! Except it doesn't. It just means you now have a bunch of recurring SaaS costs that you are locked into forever because they have your data and you still need IT people to babysit your massive cloud/SaaS sprawl. Not following fads and buzz is a huge competitive advantage in this industry. Founders and chief engineers / CTOs / CIOs take note. Just make sure you can explain why you are not using (insert latest buzzword here). The bottom line is that you should analyze the situation using your work load, your numbers, your culture, etc., and decide what works the best. Sometimes that's managed cloud. Sometimes it's unmanaged cloud. Sometimes it's bare metal. Sometimes it's on-prem. Your mileage will vary.
- bsder 5y ago
  > Founders and chief engineers / CTOs / CIOs take note. Just make sure you can explain why you are not using (insert latest buzzword here). 1) It takes amazing intestinal fortitude to fight back against the fad tide. And, when things go wrong, your decisions get the blame even if they aren't at fault. 2) As the article points out, cloud is FINE for startups and small companies. If my startup company reaches gigabucks in revenue and I now have to worry about $75 million in cloud spend, I've done my job. And then some. 3) Cloud is often about blame and liability transfer. I don't want the company website getting hacked to be my problem--I want it to be somebody else's problem. I'm willing to pay for that.
- dageshi 5y ago
  Cloud is just a big toolbox of premade tools and resources that you can play with to your hearts content with much of the friction taken out, both technical and bureaucratic. You pay a higher price for what you use than if you maintain those tools yourself... but you don't have to maintain those tools yourself. It's hardly a buzzword, it's almost defacto standard nowadays.
  - calvinmorrison 5y ago
    This. On prem required me to do so much ITIL work to get even a switch replaced, now I have a budget on azure I can do whatever the fuck I want
    - api 5y ago
      I think this is an important point. In many cases the benefit of cloud and SaaS is working around your IT department. It's a technical way of going around human management problems.
    - betaby 5y ago
      and then this happen https://twitter.com/pragmaticandy/status/1168916144121634818?lang=en https://twitter.com/pragmaticandy/status/1168916144121634818...
      - dageshi 5y ago
        sure, can happen to anyone. Still easier to shunt massive amounts of data between aws regions via amazons pipes than it is to do it yourself.
- void_mint 5y ago
  I feel like this assessment lacks a lot of nuance. I wouldn't really call "cloud" a "fad" as much as an overused option. > Cloud became the fad, and the buzz was that cloud saves money, so everyone goes cloud Cloud services _do_ save money for businesses of the appropriate size (meaning, those that can't or shouldn't be focusing on physical servers and networked hardware). > The same is happening with SaaS. SaaS eliminates the need for in-house IT! Again, SaaS _does_ eliminate the need for in-house IT _for certain classes of businesses_. > It just means you now have a bunch of recurring SaaS costs that you are locked into forever because they have your data and you still need IT people to babysit your massive cloud/SaaS sprawl. The cost of employing someone to babysit a SaaS product is much lower than the cost to employ someone to build and maintain an equivalent in-house SaaS product. If you're fine with vendor lock you're saving money. > Not following fads and buzz is a huge competitive advantage in this industry. Ignoring all nuance and claiming things that are popular are "just fads" is, to me, so much of a competitive disadvantage that it almost certainly outweights any perceived gain.
pintxo 5y ago
> [...] paradox: You’re crazy if you don’t start in the cloud; you’re crazy if you stay on it. > So what can companies do to free themselves from this paradox? As mentioned, we’re not making a case for repatriation one way or the other; rather, we’re pointing out that infrastructure spend should be a first-class metric. What do we mean by this? That companies need to optimize early, often, and, sometimes, also outside the cloud. When you’re building a company at scale, there’s little room for religious dogma.
kderbyma 5y ago
this isn't a paradox to me, this has always been the case.... big companies can optimize more. They have more non-optimal points due to sheet size alone
maerF0x0 5y ago
As I read it i couldnt help but think the following Yes, but the benefit of cloud is we only work to optimize those which gain market adoption. For every twilio there may be 100-1000 startups that did not make it, it's a good thing if those were constructed rapidly, tested for product market fit and then turned off without optimizations applied. Also if cloud adoption actually accelerates building, then it may also aid its adopters if there is a race to product market fit.
thinkingkong 5y ago
It isnt really a paradox at all. Its specifically the opportunity that companies like Oxide will go after. Basically the stage of your business determines the best course of action. Youd be insane to start with large datacenters and high capex for an undefined ROI for most projects. Similarly you dont understand your requirements or workloads yet, which would also affect the efficiency of your architecture. Whenever a rewrite happens in software there are usually massive performance gains. The same thing happens with infrastructure.
newintellectual 5y ago
"We show (using relatively conservative assumptions!) that across 50 of the top public software companies currently utilizing cloud infrastructure, an estimated $100B of market value is being lost among them due to cloud impact on margins — relative to running the infrastructure themselves." That completely overlooks the actual benefit and the reasons driving cloud adoption, which he states early on and then fails to integrate here. The real cost/benefit analysis would take into account the amount of time, money, and opportunity cost saved by using a cloud in development, a critical time that determines whether there'll actually be a profitable company eventually. Optimizing the cloud costs is certainly important, but being able to spin up highly integrated systems on demand offsets a huge amount of time and capital during development. A takeaway might be that, e.g., AWS, should offer even larger discounts for large scale operations in order to retain mature customers.
deleted 5y ago
[deleted]
peterbonney 5y ago
Am I alone in not seeing a paradox here? It's no different from a variety of things that make sense at some scales and not at others. Seems similar to office space - a 5 person company will find the economics of coworking space compelling, a 500 person company will do better with a long-term lease with a commercial landlord, and a 50,000 person company might have their own property management team in-house. That doesn't create a paradox in the commercial real estate market, just different solutions for different needs.
- void_mint 5y ago
  > Am I alone in not seeing a paradox here? It's no different from a variety of things that make sense at some scales and not at others. Yes, it's not a paradox at all. Use this thing until it doesn't make sense, and then don't use it anymore (or be comfortable with the lack of optimization). This is not a paradox.
  - lotsofpulp 5y ago
    I assume the article gets more attention if it uses the word paradox.
- dinoreic 5y ago
  I think point guy tried to make is, that ton of big companies, that would/could benefit from in-house solution still go for full cloud based one without thinking about alternatives.
- nine_k 5y ago
  The paradox is that if the coworking space were the cloud, it would just grow with you, and easily house a 50k-strong company, while you'd be paying the same (high) rate per employee. Why would this happen? Because what the company builds is a large and growing money-making contraption right inside the coworking space, also using parts of the building for critical functions, and the door is intentionally kept small. The only way to move that contraption is to dismantle it and reassamble in a much cheaper, purpose-built hangar, rebuilding some of the critical parts along the way. During all that time, the contraption would stop making money. That's the beauty of AWS business model: it's a no-brainer for a startup to use it, but the startup grows, more financially efficient infrastructure options become unattainable because of the very high cost of the move. This is the best-executed vendor lock-in I know.
StratusBen 5y ago
Disclaimer: I'm a Co-Founder at Vantage, a cloud cost platform. I also worked in public cloud for ~6 years at AWS, DigitalOcean, etc. While repatriation can make sense at a larger scale company, startups and SMBs can yield the same benefits discussed in this post by simply tracking and optimizing cloud spend. We try to make this as easy as possible for people with https://www.vantage.sh/ https://www.vantage.sh/ - where we're already helping thousands of individuals, startups, SMBs and enterprises as it relates to AWS.
gopalv 5y ago
In 2012, I was working on a migration from EC2 to on-prem & was working on infrastructure building for zCloud. The cost factor was huge, the switch out of EC2 literally paid for itself in a lot of ways. The work was very interesting, because a lot of it was actually building a private cloud for internal customers & the work primarily centered around a virtualized data-center aimed at boom-bust cycles of games (15+ million users for 6 weeks, drop to 2 million for a month and down to a million in another week). The issue is that the infrastructure cost is somewhat constant when dealing with that sort of fluctuations in revenues, so the cost to revenue ratio was unpredictable (while the cost was). So what happened in the end looked a lot like a fire-sale of hardware when the cost was unbearable, while if it was an end-user cloud, that low-demand phase would be able to cut losses as a spot instance or something. Anyway, a few years after I left, back to EC2 it is[1]. [1] - https://aws.amazon.com/solutions/case-studies/zynga/ https://aws.amazon.com/solutions/case-studies/zynga/
- jsnell 5y ago
  Why is the infrastructure cost constant? I would have thought that for gaming, at least compute and transit would scale linearly with traffic. If games have a boom bust cycle, aren't they the perfect use case for public clouds?
  - gopalv 5y ago
    The infra was on-prem, so it was planned in advance and ops folks who had on-call rotations every day etc & that was rented out by the hour to the game teams (a chargeback model). The salaries and hardware cost was paid for even if the games had a bust. The period where it worked well, the games had boom-bust in somewhat controlled fashion where farmville -> cityville -> frontierville -> fishville etc, the traffic would move around rather than die down entirely. The world turned mobile-heavy and that whole pipeline fell apart while they were restructuring into mobile games (words with friends etc), when the hardware had to be sold or the payments would start to hurt.
    - jsnell 5y ago
      Ah, I think I misread your original post. I thought you were saying that the infra costs were constant on EC2 too.
jandrewrogers 5y ago
This has been widely known for many years. The crossover point happens much sooner than I think people intuit but most companies never really measure or model it. My experience at a few different companies is that DIY infrastructure pencils out at 30-40% of the cost of the cloud. There is a learned helplessness when it comes to companies running their own data centers that has become widespread over the last decade. What used to be a fairly mechanical process has almost been mythologized as some kind of arcane art beyond the technical ability of any company that isn't Google or Amazon. Designing data center builds isn't difficult, it is a pretty straightforward albeit detail-oriented blue-collar engineering skill, but it seems few people learn it anymore.
- dilyevsky 5y ago
  Yea pretty soon we’ll have infra engs that had gone through their whole career without ever working a real server rack. I personally find that thought discomforting
  - scarface74 5y ago
    I think people who started out as assembly language programmers thought the same thing. Everything gets abstracted at some point.
    - oblio 5y ago
      No punch cards?!?
    - dilyevsky 5y ago
      The problem with that comparison is I don’t have to pay big co to be able to write code in C++
      - scarface74 5y ago
        And your on prem infrastructure people are free?
bcantrill 5y ago
If it needs to be said, this is more or less exactly the thesis behind Oxide[0], and matches what we are seeing in the market. It's certainly validating to see a VC firm echo our pitch deck back to us, even if one that (in)famously doesn't believe in hardware! ;) [0] https://news.ycombinator.com/item?id=27294471 https://news.ycombinator.com/item?id=27294471
- mtnygard 5y ago
  Congrats on the launch of your product! It's damn impressive.
- SiVal 5y ago
  Yes, but what I'm not seeing in their analysis of variable-cost cloud vs cheaper fixed-cost colo is the huge fixed cost of the personnel to manage it. Yes, I agree that switching my food-delivery business from calling for an Uber whenever I get an order to buying my own car can be much cheaper, but if I leave out the cost of hiring a full-time driver for my new car, my analysis is...incomplete. Your pitch emphasizes features that make servers easier to manage, presumably lowering (but not eliminating) the cost of personnel, so you're obviously aware of this, but I'm not seeing any "net of additional, fully-loaded personnel costs" in their analysis. It should still be cheaper above a certain scale, but the breakeven where it will make sense to "repatriate infrastructure" is much higher if you include hiring people, and will go even higher if the cloud providers run the numbers and match most of the savings with scale-based price breaks.
  - kelp 5y ago
    I don't know how every company does it. But I led the datacenter and networking teams at Square from 2011 to 2017. In that time we went from 1 DC cage with 4 server racks to 4 US DCs, one in a Japan and a couple of network pops. A bit more than 100 server racks total. My team had 2 guys doing SiteOps. They would travel to the various DCs in the Bay Area and Virginia and do all the maintenance, new installs, etc. And sometimes we'd lean on the colocation remote hands to do a few things. We had about 5 network engineers, that also handled the corporate network. (12 offices and a network backbone that connected east and west coast DCs, offices, etc). And maybe 2 SWEs who handled things like our host OS install system, etc. Basically the next layer above the hardware. So 9 people that were required to run all of that stuff. But really, NetEng ended up spending like 70% of their time on corporate network things because we'd add offices faster than datacenters. So if we focused on production only, we really needed about 6 or 7 people total. I did the math a few times (every single year) and compared our costs, including people, to the costs of moving to AWS 3 year reserve instances. Doing it ourselves was always half the price. Of course the difference here was we built on-prem from the start. So there was no repatriation that had to happen. Since then, I've been responsible for large cloud infra on all 3 major providers and learned a lot about what kind of discounts you can get when you're in the double digit millions in annual spend. I still think, at a scale of single digit thousands of servers, you'd be cheaper on-prem, fully loaded. But admittedly, I haven't run the numbers since 2017.
fnord77 5y ago
technologies like kubernetes make repatriation even simple in some cases.
- deanCommie 5y ago
  Except you miss out on all the opportunity cost of cloud-native solutions that get you moving fast for the sake of a complex convoluted mess that you need to hire subject matter experts for to wrangle, instead of focusing on your business logic. Kubernetes only makes sense from Google's perspective to sell you the managed version "from the creators of Kubernetes!" once you get tired of trying to wrestle it into submission.
  - gravypod 5y ago
    I don't think you need to "wrestle it into submission". There are many CNCF projects that work great on bare metal instances like Rook and other companies providing managed DB solutions for your hardware like KubeDB. K8s isn't that complex from a cluster operators perspective if you've managed bare metal Linux systems before. Set up some PXE + auto provisioning and you can make your node enrollment process: 1. Buy hardware from an OEM. 2. OEM ships you a piece of hardware. 3. Remote hands plug in hardware. 4. Machine PXEs, boots, and is turned into a worker on your cluster. Canonical (MaaS), VMware (vSphere), and Rancher Labs (Rancher) all have products aimed at running Kube on metal in your own datacenter. So, essentially, if you start your startup off using Kube in a managed cluster you can, at some point in time later, directly move your code off the cloud and onto your own machines for cost saving. You just need to figure out how to run your DBs and LBs but that's a solved problem as mentioned above.
  - fnord77 5y ago
    akshually we use Amazon's managed kubernetes. k8s lets us move fast - we can easily deploy new services in no time. We can also run the entire stack on our desktops in k8s, or have our laptops be a node in a mostly cloud k8s dev deployment and debug services locally.
devops000 5y ago
I don’t think is a paradox, is a standard “make or buy” decision which is different based on your company’s size and growth phase.
SirOibaf 5y ago
A bit of a weird article coming from a VC heavily invested in SaaS providers
- wmf 5y ago
  They're basically talking about helping SaaS providers reduce their IaaS spending. They're not talking about migrating off SaaS.
trjordan 5y ago
The opportunity cost is staggering. Not just of the people that do the work, but the cost of focus and top-level priorities over the years it takes to do the work. Everybody loves to cite DropBox here. The greater arc of DropBox is their product stalled, they lost the enterprise of their market to Box, and they found themselves a commodity in a commodity market. Heck, maybe they'll go back to Amazon if they keep doing things like this: https://aws.amazon.com/solutions/case-studies/dropbox-s3/ https://aws.amazon.com/solutions/case-studies/dropbox-s3/ But that's not all. It takes a top-level directive to repatriate an entire SaaS. That's at the cost of other top-level projects. It's wild to me that any company that has significant fuel left in the tank would buy back single-digit COGS percentages instead of investing in product that could add double-digit growth for several years at scale.
- redis_mlc 5y ago
  > things like this: https://aws.amazon.com/solutions/case-studies/dropbox-s3/ https://aws.amazon.com/solutions/case-studies/dropbox-s3/ Yeah but why do they have 34 PB of analytics data?
  - jeffbee 5y ago
    The only question worth asking right there. It's the story of so many companies. We were doing something incredibly stupid, and in this five-part blog series we describe a slightly different implementation of an equally stupid idea.
    - redis_mlc 5y ago
      That was my observation as a DBA. The most powerful tools we have are aggregated reports and data retention periods, the same since the 1970s. Running 10 copies of the same report (but each one with slightly different bugs) across 34 PB of data for 10 different departments is absurd, yet here we are. Specifically for Dropbox, they're not a retailer like Safeway with 10,000 products and 100's of locations to crunch product and shipping data for - Dropbox could use a single server for their entire digital products data warehouse. (I operated a data warehouse myself (literally) at Yahoo - 10 servers generating canned web reports for 500 managers. I heard they replaced it with a 1,000 node Hadoop cluster later.)
mtnygard 5y ago
Not trying to peddle cheap skepticism here. This article only looks at "seen" costs, and assumes that there are no "unseen" costs to running on-prem. Many companies do not have the operational maturity to run on-prem well. The result: high cost of operations, low availability, and large increase in time-to-value. Second unseen cost: everybody becomes their own SI. So far nobody really sells the "whole stack" for running on prem. I mean hardware, network, virtualization, application, traffic mgmt, etc. I have to buy stuff from two dozen different vendors and cobble it together into a high-labor, rickety Jenga tower of stuff.
- siddarthd2919 5y ago
  You nailed it. Unseen costs are huge and a lot of people ignore it.
  - deleted 5y ago
    [deleted]
- jandrewrogers 5y ago
  These costs are not unseen. They are straightforward ordinary costs and fully accounted for. It only seems expensive and complicated from the perspective of someone that does not have experience with it. To someone that has done it many times before it is relatively simple and almost completely mechanical to get an excellent result. It isn't rocket science, just domain expertise around things like physical infrastructure planning and supply chains that a software developer may not have encountered before but could easily learn. If a company was going to run their own data centers, they would presumably hire someone that knows what they are doing to lead the effort instead of trying to do it by trial and error.
  - oblio 5y ago
    > Many companies do not have the operational maturity to run on-prem well.
    - 5y ago
paxys 5y ago
This analysis completely skips over the "elastic" aspect of cloud infrastructure, which was a key motivator for using these providers since the very beginning and is as relevant today regardless of company size. My company (which is in fact part of the charts in the article) had every single metric across the board spike 15-20x basically overnight when the pandemic started last year in March. Our entire infrastructure burden was clicking a few buttons on the AWS console and making sure everything was provisioning and scaling as needed. If we had to send out people to buy hard drives and server racks at that time, there is no chance we would have been able to meet the extra demand. Plus, if you give me a few dozen capable engineers today I'm not going to waste their efforts on rebuilding AWS to get a best-case few percentage point return on our cloud spend. I'll launch a new product for our customers instead.
- LimaBearz 5y ago
  That’s a valid point. I don’t want to argue pedantry but I’ll say I worked some media companies that experience 5x normal volumes a few times a year during events before returning to “normal”, making AWS an obvious choice. I’ve also worked for a place so large in AWS we hit walls where we hit Amazon’s literal physical limit and had wait for them to go out and buy the hardware for us to provision. For Dropbox specifically I don’t see them being any form of special case, if it saves money and they obviously had operational experience so it makes sense
- JohnJamesRambo 5y ago
  Are you wildly overpaying though for something that is a once in a century (hopefully) event like a pandemic that shuts down the whole world?
matchagaucho 5y ago
From a Moore's Law perspective I'd like to see a true cost of ownership over time, as most infra goes obsolete in 18-24 months. Public clouds, like AWS, have cut their storage costs by more than 50% since Dropbox built their own infra in 2015.
- kelp 5y ago
  In my experience it's pretty common to stretch your infra out much longer than 18-24 months. Often finance has a 3 year depreciation schedule, and depending on your needs, you can keep existing workloads on existing hardware for many more years. The last time I ran physical infra, we had sever racks that I'd originally bought, running in production 5+ years later. Once you're past the depreciation schedule, they are basically free except for power and space. Yeah, at some point and scale, it can make sense to get new gear that is more power and space efficient to pack more into the same power and cooling budget. AWS still lets me launch c1-c6 instance types. So they still have the older generations sticking around. Yeah, the newer ones are usually more cost effective, but you do have to do work to migrate to them.
sanj 5y ago
This may be a naive question, but... Given that interactions to the cloud are through a relatively small set of known APIs, I'm surprised that there isn't there a service which replicates the APIs but with the endpoints being on "repatriated" hardware.
- kenniskrag 5y ago
  terraform does that. https://www.terraform.io/ https://www.terraform.io/
  - sanj 5y ago
    That’s clever, but not what I’m suggesting. I’m looking for a mechanism which apes existing cloud APIs directly.
    - e12e 5y ago
      Well, the thing is they're not all that similar (except maybe s3 which has been copied by competition). So terraform or similar abstractions (from pulumi to chef/ansible) are what you get. Or k8s. But there's also: https://www.eucalyptus.cloud/ https://www.eucalyptus.cloud/
- maxgashkov 5y ago
  APIs are not what's hard IMO. Creating a layer simulating S3 API is pretty straightforward. Designing and implementing a highly available and resilient storage solution on-prem is much more harder task that will take 99% of time & budget to do properly compared to a compatibility layer.
spoonjim 5y ago
I couldn’t tell, which onprem portfolio company are they pushing here?
throwawaaarrgh 5y ago
These people don't see the real value proposition of the cloud. The value is not in "scaling when you need to". Sure, that can be very handy. But that is not what 99% of companies are getting out of the cloud 99% of the time. If you self-manage, your capital investment is initially higher, and lower over time. At the same time, the effort it takes to reach the same results is always higher, and the quality of the end product may be lower, depending on how much service quality affects your product. If you pay for managed services, your initial investment is lower, and higher over time. But at the same time, you require less effort, and you get higher quality outcomes. This is obvious to anyone who has worked in the industry and done both. First, host your own service: JFrog Artifactory, Atlassian Confluence, GitLab, whatever. Now rapidly increase the demands on this service. As demand rises, quality will decrease, because it takes a lot of time, effort and expertise to build a very reliable hosted service. Now switch to a managed cloud instance. Suddenly, the service's average quality increases. Performance is steady regardless of increase in use. There are virtually no interruptions to your product or development. The impact of a service's quality and reliability has ripple effects. If poor service quality slows down development, that means development quality will go down as people cut corners to try and meet deadlines. If the service is used for production, it means your product's quality will suffer, and that effects your bottom line. So a huge amount of the actual cost is not just paying for a service, but also how much business value is generated or lost due to service quality. There is simply no way to replicate a managed service without becoming a managed service provider yourself. You have to become a whole new business within a business. It's like a yogurt company also becoming a dairy farm. Running a farm is not easy, and you will screw it up for several years. Seems obvious for farming, but for some reason people always underestimate this when it comes to technology. On paper, the Cloud's value proposition is scalability. But in practice, the true value is actually as a force-multiplier for your product's quality, reliability, and time to market. (Time to market not just being "I launched my startup" but also "I released this new feature before my competitor")
- kelp 5y ago
  Having done both, I think I mostly agree with you. And I'm a big proponent of having your engineering team focus their efforts on building and maintaining the product you are selling to customers. Avoid doing undifferentiated work. Outsource it when you can. Focus on the core product. I think this is especially true as you're scaling a company in the current SWE labor environment. If your business is growing quickly, you probably have money, and can get more funding when you need it. But hiring more SWEs is always a challenge. That is your scarce resource. Don't waste it on undifferentiated work. All that said, at larger scale and more mature businesses. I think the core thesis of the article makes sense. You may be in the slower growth part of the S curve, but you can still increase your company value and profitability substantially by increasing margins. One way do do this is to invest in cheaper infrastructure. Then you're taking on some undifferentiated work, but in service of better margins.
akh 5y ago
> By tracking cloud spend, the company enables engineers, and not just finance teams, to take ownership of cloud spend. > tie the pain directly to the folks who can fix the problem That's why we're building https://github.com/infracost/infracost https://github.com/infracost/infracost for engineering teams (free open source)
deleted 5y ago
[deleted]
liminal 5y ago
I think they understated the importance and difficulty of retaining a sufficiently redundant skilled workforce to manage the equivalent cloud infrastructure.
lowbloodsugar 5y ago
"But that’s just Dropbox." The implication here being, "Well, if Dropbox can save this much, then think about how much everyone else cans save!" But in fact the opposite is true. Dropbox sells disk storage on the cloud. For them to do so by effectively reselling someone else's disk-storage-on-the-cloud platform is obviously not high margin, and they'd be better off building it themselves. So, sure. Anyone else also offering, disk storage on the cloud, or compute clusters on the cloud, or otherwise just reselling someone else's product on the cloud, will probably have higher margins doing it themselves. But that is certainly not most companies. So, yeah, it's "just Dropbox". Disclaimer: I work for a cloud provider, but these opinions are my own.
lifeisstillgood 5y ago
One benefit of "the cloud" not mentioned here is elapsed time to acquire a new instance If you have moved to AWS / other then that time is around 5 minutes. If you are in a major fortune 500 and need a new server, quite often that time will measure in months (yes really). This simple equation just blows every other cost/benefit calculation out of the water. I may have missed it in the discussion.
- toomuchtodo 5y ago
  If you need that server today, absolutely, build an AMI and fire it up, and then place an order for a server to migrate to. It’s not binary, and it is possible between engineering and finance for balancing velocity and spend. Not everyone needs a server immediately. And not every use case will generate business value by having that server immediately available. This blog post cuts to the core of that: cloud gets you from product market fit to scale, but once at scale, it’s time to revisit capex vs opex.
- kelp 5y ago
  Yes and, it depends. Last time I was procuring racks of servers, it was about a 6 week lead time, sometimes more. So yeah, you have to do a lot more capacity planning, and probably keep capacity in reserve for things that are unexpected. I've had to go around to every engineering team and try to get them to tell me how much gear they might need in the next 1-2 quarters. They rarely know, so it's a lot of guess work. So in a situation with a quickly growing product, that grows in an unpredictable way. Or a really new product where you just have no idea what the adoption is going to look like. Then yeah, getting those new instances right now is really really important. At Segment we very often needed more instances in a huge hurry due to some customer load spike. And sometimes AWS ran out of the instances we needed in the region we were in. So we had to get creative to use other instance types. Also the cloud provider pricing structure heavily incentives you to actually do some capacity planning. On AWS, Savings Plan gives you huge savings. But if your workload is fairly predictable, then this on demand benefit isn't nearly as compelling and at the right scale, it makes sense to start buying servers to put in your own datacenter.
- jandrewrogers 5y ago
tibiahurried 5y ago
Many underestimate the complexity and the headache that comes with managing its own data center. It is easy to get a couple of racks up and running. Then you have to deal: - HW that fails, go get a new one, cost + time - backup and geographical redundancy - security - certifications - tooling to manage and forecast - black friday! scale for few days ... more HW? - etc ... It is way more convenient for the average enterprise to pay premium and be done with it. So they can focus on the actual business and not building infrastructure and tooling to manage it.
sbazerque 5y ago
It's not purely a technical matter, there are different market dynamics at play as well. If you go cloud, you get a handful of very large suppliers, that provide a lot of non-standardized services that are probably going to lock you in like hell (as the article wel says). If you build infra, you're using /mostly/ commoditized supplies and skills. If the cloud offerings where interchangeable and the industry reasonably fragmented, the margins of cloud providers would be slimmer and the paradox would probably go away (in favor of: always cloud!). See for example Porter's analysis framework [1] and how your positioning changes in the two cases. https://en.wikipedia.org/wiki/Porter%27s_five_forces_analysis https://en.wikipedia.org/wiki/Porter%27s_five_forces_analysi...
Animats 5y ago
Linden Lab, the people behind Second Life, may have made this mistake. They had leased data center space in Phoenix AZ, and owned their own machines, about 5000 of them. During 2020, the whole system was moved to AWS. Most of the servers, each of which handles an area of the virtual world, run all the time, regardless of user load. This is very different than web serving. It's the worst cost case for AWS, where the real benefits come from dynamically provisioning a changing load. AWS, used for dedicated servers, can cost 2x that of owning your own. One data center operator says 4x.[1] Before the conversion, I'd asked the responsible VP if they were sure AWS wasn't going to be more expensive. He retired when the conversion was complete. Now we hear rumors of the company finding out that AWS costs are higher than the old data center. [1] http://www.happi.io/aws-instances-versus-physical-servers/ http://www.happi.io/aws-instances-versus-physical-servers/
- JohnJamesRambo 5y ago
  I'll ask the obvious, did no one do the math on the expected difference?
  - Qworg 5y ago
    Someone likely did. They likely also retired, per the parent comment.
  - jjoonathan 5y ago
    Someone did the math of "will this help my resume," got the answer "yes," and then dropped some numbers in a spreadsheet to convince their manager to let them do it.
  - TechBro8615 5y ago
    I’m sure they did. But inevitably, someone “doing the math” on this will inject their own baseless assumptions when estimating the labor cost of maintaining an inventory of colocated or dedicated servers. I might even wager that such a comment will appear as a sibling to this one within the next 30 minutes.
    - 5y ago
ram_rar 5y ago
Cloud is not just virtualized compute, storage and network (csn) anymore. Most of the services in AWS/GCP offer a lot more than just virtualized csn. I dont think, the article factors in the cost of running a service. Its one thing to rent csn and run your own database. Its completely another to use a cloud service like dynamodb/bigtable etc. I agree, opex cost can go haywire, unless kept in check. But, its not an apples to apple comparison.
phs318u 5y ago
I think one of the benefits of cloud is (in theory) the agility that it brings. So much is abstracted away that you truly can start to focus on what's more important to your business. There's a cost to that value. So for example, while the cost to serve may be lower with your own infrastructure, it may be that the opportunity costs go up and your ability to grow that revenue stream or customer base is diminished without cloud. Another thing that came to mind reading this was - what would I need to make such a switch (from cloud to your own data centre)? And then I thought, you could provide a business doing this - commoditised hardware and services, providing the real-estate and the engineers, and more importantly, providing the same kind of service stack you find in the cloud. Then I thought, that could be a cool business for someone. And then I thought - there's nothing stopping Amazon or Microsoft or Google from offering that. I guess if you have a very clear understanding of your needs, a mature appreciation for what it will take, and a good handle on your likely growth, then taking control of it all may make sense. But for a lot of business (especially those not actually tech businesses per se), it would more likely be a return to pain.
smitty1e 5y ago
> Because this shift happens later in a company’s life, it is difficult to reverse as it’s a result of years of development focused on new features, and not infrastructure optimization. Little Miss Muffet Sits on her tuffet Watching her billing grow Her devs ever hectoring On the need for refactoring But what do these uni-brows know?
smitty1e 5y ago
Is this really a paradox? When you're small, outsource. Should the company find traction, and as you gain size, the arguments in favor of outsourcing fade, and bringing the IT in-house makes sense.
alessandroetc 5y ago
I see services like Storj the perfect hybrid model here. Companies get the security and optimizations of the cloud, while being able to simultaneously run their own storage nodes to get cost savings. Exciting times for decentralized storage!
IOT_Apprentice 5y ago
What should be considered RIGHT NOW, is that given the supply chain issues, if you are NOT on a cloud platform, good luck getting new hardware from major vendors. With AWS/Azure you can provision immediately. If you have some old hardware whose performance is impacting your production environment given the dependency on it, good luck. Firms like Dell are already stating this.
naveen99 5y ago
If cloud margins are really 50%, that seems like a case for new competition in the cloud business rather than making a data center for your own non cloud business.
whoisjuan 5y ago
This is not surprising. It's kind of obvious that cloud costs will scale more dramatically than any other cost center. So it's really an attractive idea to think that you can cut your largest bill by going on-prem. The problem with going on-prem is that for most companies this is so complex and hard to operationalize that they will likely just fail at that attempt. Cloud biggest contribution besides seamless infrastructure access, is the way it has lower the overall technical competency threshold to operate an internet business. It's really easier to find someone that knows AWS or Azure than is to find that someone who knows how to build on-prem infrastructure. I imagine that repatriation for the average large cloud business will only be possible for those who are already operating in modern infrastructure constructs like Kubernetes. If you have critical workflows build on propietary cloud services, you're probably fucked. You will have to rebuild every single service from scratch and make it scalable out of the box. If you're a multi-dimensional business that sounds like a nightmare. I honestly think that Dropbox was capable of doing it just because the nature of their business. At the primitive level you're mostly replacing your storage vendor. It happens to be that storage is their business model so it's quite evident that repatriating that critical piece of your business will yield great gains. I don't think it's that simple for other cloud businesses.

198 points by StratusBen 5y ago

api 5y ago

This isn't surprising at all. This industry is incredibly fad driven. Cloud became the fad, and the buzz was that cloud saves money, so everyone goes cloud, and then cloud eventually costs more than the original stuff did. The same is happening with SaaS. SaaS eliminates the need for in-house IT! Except it doesn't. It just means you now have a bunch of recurring SaaS costs that you are locked into forever because they have your data and you still need IT people to babysit your massive cloud/SaaS sprawl. Not following fads and buzz is a huge competitive advantage in this industry. Founders and chief engineers / CTOs / CIOs take note. Just make sure you can explain why you are not using (insert latest buzzword here). The bottom line is that you should analyze the situation using your work load, your numbers, your culture, etc., and decide what works the best. Sometimes that's managed cloud. Sometimes it's unmanaged cloud. Sometimes it's bare metal. Sometimes it's on-prem. Your mileage will vary.

bsder 5y ago
> Founders and chief engineers / CTOs / CIOs take note. Just make sure you can explain why you are not using (insert latest buzzword here). 1) It takes amazing intestinal fortitude to fight back against the fad tide. And, when things go wrong, your decisions get the blame even if they aren't at fault. 2) As the article points out, cloud is FINE for startups and small companies. If my startup company reaches gigabucks in revenue and I now have to worry about $75 million in cloud spend, I've done my job. And then some. 3) Cloud is often about blame and liability transfer. I don't want the company website getting hacked to be my problem--I want it to be somebody else's problem. I'm willing to pay for that.
dageshi 5y ago
Cloud is just a big toolbox of premade tools and resources that you can play with to your hearts content with much of the friction taken out, both technical and bureaucratic. You pay a higher price for what you use than if you maintain those tools yourself... but you don't have to maintain those tools yourself. It's hardly a buzzword, it's almost defacto standard nowadays.
- calvinmorrison 5y ago
  This. On prem required me to do so much ITIL work to get even a switch replaced, now I have a budget on azure I can do whatever the fuck I want
  - api 5y ago
    I think this is an important point. In many cases the benefit of cloud and SaaS is working around your IT department. It's a technical way of going around human management problems.
  - betaby 5y ago
    and then this happen https://twitter.com/pragmaticandy/status/1168916144121634818?lang=en https://twitter.com/pragmaticandy/status/1168916144121634818...
    - dageshi 5y ago
      sure, can happen to anyone. Still easier to shunt massive amounts of data between aws regions via amazons pipes than it is to do it yourself.
void_mint 5y ago
I feel like this assessment lacks a lot of nuance. I wouldn't really call "cloud" a "fad" as much as an overused option. > Cloud became the fad, and the buzz was that cloud saves money, so everyone goes cloud Cloud services _do_ save money for businesses of the appropriate size (meaning, those that can't or shouldn't be focusing on physical servers and networked hardware). > The same is happening with SaaS. SaaS eliminates the need for in-house IT! Again, SaaS _does_ eliminate the need for in-house IT _for certain classes of businesses_. > It just means you now have a bunch of recurring SaaS costs that you are locked into forever because they have your data and you still need IT people to babysit your massive cloud/SaaS sprawl. The cost of employing someone to babysit a SaaS product is much lower than the cost to employ someone to build and maintain an equivalent in-house SaaS product. If you're fine with vendor lock you're saving money. > Not following fads and buzz is a huge competitive advantage in this industry. Ignoring all nuance and claiming things that are popular are "just fads" is, to me, so much of a competitive disadvantage that it almost certainly outweights any perceived gain.

pintxo 5y ago

> [...] paradox: You’re crazy if you don’t start in the cloud; you’re crazy if you stay on it. > So what can companies do to free themselves from this paradox? As mentioned, we’re not making a case for repatriation one way or the other; rather, we’re pointing out that infrastructure spend should be a first-class metric. What do we mean by this? That companies need to optimize early, often, and, sometimes, also outside the cloud. When you’re building a company at scale, there’s little room for religious dogma.

kderbyma 5y ago

this isn't a paradox to me, this has always been the case.... big companies can optimize more. They have more non-optimal points due to sheet size alone

maerF0x0 5y ago

As I read it i couldnt help but think the following Yes, but the benefit of cloud is we only work to optimize those which gain market adoption. For every twilio there may be 100-1000 startups that did not make it, it's a good thing if those were constructed rapidly, tested for product market fit and then turned off without optimizations applied. Also if cloud adoption actually accelerates building, then it may also aid its adopters if there is a race to product market fit.

thinkingkong 5y ago

It isnt really a paradox at all. Its specifically the opportunity that companies like Oxide will go after. Basically the stage of your business determines the best course of action. Youd be insane to start with large datacenters and high capex for an undefined ROI for most projects. Similarly you dont understand your requirements or workloads yet, which would also affect the efficiency of your architecture. Whenever a rewrite happens in software there are usually massive performance gains. The same thing happens with infrastructure.

newintellectual 5y ago

"We show (using relatively conservative assumptions!) that across 50 of the top public software companies currently utilizing cloud infrastructure, an estimated $100B of market value is being lost among them due to cloud impact on margins — relative to running the infrastructure themselves." That completely overlooks the actual benefit and the reasons driving cloud adoption, which he states early on and then fails to integrate here. The real cost/benefit analysis would take into account the amount of time, money, and opportunity cost saved by using a cloud in development, a critical time that determines whether there'll actually be a profitable company eventually. Optimizing the cloud costs is certainly important, but being able to spin up highly integrated systems on demand offsets a huge amount of time and capital during development. A takeaway might be that, e.g., AWS, should offer even larger discounts for large scale operations in order to retain mature customers.

deleted 5y ago

[deleted]

peterbonney 5y ago

Am I alone in not seeing a paradox here? It's no different from a variety of things that make sense at some scales and not at others. Seems similar to office space - a 5 person company will find the economics of coworking space compelling, a 500 person company will do better with a long-term lease with a commercial landlord, and a 50,000 person company might have their own property management team in-house. That doesn't create a paradox in the commercial real estate market, just different solutions for different needs.

void_mint 5y ago
> Am I alone in not seeing a paradox here? It's no different from a variety of things that make sense at some scales and not at others. Yes, it's not a paradox at all. Use this thing until it doesn't make sense, and then don't use it anymore (or be comfortable with the lack of optimization). This is not a paradox.
- lotsofpulp 5y ago
  I assume the article gets more attention if it uses the word paradox.
dinoreic 5y ago
I think point guy tried to make is, that ton of big companies, that would/could benefit from in-house solution still go for full cloud based one without thinking about alternatives.
nine_k 5y ago
The paradox is that if the coworking space were the cloud, it would just grow with you, and easily house a 50k-strong company, while you'd be paying the same (high) rate per employee. Why would this happen? Because what the company builds is a large and growing money-making contraption right inside the coworking space, also using parts of the building for critical functions, and the door is intentionally kept small. The only way to move that contraption is to dismantle it and reassamble in a much cheaper, purpose-built hangar, rebuilding some of the critical parts along the way. During all that time, the contraption would stop making money. That's the beauty of AWS business model: it's a no-brainer for a startup to use it, but the startup grows, more financially efficient infrastructure options become unattainable because of the very high cost of the move. This is the best-executed vendor lock-in I know.

StratusBen 5y ago

Disclaimer: I'm a Co-Founder at Vantage, a cloud cost platform. I also worked in public cloud for ~6 years at AWS, DigitalOcean, etc. While repatriation can make sense at a larger scale company, startups and SMBs can yield the same benefits discussed in this post by simply tracking and optimizing cloud spend. We try to make this as easy as possible for people with https://www.vantage.sh/ https://www.vantage.sh/ - where we're already helping thousands of individuals, startups, SMBs and enterprises as it relates to AWS.

gopalv 5y ago

In 2012, I was working on a migration from EC2 to on-prem & was working on infrastructure building for zCloud. The cost factor was huge, the switch out of EC2 literally paid for itself in a lot of ways. The work was very interesting, because a lot of it was actually building a private cloud for internal customers & the work primarily centered around a virtualized data-center aimed at boom-bust cycles of games (15+ million users for 6 weeks, drop to 2 million for a month and down to a million in another week). The issue is that the infrastructure cost is somewhat constant when dealing with that sort of fluctuations in revenues, so the cost to revenue ratio was unpredictable (while the cost was). So what happened in the end looked a lot like a fire-sale of hardware when the cost was unbearable, while if it was an end-user cloud, that low-demand phase would be able to cut losses as a spot instance or something. Anyway, a few years after I left, back to EC2 it is[1]. [1] - https://aws.amazon.com/solutions/case-studies/zynga/ https://aws.amazon.com/solutions/case-studies/zynga/

jsnell 5y ago
Why is the infrastructure cost constant? I would have thought that for gaming, at least compute and transit would scale linearly with traffic. If games have a boom bust cycle, aren't they the perfect use case for public clouds?
- gopalv 5y ago
  The infra was on-prem, so it was planned in advance and ops folks who had on-call rotations every day etc & that was rented out by the hour to the game teams (a chargeback model). The salaries and hardware cost was paid for even if the games had a bust. The period where it worked well, the games had boom-bust in somewhat controlled fashion where farmville -> cityville -> frontierville -> fishville etc, the traffic would move around rather than die down entirely. The world turned mobile-heavy and that whole pipeline fell apart while they were restructuring into mobile games (words with friends etc), when the hardware had to be sold or the payments would start to hurt.
  - jsnell 5y ago
    Ah, I think I misread your original post. I thought you were saying that the infra costs were constant on EC2 too.

jandrewrogers 5y ago

This has been widely known for many years. The crossover point happens much sooner than I think people intuit but most companies never really measure or model it. My experience at a few different companies is that DIY infrastructure pencils out at 30-40% of the cost of the cloud. There is a learned helplessness when it comes to companies running their own data centers that has become widespread over the last decade. What used to be a fairly mechanical process has almost been mythologized as some kind of arcane art beyond the technical ability of any company that isn't Google or Amazon. Designing data center builds isn't difficult, it is a pretty straightforward albeit detail-oriented blue-collar engineering skill, but it seems few people learn it anymore.

dilyevsky 5y ago
Yea pretty soon we’ll have infra engs that had gone through their whole career without ever working a real server rack. I personally find that thought discomforting
- scarface74 5y ago
  I think people who started out as assembly language programmers thought the same thing. Everything gets abstracted at some point.
  - oblio 5y ago
    No punch cards?!?
  - dilyevsky 5y ago
    The problem with that comparison is I don’t have to pay big co to be able to write code in C++
    - scarface74 5y ago
      And your on prem infrastructure people are free?

bcantrill 5y ago

If it needs to be said, this is more or less exactly the thesis behind Oxide[0], and matches what we are seeing in the market. It's certainly validating to see a VC firm echo our pitch deck back to us, even if one that (in)famously doesn't believe in hardware! ;) [0] https://news.ycombinator.com/item?id=27294471 https://news.ycombinator.com/item?id=27294471

mtnygard 5y ago
Congrats on the launch of your product! It's damn impressive.
SiVal 5y ago
Yes, but what I'm not seeing in their analysis of variable-cost cloud vs cheaper fixed-cost colo is the huge fixed cost of the personnel to manage it. Yes, I agree that switching my food-delivery business from calling for an Uber whenever I get an order to buying my own car can be much cheaper, but if I leave out the cost of hiring a full-time driver for my new car, my analysis is...incomplete. Your pitch emphasizes features that make servers easier to manage, presumably lowering (but not eliminating) the cost of personnel, so you're obviously aware of this, but I'm not seeing any "net of additional, fully-loaded personnel costs" in their analysis. It should still be cheaper above a certain scale, but the breakeven where it will make sense to "repatriate infrastructure" is much higher if you include hiring people, and will go even higher if the cloud providers run the numbers and match most of the savings with scale-based price breaks.
- kelp 5y ago
  I don't know how every company does it. But I led the datacenter and networking teams at Square from 2011 to 2017. In that time we went from 1 DC cage with 4 server racks to 4 US DCs, one in a Japan and a couple of network pops. A bit more than 100 server racks total. My team had 2 guys doing SiteOps. They would travel to the various DCs in the Bay Area and Virginia and do all the maintenance, new installs, etc. And sometimes we'd lean on the colocation remote hands to do a few things. We had about 5 network engineers, that also handled the corporate network. (12 offices and a network backbone that connected east and west coast DCs, offices, etc). And maybe 2 SWEs who handled things like our host OS install system, etc. Basically the next layer above the hardware. So 9 people that were required to run all of that stuff. But really, NetEng ended up spending like 70% of their time on corporate network things because we'd add offices faster than datacenters. So if we focused on production only, we really needed about 6 or 7 people total. I did the math a few times (every single year) and compared our costs, including people, to the costs of moving to AWS 3 year reserve instances. Doing it ourselves was always half the price. Of course the difference here was we built on-prem from the start. So there was no repatriation that had to happen. Since then, I've been responsible for large cloud infra on all 3 major providers and learned a lot about what kind of discounts you can get when you're in the double digit millions in annual spend. I still think, at a scale of single digit thousands of servers, you'd be cheaper on-prem, fully loaded. But admittedly, I haven't run the numbers since 2017.

fnord77 5y ago

technologies like kubernetes make repatriation even simple in some cases.

deanCommie 5y ago
Except you miss out on all the opportunity cost of cloud-native solutions that get you moving fast for the sake of a complex convoluted mess that you need to hire subject matter experts for to wrangle, instead of focusing on your business logic. Kubernetes only makes sense from Google's perspective to sell you the managed version "from the creators of Kubernetes!" once you get tired of trying to wrestle it into submission.
- gravypod 5y ago
  I don't think you need to "wrestle it into submission". There are many CNCF projects that work great on bare metal instances like Rook and other companies providing managed DB solutions for your hardware like KubeDB. K8s isn't that complex from a cluster operators perspective if you've managed bare metal Linux systems before. Set up some PXE + auto provisioning and you can make your node enrollment process: 1. Buy hardware from an OEM. 2. OEM ships you a piece of hardware. 3. Remote hands plug in hardware. 4. Machine PXEs, boots, and is turned into a worker on your cluster. Canonical (MaaS), VMware (vSphere), and Rancher Labs (Rancher) all have products aimed at running Kube on metal in your own datacenter. So, essentially, if you start your startup off using Kube in a managed cluster you can, at some point in time later, directly move your code off the cloud and onto your own machines for cost saving. You just need to figure out how to run your DBs and LBs but that's a solved problem as mentioned above.
- fnord77 5y ago
  akshually we use Amazon's managed kubernetes. k8s lets us move fast - we can easily deploy new services in no time. We can also run the entire stack on our desktops in k8s, or have our laptops be a node in a mostly cloud k8s dev deployment and debug services locally.

devops000 5y ago

I don’t think is a paradox, is a standard “make or buy” decision which is different based on your company’s size and growth phase.

SirOibaf 5y ago

A bit of a weird article coming from a VC heavily invested in SaaS providers

wmf 5y ago
They're basically talking about helping SaaS providers reduce their IaaS spending. They're not talking about migrating off SaaS.

trjordan 5y ago

The opportunity cost is staggering. Not just of the people that do the work, but the cost of focus and top-level priorities over the years it takes to do the work. Everybody loves to cite DropBox here. The greater arc of DropBox is their product stalled, they lost the enterprise of their market to Box, and they found themselves a commodity in a commodity market. Heck, maybe they'll go back to Amazon if they keep doing things like this: https://aws.amazon.com/solutions/case-studies/dropbox-s3/ https://aws.amazon.com/solutions/case-studies/dropbox-s3/ But that's not all. It takes a top-level directive to repatriate an entire SaaS. That's at the cost of other top-level projects. It's wild to me that any company that has significant fuel left in the tank would buy back single-digit COGS percentages instead of investing in product that could add double-digit growth for several years at scale.

redis_mlc 5y ago
> things like this: https://aws.amazon.com/solutions/case-studies/dropbox-s3/ https://aws.amazon.com/solutions/case-studies/dropbox-s3/ Yeah but why do they have 34 PB of analytics data?
- jeffbee 5y ago
  The only question worth asking right there. It's the story of so many companies. We were doing something incredibly stupid, and in this five-part blog series we describe a slightly different implementation of an equally stupid idea.
  - redis_mlc 5y ago
    That was my observation as a DBA. The most powerful tools we have are aggregated reports and data retention periods, the same since the 1970s. Running 10 copies of the same report (but each one with slightly different bugs) across 34 PB of data for 10 different departments is absurd, yet here we are. Specifically for Dropbox, they're not a retailer like Safeway with 10,000 products and 100's of locations to crunch product and shipping data for - Dropbox could use a single server for their entire digital products data warehouse. (I operated a data warehouse myself (literally) at Yahoo - 10 servers generating canned web reports for 500 managers. I heard they replaced it with a 1,000 node Hadoop cluster later.)

mtnygard 5y ago

Not trying to peddle cheap skepticism here. This article only looks at "seen" costs, and assumes that there are no "unseen" costs to running on-prem. Many companies do not have the operational maturity to run on-prem well. The result: high cost of operations, low availability, and large increase in time-to-value. Second unseen cost: everybody becomes their own SI. So far nobody really sells the "whole stack" for running on prem. I mean hardware, network, virtualization, application, traffic mgmt, etc. I have to buy stuff from two dozen different vendors and cobble it together into a high-labor, rickety Jenga tower of stuff.

siddarthd2919 5y ago
You nailed it. Unseen costs are huge and a lot of people ignore it.
- deleted 5y ago
  [deleted]
jandrewrogers 5y ago
These costs are not unseen. They are straightforward ordinary costs and fully accounted for. It only seems expensive and complicated from the perspective of someone that does not have experience with it. To someone that has done it many times before it is relatively simple and almost completely mechanical to get an excellent result. It isn't rocket science, just domain expertise around things like physical infrastructure planning and supply chains that a software developer may not have encountered before but could easily learn. If a company was going to run their own data centers, they would presumably hire someone that knows what they are doing to lead the effort instead of trying to do it by trial and error.
- oblio 5y ago
  > Many companies do not have the operational maturity to run on-prem well.
  - 5y ago

paxys 5y ago

This analysis completely skips over the "elastic" aspect of cloud infrastructure, which was a key motivator for using these providers since the very beginning and is as relevant today regardless of company size. My company (which is in fact part of the charts in the article) had every single metric across the board spike 15-20x basically overnight when the pandemic started last year in March. Our entire infrastructure burden was clicking a few buttons on the AWS console and making sure everything was provisioning and scaling as needed. If we had to send out people to buy hard drives and server racks at that time, there is no chance we would have been able to meet the extra demand. Plus, if you give me a few dozen capable engineers today I'm not going to waste their efforts on rebuilding AWS to get a best-case few percentage point return on our cloud spend. I'll launch a new product for our customers instead.

LimaBearz 5y ago
That’s a valid point. I don’t want to argue pedantry but I’ll say I worked some media companies that experience 5x normal volumes a few times a year during events before returning to “normal”, making AWS an obvious choice. I’ve also worked for a place so large in AWS we hit walls where we hit Amazon’s literal physical limit and had wait for them to go out and buy the hardware for us to provision. For Dropbox specifically I don’t see them being any form of special case, if it saves money and they obviously had operational experience so it makes sense
JohnJamesRambo 5y ago
Are you wildly overpaying though for something that is a once in a century (hopefully) event like a pandemic that shuts down the whole world?

matchagaucho 5y ago

From a Moore's Law perspective I'd like to see a true cost of ownership over time, as most infra goes obsolete in 18-24 months. Public clouds, like AWS, have cut their storage costs by more than 50% since Dropbox built their own infra in 2015.

kelp 5y ago
In my experience it's pretty common to stretch your infra out much longer than 18-24 months. Often finance has a 3 year depreciation schedule, and depending on your needs, you can keep existing workloads on existing hardware for many more years. The last time I ran physical infra, we had sever racks that I'd originally bought, running in production 5+ years later. Once you're past the depreciation schedule, they are basically free except for power and space. Yeah, at some point and scale, it can make sense to get new gear that is more power and space efficient to pack more into the same power and cooling budget. AWS still lets me launch c1-c6 instance types. So they still have the older generations sticking around. Yeah, the newer ones are usually more cost effective, but you do have to do work to migrate to them.

sanj 5y ago

This may be a naive question, but... Given that interactions to the cloud are through a relatively small set of known APIs, I'm surprised that there isn't there a service which replicates the APIs but with the endpoints being on "repatriated" hardware.

kenniskrag 5y ago
terraform does that. https://www.terraform.io/ https://www.terraform.io/
- sanj 5y ago
  That’s clever, but not what I’m suggesting. I’m looking for a mechanism which apes existing cloud APIs directly.
  - e12e 5y ago
    Well, the thing is they're not all that similar (except maybe s3 which has been copied by competition). So terraform or similar abstractions (from pulumi to chef/ansible) are what you get. Or k8s. But there's also: https://www.eucalyptus.cloud/ https://www.eucalyptus.cloud/
maxgashkov 5y ago
APIs are not what's hard IMO. Creating a layer simulating S3 API is pretty straightforward. Designing and implementing a highly available and resilient storage solution on-prem is much more harder task that will take 99% of time & budget to do properly compared to a compatibility layer.

spoonjim 5y ago

I couldn’t tell, which onprem portfolio company are they pushing here?

throwawaaarrgh 5y ago

These people don't see the real value proposition of the cloud. The value is not in "scaling when you need to". Sure, that can be very handy. But that is not what 99% of companies are getting out of the cloud 99% of the time. If you self-manage, your capital investment is initially higher, and lower over time. At the same time, the effort it takes to reach the same results is always higher, and the quality of the end product may be lower, depending on how much service quality affects your product. If you pay for managed services, your initial investment is lower, and higher over time. But at the same time, you require less effort, and you get higher quality outcomes. This is obvious to anyone who has worked in the industry and done both. First, host your own service: JFrog Artifactory, Atlassian Confluence, GitLab, whatever. Now rapidly increase the demands on this service. As demand rises, quality will decrease, because it takes a lot of time, effort and expertise to build a very reliable hosted service. Now switch to a managed cloud instance. Suddenly, the service's average quality increases. Performance is steady regardless of increase in use. There are virtually no interruptions to your product or development. The impact of a service's quality and reliability has ripple effects. If poor service quality slows down development, that means development quality will go down as people cut corners to try and meet deadlines. If the service is used for production, it means your product's quality will suffer, and that effects your bottom line. So a huge amount of the actual cost is not just paying for a service, but also how much business value is generated or lost due to service quality. There is simply no way to replicate a managed service without becoming a managed service provider yourself. You have to become a whole new business within a business. It's like a yogurt company also becoming a dairy farm. Running a farm is not easy, and you will screw it up for several years. Seems obvious for farming, but for some reason people always underestimate this when it comes to technology. On paper, the Cloud's value proposition is scalability. But in practice, the true value is actually as a force-multiplier for your product's quality, reliability, and time to market. (Time to market not just being "I launched my startup" but also "I released this new feature before my competitor")

kelp 5y ago
Having done both, I think I mostly agree with you. And I'm a big proponent of having your engineering team focus their efforts on building and maintaining the product you are selling to customers. Avoid doing undifferentiated work. Outsource it when you can. Focus on the core product. I think this is especially true as you're scaling a company in the current SWE labor environment. If your business is growing quickly, you probably have money, and can get more funding when you need it. But hiring more SWEs is always a challenge. That is your scarce resource. Don't waste it on undifferentiated work. All that said, at larger scale and more mature businesses. I think the core thesis of the article makes sense. You may be in the slower growth part of the S curve, but you can still increase your company value and profitability substantially by increasing margins. One way do do this is to invest in cheaper infrastructure. Then you're taking on some undifferentiated work, but in service of better margins.

akh 5y ago

> By tracking cloud spend, the company enables engineers, and not just finance teams, to take ownership of cloud spend. > tie the pain directly to the folks who can fix the problem That's why we're building https://github.com/infracost/infracost https://github.com/infracost/infracost for engineering teams (free open source)

deleted 5y ago

[deleted]

liminal 5y ago

I think they understated the importance and difficulty of retaining a sufficiently redundant skilled workforce to manage the equivalent cloud infrastructure.

lowbloodsugar 5y ago

"But that’s just Dropbox." The implication here being, "Well, if Dropbox can save this much, then think about how much everyone else cans save!" But in fact the opposite is true. Dropbox sells disk storage on the cloud. For them to do so by effectively reselling someone else's disk-storage-on-the-cloud platform is obviously not high margin, and they'd be better off building it themselves. So, sure. Anyone else also offering, disk storage on the cloud, or compute clusters on the cloud, or otherwise just reselling someone else's product on the cloud, will probably have higher margins doing it themselves. But that is certainly not most companies. So, yeah, it's "just Dropbox". Disclaimer: I work for a cloud provider, but these opinions are my own.

lifeisstillgood 5y ago

One benefit of "the cloud" not mentioned here is elapsed time to acquire a new instance If you have moved to AWS / other then that time is around 5 minutes. If you are in a major fortune 500 and need a new server, quite often that time will measure in months (yes really). This simple equation just blows every other cost/benefit calculation out of the water. I may have missed it in the discussion.

toomuchtodo 5y ago
If you need that server today, absolutely, build an AMI and fire it up, and then place an order for a server to migrate to. It’s not binary, and it is possible between engineering and finance for balancing velocity and spend. Not everyone needs a server immediately. And not every use case will generate business value by having that server immediately available. This blog post cuts to the core of that: cloud gets you from product market fit to scale, but once at scale, it’s time to revisit capex vs opex.
kelp 5y ago
Yes and, it depends. Last time I was procuring racks of servers, it was about a 6 week lead time, sometimes more. So yeah, you have to do a lot more capacity planning, and probably keep capacity in reserve for things that are unexpected. I've had to go around to every engineering team and try to get them to tell me how much gear they might need in the next 1-2 quarters. They rarely know, so it's a lot of guess work. So in a situation with a quickly growing product, that grows in an unpredictable way. Or a really new product where you just have no idea what the adoption is going to look like. Then yeah, getting those new instances right now is really really important. At Segment we very often needed more instances in a huge hurry due to some customer load spike. And sometimes AWS ran out of the instances we needed in the region we were in. So we had to get creative to use other instance types. Also the cloud provider pricing structure heavily incentives you to actually do some capacity planning. On AWS, Savings Plan gives you huge savings. But if your workload is fairly predictable, then this on demand benefit isn't nearly as compelling and at the right scale, it makes sense to start buying servers to put in your own datacenter.
jandrewrogers 5y ago

tibiahurried 5y ago

Many underestimate the complexity and the headache that comes with managing its own data center. It is easy to get a couple of racks up and running. Then you have to deal: - HW that fails, go get a new one, cost + time - backup and geographical redundancy - security - certifications - tooling to manage and forecast - black friday! scale for few days ... more HW? - etc ... It is way more convenient for the average enterprise to pay premium and be done with it. So they can focus on the actual business and not building infrastructure and tooling to manage it.

sbazerque 5y ago

It's not purely a technical matter, there are different market dynamics at play as well. If you go cloud, you get a handful of very large suppliers, that provide a lot of non-standardized services that are probably going to lock you in like hell (as the article wel says). If you build infra, you're using /mostly/ commoditized supplies and skills. If the cloud offerings where interchangeable and the industry reasonably fragmented, the margins of cloud providers would be slimmer and the paradox would probably go away (in favor of: always cloud!). See for example Porter's analysis framework [1] and how your positioning changes in the two cases. https://en.wikipedia.org/wiki/Porter%27s_five_forces_analysis https://en.wikipedia.org/wiki/Porter%27s_five_forces_analysi...

Animats 5y ago

Linden Lab, the people behind Second Life, may have made this mistake. They had leased data center space in Phoenix AZ, and owned their own machines, about 5000 of them. During 2020, the whole system was moved to AWS. Most of the servers, each of which handles an area of the virtual world, run all the time, regardless of user load. This is very different than web serving. It's the worst cost case for AWS, where the real benefits come from dynamically provisioning a changing load. AWS, used for dedicated servers, can cost 2x that of owning your own. One data center operator says 4x.[1] Before the conversion, I'd asked the responsible VP if they were sure AWS wasn't going to be more expensive. He retired when the conversion was complete. Now we hear rumors of the company finding out that AWS costs are higher than the old data center. [1] http://www.happi.io/aws-instances-versus-physical-servers/ http://www.happi.io/aws-instances-versus-physical-servers/

JohnJamesRambo 5y ago
I'll ask the obvious, did no one do the math on the expected difference?
- Qworg 5y ago
  Someone likely did. They likely also retired, per the parent comment.
- jjoonathan 5y ago
  Someone did the math of "will this help my resume," got the answer "yes," and then dropped some numbers in a spreadsheet to convince their manager to let them do it.
- TechBro8615 5y ago
  I’m sure they did. But inevitably, someone “doing the math” on this will inject their own baseless assumptions when estimating the labor cost of maintaining an inventory of colocated or dedicated servers. I might even wager that such a comment will appear as a sibling to this one within the next 30 minutes.
  - 5y ago

ram_rar 5y ago

Cloud is not just virtualized compute, storage and network (csn) anymore. Most of the services in AWS/GCP offer a lot more than just virtualized csn. I dont think, the article factors in the cost of running a service. Its one thing to rent csn and run your own database. Its completely another to use a cloud service like dynamodb/bigtable etc. I agree, opex cost can go haywire, unless kept in check. But, its not an apples to apple comparison.

phs318u 5y ago

I think one of the benefits of cloud is (in theory) the agility that it brings. So much is abstracted away that you truly can start to focus on what's more important to your business. There's a cost to that value. So for example, while the cost to serve may be lower with your own infrastructure, it may be that the opportunity costs go up and your ability to grow that revenue stream or customer base is diminished without cloud. Another thing that came to mind reading this was - what would I need to make such a switch (from cloud to your own data centre)? And then I thought, you could provide a business doing this - commoditised hardware and services, providing the real-estate and the engineers, and more importantly, providing the same kind of service stack you find in the cloud. Then I thought, that could be a cool business for someone. And then I thought - there's nothing stopping Amazon or Microsoft or Google from offering that. I guess if you have a very clear understanding of your needs, a mature appreciation for what it will take, and a good handle on your likely growth, then taking control of it all may make sense. But for a lot of business (especially those not actually tech businesses per se), it would more likely be a return to pain.

smitty1e 5y ago

> Because this shift happens later in a company’s life, it is difficult to reverse as it’s a result of years of development focused on new features, and not infrastructure optimization. Little Miss Muffet Sits on her tuffet Watching her billing grow Her devs ever hectoring On the need for refactoring But what do these uni-brows know?

smitty1e 5y ago

Is this really a paradox? When you're small, outsource. Should the company find traction, and as you gain size, the arguments in favor of outsourcing fade, and bringing the IT in-house makes sense.

alessandroetc 5y ago

I see services like Storj the perfect hybrid model here. Companies get the security and optimizations of the cloud, while being able to simultaneously run their own storage nodes to get cost savings. Exciting times for decentralized storage!

IOT_Apprentice 5y ago

What should be considered RIGHT NOW, is that given the supply chain issues, if you are NOT on a cloud platform, good luck getting new hardware from major vendors. With AWS/Azure you can provision immediately. If you have some old hardware whose performance is impacting your production environment given the dependency on it, good luck. Firms like Dell are already stating this.

naveen99 5y ago

If cloud margins are really 50%, that seems like a case for new competition in the cloud business rather than making a data center for your own non cloud business.

whoisjuan 5y ago

This is not surprising. It's kind of obvious that cloud costs will scale more dramatically than any other cost center. So it's really an attractive idea to think that you can cut your largest bill by going on-prem. The problem with going on-prem is that for most companies this is so complex and hard to operationalize that they will likely just fail at that attempt. Cloud biggest contribution besides seamless infrastructure access, is the way it has lower the overall technical competency threshold to operate an internet business. It's really easier to find someone that knows AWS or Azure than is to find that someone who knows how to build on-prem infrastructure. I imagine that repatriation for the average large cloud business will only be possible for those who are already operating in modern infrastructure constructs like Kubernetes. If you have critical workflows build on propietary cloud services, you're probably fucked. You will have to rebuild every single service from scratch and make it scalable out of the box. If you're a multi-dimensional business that sounds like a nightmare. I honestly think that Dropbox was capable of doing it just because the nature of their business. At the primitive level you're mostly replacing your storage vendor. It happens to be that storage is their business model so it's quite evident that repatriating that critical piece of your business will yield great gains. I don't think it's that simple for other cloud businesses.