5 ms·

This is the biggest nasty secret about cloud (and SaaS for things you don't build in-house). When my last company transitioned to cloud, all of the on-prem fol

This is the biggest nasty secret about cloud (and SaaS for things you don't build in-house). When my last company transitioned to cloud, all of the on-prem folks had to additionally take on that responsibility. We then had to hire a bunch of new folks to ease the transition and manage the cloud pieces. And then there are new teams spun up for permissions and security. Your headcount requirements are not going to go down because of cloud. As for those that like to call DIY systems "weirdware": homegrown systems are expensive to engineer, but cheap to run. We 10x'd our cost going to 3rd party visibility and feature flagging versus the thinly staffed [1] homegrown systems. And those 3rd party systems also required a ton of engineering to support, the migrations weren't clean, and they missed a lot of key features that we had to upstream to the third parties! When SignalFx got bought out by Splunk, I've never seen an annual plan get re-wired so hastily. Top priority was moving to a new vendor, and that impacted every single team in the company. The only real tangible benefit to cloud that I've seen is that a team can instantly provision hardware, database, etc. resources without much planning. That's it. And is that worth it? (I don't really think so.) [1] Once built, half an engineering headcount per quarter to accommodate new features. Oncall rotation for one team that owned the systems. Our stuff was high resiliency and barely paged at all. No SEV1 or worse outages ever.

deanCommie 2y ago
What a short-sighted outlook 1) "homegrown systems are expensive to engineer, but cheap to run" misses the most important part - how much does it cost to maintain, and how much of a RISK is it? Homegrown probably means you're depending on tribal knowledge from a few core people who if they leave you're hosed. That's a risk. You also have to train EVERYONE you hire to learn the homegrown systems, instead of hiring people with externally transferrable skills that can come in and hit the ground running. 2) all of the on-prem folks had to additionally take on that responsibility Most transitions/migrations always end up with a period where you have both systems running. It's not surprising that at first everyone has additional responsibilities. It sounds like the "on-prem folks" either didn't have the skills to work in the cloud or refused to learn, and needed external hires. Well, that's one way to put themselves out of a job, and that would be the next step once the homegrown system is deprecated. Obviously there's a reason why "not invented here" syndrome exists. There's a reason "build vs buy" is a complex discussion because of more than just buy costs. People always prefer the home-grown thing, DIY. Engineers especially. On this site, particularly. And, also, plenty of successful businesses exist by cutting out layers and going on a shallower stack ("your margin is my opportunity"). But at the end of the day, for the vast majority of businesses, companies, and software shops, managing their own in-house infrastructure is a worse decision than paying a cloud vendor.
- aledalgrande 2y ago
  I see both points but I have to agree with this one, especially if we’re talking small companies. A startup should outsource everything that is outside their core product (but not more than that), so they can focus on making their product top class.
- jcelerier 2y ago
  > Homegrown probably means you're depending on tribal knowledge from a few core people who if they leave you're hosed. there's so much complexity and tribal knowledge with cloud deployments that if tomorrow our cloud experts leave we're also very definitely hosed too, despite everything being documented thoroughly. I'm involved in a product that leverages a cloud-based metaverse system (Mozilla Hubs) that recently had organisational changes requiring us to change our hosting approach, and it's taking the better part of a year of work to understand its logic, for something that would have been a non-issue if homegrown & self-hosted.
pphysch 2y ago
> This is the biggest nasty secret about cloud (and SaaS for things you don't build in-house). Yep. Huge amounts of marketing funds are spent tricking decision-makers into thinking that off-the-shelf, plug-and-play solutions exist for their idiosyncratic business problems and schemas. Sometimes such products exist, but they are so bloated with features to support the other 99% of customers. So you absolutely need at least one expert to unwind the complexity. It's known that "greenfield" software development is much easier than "brownfield"/migrations. It should also be widely known that brownfield SaaS integrations are troublesome, because it involves enormous software+data work to link the existing in-house interfaces to the third-party interfaces. As a rule of thumb, there is no such thing as "off-the-shelf". You WILL have to hire and build and deeply understand your business processes.
- dehrmann 2y ago
  This is what people tell themselves, but 99% of the time, what your business does and the scale it operates at aren't special.
  - adamisom 2y ago
    Special in the sense of scale does not equate to special in the sense of adapting to your business' needs/culture/sensemaking
  - lazide 2y ago
    They don’t have to be special to be bespoke. And almost all business integrations are highly bespoke.
spamizbad 2y ago
I love the term "weirdware": I've seen systems like that built (and used!) first-hand multiple times in my career. They can sometimes be very odd force multipliers in unexpected ways. My favorite example is a home-rolled payroll applications specifically for sales people from over a decade ago. When I first arrived at the org I thought they were absolutely insane to have created such a thing. But its killer feature was that it allowed our org to pay out commissions rapidly (same day) and also allowed negotiating on a per-account residual commission basis as well as giving management the option to buy-out residuals. Off-the-shelf solutions at that time kinda-sorta did this but required either massive upfront $$$ and a ton of integration work. A plucky startup couldn't afford it. This was built, tested, and rolled-out in 45 days by 2 engineers and basically allowed them to poach top sales people because they knew they could get paid faster and had more flexibility on residuals.
- davedx 2y ago
  Ha I built an internal app just like that for field sales at a mid sized company. One of the best projects I ever worked on.
- throwup238 2y ago
  That's the tradeoff everyone makes when they say something isn't in their "core competency" - they give up any real competitive advantage that area of expertise could bring them. That often makes total sense like with accounting or HR, but most companies take it way too far in engineering and internal support software. Engineers make the same mistake all the time too: we love to say that "premature optimization is the root of all evil", ignoring that the full quote is "We should forget about small efficiencies, say about 97% of the time: premature optimization is the root of all evil. _Yet we should not pass up our opportunities in that critical 3%_"
throwup238 2y ago
> The only real tangible benefit to cloud that I've seen is that a team can instantly provision hardware, database, etc. resources without much planning. That's it. And is that worth it? (I don't really think so.) Yes! Maybe not for you or me, but it’s definitely worth it to tons of organizations where everyone including engineering was beholden to old school IT departments that take months to spin up a VM after pages of bureaucratic back and forth. The cloud moves it from an IT issue to department budgeting. The other advantage of cloud was moving capex to opex which the finance people liked because *hand wave* something about amortization. Those are the two reasons most established companies moved to the cloud. It had little to do with the actual cost but how it was spent and who was in charge of doing it.
- 0cf8612b2e1e 2y ago
  where everyone including engineering was beholden to old school IT departments that take months to spin up a VM after pages of bureaucratic back and forth. The cloud moves it from an IT issue to department budgeting. My F500 company put a stop to that. There is new paperwork in place to requisition cloud resources. The bureaucracy will not be replaced.
  - ndriscoll 2y ago
    Even the startup I worked at in my last job didn't let people just spin up whatever resources they felt like in AWS. Changes had to go through architecture/security review, and I might have been the only SDE who was allowed to log into AWS at all. We were subject to SOC 2, but I imagine it's relatively common for SaaS companies to have some kind of compliance framework assuming they want F500s as customers?
- dopylitty 2y ago
  Having seen the results of everyone being able to spin up a VM (or an EMR cluster, or an EKS cluster) I can say a little more bureaucracy is probably a good thing. You end up wasting huge amounts of money on resources that are just sitting around idle (no your fancy automation to shut them down won't work because maybe that dev cluster is actually needed at 2am on Sunday. It's an org problem, not a technical problem). But more importantly you give the cloud providers an excuse to destroy another square mile of pristine forest and build a giant 4 story grey box that wastes enormous amounts of already limited water and energy.
sneak 2y ago
> Your headcount requirements are not going to go down because of cloud. Depends on how big you are. I'm one person and I do a lot of things with the cloud that I could not do if I had to rack servers (or even order dedicateds). Same goes for small 5-10 person teams that are good with Terraform. I've seen some orgs punching waaaay above their weight given their size. Not possible without classic IaaS.
- lanstin 2y ago
  Being able to put everything into (good) source with all the lessons we have about how to manage source code over time puts incredible power into teams that can handle it.
  - otabdeveloper4 2y ago
    That sort of thing is much easier to do with a standard k8s/ansible/whatever stack than having to juggle proprietary cloud provider stacks.
- throwaway29511 2y ago
  > do a lot of things with the cloud that I could not do if I had to rack servers It's not a binary choice between cloud and physical racks in a data center. There is a wide range of options. Companies like Hetzner enables you to click a button and get a dedicated instance, while Digitalocean can provide you with a VM. Both options are cheap and doesn't require much time. If you do this often, you probably also have an ansible setup to install the base software (like k8s or a simpler stack), you are ready to go within minutes. Just like you would have a terraform config for your cloud
otabdeveloper4 2y ago
"Cloud" is an accounting trick that moves CapEx to OpEx. That's it. It has no technical reason to exist.
- fragmede 2y ago
  Virtual Machines becoming sufficiently performant is a technical reason for it to exist. It's far less viable if we didn't get vmx and related extensions to the x86 command set.
- bushbaba 2y ago
  Having done both cloud and on prem. It’s soo much more than that. Pace of innovation increases with cloud as you no longer have product beholden to the infra team jira hell.
  - otabdeveloper4 2y ago
    "Infra team jira hell" exists precisely because of accounting CapEx trickery. There are no technical reasons for it to exist.

deanCommie 2y ago

What a short-sighted outlook 1) "homegrown systems are expensive to engineer, but cheap to run" misses the most important part - how much does it cost to maintain, and how much of a RISK is it? Homegrown probably means you're depending on tribal knowledge from a few core people who if they leave you're hosed. That's a risk. You also have to train EVERYONE you hire to learn the homegrown systems, instead of hiring people with externally transferrable skills that can come in and hit the ground running. 2) all of the on-prem folks had to additionally take on that responsibility Most transitions/migrations always end up with a period where you have both systems running. It's not surprising that at first everyone has additional responsibilities. It sounds like the "on-prem folks" either didn't have the skills to work in the cloud or refused to learn, and needed external hires. Well, that's one way to put themselves out of a job, and that would be the next step once the homegrown system is deprecated. Obviously there's a reason why "not invented here" syndrome exists. There's a reason "build vs buy" is a complex discussion because of more than just buy costs. People always prefer the home-grown thing, DIY. Engineers especially. On this site, particularly. And, also, plenty of successful businesses exist by cutting out layers and going on a shallower stack ("your margin is my opportunity"). But at the end of the day, for the vast majority of businesses, companies, and software shops, managing their own in-house infrastructure is a worse decision than paying a cloud vendor.

aledalgrande 2y ago
I see both points but I have to agree with this one, especially if we’re talking small companies. A startup should outsource everything that is outside their core product (but not more than that), so they can focus on making their product top class.
jcelerier 2y ago
> Homegrown probably means you're depending on tribal knowledge from a few core people who if they leave you're hosed. there's so much complexity and tribal knowledge with cloud deployments that if tomorrow our cloud experts leave we're also very definitely hosed too, despite everything being documented thoroughly. I'm involved in a product that leverages a cloud-based metaverse system (Mozilla Hubs) that recently had organisational changes requiring us to change our hosting approach, and it's taking the better part of a year of work to understand its logic, for something that would have been a non-issue if homegrown & self-hosted.

pphysch 2y ago

> This is the biggest nasty secret about cloud (and SaaS for things you don't build in-house). Yep. Huge amounts of marketing funds are spent tricking decision-makers into thinking that off-the-shelf, plug-and-play solutions exist for their idiosyncratic business problems and schemas. Sometimes such products exist, but they are so bloated with features to support the other 99% of customers. So you absolutely need at least one expert to unwind the complexity. It's known that "greenfield" software development is much easier than "brownfield"/migrations. It should also be widely known that brownfield SaaS integrations are troublesome, because it involves enormous software+data work to link the existing in-house interfaces to the third-party interfaces. As a rule of thumb, there is no such thing as "off-the-shelf". You WILL have to hire and build and deeply understand your business processes.

dehrmann 2y ago
This is what people tell themselves, but 99% of the time, what your business does and the scale it operates at aren't special.
- adamisom 2y ago
  Special in the sense of scale does not equate to special in the sense of adapting to your business' needs/culture/sensemaking
- lazide 2y ago
  They don’t have to be special to be bespoke. And almost all business integrations are highly bespoke.

spamizbad 2y ago

I love the term "weirdware": I've seen systems like that built (and used!) first-hand multiple times in my career. They can sometimes be very odd force multipliers in unexpected ways. My favorite example is a home-rolled payroll applications specifically for sales people from over a decade ago. When I first arrived at the org I thought they were absolutely insane to have created such a thing. But its killer feature was that it allowed our org to pay out commissions rapidly (same day) and also allowed negotiating on a per-account residual commission basis as well as giving management the option to buy-out residuals. Off-the-shelf solutions at that time kinda-sorta did this but required either massive upfront $$$ and a ton of integration work. A plucky startup couldn't afford it. This was built, tested, and rolled-out in 45 days by 2 engineers and basically allowed them to poach top sales people because they knew they could get paid faster and had more flexibility on residuals.

davedx 2y ago
Ha I built an internal app just like that for field sales at a mid sized company. One of the best projects I ever worked on.
throwup238 2y ago
That's the tradeoff everyone makes when they say something isn't in their "core competency" - they give up any real competitive advantage that area of expertise could bring them. That often makes total sense like with accounting or HR, but most companies take it way too far in engineering and internal support software. Engineers make the same mistake all the time too: we love to say that "premature optimization is the root of all evil", ignoring that the full quote is "We should forget about small efficiencies, say about 97% of the time: premature optimization is the root of all evil. _Yet we should not pass up our opportunities in that critical 3%_"

throwup238 2y ago

> The only real tangible benefit to cloud that I've seen is that a team can instantly provision hardware, database, etc. resources without much planning. That's it. And is that worth it? (I don't really think so.) Yes! Maybe not for you or me, but it’s definitely worth it to tons of organizations where everyone including engineering was beholden to old school IT departments that take months to spin up a VM after pages of bureaucratic back and forth. The cloud moves it from an IT issue to department budgeting. The other advantage of cloud was moving capex to opex which the finance people liked because *hand wave* something about amortization. Those are the two reasons most established companies moved to the cloud. It had little to do with the actual cost but how it was spent and who was in charge of doing it.

0cf8612b2e1e 2y ago
where everyone including engineering was beholden to old school IT departments that take months to spin up a VM after pages of bureaucratic back and forth. The cloud moves it from an IT issue to department budgeting. My F500 company put a stop to that. There is new paperwork in place to requisition cloud resources. The bureaucracy will not be replaced.
- ndriscoll 2y ago
  Even the startup I worked at in my last job didn't let people just spin up whatever resources they felt like in AWS. Changes had to go through architecture/security review, and I might have been the only SDE who was allowed to log into AWS at all. We were subject to SOC 2, but I imagine it's relatively common for SaaS companies to have some kind of compliance framework assuming they want F500s as customers?
dopylitty 2y ago
Having seen the results of everyone being able to spin up a VM (or an EMR cluster, or an EKS cluster) I can say a little more bureaucracy is probably a good thing. You end up wasting huge amounts of money on resources that are just sitting around idle (no your fancy automation to shut them down won't work because maybe that dev cluster is actually needed at 2am on Sunday. It's an org problem, not a technical problem). But more importantly you give the cloud providers an excuse to destroy another square mile of pristine forest and build a giant 4 story grey box that wastes enormous amounts of already limited water and energy.

sneak 2y ago

> Your headcount requirements are not going to go down because of cloud. Depends on how big you are. I'm one person and I do a lot of things with the cloud that I could not do if I had to rack servers (or even order dedicateds). Same goes for small 5-10 person teams that are good with Terraform. I've seen some orgs punching waaaay above their weight given their size. Not possible without classic IaaS.

lanstin 2y ago
Being able to put everything into (good) source with all the lessons we have about how to manage source code over time puts incredible power into teams that can handle it.
- otabdeveloper4 2y ago
  That sort of thing is much easier to do with a standard k8s/ansible/whatever stack than having to juggle proprietary cloud provider stacks.
throwaway29511 2y ago
> do a lot of things with the cloud that I could not do if I had to rack servers It's not a binary choice between cloud and physical racks in a data center. There is a wide range of options. Companies like Hetzner enables you to click a button and get a dedicated instance, while Digitalocean can provide you with a VM. Both options are cheap and doesn't require much time. If you do this often, you probably also have an ansible setup to install the base software (like k8s or a simpler stack), you are ready to go within minutes. Just like you would have a terraform config for your cloud

otabdeveloper4 2y ago

"Cloud" is an accounting trick that moves CapEx to OpEx. That's it. It has no technical reason to exist.

fragmede 2y ago
Virtual Machines becoming sufficiently performant is a technical reason for it to exist. It's far less viable if we didn't get vmx and related extensions to the x86 command set.
bushbaba 2y ago
Having done both cloud and on prem. It’s soo much more than that. Pace of innovation increases with cloud as you no longer have product beholden to the infra team jira hell.
- otabdeveloper4 2y ago
  "Infra team jira hell" exists precisely because of accounting CapEx trickery. There are no technical reasons for it to exist.