5 ms·

Anyone who's worked inside a large organization knows the cloud is the solution to be able to get things done and deliver value to the business. Why? On-prem

Anyone who's worked inside a large organization knows the cloud is the solution to be able to get things done and deliver value to the business. Why? On-prem solutions require I work with the infrastructure group, who wants to cookie-cutter me into their limited stable of infrastructure options; I have to work with the systems group, who gives me very limited options and configurations for the operating systems; the network group, who are very fixed in their ways and make very few concessions; and the middleware group which is what you need to connect to anything else of importance. Four groups! All with their own workflows, their own SLAs, their own exemption policies, etc. It's a nightmare to get anything done and the business is frustrated that things take so long. Now I'm on a team that does everything in AWS. We keep everything serverless so the infrastructure and systems teams can't butt in, and the cloud environment eliminates the need for interacting with the network group. I only have the middleware group to contend with. The result? We're delivering much-needed solutions far faster than any other group in the organization. They also have far fewer business impacts (read: support issues impacting operations) than the applications we have on-prem. People seriously underestimate the time and energy required to keep a highly-available, and DR capable datacenter running. They also underestimate the complications of interacting with all the teams need to keep that datacenter running. We have found that AWS is faster, cheaper, better than hosting on-prem.

piker 3y ago
So your organisation is so inefficient that it’s easier to just engage a third party to syphon off any profitability?
- belval 3y ago
  > your organisation is so inefficient Very few organizations are able to stay efficient on the long run as they scale. This is a simple matter of processes being stacked on top of processes and inter-departement frictions. That complexity is inherent to having regulations and laws that you need to comply with that become hard requirements. For a more concrete example think of HIPAA compliance, during an audit you can have everything on-prem and hope that your IT department actually understand the security and privacy boundaries or have everything in AWS and be compliant because they conform to the standard by default. > syphon off any profitability For most of those businesses software is not the profitable bit, it's a cost center like most other things that serve to support operations. In that context, migrating your code to AWS/GCP/Azure means very little cost inflation.
  - ilc 3y ago
    Just because AWS is HIPAA compliant, doesn't mean what you do there is :). But at least it solves the basics.
    - monocasa 3y ago
      I think their point is that it gives you a relatively stable env, and allows you to be the one deploying and configuring infrastructure. Anything that can be miscommunicated to client on-prem deployment staff will be, in a lot of cases breaking guarantees like HIPAA compliance.
    - jeanlou 3y ago
      That's just nitpicking
      - ilc 3y ago
        Not really. So much of HIPAA is actually getting the administration right.
  - delfinom 3y ago
    >Very few organizations are able to stay efficient on the long run as they scale Simply using AWS doesn't do anything to help with this in that same long run. Give it a decade or two and those same organizations will suddenly have multiple bureaucratic departments to approve new AWS resources and spending. All AWS is offering is a channel that a bureaucratic process hasn't been developed for _yet_ and the organizations hivemind response is to allow its usage freely because it doesn't fit in the existing form template of rules
    - pixl97 3y ago
      >Give it a decade or two A decade or two is someone else's problem.
    - jeanlou 3y ago
      Good luck maintaining your own infra, server, system middle war combo. It's a whole area in itself. Let the specialists do it and focus on your core business. Saying otherwise is just a lack of pragmatism.
- theshrike79 3y ago
  I can literally right now, sitting here in my underwear, spin up 4 clustered servers in 3 different continents with a globally replicated object storage and a master + 2 slaves setup of SQL databases on each location. And all that for less than the price of _one_ competent sysadmin's monthly salary. How would you suggest a company would match that with local servers and sysadmins? One sysadmin per 8 hour shift = 3 people for 24 hour rotation, you need 1-2 extra so that if one is sick the others don't need to pull a double. That's $400k a year alone. Then you need to get the 12 identical servers purchased and shipped around the globe to colocation. Now at least one, preferably two, of the admins needs to be a proficient DBA so they can get the replicating SQL running. Now you need to build monitoring for all of that and runbooks on what to do if crap breaks outside of office hours. ..and now a server has kicked the bucket, you need to buy a new one from $vendor, get it delivered and set up in Australia - you're in Michigan. And in some years the servers are underpowered because your business has grown, now you need to buy 12 new ones and get them installed. I can do all that from home without putting on my pants. I might need to grab a shirt if the AWS people want to hop on Zoom for something. =) This is why we use the cloud and managed services, usually it's either out of our hands anyway or already fixed when we manage to log in. For the last two biggest downtimes I've encountered even Meta's crap was offline too (FB, Instagram at least).
  - Agingcoder 3y ago
    It depends on the size of your company. If it has 200k employees, you can mutualize the salaries of your dbas. There are also intermediate steps between ´build your own dc’ and ´run everything in was’. You will build monitoring tools anyway, and will most likely have sysadmins too if your fleet is large enough etc. Cloud is not always an obvious win, and depends a lot on where you work
    - theshrike79 3y ago
      At 200k employees and being profitable, you can definitely can start investigating moving to on-prem. And most likely a lot sooner than that. BUT it depends on what field you're in. If you're managing CCPA/GDPR/HIPAA data it's a huge bureaucracy mess to get everything certified. Cloud operators usually have their documented and certified already, you just need to figure out your own part.
- jeanlou 3y ago
  I sense that you're writing this as if software is an end to itself as opposed to a mean. It's normal to engage with third party, especially for activities that are not part of your core business.
candiddevmike 3y ago
It all comes down to responsibility. In your example, you're responsible for the full stack including infrastructure (I'm assuming, if not, I feel bad for your infrastructure team). This model works for some developer teams and not for others, either due to lack of expertise, free cycles, or organizational pressure to have developers focus on code only. Define your responsibility model first, then your infrastructure consumption patterns.
- taylodl 3y ago
  You're correct, the team is taking on all that responsibility - which is another reason we use serverless architectures. We have no intention of managing servers, provisioning servers, patching servers, etc. The whole idea of compute has been completely abstracted.
tw04 3y ago
And what is the operating margin of the products your group produces? What was it prior to shifting everything to the cloud? If you can't answer those questions, you may be architecting yourself out of a job. I watched more than one company that was operating on razor-thin margins, depreciating hardware over a decade, move to the cloud and promptly bankrupt themselves. If you're in a business where you've got an additional 20% margin to piss away on someone else's gear, your logic makes a ton of sense. If you're in one that's making 5% on every widget that goes out the door, you should probably be working with your management to figure out how to remove the inefficiencies from your process rather than just moving your problem onto someone else's hardware.
- taylodl 3y ago
  We have those numbers. We reduced our costs by over 70%. We got the project approved claiming we'd reduce costs by 30%. Obviously YMMV and the savings are going to be dependent on each project, but we've had enough solutions that we've had on-prem that we've shifted to AWS and show substantial cost savings that the business is comfortable with our taking new projects and just starting out in AWS.
  - tw04 3y ago
    I'm sorry, there's simply no world in which switching to the cloud, and serverless to boot, results in a 70% cost reduction unless you so grossly over-provisioned your on-prem infrastructure that whoever architected it should no longer be employed. Cloud infrastructure is more expensive across the board, full stop. Unless you're using hollywood math to claim millions in "efficiency gains" I don't buy it.
    - fragmede 3y ago
      Millions, and 70% seems high to me, but let's look at some numbers. what is the cheapest real actual server you can buy, new, with a support contract from a reputable vendor like HP/Dell/etc? (Assume we're running both the db and web-server on one server.) And what's the cost of sitting it in a reputable colo? And then multiply that by, let's say two, to get redundancy in a us-west and us-east region. Ideally, we'd want more regions to serve the entire world, but let's just go with two for now. Let's say $10k for a server w/ support contract, x2, plus $400/mo for colo from Hurricane Electric (https://he.net/colocation.html https://he.net/colocation.html) in us-west. It's a promo deal, and only in us-west, but let's just go with that price. $400*60 for 5 years (standard lifetime of computer equipment) = $24k. So $34k per server * 2 servers / 60 months ~= $1,130/month amortized. Thankfully, we're not in cloud, so that's the cost whether it's 1 request/second, or 1,000 requests/second. A Raspberry Pi or old laptop in your basement with one consumer-grade ISP, and not factoring in the cost of electricity does not count here. Those are obviously going to be massively cheaper, but they're not remotely suitable for enterprise. Meanwhile, Lambda gives you 1M requests and 3.2 million compute-seconds a month for free. This works out to be 0.4 rps (requests/second) or 22 requests / minute if spread out evenly. Which isn't, like, a lot, but you haven't paid a single cent to AWS for this yet. Let's say you want to get to 40 rps (to keep the math easy) or 100M requests over the month. According to the AWS calculator (https://calculator.aws/#/addService/Lambda https://calculator.aws/#/addService/Lambda), this'll run you $120/month, or $120*24 ~= $3k for 2 years. But that doesn't count hosting for your static files with CloudFront, or does it count the RDS instance backing it, and we'd use AWS Cognito for user-auth since we've accepted AWS vendor lock-in. (Which isn't great, but that's the state of the industry.) CloudFront: Free Tier 1TB, 10M requests/month. 10 TB/month will run you $1k/month https://calculator.aws/#/addService/CloudFront https://calculator.aws/#/addService/CloudFront RDS: This will vary greatly. A teeny tiny db.m6g.large instance w/ 2 CPUs and 8 GiB of RAM is going to run you $250/month, but a big one fat juicy db.r5d.24xlarge will run you closer to $20k/month. https://calculator.aws/#/addService/RDSMySQL https://calculator.aws/#/addService/RDSMySQL Cognito: With 1,000 active users, this'll be $50/month; 10,000 users is $500/mo, and 100,000 users is $5k/month. Between Lambda, CloudFront, RDS, and Cognito you could easily blow past that $1,130/month estimate for a proper colo'd pair of servers, if your service gets at all popular. But if you stick within the free tier for Lambda and Cloudfront, then you're only looking at RDS and Cognito costs, which can vary greatly. Anywhere from $500/month to $26k/month, or more. Except to get the same service in a colo as a $26k/month AWS bill, you'd need to spend way more and have much more than the single server in a rack that I started with. Thankfully, with something like Equinix Smart Hands, no one has to drive to the colo to futz with a server when it develops a bad hard drive. Graph it out for your particular use case which one is better, but cloud infrastructure has the benefit of opex vs capex, as well as the opportunity cost of time. AWS Amplify will let you stand up a whole platform hacking on a Saturday before Dell can even quote you a price on a server or Equinix can answer the phone. I also did reject a raspberry pi or old laptop server early on, but that's not to be underestimated. If I've already got some server running at home for my house, running a service on that, fronted by Cloudflare can take you quite far. It really just depends on your use case.
taylodl 3y ago
Forgot to mention - if you were doing anything new, plan to spend months getting that in on-prem. They're going to run an RFP, deliberate ad nauseam, try to build a business case the tool, procure infrastructure to host it, do five year usage projections so you can measure how you're going to capitalize - it just goes on, and on. Our organization spent two years going through these gyrations for elastic search. They failed. It never happened. No other solution was brought on board, they simply couldn't justify the cost. All told they spend hundreds of thousands of dollars to come up empty-handed. Meanwhile my new team had a project where elastic search was the obvious component as part of the solution and we simply used that service on AWS. Done. It's been a while since I've worked at a startup or a mid-cap company, but for any company in the Fortune 500? Cloud is a godsend!
- robertlagrant 3y ago
  Yeah. The - possibly insoluble - problem is that if you don't go through all the approvals exercises, the people whose time would be taken up approving your thing are still there, drawing a salary. So there's no obvious cost/benefit analysis to do, or not one that's politik. But I like the idea of just doing it using cloud. Once a cloud vendor is approved, all its services are implicitly approved as well. Nice.
  - taylodl 3y ago
    > Once a cloud vendor is approved, all its services are implicitly approved as well. Nice. Exactly! When we start a new project the AWS Console is just a big tool box we open to see what's available that we can use to deliver the solution.
- zimbatm 3y ago
  Sure. Come back to us in twelve months. Either the company will have lost control of their spending and be sending 6-7 figure paychecks towards AWS. Or the ops team will have clamped down on the accounts and now you have to go trough the same process to get EC2 instances and other cloud resources. Solving cultural problems with technology generally doesn't work.
api 3y ago
Yeah, in addition to the recurring revenue aspect a major drive of cloud adoption is escaping your IT department. This suggests that the way corporations do IT is radically wrong and actively harmful.
- taylodl 3y ago
  I think it points to IT is much harder to do at scale than most people realize. There's a big difference between running a datacenter and running a server closet. Companies utilize HR services, payroll services, and legal services, amongst many other services, because that's not in their wheelhouse and a part of their core competency, so why wouldn't they utilize cloud platforms as well? Like I said, managing and running multiple data centers having high-availability and DR capability isn't easy, or cheap.
  - marcosdumay 3y ago
    The entire structure is not value oriented. The technical people want to solve technical problems and don't give a fuck about the company mission. The immediate management wants to improve the measurable recurrent tasks and don't even think about supporting anything different. The top management wants to cut jobs, minimize the need for competency, cut costs, minimize flexibility and lean time to think about non-recurrent tasks because IT is a cost center. The result is that all of those problems you point get solved. That's not the issue. But nothing of value is allowed to exist.
- tclancy 3y ago
  Probably 70% that and 30% cowboys who think security and policy rules are for people who make mistakes.
- red-iron-pine 3y ago
  if you think your IT dept is bad, wait until you see how little Amazon or Azure gives a shit
weberer 3y ago
Did you read the article? "Local" here means running on the user's computer. It is not talking about private data centers.
- taylodl 3y ago
  And there are good reasons why we abandoned that approach 20 years ago. It's a nightmare to support.
  - capableweb 3y ago
    Yup, and probably computers today is just a divergent and unique as computers were 20 years ago, so no need to ever look into this issue again. I guess that's the reason there is basically 0 good desktop applications in today's modern computing world, I certainly cannot think of a single successful desktop application. Good thinking!
  - mixmastamyk 3y ago
    It’s just a local copy of many, and can run even when offline. Like git. No nightmares.
- theshrike79 3y ago
  I still remember the time when "server" meant someone's old Dell tower we put in a closet somewhere =) I thought we figured out it's a shitty idea and moved past it?
  - imchillyb 3y ago
    The only 'shitty ideas' in business are those that cost the business and shareholders money. Cloud computing can either be a necessity for a business, or a money-hole from which is derived little to no benefit to the company. Most small businesses could thrive with a -local- small dedicated server/workstation, and locally managed network resources.
    - taylodl 3y ago
jauntywundrkind 3y ago
On-prem thankfully is changing quickly. It used to be boutique, every shop kludging together their own plan, cobbling together an assorted mix of vendor and open source. There were few macro patterns. The major change in the last 10 years is that there's now a very stable & general open-source cloud platform that covers a much wider range of the stack than what we had before, and which has achieved a critical mass of popularity. On prem is still hard. Having an organization that isn't crufty & slow, where there aren't forms in triplicate & long waits for every request, is still hard. But it's gotten a lot easier than it was to bring up, and to let users do some self service. There's a small scale org where having a small handful of clusters & letting users semi-free, mostly using ci/cd, is quite attainable. Figuring out how to scale up to serving a mid or larger size org is still quite a challenge.
- gadders 3y ago
  Yeah, I agree. I think cloud was a great forcing-function to make internal infra groups raise their game or be replaced. However, if you have I do worry that once people have fully migrated to the cloud they will have a captive audience and will start ramping up the costs.
ryandrake 3y ago
> Four groups! All with their own workflows, their own SLAs, their own exemption policies, etc. It's a nightmare to get anything done and the business is frustrated that things take so long. This seems like a wound self-inflicted by a company that can't seem to run itself efficiently. You're not describing attributes of on-prem and cloud--you're describing a dysfunctional organization.
- gochi 3y ago
  You've circled back to their point about forcing the rest of the organization into becoming cookie cutters.
ldjkfkdsjnv 3y ago
Yup. Most complaints about the cloud are people who dont understand how to use it. They have been left behind and havent reskilled. They call it "too complex". In reality, using AWS is 10000x better than some custom infrastructure.
- colechristensen 3y ago
  You have to be able to beat the cloud locally to make doing so make sense. AWS et al. set the bar and it’s difficult to beat them with your local infrastructure.
  - ldjkfkdsjnv 3y ago
    They have 10000 engineers working 50 hours a week to keep their cloud running. The thing works on the back of very hard work from very smart people.
    - colechristensen 3y ago
      You don't need that much to have a cloud with several 9's of uptime. If you've never done the work or been around it, the difficulty becomes a bit mythologized.
      - thrixton 3y ago
        Sure, but to a certain extent, that’s survivorship bias. It’s very easy to run something with several 8’s of uptime if you’re not careful. AWS has good patterns to guide you into the pit of success.
  - peteradio 3y ago
    If the thing you are trying to do isn't very complex then beating Cloud is not that difficult. Many small companies just don't need it and moving to it adds cost and complexity for what?
HumblyTossed 3y ago
Cool, cool. Now it sounds like you're doing two jobs for the price of one.
regularfry 3y ago
At my last org, the exec reaction to the cloud was to say we should build our own, AWS-compatible, in our own DC's. Yes, really.
- unethical_ban 3y ago
  I'm actually shocked that aside from S3 clones, no piece of software or open source project has attempted to replicate the software ecosystem of the cloud as it pertains to APIs and IAM.
- regularfry 3y ago
  Oh, one other thing: this was in 2019.
btbuildem 3y ago
Our org decided to take the worst of both worlds; now we have "IS requests to (I shit you not) create a branch in a repo" level of nonsense. The cliques with their turfs still exist, the hoops are still there to jump through, and the "cloud tech" is just absorbed into the corporate ineptitude mindset.
JohnMakin 3y ago
The other nice thing about primarily cloud infrastructures is, as an infra/ops guy I can build tooling and abstractions over it to allow developer and BI teams to go much faster without my help at all. An on-prem solution can allow this, but the fact that so much of the complexity is managed by the cloud provider makes it a lot faster and easier. A feature that could take months to roll out is usually offered by AWS for a price tag that's easy for the business to understand. My one irritation with stuff like AWS is that a lot of it is needlessly esoteric, to the point where I think they make their docs bad on purpose to get you to buy their enterprise support packages.
retrocryptid 3y ago
At least with on-prem, you can talk to the infrstructure team's management to negotiate adding currently non-supported features. I was hired at Amazon 8 years ago to implement a couple of features. One of them was difficult because it required communication / buy-in from multiple two pizza teams. By the time I left it was stalled. Three years ago I randomly started at an AWS customer who needed that feature as they had followed the AWS conventional wisdom for setting things up and architected themselves into a corner. I was able to craft some workarounds but they're hacky. AWS has been promising us this feature "next quarter" for the last 3 years. If you look around, you can find a number of weird corner-cases where AWS does not make it easy to do the simple thing you want to do, but you can hack around it. We went looking at GCP and Azure and there are similar issues, though in different parts of the infrastructure. Not to mention each of them change their interface frequently enough to be annoying. Thankfully, reverse incompatible changes are not at all the norm. My cloud infrastructure is replete with architectural warts to avoid quirks we didn't know existed until after we deployed. My on-prim components, though managed by a different team, do what we want them to do because the team who built them asked us what we wanted before building it. AWS weirdness accounts for about 85% of my time. On prim weirdness for 5%. But yes, local-first is obviously a non-starter.
- thrixton 3y ago
  Out of curiosity, which particular edge case was this? I’ve hit multiple oddities with cognito (only supporting custom claims in id token), lambda (consuming SQS messages without invoking the function when run in low concurrency). So there are strange edge cases, today RDS and ECS are the services of choice, I wonder where I’ll be in 12 months.
planetafro 3y ago
Curious -- in this scenario, who is doing security, governance, compliance, observability, etc...? You are probably masking a lot of benefits of a mature and competent IT team. ...or you are assuming A LOT of risk allowing velocity-driven software engineering teams to run amok.
- unethical_ban 3y ago
  My previous employer is or has already transitioned mostly to the cloud. The answer to your question is " the same teams that were doing it before, but with cloud guardrails instead of hacky bespoke solutions ". AWS IAM is baked into every single product natively. It isn't perfect and their JSON dialect is annoying at times, but having granular RBAC for storage, compute, ops, network in a single language is incredible for security. And using IaC, you can put guardrails on specific tasks that IT does often. Manual reviews become automated. It is a ton of conversion and up front work, but there are upsides. And then of course there is the instant global reliability, where a lot of formerly complicated sysops becomes automated as well Final thought: other than the hardware abstraction, everything I talked about re: IAM could be done with a local software stack, if it existed.
  - freedude 3y ago
    " the same teams that were doing it before, but with cloud guardrails instead of hacky bespoke solutions " you do realize that "cloud guardrails" often started out life as "hacky bespoke solutions". you are assuming more business risk than is necessary.
    - unethical_ban 3y ago
      As someone who did it for two years, I know not everything is perfect. But the tooling, monitoring, automation, orchestration, etc. becomes a lot easier when there are 4-5 toolsets vs. dozens. It's like taking an ops support team that is using perl, java, php, python, bash, ksh running on RHEL5 and HP-UX and getting everyone on RHEL8, terraform and Go.
stcroixx 3y ago
I see you're sold so I won't try to change your mind, but be aware your opinion is far from universal. I'd rather eat broken glass than work with proprietary cloud technologies.
r00fus 3y ago
I have direct multi-year experience where an AWS heavy client (with a strong tie to Amazon) decided to shift course because AWS spend went nearly vertical despite the velocity of features delivered, and moved to migrate all logic/data into other enterprise systems where possible. So for the first few months, it was moving all business logic to lambdas/RDS then the next year, moving it all back elsewhere - not necessarily on-prem, but anywhere except AWS. So I'd expect your cloud team to get very much like infra once cost realities become involved. It's possible your volumes/costs are much less for the features you build - in that case, kudos to designing stuff that is perf-aware - but sometimes you don't control volume and that's a cost-risk.
yonz 3y ago
Read this first: https://www.inkandswitch.com/local-first/ https://www.inkandswitch.com/local-first/ > Anyone who's worked inside a large organization knows the cloud is the solution to be able to get things done and deliver value to the business. Local first is focused on end consumer experiences and is orthogonal to the on-prem vs cloud argument; ideally there wouldn't even be a server. @pvh and i&s team have done a great job threading the needle in articulating the ideology. At a high level it is advocating for products that can provide the benefits of the cloud without being locked on or turning your phone into a paper weight if it doesn't have internet access.
- taylodl 3y ago
  The only local first we do these days is iOS apps. We develop those for our users who work outside the office in locations having spotty cell coverage.
deleted 3y ago
[deleted]
red-iron-pine 3y ago
> Anyone who's worked inside a large organization knows the cloud is the solution to be able to get things done and deliver value to the business. Nah. Multiple F500s to include manufacturing, railroad, and aircraft. Pick an industry, been around it. 3 for 3 -- large cloud rollouts clobbered the IT team and blew OpEx out of the water. Saw VPs catch an axe for it, and at least one CTO get nailed with embezzlement accusations due to a "cloud consulting" company he had ties to, and who managed to get a no-bid contract for north of 400k. Never been anywhere that it wasn't a kind-of-clusterfuck, and the only answers in reply I've seen were No True Scotsman-type "you're just not embracing the cloud and it's changes to processes bla bla bla".

piker 3y ago

So your organisation is so inefficient that it’s easier to just engage a third party to syphon off any profitability?

belval 3y ago
> your organisation is so inefficient Very few organizations are able to stay efficient on the long run as they scale. This is a simple matter of processes being stacked on top of processes and inter-departement frictions. That complexity is inherent to having regulations and laws that you need to comply with that become hard requirements. For a more concrete example think of HIPAA compliance, during an audit you can have everything on-prem and hope that your IT department actually understand the security and privacy boundaries or have everything in AWS and be compliant because they conform to the standard by default. > syphon off any profitability For most of those businesses software is not the profitable bit, it's a cost center like most other things that serve to support operations. In that context, migrating your code to AWS/GCP/Azure means very little cost inflation.
- ilc 3y ago
  Just because AWS is HIPAA compliant, doesn't mean what you do there is :). But at least it solves the basics.
  - monocasa 3y ago
    I think their point is that it gives you a relatively stable env, and allows you to be the one deploying and configuring infrastructure. Anything that can be miscommunicated to client on-prem deployment staff will be, in a lot of cases breaking guarantees like HIPAA compliance.
  - jeanlou 3y ago
    That's just nitpicking
    - ilc 3y ago
      Not really. So much of HIPAA is actually getting the administration right.
- delfinom 3y ago
  >Very few organizations are able to stay efficient on the long run as they scale Simply using AWS doesn't do anything to help with this in that same long run. Give it a decade or two and those same organizations will suddenly have multiple bureaucratic departments to approve new AWS resources and spending. All AWS is offering is a channel that a bureaucratic process hasn't been developed for _yet_ and the organizations hivemind response is to allow its usage freely because it doesn't fit in the existing form template of rules
  - pixl97 3y ago
    >Give it a decade or two A decade or two is someone else's problem.
  - jeanlou 3y ago
    Good luck maintaining your own infra, server, system middle war combo. It's a whole area in itself. Let the specialists do it and focus on your core business. Saying otherwise is just a lack of pragmatism.
theshrike79 3y ago
I can literally right now, sitting here in my underwear, spin up 4 clustered servers in 3 different continents with a globally replicated object storage and a master + 2 slaves setup of SQL databases on each location. And all that for less than the price of _one_ competent sysadmin's monthly salary. How would you suggest a company would match that with local servers and sysadmins? One sysadmin per 8 hour shift = 3 people for 24 hour rotation, you need 1-2 extra so that if one is sick the others don't need to pull a double. That's $400k a year alone. Then you need to get the 12 identical servers purchased and shipped around the globe to colocation. Now at least one, preferably two, of the admins needs to be a proficient DBA so they can get the replicating SQL running. Now you need to build monitoring for all of that and runbooks on what to do if crap breaks outside of office hours. ..and now a server has kicked the bucket, you need to buy a new one from $vendor, get it delivered and set up in Australia - you're in Michigan. And in some years the servers are underpowered because your business has grown, now you need to buy 12 new ones and get them installed. I can do all that from home without putting on my pants. I might need to grab a shirt if the AWS people want to hop on Zoom for something. =) This is why we use the cloud and managed services, usually it's either out of our hands anyway or already fixed when we manage to log in. For the last two biggest downtimes I've encountered even Meta's crap was offline too (FB, Instagram at least).
- Agingcoder 3y ago
  It depends on the size of your company. If it has 200k employees, you can mutualize the salaries of your dbas. There are also intermediate steps between ´build your own dc’ and ´run everything in was’. You will build monitoring tools anyway, and will most likely have sysadmins too if your fleet is large enough etc. Cloud is not always an obvious win, and depends a lot on where you work
  - theshrike79 3y ago
    At 200k employees and being profitable, you can definitely can start investigating moving to on-prem. And most likely a lot sooner than that. BUT it depends on what field you're in. If you're managing CCPA/GDPR/HIPAA data it's a huge bureaucracy mess to get everything certified. Cloud operators usually have their documented and certified already, you just need to figure out your own part.
jeanlou 3y ago
I sense that you're writing this as if software is an end to itself as opposed to a mean. It's normal to engage with third party, especially for activities that are not part of your core business.

candiddevmike 3y ago

It all comes down to responsibility. In your example, you're responsible for the full stack including infrastructure (I'm assuming, if not, I feel bad for your infrastructure team). This model works for some developer teams and not for others, either due to lack of expertise, free cycles, or organizational pressure to have developers focus on code only. Define your responsibility model first, then your infrastructure consumption patterns.

taylodl 3y ago
You're correct, the team is taking on all that responsibility - which is another reason we use serverless architectures. We have no intention of managing servers, provisioning servers, patching servers, etc. The whole idea of compute has been completely abstracted.

tw04 3y ago

And what is the operating margin of the products your group produces? What was it prior to shifting everything to the cloud? If you can't answer those questions, you may be architecting yourself out of a job. I watched more than one company that was operating on razor-thin margins, depreciating hardware over a decade, move to the cloud and promptly bankrupt themselves. If you're in a business where you've got an additional 20% margin to piss away on someone else's gear, your logic makes a ton of sense. If you're in one that's making 5% on every widget that goes out the door, you should probably be working with your management to figure out how to remove the inefficiencies from your process rather than just moving your problem onto someone else's hardware.

taylodl 3y ago
We have those numbers. We reduced our costs by over 70%. We got the project approved claiming we'd reduce costs by 30%. Obviously YMMV and the savings are going to be dependent on each project, but we've had enough solutions that we've had on-prem that we've shifted to AWS and show substantial cost savings that the business is comfortable with our taking new projects and just starting out in AWS.
- tw04 3y ago
  I'm sorry, there's simply no world in which switching to the cloud, and serverless to boot, results in a 70% cost reduction unless you so grossly over-provisioned your on-prem infrastructure that whoever architected it should no longer be employed. Cloud infrastructure is more expensive across the board, full stop. Unless you're using hollywood math to claim millions in "efficiency gains" I don't buy it.
  - fragmede 3y ago
    Millions, and 70% seems high to me, but let's look at some numbers. what is the cheapest real actual server you can buy, new, with a support contract from a reputable vendor like HP/Dell/etc? (Assume we're running both the db and web-server on one server.) And what's the cost of sitting it in a reputable colo? And then multiply that by, let's say two, to get redundancy in a us-west and us-east region. Ideally, we'd want more regions to serve the entire world, but let's just go with two for now. Let's say $10k for a server w/ support contract, x2, plus $400/mo for colo from Hurricane Electric (https://he.net/colocation.html https://he.net/colocation.html) in us-west. It's a promo deal, and only in us-west, but let's just go with that price. $400*60 for 5 years (standard lifetime of computer equipment) = $24k. So $34k per server * 2 servers / 60 months ~= $1,130/month amortized. Thankfully, we're not in cloud, so that's the cost whether it's 1 request/second, or 1,000 requests/second. A Raspberry Pi or old laptop in your basement with one consumer-grade ISP, and not factoring in the cost of electricity does not count here. Those are obviously going to be massively cheaper, but they're not remotely suitable for enterprise. Meanwhile, Lambda gives you 1M requests and 3.2 million compute-seconds a month for free. This works out to be 0.4 rps (requests/second) or 22 requests / minute if spread out evenly. Which isn't, like, a lot, but you haven't paid a single cent to AWS for this yet. Let's say you want to get to 40 rps (to keep the math easy) or 100M requests over the month. According to the AWS calculator (https://calculator.aws/#/addService/Lambda https://calculator.aws/#/addService/Lambda), this'll run you $120/month, or $120*24 ~= $3k for 2 years. But that doesn't count hosting for your static files with CloudFront, or does it count the RDS instance backing it, and we'd use AWS Cognito for user-auth since we've accepted AWS vendor lock-in. (Which isn't great, but that's the state of the industry.) CloudFront: Free Tier 1TB, 10M requests/month. 10 TB/month will run you $1k/month https://calculator.aws/#/addService/CloudFront https://calculator.aws/#/addService/CloudFront RDS: This will vary greatly. A teeny tiny db.m6g.large instance w/ 2 CPUs and 8 GiB of RAM is going to run you $250/month, but a big one fat juicy db.r5d.24xlarge will run you closer to $20k/month. https://calculator.aws/#/addService/RDSMySQL https://calculator.aws/#/addService/RDSMySQL Cognito: With 1,000 active users, this'll be $50/month; 10,000 users is $500/mo, and 100,000 users is $5k/month. Between Lambda, CloudFront, RDS, and Cognito you could easily blow past that $1,130/month estimate for a proper colo'd pair of servers, if your service gets at all popular. But if you stick within the free tier for Lambda and Cloudfront, then you're only looking at RDS and Cognito costs, which can vary greatly. Anywhere from $500/month to $26k/month, or more. Except to get the same service in a colo as a $26k/month AWS bill, you'd need to spend way more and have much more than the single server in a rack that I started with. Thankfully, with something like Equinix Smart Hands, no one has to drive to the colo to futz with a server when it develops a bad hard drive. Graph it out for your particular use case which one is better, but cloud infrastructure has the benefit of opex vs capex, as well as the opportunity cost of time. AWS Amplify will let you stand up a whole platform hacking on a Saturday before Dell can even quote you a price on a server or Equinix can answer the phone. I also did reject a raspberry pi or old laptop server early on, but that's not to be underestimated. If I've already got some server running at home for my house, running a service on that, fronted by Cloudflare can take you quite far. It really just depends on your use case.

taylodl 3y ago

Forgot to mention - if you were doing anything new, plan to spend months getting that in on-prem. They're going to run an RFP, deliberate ad nauseam, try to build a business case the tool, procure infrastructure to host it, do five year usage projections so you can measure how you're going to capitalize - it just goes on, and on. Our organization spent two years going through these gyrations for elastic search. They failed. It never happened. No other solution was brought on board, they simply couldn't justify the cost. All told they spend hundreds of thousands of dollars to come up empty-handed. Meanwhile my new team had a project where elastic search was the obvious component as part of the solution and we simply used that service on AWS. Done. It's been a while since I've worked at a startup or a mid-cap company, but for any company in the Fortune 500? Cloud is a godsend!

robertlagrant 3y ago
Yeah. The - possibly insoluble - problem is that if you don't go through all the approvals exercises, the people whose time would be taken up approving your thing are still there, drawing a salary. So there's no obvious cost/benefit analysis to do, or not one that's politik. But I like the idea of just doing it using cloud. Once a cloud vendor is approved, all its services are implicitly approved as well. Nice.
- taylodl 3y ago
  > Once a cloud vendor is approved, all its services are implicitly approved as well. Nice. Exactly! When we start a new project the AWS Console is just a big tool box we open to see what's available that we can use to deliver the solution.
zimbatm 3y ago
Sure. Come back to us in twelve months. Either the company will have lost control of their spending and be sending 6-7 figure paychecks towards AWS. Or the ops team will have clamped down on the accounts and now you have to go trough the same process to get EC2 instances and other cloud resources. Solving cultural problems with technology generally doesn't work.

api 3y ago

Yeah, in addition to the recurring revenue aspect a major drive of cloud adoption is escaping your IT department. This suggests that the way corporations do IT is radically wrong and actively harmful.

taylodl 3y ago
I think it points to IT is much harder to do at scale than most people realize. There's a big difference between running a datacenter and running a server closet. Companies utilize HR services, payroll services, and legal services, amongst many other services, because that's not in their wheelhouse and a part of their core competency, so why wouldn't they utilize cloud platforms as well? Like I said, managing and running multiple data centers having high-availability and DR capability isn't easy, or cheap.
- marcosdumay 3y ago
  The entire structure is not value oriented. The technical people want to solve technical problems and don't give a fuck about the company mission. The immediate management wants to improve the measurable recurrent tasks and don't even think about supporting anything different. The top management wants to cut jobs, minimize the need for competency, cut costs, minimize flexibility and lean time to think about non-recurrent tasks because IT is a cost center. The result is that all of those problems you point get solved. That's not the issue. But nothing of value is allowed to exist.
tclancy 3y ago
Probably 70% that and 30% cowboys who think security and policy rules are for people who make mistakes.
red-iron-pine 3y ago
if you think your IT dept is bad, wait until you see how little Amazon or Azure gives a shit

weberer 3y ago

Did you read the article? "Local" here means running on the user's computer. It is not talking about private data centers.

taylodl 3y ago
And there are good reasons why we abandoned that approach 20 years ago. It's a nightmare to support.
- capableweb 3y ago
  Yup, and probably computers today is just a divergent and unique as computers were 20 years ago, so no need to ever look into this issue again. I guess that's the reason there is basically 0 good desktop applications in today's modern computing world, I certainly cannot think of a single successful desktop application. Good thinking!
- mixmastamyk 3y ago
  It’s just a local copy of many, and can run even when offline. Like git. No nightmares.
theshrike79 3y ago
I still remember the time when "server" meant someone's old Dell tower we put in a closet somewhere =) I thought we figured out it's a shitty idea and moved past it?
- imchillyb 3y ago
  The only 'shitty ideas' in business are those that cost the business and shareholders money. Cloud computing can either be a necessity for a business, or a money-hole from which is derived little to no benefit to the company. Most small businesses could thrive with a -local- small dedicated server/workstation, and locally managed network resources.
  - taylodl 3y ago

jauntywundrkind 3y ago

On-prem thankfully is changing quickly. It used to be boutique, every shop kludging together their own plan, cobbling together an assorted mix of vendor and open source. There were few macro patterns. The major change in the last 10 years is that there's now a very stable & general open-source cloud platform that covers a much wider range of the stack than what we had before, and which has achieved a critical mass of popularity. On prem is still hard. Having an organization that isn't crufty & slow, where there aren't forms in triplicate & long waits for every request, is still hard. But it's gotten a lot easier than it was to bring up, and to let users do some self service. There's a small scale org where having a small handful of clusters & letting users semi-free, mostly using ci/cd, is quite attainable. Figuring out how to scale up to serving a mid or larger size org is still quite a challenge.

gadders 3y ago
Yeah, I agree. I think cloud was a great forcing-function to make internal infra groups raise their game or be replaced. However, if you have I do worry that once people have fully migrated to the cloud they will have a captive audience and will start ramping up the costs.

ryandrake 3y ago

> Four groups! All with their own workflows, their own SLAs, their own exemption policies, etc. It's a nightmare to get anything done and the business is frustrated that things take so long. This seems like a wound self-inflicted by a company that can't seem to run itself efficiently. You're not describing attributes of on-prem and cloud--you're describing a dysfunctional organization.

gochi 3y ago
You've circled back to their point about forcing the rest of the organization into becoming cookie cutters.

ldjkfkdsjnv 3y ago

Yup. Most complaints about the cloud are people who dont understand how to use it. They have been left behind and havent reskilled. They call it "too complex". In reality, using AWS is 10000x better than some custom infrastructure.

colechristensen 3y ago
You have to be able to beat the cloud locally to make doing so make sense. AWS et al. set the bar and it’s difficult to beat them with your local infrastructure.
- ldjkfkdsjnv 3y ago
  They have 10000 engineers working 50 hours a week to keep their cloud running. The thing works on the back of very hard work from very smart people.
  - colechristensen 3y ago
    You don't need that much to have a cloud with several 9's of uptime. If you've never done the work or been around it, the difficulty becomes a bit mythologized.
    - thrixton 3y ago
      Sure, but to a certain extent, that’s survivorship bias. It’s very easy to run something with several 8’s of uptime if you’re not careful. AWS has good patterns to guide you into the pit of success.
- peteradio 3y ago
  If the thing you are trying to do isn't very complex then beating Cloud is not that difficult. Many small companies just don't need it and moving to it adds cost and complexity for what?

HumblyTossed 3y ago

Cool, cool. Now it sounds like you're doing two jobs for the price of one.

regularfry 3y ago

At my last org, the exec reaction to the cloud was to say we should build our own, AWS-compatible, in our own DC's. Yes, really.

unethical_ban 3y ago
I'm actually shocked that aside from S3 clones, no piece of software or open source project has attempted to replicate the software ecosystem of the cloud as it pertains to APIs and IAM.
regularfry 3y ago
Oh, one other thing: this was in 2019.

btbuildem 3y ago

Our org decided to take the worst of both worlds; now we have "IS requests to (I shit you not) create a branch in a repo" level of nonsense. The cliques with their turfs still exist, the hoops are still there to jump through, and the "cloud tech" is just absorbed into the corporate ineptitude mindset.

JohnMakin 3y ago

The other nice thing about primarily cloud infrastructures is, as an infra/ops guy I can build tooling and abstractions over it to allow developer and BI teams to go much faster without my help at all. An on-prem solution can allow this, but the fact that so much of the complexity is managed by the cloud provider makes it a lot faster and easier. A feature that could take months to roll out is usually offered by AWS for a price tag that's easy for the business to understand. My one irritation with stuff like AWS is that a lot of it is needlessly esoteric, to the point where I think they make their docs bad on purpose to get you to buy their enterprise support packages.

retrocryptid 3y ago

At least with on-prem, you can talk to the infrstructure team's management to negotiate adding currently non-supported features. I was hired at Amazon 8 years ago to implement a couple of features. One of them was difficult because it required communication / buy-in from multiple two pizza teams. By the time I left it was stalled. Three years ago I randomly started at an AWS customer who needed that feature as they had followed the AWS conventional wisdom for setting things up and architected themselves into a corner. I was able to craft some workarounds but they're hacky. AWS has been promising us this feature "next quarter" for the last 3 years. If you look around, you can find a number of weird corner-cases where AWS does not make it easy to do the simple thing you want to do, but you can hack around it. We went looking at GCP and Azure and there are similar issues, though in different parts of the infrastructure. Not to mention each of them change their interface frequently enough to be annoying. Thankfully, reverse incompatible changes are not at all the norm. My cloud infrastructure is replete with architectural warts to avoid quirks we didn't know existed until after we deployed. My on-prim components, though managed by a different team, do what we want them to do because the team who built them asked us what we wanted before building it. AWS weirdness accounts for about 85% of my time. On prim weirdness for 5%. But yes, local-first is obviously a non-starter.

thrixton 3y ago
Out of curiosity, which particular edge case was this? I’ve hit multiple oddities with cognito (only supporting custom claims in id token), lambda (consuming SQS messages without invoking the function when run in low concurrency). So there are strange edge cases, today RDS and ECS are the services of choice, I wonder where I’ll be in 12 months.

planetafro 3y ago

Curious -- in this scenario, who is doing security, governance, compliance, observability, etc...? You are probably masking a lot of benefits of a mature and competent IT team. ...or you are assuming A LOT of risk allowing velocity-driven software engineering teams to run amok.

unethical_ban 3y ago
My previous employer is or has already transitioned mostly to the cloud. The answer to your question is " the same teams that were doing it before, but with cloud guardrails instead of hacky bespoke solutions ". AWS IAM is baked into every single product natively. It isn't perfect and their JSON dialect is annoying at times, but having granular RBAC for storage, compute, ops, network in a single language is incredible for security. And using IaC, you can put guardrails on specific tasks that IT does often. Manual reviews become automated. It is a ton of conversion and up front work, but there are upsides. And then of course there is the instant global reliability, where a lot of formerly complicated sysops becomes automated as well Final thought: other than the hardware abstraction, everything I talked about re: IAM could be done with a local software stack, if it existed.
- freedude 3y ago
  " the same teams that were doing it before, but with cloud guardrails instead of hacky bespoke solutions " you do realize that "cloud guardrails" often started out life as "hacky bespoke solutions". you are assuming more business risk than is necessary.
  - unethical_ban 3y ago
    As someone who did it for two years, I know not everything is perfect. But the tooling, monitoring, automation, orchestration, etc. becomes a lot easier when there are 4-5 toolsets vs. dozens. It's like taking an ops support team that is using perl, java, php, python, bash, ksh running on RHEL5 and HP-UX and getting everyone on RHEL8, terraform and Go.

stcroixx 3y ago

I see you're sold so I won't try to change your mind, but be aware your opinion is far from universal. I'd rather eat broken glass than work with proprietary cloud technologies.

r00fus 3y ago

I have direct multi-year experience where an AWS heavy client (with a strong tie to Amazon) decided to shift course because AWS spend went nearly vertical despite the velocity of features delivered, and moved to migrate all logic/data into other enterprise systems where possible. So for the first few months, it was moving all business logic to lambdas/RDS then the next year, moving it all back elsewhere - not necessarily on-prem, but anywhere except AWS. So I'd expect your cloud team to get very much like infra once cost realities become involved. It's possible your volumes/costs are much less for the features you build - in that case, kudos to designing stuff that is perf-aware - but sometimes you don't control volume and that's a cost-risk.

yonz 3y ago

Read this first: https://www.inkandswitch.com/local-first/ https://www.inkandswitch.com/local-first/ > Anyone who's worked inside a large organization knows the cloud is the solution to be able to get things done and deliver value to the business. Local first is focused on end consumer experiences and is orthogonal to the on-prem vs cloud argument; ideally there wouldn't even be a server. @pvh and i&s team have done a great job threading the needle in articulating the ideology. At a high level it is advocating for products that can provide the benefits of the cloud without being locked on or turning your phone into a paper weight if it doesn't have internet access.

taylodl 3y ago
The only local first we do these days is iOS apps. We develop those for our users who work outside the office in locations having spotty cell coverage.

deleted 3y ago

[deleted]

red-iron-pine 3y ago

> Anyone who's worked inside a large organization knows the cloud is the solution to be able to get things done and deliver value to the business. Nah. Multiple F500s to include manufacturing, railroad, and aircraft. Pick an industry, been around it. 3 for 3 -- large cloud rollouts clobbered the IT team and blew OpEx out of the water. Saw VPs catch an axe for it, and at least one CTO get nailed with embezzlement accusations due to a "cloud consulting" company he had ties to, and who managed to get a no-bid contract for north of 400k. Never been anywhere that it wasn't a kind-of-clusterfuck, and the only answers in reply I've seen were No True Scotsman-type "you're just not embracing the cloud and it's changes to processes bla bla bla".