5 ms·
Admins wonder if the cloud was such a good idea after all
- CuriouslyC 2y agoVPS are good, but the fact that you have to enable 5 different services to do stuff on most clouds and deal with semi-opaque billing is pretty annoying as someone who lived through the colo and dedicated hosting days.
- ecmascript 2y agoI have never understood the rush for "the cloud". I run all my shit on a VPS (which could be called a cloud) or a dedicated rented server but that is so easy to setup and I can run all projects on the same server. Easy, simple and if I need to scale I just rent a bigger server. Scaling vertically is easy, scaling horizontally is hard. Most people never need to scale horizontally but does so anyway because they think they do. You also get like 10x the perf for the same money. Using SQLite makes it easy to have backups and even time-specific testing databases.
- Jcampuzano2 2y agoBecause most companies/startups have this in-built assumption that they will eventually grow like wildfire. Obviously this is because they are selling to investors who want this to happen so they can cash out on their unicorn. So they just build it in the cloud in the first place with that in mind. Now in some miniscule amount of cases this is true and probably did help some people who's business 100x'd overnight, but in the vast majority of cases your business just will never get to the point it needs to be "cloud scale" in the first place. Nevermind accidentally shooting yourself in the foot with a recursive lambda here and there in certain instances or a misconfuguration causing a huge bill. Edit: Another is because lots of companies who do actually end up succeeding negotiate a shit-load of credits with cloud providers so they can basically grow their business for free for a while. That is until those credits run out and they get hit with the actual costs.
- anonylizard 2y agoFor most tech startups, when the growth period hits, its vital there must be NO DELAY to scale. Miss the timing and its permanently gone. If word of mouth leads to a sudden surge of orders, you must be able to supply the good, or you get very angry customers who walk straight into the competitors door. Most of the value generated by startups is highly concentrated in the few that succeed. So naturally the industry should optimize itself to go big or go home, not penny pinch. That's also, why they hire expensive engineers rather than offshored developers, because speed matters more than cost. As for non-tech companies. Their demand is more stable in the long run, but they are not tolerant of outages. Amazon cannot have its servers go down during a big sale, too many physical ongoing costs that gets wasted for every second the central nervous system is down. So the cloud is good for its reliability.
- cdavid 2y agoFor large companies, a big reason is to transform capex into opex, and the predictability. Moreover, large organizations tend to favor predictivity over levels, i.e. are ok to increase average if variance is decreased.
- RajT88 2y agoThis. The beginning of my career on cloud was a POC, where the director shared with me this was a major driving factor (capex > opex), as well as some of the fringe benefits. I got to see close up that a team of devs ran their whole solution (with a bunch of paying customers and everything) in the cloud, because cloud automation was good enough that they didn't need dedicated ops people. Now I work for a cloud provider. I can't say that if I was running a business that I'd build it cloud-first instead of OnPrem. Certain use cases, sure. If I didn't need a lot of horsepower, I might build it on a cluster of VM's with some segmentation of duties - not quite microservices, not quite a monolith. Most likely if I was hosting on the cloud, I'd use the provider I work for, just because I know the system and how to get things done and how to talk to support. I will say though - learning the ins and outs of cloud computing has made for a great career. Challenging, but lucrative. FTA: > Microsoft and Google decided not to officially comment on the survey's findings. However, a representative for one of the hyperscalers retorted that the figures seemed cherry-picked and pointed out that, as an example, customers using reserved instances could realize significant savings. Reserved instances are a thing for sure. There's lots of other ways you can control cloud spend (enterprise agreements, dev/test subscriptions, spot instances, automated shut down / scale down, etc.) - it's enough complexity by itself that big companies hire entire teams of people to just work on tracking, projecting and controlling cloud costs.
- zevv 2y agoOne recurring pattern I have seen at multiple customer sites is that scaling makes the engineers lazy to optimize. One production performance calamity requires the team to add CPU as a quick fix, and from that time on the baseline for the product's requirement has been set to the new number of CPUs. "Back in the olden days", if your product was slow but the number of CPUs was fixed (or could not be increased instantly), the solution was to go and fix your code. Basic system level skills are now no longer taught or practiced at the appropriate levels, so teams end up without engineers who actually know how to profile and optimize. The cloud providers are the big winners here.
- mgkimsal 2y ago"scaling makes the engineers lazy to optimize" I've lost track of the times I've heard "compute is cheap! engineers are expensive!" Except... that compute cost will live forever. The time it takes someone to debug a bad loop or poor query is at worst a one time cost. Longer term, it may even make other stuff faster in the future.
- Ekaros 2y agoLooking at some numbers on cloud, yeah I don't believe in that statement anymore. End device compute might be cheap, but cloud services certainly are not.
- mgkimsal 2y agoWe've also seen a rise (or maybe I just notice it more) of stories of "I changed these 3 lines in 5 minutes and saved my company $40k/month!" And then you'll get responses like "pfft.... that's hardly the cost of one part time FAANG person who makes $680k/year - what's the point?" And around we go...
- jazz9k 2y agoBecause it's cheaper now to throw hardware at a problem than actually try to fix the code or root of the issue. This wasn't the case a couple of decades ago. It also means less engineers are needed for most companies.
- benterix 2y agoThe initial rush for what is now called the public cloud was caused by many factors, the main ones being "you don't need to manage things as we will do it for you" (=you save on operations) and "you can scale up and down how much you want without any commitment" (=you save on recurring costs) with success stories like the NYT resizing tons of images fast without having to rent servers, configure them etc. Fast forward and what we now have is a terribly complex beast with nets of dependencies that got developed partly by marketing and product teams, partly by demands of larger customers. And it's more or less clear that if you are small, you will be much better off using VPS (that's why Amazon decided to offer Lightsail), and if you are very big, you will save a lot of money moving at least part of your infra away from the public cloud. But what remains is a large part of the market: medium sized businesses and large organizations that depend on the public cloud for many reasons. But they are not stupid: once a project becomes expensive, someone starts asking questions. And after you've exhausted the path of reserved instances, spot etc., and still burn a lot of money with not-always-stellar performance, you'll find a way of moving these workloads where it makes business sense.
- bluGill 2y agoIf you can run everything on just a couple servers then the cloud doesn't make sense. There are a lot of companies that have tens of thousands of servers scattered around the world. just tracking them is a headache and you can offload a lot other overhead as well. Remember you are paying for the people full time to be in that computer room either if you do it in the cloud or not, but the cloud allows sharing those costs better when you don't need the computer.
- arkh 2y agoThe value proposition of "the cloud" is to ease all the "not important at first but you'll need one day" things: logging, alerting, availability, backups, SSO etc which usually requires different know-how from what devs have. But it has become so complex that instead of an OPS team you now have a Cloud team. With a huge wallet. Best of both worlds is to setup your own Cloud on multiple VPS which is relatively easy nowadays: HAproxy, Rancher, Kubernetes, , Keycloack, Openwhisk, Gitlab, Harbor, Opentelemtry + Prometheus + Grafana and your devops will feel right at home.
- crowcroft 2y agoFor most people I agree. When AWS was first getting big though they solved genuine, really hard problems for a lot of organisations that were large or growing quickly. NVME drives didn't exist, SSDs were expensive and a lot of servers still had spinning SAS drives – A little box with some ram and some NVME drives didn't scale as ridiculously far as they do now. I do think as computers keep getting faster and smaller the number of use cases that need a 'cloud' shrink very quickly though.
- benterix 2y agoThe problem is, the larger your organization is, the more difficult is to break free from vendor lock-in.
- flerchin 2y agoAt least for us, the promise was not a lower bill, but faster time to market. The requests to on-prem engineers are just _not a thing_, which is much faster for our developers.
- this_user 2y agoThat that sounds like you are solving your internal organisational dysfunctions by outsourcing the function rather than trying to fix the organisation.
- AlexandrB 2y agoRightly so. Unless you're the CEO/CTO you're not going to be able to fix something like this.
- hyperpape 2y agoThey are, and it is by far the most effective approach for most organizations.
- Normal_gaussian 2y agoNot managing your own physical infrastructure is not a dysfunction; unless you have a very specific hardware requirement, or are very large, on-prem is just never going to be cost effective. Where it is cost-effective, it can still be incredibly strategically limiting.
- gorjusborg 2y agoI understand your point of view, but the BigOrgs are strange. They say they want to move fast and succeed, but they are also magnets for people who would much rather accumulate money/power through politics than work. This often sets up perverse incentives to do the opposite of what would make sense. For instance, the IT department manager claims to need more headcount and money to fund it, because they believe that having more budget and employees shows how important their function is. Engineering requests a new VM, and the request sits and sits and sits, so long that you start believing that they are overloaded. Engineering complains that it is taking too long to provision machines, and IT (for the 100th time) says they need more headcount to keep up with internal demand. To fix the problem, management grants one or two hires, things get marginally better for a while, until the cycle repeats. In this scenario, Engineering is trying to get their product completed, and looks bad because the schedule is slipping. If you are the Engineering manager, do you care more about fixing the organization, or sidestepping the issue by swiping a credit card? Fixing things gets political fast, spending money less so.
- ziddoap 2y agoThis article would be much more interesting if it compared the cost increase of cloud vs. the cost increase of on-prem. It's not like on-prem costs have magically stayed the same while cloud costs increase.
- duxup 2y agoOr the added costs of on-prem that you're not dealing with in the cloud. Not to say it's a slam dunk either way, you gotta compare, but on-prem is not without its own unique requirements.
- luma 2y agoLike-for-like, those costs go down over time and have since we started building datacenters. Electric costs roughly track inflation, but the price of a TB of storage or 10GB switch port or a 24 core CPU goes down over time. That's why it's hard to align the price of cloud services, the underlying fundamentals are understood and they generally track downward, yet somehow AWS and Azure and GC etc all find ways to increase their pricing.
- ajsnigrutin 2y agoThere's still the middle ground, either virtual or real machines hosted somewhere else. Pay fixed monthly amount, get a machine somewhere, don't have to deal with heating, cooling, power, internet, anything, but still noone will charge you anything "per database lookup", per IO cycles, etc.
- draw_down 2y ago[dead]
- miah_ 2y agoBased on my exexperience: In some cases yes, in many others no.
- alphazard 2y agoThis is just the tradeoff between capex and opex, debated since you had to choose between borrowing or buying your neighbor's sharpest rock.
- pino82 2y agoThe 'cloud' always was - at its core - a mechanism to take your sovereignty away. Like the social media or IM walled gardens that you like. And like a lot of things that came up in the last ~15 years. Subscription models as well... All that will not fail, because there are strong forces that explicitly want this to happen. But it's always nice to hear that there are at least problems...
- anonylizard 2y agoYour power, water and gas supply also come from the 'cloud'. If anything, localized computing devices (Phones + PCs) are significantly more popular than localized power generation/water treatment (garage diesel generator/wells). Historically, the main economic impetus for centralized government, is to coordinate the construction of irrigation infrastructure. The more dependent a society is on one big river (Ancient China/Egypt), the more centralized it becomes.
- rollcat 2y agoWe need to first differentiation between "public compute cloud" (AWS, GCP, Azure) and "cloud services" (GMail, Dropbox, iCloud, etc). The fine article is concerned with the former exclusively, and I think mixing in the latter dilutes the point it's trying to make. So unless your personal homelab needs to be hyperscalar, you don't need a public cloud; your needs will be served perfectly by a $5/mo VPS/PaaS, and/or a LACK rack. Do companies need sovereignty? No, companies need to make money, that's their primary and main reason to exist. The choice of public cloud vs dedicated servers vs colocation vs on-prem vs etc amounts to whatever best fits your operating model. The public cloud was overhyped (like everything in tech), but smart companies chose the right tool to do the job.
- mintaka5 2y ago[dead]
- makestuff 2y agoI think the cloud is great; however, most people go way overboard for what they need. You don't need 15 lambdas, 4 micro-services, dynamoDB, API Gateway, etc. to build your internal app that gets 10 TPS. Most engineers over build things because it is how you show "complexity" to get promoted. When in reality a monolith running on some bare metal will work just fine. I agree with others that the cloud vendors make it complex just to setup a simple service and push you to have these complex architectures. While they are beneficial for availability reasons, I question how often it is really needed.
- api 2y agoThe cloud is absurdly overpriced. The margins are huge for cloud companies, especially on bandwidth. The gains brought by Moore’s law are pocketed by could providers and barely passed on because once you go “cloud native” they have fairly strong lock in. Even if you can move your work load doing so is painful and usually requires downtime. It’s also designed to encourage and then monetize a certain style of complex, over-architected, high resource load development: microservices, use of single-cloud-proprietary services, tons of loosely coupled systems, etc. Cloud native is the J2EE of the present era. This is an anti pattern that keeps happening over and over in our industry. It’s partly a result of the middlebrow meme— that mid developers love complexity— and partly something that is pushed because someone (consultants, cloud providers) are profiting. Edit: I don’t mean to blast mid developers too much. One must often journey through the middlebrow meme to make it to the hooded dude on the right. The meme describes a learning trajectory. But we should try not to let the complexity explosion of mid developers become industry norm.
- HPsquared 2y agoI think it's another case of a company's organizational structure being reflected in the code. Conway's Law. Modern businesses are full of outsourcing, temporary contracts and overdone management structures. This the shape of the software they produce.
- mwcampbell 2y agoDo you count VPSes from Vultr or the like as "the cloud" for this purpose? And, incidentally, are there any VPS providers that you would recommend over Vultr when one doesn't need a whole dedicated machine in a given location?
- idisagree 2y agoOh wow, imagine that, a forced meme in tech pushed primarily by marketers, this is totally not a trend at all. /sarcasm this is like when UX designers mess with a GUI just because they need to make themselves look useful on occasion.
- lr4444lr 2y agoHow about the cost of needing to hire more admins to handle on-premises machines, and their time spent trying to save data from faulty end-user machines instead of just replacing them and thinking twice about it?
- ocdtrekkie 2y agoEither you pay admins to manage your on-premise systems, or you both pay admins to operate your cloud systems and also pay cloud providers for their admins to provide the cloud systems your admins manage.
- rdtsc 2y agoExactly. Now you’re paying admins and paying Bezos.
- unethical_ban 2y agoThe scale of cloud means the "pay cloud providers for their admins" is pretty cheap. I don't pay someone to knit my clothes, I buy shirts at the store.
- consteval 2y agoSure, but that's only a fraction of the equation. You need cloud people your side too, otherwise how do you build the software?
- olix0r 2y agoWhile I think civo are doing some pretty novel things to allow kubernetes to run cheaply for small clusters, it’s a mistake to take a vendor’s “report” at face value. It’s a press release.
- Fokamul 2y agoMove everything to cloud, let US companies pay extra, and in the cloud company outsource everything to Pakistan.
- OrvalWintermute 2y agoI remember when I first evaluated “regulated Cloud”. Terramark - your $10K per month per server (of very modest specs) for a NIST 800-53 High, convinced me of the fact that much cloud is a giant scam.
- benterix 2y agoThis opinion poll was funded by Civo. I remember they advertised everywhere how fast their K8s startup times are: https://www.civo.com/kubernetes https://www.civo.com/kubernetes So I decided to try it to see it with my own eyes (EKS takes often half an hour or so...) and lo and behold, they use k3s! I mean, there is nothing wrong with that, but it's not mentioned anywhere on their product page so that was a bit misleading. In any case they have a verification process that I haven't managed to pass. I was surprised because I used the same credit card for Gogle and AWS, but for some reason they closed my account.
- marvinblum 2y agoI've been running Pirsch [0] on Hetzner Cloud for 3 1/2 years now on a self-hosted HashiCorp Nomad cluster. It has been super stable and very cost-effective. The Hetzner VMs are really cheap and a lot more capable at the same time. You can find everything on our blog article [1]. In front of it are two Caddy load balancers, also running on VMs (Hetzner offers load balancers, but we wanted to support custom domains [2]). The databases are running on root servers though, to get the maximum out of them. When we started, we used Google Cloud. I think we would be paying at least $2,500/month for the same setup, compared to ~$400 now. And I have happily managed this all by myself. I think a proper sysadmin for a larger corp is worth the cost! [0] https://pirsch.io https://pirsch.io [1] https://pirsch.io/blog/techstack/ https://pirsch.io/blog/techstack/ [2] https://pirsch.io/blog/how-we-use-caddy-to-provide-custom-domains-for-our-clients/ https://pirsch.io/blog/how-we-use-caddy-to-provide-custom-do...
- LifeOverIP 2y agoI'm curious, how do you routinely test your backup and restore strategy / scripts?
- marvinblum 2y agoWe back up simple dumps to a Hetzner storage box (5€/month for 1 TB), which is fine for the moment. I download and restore the dumps locally regularly. In the future, we will need a more robust and scalable solution of course.
- sghiassy 2y agoEveryone loves the cloud, until they get the bill
- 867-5309 2y agopay the bill on commencement writing a blank cheque to begin with is just an easy invitation for 'surprise' billing swindlers
- avereveard 2y agoyeah, that wonderful time when you needed a three year contract + support + licenses to get a startup going off ground sure cloud benefit might not look like a lot if you are an enterprise customer, but for startup, along with open source software, it changed the game and is still an enabler of what you see today in term of supporting startups from bootstrapping all the way to unicorn.
- bilsbie 2y agoI’ve been out of the game for a while. What’s something that’s most like heroku today? You just upload your code and they handle everything? I don’t want to be worried about applying OS updates, etc.
- OptionOfT 2y agoThe cloud, while expensive, gives flexibility. We had x machines running at a space where you rack machines. First it took 3 weeks to decide what we needed, then 3 weeks to get a quote for new machines, then 3 weeks of negotiating and penny pinching because this was a 3 year lease. Then it took 3 months for them to get the machines in there. In the cloud it's 3 clicks and you have the servers, and even better, scale on demand once you have good metrics in place. Flexibility has a cost. But not being able to do anything also has a cost.
- deleted 2y ago[deleted]
- teddyh 2y agoRelevant: The Cloud Fugitive; interview with David Heinemeier Hansson: <https://www.youtube.com/watch?v=a30vFpSaoZg https://www.youtube.com/watch?v=a30vFpSaoZg>
- bob1029 2y agoI've always found the best cloud path to stick with basic VM/VPS offerings, blob storage, DNS, CDN and IdP. Getting into things like functions as a service has always felt a little too hot for me. I don't mind integrating with cloud services though (e.g. S3-compatible APIs). Hybrid seems to be most compelling if you can keep the monster in the box. Sign up for something like Azure or AWS to get at their IdP, DNS and CDN but keep all your actual machines in a cheaper provider (hetzner, et. al.).
- acdha 2y agoMy rule is to use services which could easily be replaced. For example, on AWS using RDS is a gimme because you can always spin up your own database with the same engine but not having to spend months building the equivalent battle-tested configuration is a pure win. ECS Fargate is great because you can run containers for at one order of magnitude lower ops investment than Kubernetes but if you had to leave, it’s vanilla containers on Linux. Something like DynamoDB, though, I’d want to think about what a migration plan would look like since databases are core to your application but the semantics are not as interchangeable as SQL databases are.
- deleted 2y ago[deleted]
- Haammamajoe 2y agoCloud enabled a lot for small companies. Instead of having to hire a full sysadmin and buying infrastructure, you just click a server, enable snapshoting and you are already 10x better of of what you had before. The biggest issue with these 'cloud' discussions and the pricing is simply solved: EITHER you have a good team who understands it and can and should determine if you are better of onprem or on cloud, or you do not have this team and you are not wasting money you are just paying for a lot better system which you would otherwise never had. I have seen plenty of tremendes shitty setups in small companies from people who should know better. Databases reachable on the internet, slow ticket systems for getting hardware, costly upgrade prices (and partially internal cost centers were an server upgrade costs a few k). You use too many slow lambdas? your Database costs tons of money? You can't control your micro services anymore? And you think your company would have been able to setup a better thing on prem without the help of cloud? Never
- jjav 2y ago> Instead of having to hire a full sysadmin You need the same sysadmins (called something else to be trendy, but basically same role) either way.
- Haammamajoe 2y agoWe did not had them. It was maintained by us the development team. Why? Because it was a lot less effort to just keep a few servers running (no hardware, etc.) and its not like a dev can't do anything system related.
- jjav 2y ago> and its not like a dev can't do anything system related I'm glad to read this! Unfortunately the industry has become so fragmented that a very significant percentage of younger developers no longer have any knowledge nor any interest in understanding how systems work. There is this idea that a developer only writes code and has no need to understand threads or processes or userspace vs. kernel or networking or anything at all other than the language syntax. It is very sad.
- mega_dingus 2y agoIt's about TCO and time to market. Depending on what you're building, it's all YMMV. This article just quotes some individual costs As somebody who worked at a mid-tier company trying to run databases, I can attest that RDS was a godsend. Trying to hire both a DBA and an Ops team that knew how to write the chef cookbooks for a proper multi-node cluster postgres was a nightmare. Like, we never succeeded, and something that's ootb with RDS You must not forget that also, many (most?) companies that run things themselves do not do it right. Like, with proper off-site backups that you're regularly testing and know you have options to easily spin up replicas or restore point-in-time backups Jeff Atwood's been saying this from the initial SO podcasts from 2008. If you have the right people who are motivated, and provide the right equipment and resources, you've always had the opportunity to have lower TCO doing it yourself I have since moved on to a small top-tier company and still prefer to "outsource" my DBA work by way of using Aurora. Yes Aurora is more expensive. No, I don't have the mental or monetary budget to hire up a proper ops team. I know my limits
- moribvndvs 2y agoThe place I work at now wouldn’t exist without the cloud. We simply don’t have the money or manpower at this stage to run everything in house relative to the growth. That being said, a design philosophy I’ve imposed is KISS with largely off the shelf OSS solutions, that way we have the ability to move the software elsewhere in the future (I’d love to run in house if we get big enough). Of course, nothing is that simple but it’s much harder if not effectively impossible with something built significantly out of vendor-specific libraries and platforms. I don’t mind managed solutions as long they are completely swappable. Unfortunately, the people before me used DynamoDB as our persistence layer, but I digress.