17 ms·

I'm so surprised there is so much pushback against this.. AWS is extremely expensive. The use cases for setting up your system or service entirely in AWS are mo

I'm so surprised there is so much pushback against this.. AWS is extremely expensive. The use cases for setting up your system or service entirely in AWS are more rare than people seem to realise. Maybe I'm just the old man screaming at cloud (no pun intended) but when did people forget how to run a baremetal server ? > We have 730+ days with 99.993% measured availability and we also escaped AWS region wide downtime that happened a week ago. This is a very nice brag. Given they are using their ddos protection ingress via CloudFlare there is that dependancy, but in that case I can 100% agree than DNS and ingress can absolutely be a full time job. Running some microservices and a database absolutely is not. If your teams are constantly monitoring and adjusting them such as scaling, then the problem is the design. Not the hosting. Unless you're a small company serving up billions of heavy requests an hour, I would put money on the bet AWS is overcharging you.

JCM9 1y ago
As the author points out AWS can provide a few things that you wouldn’t want to try and replicate (like CloudFront) but for most other things you’re very much correct. AWS is ultimately very expensive for what it is. The complicated billing that’s full of surprises also makes cost management a head-banging experience.
- tyingq 1y ago
  Fair, though using AWS solely for CloudFront would mean you should compare to Cloudflare, Akamai, Fastly, etc. I'm not sure if the value prop for it looks so great if you don't include the "integrated with your other AWS stuff" benefit.
  - vidarh 1y ago
    I mean, AWS egress is so expensive that I'd put something else in front of it for anyone who has any decent amount of traffic.
  - JCM9 1y ago
    Agree, CloudFront isn’t super competitive with CDN focused vendors. It’s basically the “well you’re already on AWS so may as well just use this” play.
jagged-chisel 1y ago
Forget? You have to hire people for that. We are a software organization. We build software. If we rent in the cloud, there is less HR hassle - hiring, raises, bonuses, benefits, firing … none of that headache involved with the cloud. Technically? Totally doable. But the owners prefer renting in the cloud over the people-related issues of hiring.
- sieabahlpark 1y ago
  [dead]
- bilekas 1y ago
  > Forget? You have to hire people for that. We are a software organization. We build software. You don't need to hire dedicated people full time. It could even be outsourced and then a small contract for maintenance. It's the same argument you could say for "accounting persons", or "HR persons" - "We are a software organisation!" - Personally I don't buy the argument.
  - Foobar8568 1y ago
    Outsourcing and cloud cost are always underestimated.
  - deleted 1y ago
    [deleted]
  - theta_d 1y ago
    > It could even be outsourced and then a small contract for maintenance. Yeah, those people we outsourced to happen to work at AWS.
    - vidarh 1y ago
      They don't though. You still need devops when you use AWS, and most organisations end up needing more time spent on devops when they use AWS.
- chasd00 1y ago
fabian2k 1y ago
A large part of the different views on this topic are due to the way people estimate the amount of saved effort and money because you're pushing some admin duties to the cloud provider instead of doing this yourself. And people come to vastly different conclusions on this aspect. It's also that the requirements vary a lot, discussions here on HN often seem to assume that you need HA and lots of scaling options. That isn't universally true.
- nicce 1y ago
  > A large part of the different views on this topic are due to the way people estimate the amount of saved effort and money because you're pushing some admin duties to the cloud provider instead of doing this yourself. And people come to vastly different conclusions on this aspect This applies only if you had an extra customer that pays the difference. Basically argument only holds if you can’t take more customers because upkeeping the infrastructure takes too much time or you need to hire extra person which takes more money than AWS bill difference.
- tstrimple 1y ago
  > discussions here on HN often seem to assume that you need HA and lots of scaling options. Funny how our perceptions differ. I seem to mostly see people saying all you need is a cheap Hetzner instance and postgres to solve all technical problems. We clearly all have different working environments and requirements. That's I roll my eyes at the suggestions in threads I see of going all in on colo. My last two major cloud migrations were due to colo facilities shutting down. They were getting kicked out and had a deadline. In one of the cases, the company I was working with was the second largest client at the colo but when the largest client decided to pull out the owners decided the economics of running the datacenter didn't make sense to them anymore. Switching colo facilities when you have a few servers isn't a big deal. It's annoying but manageable. When you have hundreds to thousands of servers, it becomes a major operational risk and is enormously disruptive to business as usual.
fulafel 1y ago
The direct cost is the easy part. The more insidious part is that you're now cultivating a growing staff of technologists whose careers depend on doing things the AWS way, getting AWS certified to ensure they build your systems the AWS Well Architected Way instead of thinking themselves, and can upsell you on AWS lock-in solutions using AWS provided soundbites and sales arguments. ("Shall we make the app very resilient to failure? Yes running on multiple regions makes the AWS bill bigger but you'll get much fewer outages, look at all this technobabble that proves it") And of course AWS lock-in services are priced to look cheaper compared to their overpricing of standard stuff[1] - if you just spend the engineering effort and IaC coding effort to move onto them, this "savings" can be put to more AWS cloud engineering effort which again makes your cloud eng org bigger and more important. [1] (For example implementing your app off containers to Lambda, or the db off PostgreSQL to DynamoDB etc)
- Hilift 1y ago
  > The direct cost is the easy part I don't think it is easy. I see most organizations struggle with the fact that everything is throttled in the cloud. CPU, storage, network. Tenants often discover large amounts of activity they were previously unaware of, that contributes to the usage and cost. And there may be individuals or teams creating new usages that are grossly impacting their allocation. Did you know there is a setting in MS SQL Server that impacts performance by an order of magnitude when sending/receiving data from the Cloud to your on-premises servers? It's the default in the ORM generated settings. Then you can start adding in the Cloud value, such as incomprehensible networking diagrams that are probably non-compliant in some way (guess which ones!), and security? What is it?
  - m-gasser 1y ago
    > Did you know there is a setting in MS SQL Server that impacts performance by an order of magnitude when sending/receiving data from the Cloud to your on-premises servers? It's the default in the ORM generated settings. Sounds interesting, which setting is that?
    - 1y ago
ownagefool 1y ago
The consequence of running ingress and DNS poorly is downtime. The consequence of running a database poorly is lost data. At the end of the day they're all just processes on a machine somewhere, none of it is particularly difficult, but storing, protecting, and traversing state is pretty much _the_ job and I can't really see how you'd think ingress and DNS would be more work than the datastores done right. Now with AWS, I have a SaaS that makes 6 figures and the AWS bill is <$1000 a month. I'm entirely capable of doing this on-prem, but the vast majority of the bill is s3 state, so what we're actually talking about is me being on-call for an object store and a database, and the potential consequences of doing so. With all that said, there's definitely a price point and staffing point where I will consider doing that, and I'm pretty down for the whole on-prem movement generally.
- vidarh 1y ago
  I'm generally strongly in favour of bare metal (not so much actually on prem) but your case is one of the rare cases wher AWS makes sense. Even for cheap setups like that, bare metal could likely be cheaper even factoring in someone on call to handle issues for you, but the amounts are so small it's a perfectly reasonable choice to just pick whatever you're comfortable with. That's the sweet spot for AWS customers. Not so much for AWS. The key thing for AWS is trying to get you locked in by "helping you" depend on services that are hard to replicate elsewhere, so that if your costs grow to a point where moving elsewhere is worth it, it's hard for you to do so.
neves 1y ago
It's always nice to remember that AWS is responsible for 70% of Amazon profits.
- vidarh 1y ago
  As Jeff Bezos has been quoted as saying "your margin is my opportunity"... The biggest difficulty in eating into AWS market share is that believing it is cheap has become religion.
mberning 1y ago
It’s expensive and the “design” of the services, if you could call it that, is such that you are forced to pay a lot, or play a lot of games to get around it. If you are going to spend your engineering time working around their ridiculous pricing schemes, you might as well spend the money on building things out yourself. Perfect example - MSK. The brokers are config locked at certain partition counts, even if your CPU is 5%. But their MSK replicator is capped on topic count. So now I have to work around topic counts at the cluster level, and partition counts at the broker level. Neither of which are inherent limits in the underlying technologies (kafka and mirrormaker)
vb-8448 1y ago
> Maybe I'm just the old man screaming at cloud (no pun intended) but when did people forget how to run a baremetal server ? It's a way to "commoditize" engineers. You can run on premise or mixed infra better and cheaper, but only if you know what you are doing. This requires experienced guys and doesn't work with new grad hired by big cons and sold ad "cloud experts".
- calgoo 1y ago
  Also, when something breaks, you are responsible. If you put it in AWS like everyone else and it breaks, then its their problem not yours. We will still implement workarounds and fixes when it happens, but we are not responsible. Basic enterprise rules these days is to always pay someone else to be responsible.
  - vidarh 1y ago
    Unless you put someone on retainer to be responsible, which you can do cheaper than to keep your AWS setup from breaking... (I do that for people; my AWS using customers consistently end up needing more help)
    - wredcoll 1y ago
      The point isn't cost, it's dodging responsibility.
      - vidarh 1y ago
        You can dodge responsibility equally well by outsourcing to people who'll run your bare metal setup for you. We exist from small consultancies like mine to huge multinationals.
  - vb-8448 1y ago
    Actually nothing new here, this was the same in the pre-cloud era where everyone in enterprises prefer big names(ibm, microsoft, oracle, ecc) to pass the responsibility to them in case of failures ... aka "nobody get fired because of buying IBM"
esskay 1y ago
> I'm so surprised there is so much pushback against this I'm not. It seems to be happening a lot. Any time a topic about not using AWS comes up here, or on Reddit there a sudden surge of people appearing out of nowhere shouting down anyone who suggests other options. It's honestly starting to feel like paid shilling.
- 7thaccount 1y ago
  I think some of that is a certain group of people will do anything to play with the new shiny stuff. In my org it's cloud and now GPU. The cloud stuff is extremely expensive and doesn't work any better than our existing solutions. Like a commentator said below, it's insidious as your entire organization later becomes dependent on that. If you buy a cloud solution, you're also stuck with the vendor deciding to double the cost of the product once you're locked in. The GPU stuff is annoying as all of our needs are fine with normal CPU workloads today. There are no performance issues, so again...what's the point? Well... somebody wants to play with GPUs I guess.
  - ghaff 1y ago
    Resume-driven development. It's probably pretty much always been a thing.
- TheCondor 1y ago
  It’s the current version of CCIE or some of the other certs. People pay money to learn how to operate AWS, other thing erode the value of their investment.
- Spooky23 1y ago
  I don’t think it’s paid shilling, it’s dogma that reflects where people are working here. The individual engineers are hammers and AWS is the nail. AWS/Azure/GCP is great, but like any tool or platform you need to do some financial/process engineering to make an optimal choice. For small companies, time to market is often key, hence AWS. Once you’re a little bigger, you may develop frameworks to operate efficiently. I have apps that I run in a data center because they’d cot 10-20x at a cloud provider. Conversely, I have apps that get more favorable licensing terms in AWS that I run there, even though the compute is slower and less efficient. You also have people who treat AWS with the old “nobody gets fired for buying IBM” mentality.
realitysballs 1y ago
For my org. I don’t have budget for a dedicated in-house opsec team, so if I on-prem it triggers additional salary burden for security . How would I overcome this?
- Msurrow 1y ago
  Familiarize yourself with your company’s decision process on strategic decisions like this. Ensure you have a way to submit a proposal for a decision on making the change (or find someone who has that access to sponsor your proposal), build a business case that shows cost of opsec team, hardware and everything else is lower than AWS (or if cost is higher then some other business value is gained from making the change — currently digital sovereignty could be a strong argument if you are EU based). If you cant build a positive business case then its not the correct move. Cash is king. Sadly.
- Ensorceled 1y ago
  You can't. That's the use case FOR AWS/GCP. Once the differential between having a in-house team and the AWS premium becomes positive is when you make the switch. A lot of the discussion here is that the cost of the in-house team is less than people think. For instance: at a former gig, we used a service in the EU that handled weekends, holidays and night time issues and escalated to our team as needed. It was pretty cheap, approximately $10K monthly fee for availability and hourly rate when there were any issues to be resolved. There were a few mornings I had an email with a post-mortem report and an invoice for a hundred euros or so. We came pretty close to 5 9's uptime but we didn't have to worry about SLA's or anything.
  - spwa4 1y ago
    There is also the factor that the idea that you don't need administrators for AWS is bullshit. Cool idea, bro. Go to your favorite jobs portal. Search for "devops" ... 1000s of jobs. I click on the first link. Well, well, they have a whole team doing "devops administration" on AWS and require extra people. So not having the money for an in-house team ... no AWS for you. I've worked for 2 large-ish firms in the past 3 years. One huge telco, one "medium" telco (still 100s of people). BOTH had a team just for AWS IAM administration. Only for that one thing, because that was company-wide (and was regularly demonstrated to be a single point of failure). And they had AWS administrator teams, yes teams, for every department (even HR had one, though in the medium telco all management had a shared team, but the networking and development departments still had their own AWS teams, who, btw, also did IAM. The company-wide IAM team maintained an AWS IAM and some solution they'd bought that also worked for their windows domain and ticketing system (I hate you IBM remedy), and eqiupment ordering portal and ...) AND there were "devops" positions on every development team, and on the network engineering team, and even a small one for the building "technics" team. Oh and they both had an internal cluster on top of AWS, part on-premise, part rented DC space, which did at least half the compute work (but presumably a lot less of the weird edge-cases), that one ran the company services that are just insane on AWS like any kind of video.
yomismoaqui 1y ago
> Maybe I'm just the old man screaming at cloud (no pun intended) but when did people forget how to run a baremetal server ? We should coin the term "Cloud Learned Helplessness"
vidarh 1y ago
There is this belief that it is not extremely expensive and/or that the ops cost of bare metal will outpace it. It is a belief, and it is very rarely supported by facts. Having done consulting in this space for a decade, and worked with containerised systems since before AWS existed, my experience is that managing an AWS system is consistently more expensive and that in fact the devops cost is part of what makes AWS an expensive option.
steelegbr 1y ago
AWS may be overcharging but it's a balancing act. Going on-prem (well, shared DC) will be cheaper but comes with requirements for either jack of all trades sysadmins or a bunch of specialists. It can work well if your product is simple and scalable. A lot of places quietly achieve this. That said, I've seen real world scenarios where complexity is up the wazoo and an opex cost focus means you're hiring under skilled staff to manage offerings built on components with low sticker prices. Throw in a bit of the old NIH mindset (DIY all the things!) and it's large blast radii with expensive service credits being dished out to customers regularly. On a human factors front your team will be seeing countless middle of the night conference calls. While I'm not 100% happy with the AWS/Azure/GCP world, the reality is that on-prem skillsets are becoming rarer and more specialist. Hiring good people can be either really expensive or a bit of a unicorn hunt.
- PenguinCoder 1y ago
  I'm proudly 100% on prem Linux sys admin. There are not openings for my skills and they do not pay as well as whatever cloud hotness is "needed".
  - whstl 1y ago
    That's the crazy thing. Most AWS-only Ops engineers I know are making bank and in high demand, and Ops teams are always HUGE in terms of headcount outside of startups. The "AWS is cheaper" thing is the biggest grift in our industry.
    - hedora 1y ago
      I wonder how vibe coding will impact this. You can easily get your service up by asking claude code or whatever to just do it It produces aws yaml that’s better than many devops people I’ve worked with. In other words, it absolutely should not be trusted with trivial tasks, but you could easily blow $100K’s per year for worse.
      - throwforfeds 1y ago
speleding 1y ago
The complexity of AWS versus bare metal depends on what you are doing. Setting up an apache app server: just as easy on bare metal. Setting up high availability MySQL with hot failover: much easier on AWS. And a lot of businesses need a highly available database.
- spwa4 1y ago
  A high availability MySQL server on AWS is about the same difficulty as on your own kubernetes instance (I've got a play one on one of those $100 N100 machines, got one with 16G mem). Then: helm repo add mariadb-operator https://mariadb-operator.github.io/mariadb-operator helm install mariadb-operator mariadb-operator/mariadb-operator And then you can just provision MariaDB "kind", ie. you kubectl apply with something specifying database name, maximum memory, type of high availability (single primary or multimaster) and secret reference and there you go: new database, ready to be plugged into other pods.
  - papichulo2023 1y ago
    Dont you need ECC in your db nodes?
    - dd_xplore 1y ago
      N100 supports DDR5 memory (although 1 channel) but I believe DDR5 has some error correction... May not be full ECC
    - 1oooqooq 1y ago
      amazing how nobody even know about ECC these days. see so many series B+ companies running DB and storage without a care in the world.
    - spwa4 1y ago
      N100 is my homelab, for playing. For instance I have a kubernetes cluster running KubeVirt, which runs 5 VMs, which ... have a kubernetes installation (so I have multiple worker nodes doing a "distributed filesystem" all of which is resharing disks from the same SSD). My production servers are generally older Xeons with ECC ram, which are also running kubernetes.
izacus 1y ago
A lot of people here have built their whole professional careers around knowing AWS and deploying to it. Moving away is an existential issue for them - this is why there's such pushback. A huge % of new developer and devops generation doesn't know anything about deploying software on bare metal or even other clouds and they're terrified about being unemployed.
- goalieca 1y ago
  meanwhile skills in operating systems, networking, and optimization are declining. Every system i've seen in the last 10 years or so has left huge cash on the table by not being aware of the basics.
  - snoman 1y ago
    That could have more to do with containerization than the cloud - and that was a goal if I recall.
Aurornis 1y ago
> I'm so surprised there is so much pushback against this.. AWS is extremely expensive. I see more comments in favor than pushing back. The problem I have with these stories is the confirmation bias that comes with them. Going self-hosted or on-premises does make sense in some carefully selected use cases, but I have dozens of stories of startup teams spinning their wheels with self-hosting strategies that turn into a big waste of time and headcount that they should have been using to grow their businesses instead. The shared theme of all of the failure stories is missing the true cost of self-hosting: The hours spent getting the servers just right, managing the hosting, debating the best way to run things, and dealing with little issues add up but are easily lost in the noise if you’re not looking closely. Everyone goes through a honeymoon phase where the servers arrive and your software is up and running and you’re busy patting yourselves on the back about how you’re saving money. The real test comes 12 months later when the person who last set up the servers has left for a new job and the team is trying to do forensics to understand why the documentation they wrote doesn’t actually match what’s happening on the servers, or your project managers look back at the sprints and realize that the average time spent on self-hosting related tasks and ideas has added up to a lot more than anyone would have guessed. Those stories aren’t shared as often. When they are, they’re not upvoted. A lot of people in my local startup scene have sheepish stories about how they finally threw in the towel on self-hosting and went to AWS and got back to focusing on their core product. Few people are writing blog posts about that because it’s not a story people want to hear. We like the heroic stories where someone sets up some servers and everything just works perfectly and there are no downsides. You really need to weigh the tradeoffs, but many people are not equipped to do that. They just think their chosen solution will be perfect and the other side will be the bad one.
- DrewADesign 1y ago
  > The shared theme of all of the failure stories is missing the true cost of self-hosting: The hours spent getting the servers just right, managing the hosting, debating the best way to run things, and dealing with little issues add up but are easily lost in the noise if you’re not looking closely. What the modern software business seems to have lost is the understanding that ops and dev are two different universes. DevOps was a reaction to the fact that even outsourcing ops to AWS doesn’t entirely solve all of your ops problems and the role is absolutely no substitute for a systems administrator. Having someone that helps derive the requirements for your infrastructure, then designs it, builds it , backs it up, maintains it, troubleshoots it, monitors performance, determines appropriate redundancy, etc. etc. etc. and then tells the developers how to work with it is the missing link. Hit-by-a-bus documentation, support and update procedures, security incident response… these are all problems we solved a long time ago, but sort of forgot about moving everything to cloud architecture.
zjaffee 1y ago
AWS (along with the vast majority of B2B services in the software development industry) is good because it allows you to focus on building your product or business without needing to worry about managing servers nearly as much. The problems here are no different than using SaaS anywhere else in a business, you can also run all your sales tracking through excel, it's just that once you have more than a few people doing sales that becomes a major bottleneck the same way not having an easier to manage infrastructure system.
maccard 1y ago
I work for a small company owned by a huge company. We are entirely independent except for purchasing, IT, and budget approval. We run our CI on AWS, and it’s slow and flaky for a variety of reasons (compiling large c++ projects combined with instance type pressure). It’s also expensive. We planned a migration to move from 4OD instances to one on prem machine and we guessed we’d save $1000/mo, our builds would be faster and we’d have less failures due to capacity issues. We even had a spare workstation and a rack in the office that so the capex was 0. I plugged the machine into the rack and no internet connectivity. Put in an IT ticket which took 2 days for a reply, only to be told that this was an unauthorised machine and needed to be imaged by IT. The back and forth took 4 weeks, multiple meetings and multiple approvals. My guess is that 4 people spent probably 10 hours arguing whether we should do this or not. On AWS I can write a python script and have a running windows instance in 15 minutes.
- wredcoll 1y ago
  This is the root success of aws, it lets internal teams bypass sysadmin departments.
  - ghaff 1y ago
    Working around official IT was certainly a significant factor early on. I'm less convinced it is nearly as big a driver (or a downside depending on your perspective) today.
    - whstl 1y ago
      Especially considering that outside of startups (where approval would be fast with or without cloud), virtual infrastructure also got its own bureaucratic process.
      - ghaff 1y ago
        A lot of people forget that, when server virtualization was still gaining momentum in a lot of circles, it wasn't uncommon at less technically savvy customers--say a regional bank at the time--to be told that it might take 2 months to provision a new server.
UltraSane 1y ago
I'm not going to argue that AWS can be expensive but in my experience its biggest advantage is SPEED. In every company I worked for that ran their own data centers ever damn thing took FOREVER. new servers took months to buy and rack. any network change like a new VLAN took days to weeks. It was so annoying. But in AWS almost anything is just an API call and a few minutes at most from being enabled. It is so much more productive.
SJC_Hacker 1y ago
> I'm so surprised there is so much pushback against this.. AWS is extremely expensive. The use cases for setting up your system or service entirely in AWS are more rare than people seem to realise. Maybe I'm just the old man screaming at cloud (no pun intended) but when did people forget how to run a baremetal server ? Long term yes you can save money rolling your own. But with cloud you can get something up and running within maybe a few days, sometimes even faster. Often with built in scalability. This is a much easier sell to the non-tech (i.e., money) people. If the project continues, the path of least resistance is often to just continue with the cloud solution. At a certain point, there will be so much tech debt that any savings from long term costs from the traditional on-premises, co-location or managed hosting, are vastly by the cost of migration.
comprev 1y ago
I'm on a Platform team of <8 people and only 3 of us (most experienced too) come from sysadmin backgrounds. The rest have only ever known containers/cloud and never touched (both figuratively and literally :-) bare metal servers in their careers. They've never used tools like Ansible (or Anaconda) or been in situations where they couldn't destroy the container and start afresh instantly.
eek2121 1y ago
I once moved a small site from AWS to Digital Ocean + Cloudflare. $100-$300 on AWS -> $35/mo for DO + CF. Coincidentally, AWS had an outage soon after, which was avoided thanks to the move. I have used DO for both clients and myself, and have not had any huge problems with them.
rdtsc 1y ago
> I'm so surprised there is so much pushback against this.. AWS is extremely expensive. Basic rationalization. People will go to extraordinary lengths to justify and defend the choices they made. It's a defense mechanism: if they spent millions on AWS they are not going to sit idly while HN discusses saving hundreds of thousands with everyone nodding and agreeing. It's important for their own sanity to defend the choice they made.
- TacticalCoder 1y ago
  [dead]
j45 1y ago
The cloud is incredibly profitable for the efficiencies and improvements its introduced and held onto. Easy to push back against what is now the unknown (bare metal), when the layers extending bare metal to cloud service have become better and better, as well as more accessible.
dumbledoren 1y ago
> when did people forget how to run a baremetal server ? Bigger question: When did people forget that doing that is much easier than AWS...
citizenpaul 1y ago
The "value add" of AWS has never been what it can do or does. It has always appealed to weak/incompetent/sociopathic managers and execs desire to not have to deal with capable employees. As far as they are concerned AWS is taking care of computing AND hiring for them. I've never worked anywhere that at least some sort of power holder would instantly go to consultants or outsourcing rather than in house because they believe that if you work for the company you must be incompetent, dumb or below average. If you don't work for them you must be exceptional.
_ea1k 1y ago
> I'm so surprised there is so much pushback against this.. Same, this trend towards "AWS all the things" has really amazed me. We've all mocked small companies copying big companies by trying to make their app super-duper scalable from the very start. After all, everyone things they are the next google, despite their 5 total users right now. But this is really the opposite. AWS is phenomenal for the startup that would readily trade high opex for lower capex. Servers aren't the cheapest things in the world to buy and they depreciate. It makes total sense for startups to start this way. But why are big companies, with an actual budget for staff, copying the behavior of their favorite startups?
- guax 1y ago
  Opex looks nicer on the sheets than capex for large deployments. Incredible high investment from AWS on luring in C level with "white-papers" and promises of cost and governance magical revolutions. I've heard the promise of cheaper, faster where you can focus on "innovation". I am yet to see any of it become a reality.
mk89 1y ago
How would you do multi-region deployments with your own DC? This is an issue for several companies that start small and within 5 years they find the need to expand abroad. Be it for data sovereignty or so, which is becoming more important than ever in the last 10 years. Duplicating a region is "a few clicks away" on AWS. This is what the provider enables you to do. This and a lot of other things. And for such things, yes, you gotta pay.
- Hikikomori 1y ago
  I mean its not that complicated. Rent space in another location, get separate fibers/wavelength between them, redundant internet connection. But if you're in a growth/startup phase it doesn't make much sense to spend engineering time on this, not that multi region setups in Aws is one button either. Once you're past that and paying aws a million per week or so I think it can make sense to offload expensive services to your own hardware.
axegon_ 1y ago
I am not - I hate AWS(and cloud in general) with a passion - overpriced, you are getting locked in by a closed ecosystem the moment you say "hey this feature is neat it will save me so much work", only to realize that you are stuck paying for it for years if you decide to move away from it. But people are inclined to jump on a hype train and become evangelists for life. Truth is AWS(or GCP or Azure or anything else) is a viable option in two cases: 1. You are making a product with 3 friends on evenings and you want to ship asap without having the capacity to invest and setup infrastructure. 2. You are a huge corporation with tens of thousands of employees and hardware needs that you simply cannot source yourself easily or sort out the collocation of the hardware. Everyone else - get a dozen second-hand servers, shove them in a rack in a data center and you will own the hardware and everything associated with it at half the price of what you'd be paying AWS in a year.
TacticalCoder 1y ago
[dead]
kelnos 1y ago
> when did people forget how to run a baremetal server ? I don't think people have forgotten, but I think Amazon has done an amazingly excellent job of marketing and developer relations over the years, to the point that they've convinced most developers that doing your own thing is 1) a lot of expensive, specialized work, and 2) actively dangerous for your business, whether it's because of security, uptime, or some other ops bogeyman of the day. Note that I said "developers". Most developers are not sysadmins or IT operations people. Most of them have never set up Linux on a desktop or laptop, let alone on real server hardware (that they've also set up themselves). Most of them never had the chance to forget how to run a bare-metal server; they never knew how in the first place. (Hell, I've been running desktop Linux for 25+ years, and I don't think I've ever set up Linux on actual server hardware. Closest I've come is bare-metal Solaris, but that was like 25 years ago.) "DevOps" today usually means that you know how to run a CLI tool or drive a web interface to deploy your automatically-built container artifact to some cloud-based production system that someone else manages, hiding the details from you. (This bit also can be true for shops that run on bare metal, depending on how advanced their own sysadmin/ops team is.) While developers are often not decision-makers in a larger org, they can be at smaller orgs, and once those developers get on the cloud, you probably will stay on the cloud (companies like OneUptime are the exception, not the rule), even if you've gotten much larger and it's stupid expensive to continue running that way.
deleted 1y ago
[deleted]
hoppp 1y ago
AWS spends a lot in educating developers to use their service and it's a working strategy. People simply don't believe bare metal is better because of this conditioning towards the cloud. Every company should strive for self-sufficiency, but this idea is not that widespread in the software industry.
rasjani 1y ago
> when did people forget how to run a baremetal server ? My opinion on this: docker sort of changed the game here. It sort of enabled a lot of people to get a "new and fresh" level of abstraction to not bother about bare metal. As an example, I work in company where most consultants are doing DevOps and k8 is big part of that. What made me consider that? I've been told multiple times that "you know your stuff" when I mention some kernel or userland feature that container approach provides.

JCM9 1y ago

As the author points out AWS can provide a few things that you wouldn’t want to try and replicate (like CloudFront) but for most other things you’re very much correct. AWS is ultimately very expensive for what it is. The complicated billing that’s full of surprises also makes cost management a head-banging experience.

tyingq 1y ago
Fair, though using AWS solely for CloudFront would mean you should compare to Cloudflare, Akamai, Fastly, etc. I'm not sure if the value prop for it looks so great if you don't include the "integrated with your other AWS stuff" benefit.
- vidarh 1y ago
  I mean, AWS egress is so expensive that I'd put something else in front of it for anyone who has any decent amount of traffic.
- JCM9 1y ago
  Agree, CloudFront isn’t super competitive with CDN focused vendors. It’s basically the “well you’re already on AWS so may as well just use this” play.

jagged-chisel 1y ago

Forget? You have to hire people for that. We are a software organization. We build software. If we rent in the cloud, there is less HR hassle - hiring, raises, bonuses, benefits, firing … none of that headache involved with the cloud. Technically? Totally doable. But the owners prefer renting in the cloud over the people-related issues of hiring.

sieabahlpark 1y ago
[dead]
bilekas 1y ago
> Forget? You have to hire people for that. We are a software organization. We build software. You don't need to hire dedicated people full time. It could even be outsourced and then a small contract for maintenance. It's the same argument you could say for "accounting persons", or "HR persons" - "We are a software organisation!" - Personally I don't buy the argument.
- Foobar8568 1y ago
  Outsourcing and cloud cost are always underestimated.
- deleted 1y ago
  [deleted]
- theta_d 1y ago
  > It could even be outsourced and then a small contract for maintenance. Yeah, those people we outsourced to happen to work at AWS.
  - vidarh 1y ago
    They don't though. You still need devops when you use AWS, and most organisations end up needing more time spent on devops when they use AWS.
chasd00 1y ago

fabian2k 1y ago

A large part of the different views on this topic are due to the way people estimate the amount of saved effort and money because you're pushing some admin duties to the cloud provider instead of doing this yourself. And people come to vastly different conclusions on this aspect. It's also that the requirements vary a lot, discussions here on HN often seem to assume that you need HA and lots of scaling options. That isn't universally true.

nicce 1y ago
> A large part of the different views on this topic are due to the way people estimate the amount of saved effort and money because you're pushing some admin duties to the cloud provider instead of doing this yourself. And people come to vastly different conclusions on this aspect This applies only if you had an extra customer that pays the difference. Basically argument only holds if you can’t take more customers because upkeeping the infrastructure takes too much time or you need to hire extra person which takes more money than AWS bill difference.
tstrimple 1y ago
> discussions here on HN often seem to assume that you need HA and lots of scaling options. Funny how our perceptions differ. I seem to mostly see people saying all you need is a cheap Hetzner instance and postgres to solve all technical problems. We clearly all have different working environments and requirements. That's I roll my eyes at the suggestions in threads I see of going all in on colo. My last two major cloud migrations were due to colo facilities shutting down. They were getting kicked out and had a deadline. In one of the cases, the company I was working with was the second largest client at the colo but when the largest client decided to pull out the owners decided the economics of running the datacenter didn't make sense to them anymore. Switching colo facilities when you have a few servers isn't a big deal. It's annoying but manageable. When you have hundreds to thousands of servers, it becomes a major operational risk and is enormously disruptive to business as usual.

fulafel 1y ago

The direct cost is the easy part. The more insidious part is that you're now cultivating a growing staff of technologists whose careers depend on doing things the AWS way, getting AWS certified to ensure they build your systems the AWS Well Architected Way instead of thinking themselves, and can upsell you on AWS lock-in solutions using AWS provided soundbites and sales arguments. ("Shall we make the app very resilient to failure? Yes running on multiple regions makes the AWS bill bigger but you'll get much fewer outages, look at all this technobabble that proves it") And of course AWS lock-in services are priced to look cheaper compared to their overpricing of standard stuff[1] - if you just spend the engineering effort and IaC coding effort to move onto them, this "savings" can be put to more AWS cloud engineering effort which again makes your cloud eng org bigger and more important. [1] (For example implementing your app off containers to Lambda, or the db off PostgreSQL to DynamoDB etc)

Hilift 1y ago
> The direct cost is the easy part I don't think it is easy. I see most organizations struggle with the fact that everything is throttled in the cloud. CPU, storage, network. Tenants often discover large amounts of activity they were previously unaware of, that contributes to the usage and cost. And there may be individuals or teams creating new usages that are grossly impacting their allocation. Did you know there is a setting in MS SQL Server that impacts performance by an order of magnitude when sending/receiving data from the Cloud to your on-premises servers? It's the default in the ORM generated settings. Then you can start adding in the Cloud value, such as incomprehensible networking diagrams that are probably non-compliant in some way (guess which ones!), and security? What is it?
- m-gasser 1y ago
  > Did you know there is a setting in MS SQL Server that impacts performance by an order of magnitude when sending/receiving data from the Cloud to your on-premises servers? It's the default in the ORM generated settings. Sounds interesting, which setting is that?
  - 1y ago

ownagefool 1y ago

The consequence of running ingress and DNS poorly is downtime. The consequence of running a database poorly is lost data. At the end of the day they're all just processes on a machine somewhere, none of it is particularly difficult, but storing, protecting, and traversing state is pretty much _the_ job and I can't really see how you'd think ingress and DNS would be more work than the datastores done right. Now with AWS, I have a SaaS that makes 6 figures and the AWS bill is <$1000 a month. I'm entirely capable of doing this on-prem, but the vast majority of the bill is s3 state, so what we're actually talking about is me being on-call for an object store and a database, and the potential consequences of doing so. With all that said, there's definitely a price point and staffing point where I will consider doing that, and I'm pretty down for the whole on-prem movement generally.

vidarh 1y ago
I'm generally strongly in favour of bare metal (not so much actually on prem) but your case is one of the rare cases wher AWS makes sense. Even for cheap setups like that, bare metal could likely be cheaper even factoring in someone on call to handle issues for you, but the amounts are so small it's a perfectly reasonable choice to just pick whatever you're comfortable with. That's the sweet spot for AWS customers. Not so much for AWS. The key thing for AWS is trying to get you locked in by "helping you" depend on services that are hard to replicate elsewhere, so that if your costs grow to a point where moving elsewhere is worth it, it's hard for you to do so.

neves 1y ago

It's always nice to remember that AWS is responsible for 70% of Amazon profits.

vidarh 1y ago
As Jeff Bezos has been quoted as saying "your margin is my opportunity"... The biggest difficulty in eating into AWS market share is that believing it is cheap has become religion.

mberning 1y ago

It’s expensive and the “design” of the services, if you could call it that, is such that you are forced to pay a lot, or play a lot of games to get around it. If you are going to spend your engineering time working around their ridiculous pricing schemes, you might as well spend the money on building things out yourself. Perfect example - MSK. The brokers are config locked at certain partition counts, even if your CPU is 5%. But their MSK replicator is capped on topic count. So now I have to work around topic counts at the cluster level, and partition counts at the broker level. Neither of which are inherent limits in the underlying technologies (kafka and mirrormaker)

vb-8448 1y ago

> Maybe I'm just the old man screaming at cloud (no pun intended) but when did people forget how to run a baremetal server ? It's a way to "commoditize" engineers. You can run on premise or mixed infra better and cheaper, but only if you know what you are doing. This requires experienced guys and doesn't work with new grad hired by big cons and sold ad "cloud experts".

calgoo 1y ago
Also, when something breaks, you are responsible. If you put it in AWS like everyone else and it breaks, then its their problem not yours. We will still implement workarounds and fixes when it happens, but we are not responsible. Basic enterprise rules these days is to always pay someone else to be responsible.
- vidarh 1y ago
  Unless you put someone on retainer to be responsible, which you can do cheaper than to keep your AWS setup from breaking... (I do that for people; my AWS using customers consistently end up needing more help)
  - wredcoll 1y ago
    The point isn't cost, it's dodging responsibility.
    - vidarh 1y ago
      You can dodge responsibility equally well by outsourcing to people who'll run your bare metal setup for you. We exist from small consultancies like mine to huge multinationals.
- vb-8448 1y ago
  Actually nothing new here, this was the same in the pre-cloud era where everyone in enterprises prefer big names(ibm, microsoft, oracle, ecc) to pass the responsibility to them in case of failures ... aka "nobody get fired because of buying IBM"

esskay 1y ago

> I'm so surprised there is so much pushback against this I'm not. It seems to be happening a lot. Any time a topic about not using AWS comes up here, or on Reddit there a sudden surge of people appearing out of nowhere shouting down anyone who suggests other options. It's honestly starting to feel like paid shilling.

7thaccount 1y ago
I think some of that is a certain group of people will do anything to play with the new shiny stuff. In my org it's cloud and now GPU. The cloud stuff is extremely expensive and doesn't work any better than our existing solutions. Like a commentator said below, it's insidious as your entire organization later becomes dependent on that. If you buy a cloud solution, you're also stuck with the vendor deciding to double the cost of the product once you're locked in. The GPU stuff is annoying as all of our needs are fine with normal CPU workloads today. There are no performance issues, so again...what's the point? Well... somebody wants to play with GPUs I guess.
- ghaff 1y ago
  Resume-driven development. It's probably pretty much always been a thing.
TheCondor 1y ago
It’s the current version of CCIE or some of the other certs. People pay money to learn how to operate AWS, other thing erode the value of their investment.
Spooky23 1y ago
I don’t think it’s paid shilling, it’s dogma that reflects where people are working here. The individual engineers are hammers and AWS is the nail. AWS/Azure/GCP is great, but like any tool or platform you need to do some financial/process engineering to make an optimal choice. For small companies, time to market is often key, hence AWS. Once you’re a little bigger, you may develop frameworks to operate efficiently. I have apps that I run in a data center because they’d cot 10-20x at a cloud provider. Conversely, I have apps that get more favorable licensing terms in AWS that I run there, even though the compute is slower and less efficient. You also have people who treat AWS with the old “nobody gets fired for buying IBM” mentality.

realitysballs 1y ago

For my org. I don’t have budget for a dedicated in-house opsec team, so if I on-prem it triggers additional salary burden for security . How would I overcome this?

Msurrow 1y ago
Familiarize yourself with your company’s decision process on strategic decisions like this. Ensure you have a way to submit a proposal for a decision on making the change (or find someone who has that access to sponsor your proposal), build a business case that shows cost of opsec team, hardware and everything else is lower than AWS (or if cost is higher then some other business value is gained from making the change — currently digital sovereignty could be a strong argument if you are EU based). If you cant build a positive business case then its not the correct move. Cash is king. Sadly.
Ensorceled 1y ago
You can't. That's the use case FOR AWS/GCP. Once the differential between having a in-house team and the AWS premium becomes positive is when you make the switch. A lot of the discussion here is that the cost of the in-house team is less than people think. For instance: at a former gig, we used a service in the EU that handled weekends, holidays and night time issues and escalated to our team as needed. It was pretty cheap, approximately $10K monthly fee for availability and hourly rate when there were any issues to be resolved. There were a few mornings I had an email with a post-mortem report and an invoice for a hundred euros or so. We came pretty close to 5 9's uptime but we didn't have to worry about SLA's or anything.
- spwa4 1y ago
  There is also the factor that the idea that you don't need administrators for AWS is bullshit. Cool idea, bro. Go to your favorite jobs portal. Search for "devops" ... 1000s of jobs. I click on the first link. Well, well, they have a whole team doing "devops administration" on AWS and require extra people. So not having the money for an in-house team ... no AWS for you. I've worked for 2 large-ish firms in the past 3 years. One huge telco, one "medium" telco (still 100s of people). BOTH had a team just for AWS IAM administration. Only for that one thing, because that was company-wide (and was regularly demonstrated to be a single point of failure). And they had AWS administrator teams, yes teams, for every department (even HR had one, though in the medium telco all management had a shared team, but the networking and development departments still had their own AWS teams, who, btw, also did IAM. The company-wide IAM team maintained an AWS IAM and some solution they'd bought that also worked for their windows domain and ticketing system (I hate you IBM remedy), and eqiupment ordering portal and ...) AND there were "devops" positions on every development team, and on the network engineering team, and even a small one for the building "technics" team. Oh and they both had an internal cluster on top of AWS, part on-premise, part rented DC space, which did at least half the compute work (but presumably a lot less of the weird edge-cases), that one ran the company services that are just insane on AWS like any kind of video.

yomismoaqui 1y ago

> Maybe I'm just the old man screaming at cloud (no pun intended) but when did people forget how to run a baremetal server ? We should coin the term "Cloud Learned Helplessness"

vidarh 1y ago

There is this belief that it is not extremely expensive and/or that the ops cost of bare metal will outpace it. It is a belief, and it is very rarely supported by facts. Having done consulting in this space for a decade, and worked with containerised systems since before AWS existed, my experience is that managing an AWS system is consistently more expensive and that in fact the devops cost is part of what makes AWS an expensive option.

steelegbr 1y ago

AWS may be overcharging but it's a balancing act. Going on-prem (well, shared DC) will be cheaper but comes with requirements for either jack of all trades sysadmins or a bunch of specialists. It can work well if your product is simple and scalable. A lot of places quietly achieve this. That said, I've seen real world scenarios where complexity is up the wazoo and an opex cost focus means you're hiring under skilled staff to manage offerings built on components with low sticker prices. Throw in a bit of the old NIH mindset (DIY all the things!) and it's large blast radii with expensive service credits being dished out to customers regularly. On a human factors front your team will be seeing countless middle of the night conference calls. While I'm not 100% happy with the AWS/Azure/GCP world, the reality is that on-prem skillsets are becoming rarer and more specialist. Hiring good people can be either really expensive or a bit of a unicorn hunt.

PenguinCoder 1y ago
I'm proudly 100% on prem Linux sys admin. There are not openings for my skills and they do not pay as well as whatever cloud hotness is "needed".
- whstl 1y ago
  That's the crazy thing. Most AWS-only Ops engineers I know are making bank and in high demand, and Ops teams are always HUGE in terms of headcount outside of startups. The "AWS is cheaper" thing is the biggest grift in our industry.
  - hedora 1y ago
    I wonder how vibe coding will impact this. You can easily get your service up by asking claude code or whatever to just do it It produces aws yaml that’s better than many devops people I’ve worked with. In other words, it absolutely should not be trusted with trivial tasks, but you could easily blow $100K’s per year for worse.
    - throwforfeds 1y ago

speleding 1y ago

The complexity of AWS versus bare metal depends on what you are doing. Setting up an apache app server: just as easy on bare metal. Setting up high availability MySQL with hot failover: much easier on AWS. And a lot of businesses need a highly available database.

spwa4 1y ago
A high availability MySQL server on AWS is about the same difficulty as on your own kubernetes instance (I've got a play one on one of those $100 N100 machines, got one with 16G mem). Then: helm repo add mariadb-operator https://mariadb-operator.github.io/mariadb-operator helm install mariadb-operator mariadb-operator/mariadb-operator And then you can just provision MariaDB "kind", ie. you kubectl apply with something specifying database name, maximum memory, type of high availability (single primary or multimaster) and secret reference and there you go: new database, ready to be plugged into other pods.
- papichulo2023 1y ago
  Dont you need ECC in your db nodes?
  - dd_xplore 1y ago
    N100 supports DDR5 memory (although 1 channel) but I believe DDR5 has some error correction... May not be full ECC
  - 1oooqooq 1y ago
    amazing how nobody even know about ECC these days. see so many series B+ companies running DB and storage without a care in the world.
  - spwa4 1y ago
    N100 is my homelab, for playing. For instance I have a kubernetes cluster running KubeVirt, which runs 5 VMs, which ... have a kubernetes installation (so I have multiple worker nodes doing a "distributed filesystem" all of which is resharing disks from the same SSD). My production servers are generally older Xeons with ECC ram, which are also running kubernetes.

izacus 1y ago

A lot of people here have built their whole professional careers around knowing AWS and deploying to it. Moving away is an existential issue for them - this is why there's such pushback. A huge % of new developer and devops generation doesn't know anything about deploying software on bare metal or even other clouds and they're terrified about being unemployed.

goalieca 1y ago
meanwhile skills in operating systems, networking, and optimization are declining. Every system i've seen in the last 10 years or so has left huge cash on the table by not being aware of the basics.
- snoman 1y ago
  That could have more to do with containerization than the cloud - and that was a goal if I recall.

Aurornis 1y ago

> I'm so surprised there is so much pushback against this.. AWS is extremely expensive. I see more comments in favor than pushing back. The problem I have with these stories is the confirmation bias that comes with them. Going self-hosted or on-premises does make sense in some carefully selected use cases, but I have dozens of stories of startup teams spinning their wheels with self-hosting strategies that turn into a big waste of time and headcount that they should have been using to grow their businesses instead. The shared theme of all of the failure stories is missing the true cost of self-hosting: The hours spent getting the servers just right, managing the hosting, debating the best way to run things, and dealing with little issues add up but are easily lost in the noise if you’re not looking closely. Everyone goes through a honeymoon phase where the servers arrive and your software is up and running and you’re busy patting yourselves on the back about how you’re saving money. The real test comes 12 months later when the person who last set up the servers has left for a new job and the team is trying to do forensics to understand why the documentation they wrote doesn’t actually match what’s happening on the servers, or your project managers look back at the sprints and realize that the average time spent on self-hosting related tasks and ideas has added up to a lot more than anyone would have guessed. Those stories aren’t shared as often. When they are, they’re not upvoted. A lot of people in my local startup scene have sheepish stories about how they finally threw in the towel on self-hosting and went to AWS and got back to focusing on their core product. Few people are writing blog posts about that because it’s not a story people want to hear. We like the heroic stories where someone sets up some servers and everything just works perfectly and there are no downsides. You really need to weigh the tradeoffs, but many people are not equipped to do that. They just think their chosen solution will be perfect and the other side will be the bad one.

DrewADesign 1y ago
> The shared theme of all of the failure stories is missing the true cost of self-hosting: The hours spent getting the servers just right, managing the hosting, debating the best way to run things, and dealing with little issues add up but are easily lost in the noise if you’re not looking closely. What the modern software business seems to have lost is the understanding that ops and dev are two different universes. DevOps was a reaction to the fact that even outsourcing ops to AWS doesn’t entirely solve all of your ops problems and the role is absolutely no substitute for a systems administrator. Having someone that helps derive the requirements for your infrastructure, then designs it, builds it , backs it up, maintains it, troubleshoots it, monitors performance, determines appropriate redundancy, etc. etc. etc. and then tells the developers how to work with it is the missing link. Hit-by-a-bus documentation, support and update procedures, security incident response… these are all problems we solved a long time ago, but sort of forgot about moving everything to cloud architecture.

zjaffee 1y ago

AWS (along with the vast majority of B2B services in the software development industry) is good because it allows you to focus on building your product or business without needing to worry about managing servers nearly as much. The problems here are no different than using SaaS anywhere else in a business, you can also run all your sales tracking through excel, it's just that once you have more than a few people doing sales that becomes a major bottleneck the same way not having an easier to manage infrastructure system.

maccard 1y ago

I work for a small company owned by a huge company. We are entirely independent except for purchasing, IT, and budget approval. We run our CI on AWS, and it’s slow and flaky for a variety of reasons (compiling large c++ projects combined with instance type pressure). It’s also expensive. We planned a migration to move from 4OD instances to one on prem machine and we guessed we’d save $1000/mo, our builds would be faster and we’d have less failures due to capacity issues. We even had a spare workstation and a rack in the office that so the capex was 0. I plugged the machine into the rack and no internet connectivity. Put in an IT ticket which took 2 days for a reply, only to be told that this was an unauthorised machine and needed to be imaged by IT. The back and forth took 4 weeks, multiple meetings and multiple approvals. My guess is that 4 people spent probably 10 hours arguing whether we should do this or not. On AWS I can write a python script and have a running windows instance in 15 minutes.

wredcoll 1y ago
This is the root success of aws, it lets internal teams bypass sysadmin departments.
- ghaff 1y ago
  Working around official IT was certainly a significant factor early on. I'm less convinced it is nearly as big a driver (or a downside depending on your perspective) today.
  - whstl 1y ago
    Especially considering that outside of startups (where approval would be fast with or without cloud), virtual infrastructure also got its own bureaucratic process.
    - ghaff 1y ago
      A lot of people forget that, when server virtualization was still gaining momentum in a lot of circles, it wasn't uncommon at less technically savvy customers--say a regional bank at the time--to be told that it might take 2 months to provision a new server.

UltraSane 1y ago

I'm not going to argue that AWS can be expensive but in my experience its biggest advantage is SPEED. In every company I worked for that ran their own data centers ever damn thing took FOREVER. new servers took months to buy and rack. any network change like a new VLAN took days to weeks. It was so annoying. But in AWS almost anything is just an API call and a few minutes at most from being enabled. It is so much more productive.

SJC_Hacker 1y ago

> I'm so surprised there is so much pushback against this.. AWS is extremely expensive. The use cases for setting up your system or service entirely in AWS are more rare than people seem to realise. Maybe I'm just the old man screaming at cloud (no pun intended) but when did people forget how to run a baremetal server ? Long term yes you can save money rolling your own. But with cloud you can get something up and running within maybe a few days, sometimes even faster. Often with built in scalability. This is a much easier sell to the non-tech (i.e., money) people. If the project continues, the path of least resistance is often to just continue with the cloud solution. At a certain point, there will be so much tech debt that any savings from long term costs from the traditional on-premises, co-location or managed hosting, are vastly by the cost of migration.

comprev 1y ago

I'm on a Platform team of <8 people and only 3 of us (most experienced too) come from sysadmin backgrounds. The rest have only ever known containers/cloud and never touched (both figuratively and literally :-) bare metal servers in their careers. They've never used tools like Ansible (or Anaconda) or been in situations where they couldn't destroy the container and start afresh instantly.

eek2121 1y ago

I once moved a small site from AWS to Digital Ocean + Cloudflare. $100-$300 on AWS -> $35/mo for DO + CF. Coincidentally, AWS had an outage soon after, which was avoided thanks to the move. I have used DO for both clients and myself, and have not had any huge problems with them.

rdtsc 1y ago

> I'm so surprised there is so much pushback against this.. AWS is extremely expensive. Basic rationalization. People will go to extraordinary lengths to justify and defend the choices they made. It's a defense mechanism: if they spent millions on AWS they are not going to sit idly while HN discusses saving hundreds of thousands with everyone nodding and agreeing. It's important for their own sanity to defend the choice they made.

TacticalCoder 1y ago
[dead]

j45 1y ago

The cloud is incredibly profitable for the efficiencies and improvements its introduced and held onto. Easy to push back against what is now the unknown (bare metal), when the layers extending bare metal to cloud service have become better and better, as well as more accessible.

dumbledoren 1y ago

> when did people forget how to run a baremetal server ? Bigger question: When did people forget that doing that is much easier than AWS...

citizenpaul 1y ago

The "value add" of AWS has never been what it can do or does. It has always appealed to weak/incompetent/sociopathic managers and execs desire to not have to deal with capable employees. As far as they are concerned AWS is taking care of computing AND hiring for them. I've never worked anywhere that at least some sort of power holder would instantly go to consultants or outsourcing rather than in house because they believe that if you work for the company you must be incompetent, dumb or below average. If you don't work for them you must be exceptional.

_ea1k 1y ago

> I'm so surprised there is so much pushback against this.. Same, this trend towards "AWS all the things" has really amazed me. We've all mocked small companies copying big companies by trying to make their app super-duper scalable from the very start. After all, everyone things they are the next google, despite their 5 total users right now. But this is really the opposite. AWS is phenomenal for the startup that would readily trade high opex for lower capex. Servers aren't the cheapest things in the world to buy and they depreciate. It makes total sense for startups to start this way. But why are big companies, with an actual budget for staff, copying the behavior of their favorite startups?

guax 1y ago
Opex looks nicer on the sheets than capex for large deployments. Incredible high investment from AWS on luring in C level with "white-papers" and promises of cost and governance magical revolutions. I've heard the promise of cheaper, faster where you can focus on "innovation". I am yet to see any of it become a reality.

mk89 1y ago

How would you do multi-region deployments with your own DC? This is an issue for several companies that start small and within 5 years they find the need to expand abroad. Be it for data sovereignty or so, which is becoming more important than ever in the last 10 years. Duplicating a region is "a few clicks away" on AWS. This is what the provider enables you to do. This and a lot of other things. And for such things, yes, you gotta pay.

Hikikomori 1y ago
I mean its not that complicated. Rent space in another location, get separate fibers/wavelength between them, redundant internet connection. But if you're in a growth/startup phase it doesn't make much sense to spend engineering time on this, not that multi region setups in Aws is one button either. Once you're past that and paying aws a million per week or so I think it can make sense to offload expensive services to your own hardware.

axegon_ 1y ago

I am not - I hate AWS(and cloud in general) with a passion - overpriced, you are getting locked in by a closed ecosystem the moment you say "hey this feature is neat it will save me so much work", only to realize that you are stuck paying for it for years if you decide to move away from it. But people are inclined to jump on a hype train and become evangelists for life. Truth is AWS(or GCP or Azure or anything else) is a viable option in two cases: 1. You are making a product with 3 friends on evenings and you want to ship asap without having the capacity to invest and setup infrastructure. 2. You are a huge corporation with tens of thousands of employees and hardware needs that you simply cannot source yourself easily or sort out the collocation of the hardware. Everyone else - get a dozen second-hand servers, shove them in a rack in a data center and you will own the hardware and everything associated with it at half the price of what you'd be paying AWS in a year.

TacticalCoder 1y ago

[dead]

kelnos 1y ago

> when did people forget how to run a baremetal server ? I don't think people have forgotten, but I think Amazon has done an amazingly excellent job of marketing and developer relations over the years, to the point that they've convinced most developers that doing your own thing is 1) a lot of expensive, specialized work, and 2) actively dangerous for your business, whether it's because of security, uptime, or some other ops bogeyman of the day. Note that I said "developers". Most developers are not sysadmins or IT operations people. Most of them have never set up Linux on a desktop or laptop, let alone on real server hardware (that they've also set up themselves). Most of them never had the chance to forget how to run a bare-metal server; they never knew how in the first place. (Hell, I've been running desktop Linux for 25+ years, and I don't think I've ever set up Linux on actual server hardware. Closest I've come is bare-metal Solaris, but that was like 25 years ago.) "DevOps" today usually means that you know how to run a CLI tool or drive a web interface to deploy your automatically-built container artifact to some cloud-based production system that someone else manages, hiding the details from you. (This bit also can be true for shops that run on bare metal, depending on how advanced their own sysadmin/ops team is.) While developers are often not decision-makers in a larger org, they can be at smaller orgs, and once those developers get on the cloud, you probably will stay on the cloud (companies like OneUptime are the exception, not the rule), even if you've gotten much larger and it's stupid expensive to continue running that way.

deleted 1y ago

[deleted]

hoppp 1y ago

AWS spends a lot in educating developers to use their service and it's a working strategy. People simply don't believe bare metal is better because of this conditioning towards the cloud. Every company should strive for self-sufficiency, but this idea is not that widespread in the software industry.

rasjani 1y ago

> when did people forget how to run a baremetal server ? My opinion on this: docker sort of changed the game here. It sort of enabled a lot of people to get a "new and fresh" level of abstraction to not bother about bare metal. As an example, I work in company where most consultants are doing DevOps and k8 is big part of that. What made me consider that? I've been told multiple times that "you know your stuff" when I mention some kernel or userland feature that container approach provides.