16 ms·

Moving from AWS to Bare-Metal saved us $230k per year

• teenXTY 3y ago
  [flagged]
• not_your_vase 3y ago
  How are such savings not obvious after putting the amounts in an Excel sheet, and spending an hour over it (and most importantly doing this before spending half a million/year on AWS)?
  • Spivak 3y ago
    I would be surprised if people didn't know that coloing was cheaper. I certainly evangelize it for workloads that are particularly expensive on AWS. It's not entirely without downsides though and I think many shops are willing to pay more for a different set of them. It is incredibly rewarding work though. You get to do magic. * You do need more experienced people, there's no way around it and the skills are hard to come by sometimes. We spent probably 3 years looking to hire a senior dba before we found one. Networking people are also unicorns. * Having to deal with the full full stack is a lot more work and needing manage IRL hardware is a PITA. I hated driving 50 miles to swap some hard drives. Rather than using those nice cloud APIs you are also on the other side implementing them. And all the VM management software sucks in their own unique ways. * Storage will make you lose sleep. Ceph is a wonder of the technological world but it will also follow you in a dark alleyway and ruin your sleep. * Building true redundancy is harder than you think it should be. "What if your ceph cluster dies?" "What if your ESXi shits the bed?" "What if Consul?" Setting things up so that you don't accidentally have single points of failure is tedious work. * You have to constantly be looking at your horizons. We made a stupid little doomsday clock web app that we put all the "in the next x days/weeks/months we have to do x or we'll have an outage." Because it will take more time than you think it should to buy equipment.
    • theLiminator 3y ago
      It's great when you don't need instant elasticity and traffic is very predictable. I think it's very useful for batch processing, especially owning a GPU cluster could be great for ML startups. Hybrid cloud + bare metal is probably the way to go (though that does incur the complexity of dealing with both, which is also hard).
    • deleted 3y ago
      [deleted]
  • mschuster91 3y ago
    > and most importantly doing this before spending half a million/year on AWS AWS is... incentivizing scope creep, to put it mildly. In ye olde days, you had your ESXi blades, and if you were lucky some decent storage attached to it, and you gotta made do with what you had - if you needed more resources, you'd have to go through the entire usual corporate bullshit. Get quotes from at least three comparable vendors, line up contract details, POs, get approval from multiple levels... Now? Who cares if you spin up entire servers worth of instances for feature branch environments, and look, isn't that new AI chatbot something we could use... you get the idea. The reason why cloud (not just AWS) is so popular in corporate hellscapes is because it eliminates a lot of the busybody impeders. Shadow IT as a Service.
    • tqi 3y ago
      Those busybodies are also there to keep rogue engineers from burning money on useless features (like AI chat bots) that only serve to bolster their promo packet...
      • mschuster91 3y ago
        That depends on how the incentive structures for your corporate purchase department are set up - and there's really a ton of variance there, with results ranging from everyone being happy in the best case to frustrated employees quitting in droves or the company getting burned at employer rating portals.
        • tqi 3y ago
          > That depends on how the incentive structures for your corporate purchase department are set up Sure, but that seems orthogonal to the pros and cons of having more layers of oversight (busybodies, to use your term) on infra spend. Badly run companies are badly run, and I don't think having the increased flexibility that comes from cloud providers changes that.
      • withinboredom 3y ago
        I literally started laughing at this. I worked at a bare-metal shop fairly recently and a guy on my team used a corporate credit card to set up an AWS account and create an AI chatbot. The dude nearly got fired, but your comment hit the spot. You made my night, thank you.
  • ldargin 3y ago
    Bare-metal solutions save money, but are costly in terms of development time and lost agility. Basically, they have much more friction.
    • withinboredom 3y ago
      huh, say wut? I guess before Amazon invented "the cloud" there wasn't any software companies...
      • threeseed 3y ago
        AWS isn't just IaaS they are PaaS. So it's a fact that for most use cases it will be significantly easier to manage than bare metal. Because much of it is being managed for you e.g. object store, databases etc.
        • withinboredom 3y ago
          Setting up k3s: 2 hours Setting up Garage for obj store: 1 hour. Setting up Longhorn for storage: .25 hour. Setting up db: 30 minutes. Setting up Cilium with a pool of ips to use as a lb: 45 mins. All in: ~5 hours and I'm ready to deploy and spending 300 bucks a month, just renting bare metal servers. AWS, for far less compute and same capabilities: approximately 800-1000 bucks a month, and takes about 3 hours -- we aren't even counting egress costs yet. So, for two extra hours on your initial setup, you can save a ridiculous amount of money. Maintenance is actually less work than AWS too. (source: I'm working on a youtube video)
          • threeseed 3y ago
            You should stick to making Youtube videos then. Because there is a world of difference between installing some software and making it robust enough to support a multi-million dollar business. I would be surprised if you can setup and test a proper highly-available database with automated backup in < 30 mins.
            • 3y ago
  • RhodesianHunter 3y ago
    Because the amount of time your engineers will spend maintaining your little herd of pet servers, and the opportunity cost of not being able to spin up manged service X to try an experiment, are not measurable.
    • yjftsjthsd-h 3y ago
      > maintaining your little herd of pet servers You know bare metal can be an automated fleet of cattle too, right?
    • withinboredom 3y ago
      Have you ever heard of pxe boot? You should check out Harvester made by Rancher (IIRC). Basically, manage bare metal machines using standard k8s tooling.
    • deleted 3y ago
      [deleted]
  • dilyevsky 3y ago
    Cloud is putting spend decisions into individual EMs’ or even devs’ hands. With baremetal one team (“infra” or whatever) will own all compute and thus spend decisions need to be justified by EMs which they usually dont like ;)
• hipadev23 3y ago
  They were paying on-demand ec2 prices and reserved instances alone would save them ~35%, savings plan even more which would apply to egress and storage costs too. Anyway, they're still saving a lot more (~55%), but it's not nearly as egregious of a difference.
  • RhodesianHunter 3y ago
    Right. Now what's the developer man-hour cost of the move? Unless their product is pretty static and not seeing much development, they're probably in the negative.
    • manvillej 3y ago
      when we don't optimize for cloud and look at it from this angle and squint, it looks like we're saving money!
      • deleted 3y ago
        [deleted]
    • zer00eyz 3y ago
      It's not a very good question: they still have aws compatibility as their fail over/backup (should be live but that's another matter...) What's capex vs opex now? Thats 150k of depreciable assets, probably ones that will be available for use long after all the current staff depart. Everyone forgets what WhatsApp did with few engineers and less hardware, there's probably more than enough room for them to grow, and they have space to increase capacity. The cloud has a place, but candidly so does a Datacenter and ownership.
      • RhodesianHunter 3y ago
        > they still have aws compatibility as their fail over/backup Now you're maintaining two tiers. That's more work, not less.
    • potatopatch 3y ago
• 4death4 3y ago
  Cool, so you can hire one additional engineer. Are you sure your bare metal setup will occupy less than a single engineer’s time?
  • VBprogrammer 3y ago
    I'm not so sure it's a zero sum kind of thing. Yes, it seems likely that they are paying at least one full time employee to maintain their production environment. At the same time, AWS isn't without it's complications, there are people who are employed specifically to babysit it.
    • 4death4 3y ago
      Sure people need to babysit AWS resources, but those resources still exist and need babysitting. The bare metal approach is purely additive. E.g. if you need someone to babysit EKS, then you also need someone to babysit your bare metal k8s setup, but likely the bare metal k8s setup is even more work.
  • r2_pilot 3y ago
    Ha, while I'd work for that salary, even half of it would almost double what I make as a sysadmin. I guess I work here for the mission. Plus I don't have to deal with cloud services except for when the external services go down. Our stuff keeps running though.
    • 4death4 3y ago
      I’m including overhead in that number. FWIW I know many ICs earning over double that number, not even including overhead.
      • r2_pilot 3y ago
        >FWIW I know many ICs earning over double that number, not even including overhead. Keep rubbing salt lol I live in a low cost area though. It's even pleasant some times of the year.
• jscheel 3y ago
  So, they essentially saved the cost of one good engineer. Question is, are they spending 1 man-year of effort to maintain this setup themselves? If not, they made the right choice. Otherwise, it’s not as clear cut.
  • jacquesm 3y ago
    That depends on where they are located. Good engineers aren't 230K$ / year everywhere.
    • corobo 3y ago
      Americans get paid so much, got dayum. Half that and half it again and I'd still be looking at a decent raise lmao
      • m00x 3y ago
        1-2 weeks vacation if lucky, no overtime pay, 7/24 on-call (spread through team), many startups do 10-12h/day (and amazon), minus healthcare co-pay, etc.
        • corobo 3y ago
          That (and the other replies, I don't want to spam up the joint replying to all) is a fair point Just to mention, I hope I didn't come across as saying paid too much. I'm always astounded as to how much cash is thrown about over the pond is all. Go get that bag, hell yeah! :)
      • uoaei 3y ago
        That's about senior-level compensation even among most companies in the Bay Area. Only the extreme outliers with good performance on the stock market can be said to be significantly higher in TC. Edit: even then, TC is tied to how the stock market is doing, and not paid out by the company directly, so it only makes sense to compare with base wage plus benefits.
• keremkacel 3y ago
  And now their blog won’t load
• mike_d 3y ago
  A lot of comments here seem to be along the lines of "you can hire one more engineer," but given the current economic situation remember that might be "keep one more engineer." Would you lay off someone on your team to keep AWS? Keeping a few racks of servers happily humming along isn't the massive undertaking that most people here seem to think it is. I think lots of "cloud native" engineers are just intimidated by having to learn lower levels to keep things running.
  • bee_rider 3y ago
    Especially given the low unemployment rate, laying somebody off seems quite risky, if it doesn’t work out you’ll have trouble hiring some replacement I guess.
    • cj 3y ago
      The current hiring market in tech is the easiest (for employers) than it has been in a really long time. It used to take 3-4 months to fill a role. In the current market it's more like 2-4 weeks.
      • sgarland 3y ago
        Not for SRE and DBRE lol. 4-6 months easy. I made the correct career choice. Downturn? What downturn?
  • unglaublich 3y ago
    > I think lots of "cloud native" engineers are just intimidated by having to learn lower levels to keep things running. Rightly so, because they're cloud native engineers, not system administrators. They're intimidated by the things they don't know. It'll be a very individual calculation whether it's worth it for your enterprise to organize and maintain hardware yourself, or isn't.
    • moduspol 3y ago
      There's certainly no shortage of sysadmins comfortable with their on-prem skillsets that have their heads in the sand about the cloud. And there are plenty of us who've spent time managing hardware and physical networks, transitioned to cloud, and are very happy to not be looking back.
• candiddevmike 3y ago
  Would anyone be interested in an immutable OS with built in configuration management that works the same (as in the same image) in the cloud and on premise (bare metal, PXE, or virtual)? Basically using this image you could almost guarantee everything runs the same way.
  • quillo_ 3y ago
    Yes - I would be interested :) my issue is that there is a mixed workload of centralised cloud compute and physical hardware in strange locations. I want something like Headscale as a global mesh control plane and some mechanism for deploying immutable flatcar images that hooks into IAM (for cloud) and TPM (for BM) as a system auth mechanism.
    • candiddevmike 3y ago
      My email is in my profile if you want to discuss this more...!
  • dilyevsky 3y ago
    This already exists - https://docs.fedoraproject.org/en-US/fedora-coreos/bare-metal/ https://docs.fedoraproject.org/en-US/fedora-coreos/bare-meta...
    • lijok 3y ago
      I thought coreos was dead
      • dilyevsky 3y ago
        Fedora CoreOS and Flatcar are fedora and gentoo based descendants and are actively maintained Last fcos release was a week ago - https://fedoraproject.org/coreos/release-notes?arch=x86_64&stream=stable https://fedoraproject.org/coreos/release-notes?arch=x86_64&s...
• andrewstuart 3y ago
  The key point is per year - ongoing saving every year.
• dankwizard 3y ago
  And these savings will be passed down to your customers too? Or....?
  • willsmith72 3y ago
    why should they be?
    • timbit42 3y ago
      Only if they have a competitor with a better deal.
• alex_lav 3y ago
  Curious how much was spent on the migration? I skimmed but didn't see that number. > Server Admins: When planning a transition to bare metal, many believe that hiring server administrators is a necessity. While their role is undeniably important, it’s worth noting that a substantial part of hardware maintenance is actually managed by the colocation facility. In the context of AWS, the expenses associated with employing AWS administrators often exceed those of Linux on-premises server administrators. This represents an additional cost-saving benefit when shifting to bare metal. With today’s servers being both efficient and reliable, the need for “management” has significantly decreased. This feels like a "famous last words" moment. Next year there'll be 400k in "emergency Server Admin hire" budget allocated.
• nwmcsween 3y ago
  I would go with managed bare-metal, it's a step up from unmanaged bare metal cost wise but saves you on headaches from memory, storage, network, etc issues.
• lgkk 3y ago
  I’m sure if the stack is simple enough it’s non trivial for most senior plus engineers to figure out the infrastructure. I’ve definitely seen a lot of over engineered solutions in the chase of some ideals or promotions.
• yieldcrv 3y ago
  I had an out of touch cofounder a few years back, he had asked me why the coworking space’s hours were the way they were, before interjecting that companies were probably managing their servers up there at the those later hours like, talk about decades removed! no, nobody has their servers in the coworking space anymore sir. nice to see people attempting a holistic solution to hosting though. with containerization redeploying anywhere on anything shouldn't be hard.
• maximusdrex 3y ago
  It feels like every comment on this article didn’t read past the first paragraph. Every comment I see is talking about how they likely barely made any money on the transition once all costs are factored in, but they explicitly stated a critical business rationale behind the move that remains true regardless of how much money it cost them to transition. Since they needed to function even when AWS is down, it made sense for them to transition even if it cost them more. This may increase the cost of running their service (though probably not) but it could made it more reliable, and therefore a better solution, making them more down the line.
  • threeseed 3y ago
    > Since they needed to function even when AWS is down AWS as a whole has never been down. It's Cloud 101 to architect your platform to operate across multiple availability zones (data centres). Not only to insulate against data centre specific issues e.g. fire, power. But also AWS backplane software update issues or cascading faults. If you read what they did it's actually worse than AWS because their Kubernetes control plane isn't highly-available.
    • wbsun 3y ago
      People often learn the lessons in a hard way: they will keep saving 230k/yr until one day their non-HA bare-metal is down and major customers retreat.
      • christophilus 3y ago
        > We have a ready to go backup cluster on AWS that can spin up in under 10 minutes if something were to happen to our co-location facility. Sounds like they already have their bases covered.
        • threeseed 3y ago
          Still need to synchronise data, update DNS records, wait for TTLs to expire. HA architectures exist for a reason because that last step is a massive headache.
• didip 3y ago
  The post is super light on details, it's hard to visualize if it's worth it or not. For examples: - How much data are they working with? What's the traffic shape? Using NFS makes me think that they don't have a lot of data. - What happened when their customers accidentally sent too much events? Will they simply drop the payload? In bare-metal they lose the ability to auto-scale quickly. - Are they using S3 or not, if they are, did they move that as well to their own Ceph cluster? - What's the RDBMS setup? Are they running their own DB proxy that can handle live switch-over and seamless upgrade? - What's the details on the bare metal setup? Is everything redundant? How quickly can they add several racks in one go? What's included as a service from their co-lo provider?
  • oxfordmale 3y ago
    It is not unlikely an AWS to GCP migration would have saved them significant money too, in the sense that they likely reviewed and right sizes different systems. I also would love to see a comparison done by a financial planning analyst to ensure no cost centres are missed. On prem is cheaper but only by 30 to 50%. That is the premium you pay for flexibility, which you can partly mitigate by purchasing reserved instance for multiple years.
    • threeseed 3y ago
      > On prem is cheaper but only by 30 to 50% Depending on use case. If you have traffic which isn't consistent 24/7 then AWS Spot instances with Gravitron CPUs will be cheaper than on-premise. Because you have the ability to in real-time scale your infrastructure up/down.
      • Thristle 3y ago
        Its 2 seperate issues: fluctuation in traffic is handled by auto scaling Saving money on stateless (or short start times) services is done with spots
• mensetmanusman 3y ago
  nines of up time?
  • avereveard 3y ago
    I don't think they care since they moved to a single rack single colocation setup, no idea who saw that deployment plan and said "yeah this is sensible I'll approve it." They're selling an observability solution...
• renecito 3y ago
  now a set of linux machines is considered bare-metal? I was under the impression that bare-metal means "no OS".
  • jdoss 3y ago
    The term bare-metal in this context means installing and managing Linux (or your OS of choice) on hardware directly without a Hypervisor is considered a bare-metal deployment. It's never meant no operating system.
  • _joel 3y ago
    In this context it's running without the use of virtualisation
• w4f7z 3y ago
  ServeTheHome has also written[0] a bit about the relative costs of AWS vs. colocation. They compare various reserved instance scenarios and include the labor of colocation. TL;DR: it's still far cheaper to colo. [0] https://www.servethehome.com/falling-from-the-sky-2020-self-hosting-still-pays/ https://www.servethehome.com/falling-from-the-sky-2020-self-...
• 0xbadcafebee 3y ago
  Moving from buying Ferraris to Toyota Camrys would save a lot of money too. These stories are always bs blog spam by companies trying to pretend they pulled off some amazing new hack. In reality they were burning cash because they hadn't the faintest idea how to control their spend. When we were utilizing AWS, our setup consisted of a 28-node managed Kubernetes cluster. Each of these nodes was an m7a EC2 instance. With block storage and network fees included, our monthly bills amounted to $38,000+ The hell were you doing with 28 nodes to run an uptime tracking app? Did you try just running it on like, 3 nodes, without K8s? When compared to our previous AWS costs, we’re saving over $230,000 roughly per year if you amortize the cap-ex costs of the server over 5 years. Compared to a 5-year AWS savings plan? Probably not. On top of this, they somehow advertise using K8s as a simplification? Let's reign in our spend, not only by abandoning the convenience of VMs and having to do more maintenance, but let's require customers use a minimum of 3 nodes and a dozen services to run a dinky uptime tracking app. This meme must be repeating itself due to ignorance. The CIOs/CTOs have no clue how to control spend in the cloud, so they rake up huge bills and ignore it "because we're trying to grow quickly!" Then maybe they hire someone who knows the Cloud, but they tell them to ignore the cost too. Finally they run out of cash because they weren't watching the billing, so they do the only thing they are technically competent enough to do: set up some computers and install Linux, and write off the cost as cap-ex. Finally they write a blog post in order to try to gain political cover for why they burned through several headcount worth of funding on nothing.
  • Nextgrid 3y ago
    > The hell were you doing with 28 nodes to run an uptime tracking app? To be fair, considering the pocket-calculator-grade performance you get from AWS (along with terrible IO performance compared to direct-attach NVME) I can totally understand they’d need 28 nodes to run something that would run on a handful of real, uncontended bare-metal hosts.
    • m00x 3y ago
      What are you talking about. That's crazy spend for that app. AWS isn't that bad.
• thenayr 3y ago
  [dead]
• boiler_up800 3y ago
  I’d say $500k per year on AWS is kind of within a dead man’s zone where if you’re not expecting that spend to grow significantly and your infra is relatively simple, migrating off may actually make sense. On the other hand maintaining $100K a year of spend on AWS is unlikely to be worth the effort of optimizing and maintaining $1M+ on AWS probably means the usage patterns are such that the cloud is cheaper and easier to maintain.
  • dvfjsdhgfv 3y ago
    In my experience amounts are meaningless, what counts is what kinds of services you need most. In my current org we use all 3 major public clouds + on-on prem services, carefully planning what should go where and why.
• m3kw9 3y ago
  Actually saves less if you spread the development of transitions and op costs over 5 years. Hidden costs
• monlockandkey 3y ago
  I've said this before, unless you are using specific AWS services, I think it is a fools errand to use it. Compute, storage, database, networking. You would be better off using Digital Ocean, Linode Vultr etc. so much cheaper than AWS, lots of bandwidth included rather than the extortionate $0.08 GB egress. Compute is the same story. 2 VCPU, 4GB VPS is ~$24 using a VPS. The equivalent instances (after navigating the obscured pricing and naming scheme), is the c6g.large is double the price at $50. This is the happy middle ground between bare metal and AWS.
• greyface- 3y ago
  Weren't you searching for a colo provider just yesterday? That was a quick $230k! https://news.ycombinator.com/item?id=38275614 https://news.ycombinator.com/item?id=38275614
  • avereveard 3y ago
    Oof this should be higher up things didn't quite add up before and now it's even more fishy
    • tinix 3y ago
      that's asking for the US, specifically. looks to me like they did this in Europe previously, and they are looking to do the same in the US now.
  • muttantt 3y ago
    They probably are looking for one legitimately. It appears they now use Scaleway dedicated bare metal servers
  • lkt 3y ago
    Apparently the move in this article is for their EU operations, now they're looking for a US colo provider to do the same there. https://news.ycombinator.com/item?id=38280506 https://news.ycombinator.com/item?id=38280506
• dilyevsky 3y ago
  Anyone can comment on server lifetime of 5 years? I would think it’s on the order of 8-10 years these days? Cores don't get that much faster you just get more of them, etc
  • m00x 3y ago
    Server hardware is pricey, and if it's not pricey it fails often. 5 years would be a very good lifetime for a heavy use server, especially HDs.
    • dilyevsky 3y ago
      Ok that is just fud and i was asking about servers themselves - disks can be swapped. > 5 years would be a very good lifetime for a heavy use server, especially HDs. Read Backblaze report - a lot of their HDs are over 8yo and the afr is less than 2%. SSDs will actually fail faster under heavy write load around 5 years yes
      • m00x 3y ago
        Yeah, it's easy to swap out. I'm just pointing out that it's not the same as having a gaming PC at home.
• Havoc 3y ago
  Cheaper price, lower redundancy: >single rack configuration at our co-location partner I've got symetrical gigabit static ipv4 at home...so can murder commercial offerings out there on bang/buck for many things. Right up until you factor in reliability and redundancy.
  • icedchai 3y ago
    Me too. I have an entire /24 routed to my home data center!
• FuriouslyAdrift 3y ago
  The math I have always seen is cloud is around 2.5x more expensive than on-prem UNLESS you can completely re-architect your infra to be cloud native. Lift and shift is brutal and doesn't make a lot of sense.
  • dvfjsdhgfv 3y ago
    > The math I have always seen is cloud is around 2.5x more expensive than on-prem UNLESS you can completely re-architect your infra to be cloud native. And at this point you are completely locked in.
• allenrb 3y ago
  Cue the inevitable litany of reasons why it is wrong to move out of “the cloud” in 3… 2… 1…
  • deleted 3y ago
    [deleted]
• dstainer 3y ago
  In my opinion the story here is that AWS allowed them to quickly build and prove out a business idea. This in turn afforded them the luxury to make this kind of switch. Cloud did it's job.
  • notnmeyer 3y ago
    this is a very good take on the situation.
  • dilyevsky 3y ago
    It's good take and one that's important to keep in mind when you're engineering your stack to be fully dependent on your chosen cloud vendor
• tonymet 3y ago
  what is the market distortion that allows AWS margins to remain so high? there are two major competitors (Azure, GCP) and dozens of minor ones. It seems crazy to me that the two options are AWS vs bare metal to save that much money. Why not a moderate solution?
  • 999900000999 3y ago
    Then you need to train people on Azure, GCP , etc. It's probably easier to optimize your stack TBH. I can't wait until I get a chance to use ARM on AWS.
  • tutfbhuf 3y ago
    AWS, Azure, and GCP are already competing in terms of pricing and customer acquisition. If you mention to Google that you are considering moving your multi-million dollar project to AWS due to a better price point, you will almost certainly receive a counteroffer. However, it's not just the big players in the game; there are also many smaller cloud providers like DigitalOcean, which are indeed a bit cheaper, but not as affordable as bare-metal solutions (e.g., Hetzner). I came to the conclusion that when you factor everything in—the time it takes to maintain such a massive infrastructure operating smoothly across the globe, investment in the further development of cloud services, employing security experts, paying developers competitive salaries, and of course aiming for a profit margin for the company—you end up with the prices that are evident among the major cloud providers. It simply isn't feasible to offer these services for much less (there are a few exceptions people rightly complain about, such as stupidly high egress costs). It has reached a point where Google, for example, has attempted to undercut prices to such a degree that their cloud operations have been running at a loss in the past[^1]. [^1]: https://www.ciodive.com/news/google-cloud-revenue-Q2-2022/628283/ https://www.ciodive.com/news/google-cloud-revenue-Q2-2022/62... (read last paragraph)
• notnmeyer 3y ago
  bare metal is cheaper than aws?! wild! edit: when you selectively choose your data points, and ignore human and migration costs.
• bananapub 3y ago
  yes you should do your own cost optimisation, which also has it's own cost.
• chx 3y ago
  If you are not using cloud services just virtual machines that's rarely worth it. And for a lot of people renting dedicated servers is the best business decision. It can be so much cheaper than EC2 that you solve your normal scalability problem by simply having a large amount of excess capacity. As you grow, some dedicated providers will give you an API to spin up a server in <120 seconds so even adding capacity quickly becomes possible. If your load is extremely spikey then this is not going to work but most people doesn't deal with loads like that -- or have other mitigation strategies. Always keep an eye on your business goals and not on the hype. For oneuptime obviously downtime is a huge problem but you'd be surprised for how many businesses it's much cheaper to be down for a few minutes here and there than engineering a complex HA mechanism. The aforementioned spiking problem often can be solved cheaply by degrading the hot pages to static and serving them from CDN (if you have a mechanism for doing this of course). And so forth. Remember KISS.
• mercwear 3y ago
  Who would have thought that vendor lock in results in absorbent spend!
  • moduspol 3y ago
    What vendor lock-in? They built their stack from the beginning to allow for hosting on bare metal, and then they made that transition.
• muttantt 3y ago
  Looks like they moved to Scaleway bare metal
• 1letterunixname 3y ago
  Disclaimer: Former AWS consultant from banking/Wall St. It's been a big "duh" for 20+ years. Large-scale, consistent loads aren't suited to cloud infrastructure. Mostly its shops that don't care about costs, don't know any better, or lack technical capabilities outsource most of their infrastructure. The use-cases for *aaS are: - Early startups - Beginning projects - Prototyping - Peaky loads on-demand or one-of - Evade corporate IT department AWS and VPSes can also be ill-suited for personal use if you live in a major city with bottom tier, cheap datacenters that can rent a 1 GbE uplink, a PDU plug, and 4U. For anything substantial, it's not hard to lease some dark fiber and run (E)BGP. https://www.ripe.net/manage-ips-and-asns/as-numbers/request-an-as-number https://www.ripe.net/manage-ips-and-asns/as-numbers/request-...
• bassiek 3y ago
  Very nice ! (Hard time finding the author/contact info though...) At "Storage and LoadBalancers" the NFS link point to https://microk8s.io/docs/nfs https://microk8s.io/docs/nfs Should be https://microk8s.io/docs/addon-nfs https://microk8s.io/docs/addon-nfs
• throwaway2990 3y ago
  Oh look at that. Another company who doesn’t know how to use cloud complaining about paying too much.
• turtlebits 3y ago
  Sorry, but if you're mainly using AWS for compute (ie EC2) and scale minimally you're doing it wrong and probably burning a whole lot of money.
• accountinhn 3y ago
  Can anyone explain or shed more light on how the "remote hands" part the support would work in bare metal server scenario? Would they run commands you give (or) will they follow your runbook (or) something else (or) all of the above?
  • mike_d 3y ago
    There are two types of remote hands: dumb and smart (under various names) Most colocation facilities include dumb hands for free. Push a button, tell me what lights are on, plug in a monitor and tell me what it says, move the network cable from port 14 to 15, replace the drive with the spare sitting in the rack, etc. Smart hands are billed around $100-$300/hr or a flat rate per task from a menu. Write an image to a USB stick and reinstall the OS on a server. Unrack and replace a switch. Figure out which drive has failed in the server and replace it. etc. I've ran computers in datacenters for 20+ years and maybe used smart hands 1 or 2 times.
• 1vuio0pswjnm7 3y ago
  These real world stories are always fun to hear because they trigger such defensiveness from for those who are invested in "cloud computing". This so-called cloud computing seems to have a strong need for a cheering section (hype) and a jeering section (sniping anyone who questions its value). Bring on the jeers.
  • kikimora 3y ago
    I look at this differently. On-prem proponents always fail to account for hard problems they absolutely have to solve, like tamper protected access logs, centralized server access controls, replacing failed hardware, backups, secure networking between servers in a collocation (are you sure a neighbor droplet cannot do an arp poisoning and hijack the traffic?). All this will bite them sooner or later, especially if they operate $250k worth of hardware. But because it does not bother them at the moment they can claim they don’t need this or it is not a problem thus rendering cloud as expensive.
• kaydub 3y ago
  We're in the business of making money. Our product makes the money. Not our skills and abilities to manage hardware and IT infrastructure. They have a decent business case, but I don't feel like they executed well to meet the real objective. They don't want to be in AWS since they're an uptime monitor and they want to alert on downtime on AWS. But they have a single rack of servers in a single location. A cold standby in AWS doesn't mean a ton unless they're testing their failovers there... which comes at quite the cost. I've worked on-prem before. Now I work somewhere that's 100% AWS and cloud native. You'd have to pay me quite a bit more to go back to on-prem. You'd have to pay me quite a bit more to go somewhere not using one of the 3 major clouds with all their vendor specific technologies. The speed is invaluable to a business. It's better to have elevated spend while trying to find good product market fit. I didn't understand this until I worked somewhere with a product the market wanted. > 50% growth for half a decade wouldn't have been possible on-prem. > 25% growth for a full decade wouldn't have been possible on-prem. I've been with my current company from $80MM ARR to $550MM ARR. We've never breached more than 1.5% of income on total cloud spend. We've been told that's the lowest they've seen by everyone from AWS TAMs to VC/PE people. It's because we're cloud native, we've always been cloud native, and we're always going to be cloud native. You've gotta get over "vendor lock-in". With our agility it's not really a thing. We could move to another cloud or on-prem if we really wanted. Wouldn't be a huge problem moving things service by service over... though we've had a few more recent changes that would be troublesome, we'd be able to work around them because we're architected well.
• nunez 3y ago
  Are they accounting for all of the labor costs of managing stuff in their stack that they previously deferred to AWS, like OS imaging/patching, backups, and building your own external services instead of whatever AWS had available?
• lijok 3y ago
  Now get rid of k8s, put the system back together into a monolith, optimize it, get 3 mac minis with gigabit ethernet and save yourself another 65k. It's an uptime monitoring tool...
• darthrupert 3y ago
  If you're using AWS primarily as a place to run virtual machines, you probably should be somewhere else. The largest benefits appear when you primarily use cloud-native things: lambdas, sqs, dynamodb, etc. Of course these will constitute a vendor lock-in, but that's usually an acceptable compromise.
• m00x 3y ago
  So the salary of 1 dev?
• toasted-subs 3y ago
  I'm in disbelief at how much people pay for aws. At a startup the costs were close to $500 per dev account. Just to keeping a few lambda functions and dynamo tables. Rolling a droplet, load balancer and database costs like $30.