25 ms·

Self-Hosting Still Pays

382 points by emperor_ 6y ago

deleted 6y ago
[deleted]
munk-a 6y ago
Wait - this article appears to compare hardware costs of self-hosting against the total cost of using AWS. It's missing the most expensive component (labour) which just so happens to be extremely expensive to scale.
- shiftpgdn 6y ago
  AWS doesn't manage itself. You still need someone to control your AWS instances.
  - yowlingcat 6y ago
    The whole point of using the AWS ecosystem is generally to use as many of the managed services as possible so as to minimize the latter. If you're using it to be multi-cloud portable, then yes, your point stands -- but then it brings into question whether you're exploiting its capabilities to their full potential.
  - threatofrain 6y ago
    AWS just needs to shave off a few engineers, and at least it has shaved off the need for hardware competency.
  - TomBombadildoze 6y ago
    Of course. However, you don't need someone to provision and monitor power, connectivity, datacenter security, and all the responsibilities AWS absolutely prices in for you. You also substantially reduce your effort and cost toward capacity planning and managing depreciation. You pay for consumption of _all the things_ required to operate compute infrastructure, plus their margin. There are certainly companies and problems for which colocating still makes more sense than the cloud but 9 out of 10 of these articles completely gloss over most of the costs. This article is one of them.
    - riku_iki 6y ago
      > you don't need someone to provision and monitor power, connectivity, datacenter security this is all included into colocation service?
  - Rantenki 6y ago
    That's false equivalence. If your instance dies, you don't need to drive over to the colo, plug into a KVM, diagnose, and then pull and re-rack a new server. They do all that stuff, and it's worth the $$$. Also, you'll never get into a situation on AWS where you have a 3 day lead-time for a replacement GBIC, or NVMe drive, or seomthing, while your customers scream and bail out for a competitor that isn't down. Yes, you need to manage the infrastructure, but now it's a software configuration task instead of a physical maintenance one.
    - shiftpgdn 6y ago
      99.9% of colo facilities offer 24/7 remote hands. With the extreme cost savings of colo you could build in very significant redundancy, which is to say nothing of how infrequently modern server hardware fails.
    - anonfornoreason 6y ago
      I manage a rack of 12 servers, I have 110 hard drives in that rack. I've been managing this rack for a decade. I occasionally go to the datacenter to swap out a machine that is end of life, and I go every 3-6 months to swap out a failing drive or two. I'd estimate I spend < 20 hours a year at the datacenter. Everything else is almost 1:1 with what it would take to manage that using AWS. Obvious anecdata here, but it's not hard to run a redundant physical infrastructure that is low maintenance. Obviously bad luck is a thing, but with the correct planning and setup, physical infrastructure isn't that big of a deal for most web businesses.
      - munk-a 6y ago
        That's true - I was comparing managing a server rack in the office building against AWS - not comparing the relatively light hardware interaction needs when it comes to co-location.
        addicted 6y ago
        I'm pretty sure the original article is talking about Colo.
- 02020202 6y ago
  that it the most used argument these days...but it's bs. if i use bare metal, the purchase is equivalent of 2-3 months of cloud pricing and the rest 9-10 months can be spent on admin/tech...the catch is that if you start scaling this up(ie. you get 10 powerful servers), bare metal becomes cheaper by thousands of percent yoy. the compounded savings are through the roof. cloud just has better PR, that's all. cloud is great for development, tinkering and figuring out the scale. after that, bare metal is king. and with docker, ansible/salt, nomad... there is zero need for over-engineered and over-complicated and over-paid cloud crap that sales people force down your throat day after day.
- Patrick-STH 6y ago
  Labor is included and discussed both for my time as well as remote hands we pay for. It is very low at this point.
  - ericpauley 6y ago
    Are you including having someone on call who can drive to the colo on a moment's notice? Unless your hardware is redundant and remotely swappable then you need to be ready for physical disaster response. Cloud providers abstract this away for a price.
    - kayson 6y ago
      That's what he means by "remote hands"
      - ericpauley 6y ago
        Not true. In the article remote hands refers to actual hours of maintenance performed ($640 for 4 hours). The cost of having a human on call to solve hardware problems is far higher than that.
        welterde 6y ago
        There is no need to have anyone on call yourself to drive to the colo, since any decent colocation provider will have staff on-site 24/7 to perform any physical intervention required. Part of that you pay through your colocation bill and part you pay by the hour (depending on your contract/provider).
        ericpauley 6y ago
        They're doing their own routing. Will the colo tech fix their iptables if they mess them up and can't ssh?
        6y ago
shiftpgdn 6y ago
You don't even need to buy a new EPYC or Xeon Scalable system. A 20 core Xeon v3/v4 system with 128gb of ram can be had off lease for $1000. Psychz.net provides 1u colocation with 1/Gbs link (10/Gbs burst) for $80/mo. Equivalent EC2 is $300-600/month with no better reliability. Plus you have to deal with noisy neighbors and bandwidth overages. Build your baseline usage in colo and burst into AWS unless you like buying Bezos more houses.
- KingMachiavelli 6y ago
  > No better reliability? Pretty sure EC2 instances and EBS volumes have a lot more redundency than a single server. You really need two colocated servers to replace a single EC2 instance. Still probably cheaper but also a larger time investment. If the difference between AWS vs colocation is an additional FTE then AWS is cheaper.
  - WhatIsDukkha 6y ago
    I'm always nonplussed by the "additional FTE" argument, its obscenely overestimated - """We are going to say we used four hours of labor. This includes drive time to the primary data center. Since it is far away (18-20 minute drive), we actually did not go there for several quarters. So over 32 months, we had budgeted $640 for remote hands. We effectively either paid $160/ hr or paid less than that even rounding up to four hours. """ Software and systems configuration wise you aren't really going to be doing much different timewise then you would by doing loop de loops with AWS configuration stuff anyway. Tftp etc is just not that tough.
    - unethical_ban 6y ago
      About 7 years ago we brought in the ELK stack for security log ingestion and basic analytics. It took at least three FTEs to maintain the cluster, along with constant issues of queries crashing clusters, or provisioning/building/repairing the dozens of servers and storage units to handle the several terabytes per day of ingest. To be able to throw that behind an infinite, horizontally scalable mechanism would have saved us a lot of pain and troubleshooting.
yowlingcat 6y ago
I wondered whether this comparison (like many others like it) would /finally/ bake cluster operational costs into the calculus. Not this time, it looks like. To raise a more constructive point. Looking at [1] (an earlier post in the series), it looks like their workload is WordPress + VBulletin forums, and "pets" not "cattle" -- I wonder how much more they'd be able to get from separating their stateful and stateless layers more cleanly, and using some of the more powerful (and cheaper) AWS primitives for serving traffic that involves the latter. Why do they need such beefy instances for essentially serving content? The truth is, it's because the platforms they're using (vBulletin + WP) exist, they're very powerful and they get the job done. This leads me to believe that the operational UX for running these kinds of common applications (which compose most of the internet) on AWS without highly technical supervision is not great. [1] https://www.servethehome.com/falling-sky-part-3-evaluating-amazon-ec2-vps-dedicated-colocation-options/ https://www.servethehome.com/falling-sky-part-3-evaluating-a...
Zelphyr 6y ago
I see so often developers state something along the lines of, "We need to go with AWS!" (sometimes substituting AWS with GCP or Azure) and the reason they always give is "scalability". 99% of the time, what they're building not only doesn't need to scale to that degree but they somehow seem to think that it's always inherently cheaper. Meanwhile, when I build something for a client my first go-to is something like DigitalOcean or Amazon LightSail. I know, LightSail is still AWS. But it's not going to automatically increase the bill by 300% because someone accidentally toggled something in AWS's byzantine admin console. When ("if" is more often the case) that client starts to see the kinds of traffic that necessitates a full-scale AWS build out then I'll advise them of their options. By that point, they're actually generating real revenue and, as importantly, they didn't pay me to prematurely optimize a product that wasn't yet generating any revenue, which helped them succeed.
- hangonhn 6y ago
  I think your sort of thinking requires a certain level of maturity and experience. I see the opposite with with some engineers who always have this illusion that the first or even second version of their project will be the final one for all times so they build it to scale (which actually doesn't even really scale because they don't know where the real bottlenecks are) and highly configurable. I've seen way, way more projects bite the dust because of poor product design or product market fit than the inability to scale. I've definitely seen projects that can't scale but at least from an engineering point of view, those problems are way more tractable than people not wanting or needing what you've created. As I've gotten more experienced, I've realized one of the great skills a good engineer has is the ability to change/fix/scale a project once it is up and running and while people are using it.
  - bob1029 6y ago
    > I've realized one of the great skills a good engineer has is the ability to change/fix/scale a project once it is up and running and while people are using it. This is by far the most important lesson I have learned this decade. Anyone can pull the rug out on iteration 1 and start a completely new attempt in a 2nd iteration bucket. It takes determined engineering efforts and talent to iterate on top of the existing code base while not impacting the ability to deliver ongoing feature support to production. We have historically tried to do complete rewrites with bombastic claims like "We can do it right this time". Well, we did it "right" 3 times until we decided that restarting from zero every time is a bad way to make forward progress. I will advocate for a clean restart if you are just beginning or it is clear you have totally missed the domain model target. If you have been in production for 3 years, maybe consider a more conservative iteration approach. The best analogy I can come up with is the Ship of Theseus thought experiment. Rebuilding the ship from the inside while you are sailing it to the new world will get you there much faster than if you have to restart in Spain every time someone is in a bad mood or doesn't like how the sails on the boat are arranged.
Rantenki 6y ago
TIL: AWS is only expensive if your time has no value. Article neglected to add the cost of labour for setup (edit: aside of racking), maintenance (security updates on infrastructure, etc), and disaster response. It also missed the cost of downtime based on projected MTTR for a failure. The word "fail" only appears once in the page, and not in regards to recovery, only preventative replacement. It's absolutely true that colo is cheaper than AWS for projects that can absorb the labour and failure related costs, but that usually is only true for hobby or very small business services.
- riku_iki 6y ago
  It also depends on how computationally intensive your application is, AWS premium can be higher than labor cost if you need large scale data crunching.
  - Rantenki 6y ago
    That's definitely true. It still makes sense to run your own AI/ML servers on-prem if you're saturating some $20k/each GPU boxes, for example. But ... most web applications are still IO bound, and labour is surprisingly expensive.
  - heipei 6y ago
    Or if you need to egress a lot of data to the Internet. Or send a lot of data between different AZs. Or between Regions. Or anywhere really. Plus now you have to get a PhD in AWS Billing first: https://twitter.com/QuinnyPig/status/1155838435065655296/photo/1 https://twitter.com/QuinnyPig/status/1155838435065655296/pho... and https://twitter.com/QuinnyPig/status/1172239124251709449/photo/1 https://twitter.com/QuinnyPig/status/1172239124251709449/pho...
- shiftpgdn 6y ago
  Do you actually use AWS or are you just parroting something you've read? Do you understand that with AWS you still have to setup instances, manage security updates, permission and deal with disaster response (US-East-1 outage?) If you've run anything at any scale for any length of time you'll know that EC2 hosts frequently get rebooted with short notice.
honkycat 6y ago
Good luck hiring the professionals needed to run a bunch of co-located servers in 2020.
- nikisweeting 6y ago
  Maybe this is my old-school sysadmin showing, but how is managing a rack server more difficult than managing an EC2 instance? The hardware part is not difficult, if you've built a PC you can set up a rack server (it's arguably easier because they're built to assemble without needing tools), and the software is all Linux on both, no? Or do you mean hiring on-site engineers within driving distance of your rack servers is difficult?
  - yowlingcat 6y ago
    The pieces aren't difficult in isolation, but the combined logistical total cost of operational ownership is high and includes huge tail risks. When your metal fails, it stays down until physically fixed. Depending on your business, that's either acceptable or not.
    - nikisweeting 6y ago
      We mitigate that risk by having our backup servers and database replicas in the cloud, but the big beefy primary machines are bare metal run on cheap Quebec power w/ unlimited 1Gbps residential bandwidth.
  - mekster 6y ago
    For those generations of people who grew up by being able to spin up servers with a few mouse clicks, doing that manually seems like a whole level of professional skills.
    - nikisweeting 6y ago
      I feel almost the opposite. The AWS console seems so chaotic and needlessly complex to me compared to the simple familiarity of plugging in a hardware server. Maybe I'm an old geezer already but I'm not even that old!
bpodgursky 6y ago
Even ignoring labor, none of these analyses ever consider BC/DR or backup costs. S3 for backups is the single biggest ease-of-mind feature AWS provides, IMO. I do not want all my data sitting in a single colocation facility without backups. If I run a database on RDS, maybe it costs more than a bare-metal server, but I can replicate the automatic hourly backups to as many regions as I want. Incredible peace of mind over a self-managed facility.
- Patrick-STH 6y ago
  Those are included in our colo costs with a second data center where we have a smaller DR cluster.
  - teh_klev 6y ago
    It never fails to amaze me how many HN contributors don't seem to read the linked articles yet seem compelled to tell you off for things you've already accounted for, explained and justified.
    - bpodgursky 6y ago
      There's absolutely nothing in the article (which I did read) about it. If you yourself had read the article, you would know that it was not explained, accounted for, or justified. There is a single vague reference to a "primary DC". There is no discussion at all of backups or disaster recovery.
    - renewiltord 6y ago
      Mind quoting the section in the linked article that accounts for this, explains it, and justifies it?
    - mekster 6y ago
      It's because comments are usually far more intelligent and interesting to read and I set my app to open up comments than the story by default. I don't blame the others doing the same.
jedberg 6y ago
These calculations never account for capital overhead of having spares on hand. That cost is part of what you pay for on Amazon. If the hardware breaks on Amazon, you just switch hardware. If hardware breaks in your datacenter, it's an emergency and you better have spare capacity or extra hardware laying around.
- Patrick-STH 6y ago
  We include in our "Hardware Costs" having extra node and spares. For example, we keep a full spare node in the DC as well as spares.
sabalaba 6y ago
Always love Patrick's posts. Lambda has built a business on moving people from the cloud to on-prem and reducing the cost of cloud AI infrastructure. As some of the other comments have pointed out, for machine learning applications with GPUs, it almost always makes sense to run your workloads on prem, STH is showing that it can also true for traditional CPU workloads. For example, we're able to provide GPU instances (https://lambdalabs.com/service/gpu-cloud https://lambdalabs.com/service/gpu-cloud) that are half the hourly cost of AWS hourly on-demand pricing. How? Because there are huge markups on clouds services. We've done such extensive benchmarking and TCO analysis and the jury is out: it's simply less expensive to run on-prem. You're just paying for convenience when using a cloud. GPU or otherwise. Sources: - https://lambdalabs.com/gpu-benchmarks https://lambdalabs.com/gpu-benchmarks - https://lambdalabs.com/blog/hyperplane-16-infiniband-cluster-total-cost-of-ownership/ https://lambdalabs.com/blog/hyperplane-16-infiniband-cluster...
jeffbee 6y ago
I have a feeling you could host that site on a PC at home for even less. It looks like their forum gets < 100 posts per day. I can't imagine they're hitting much more than 1 QPS.
- anaganisk 6y ago
  Its not always an option, because of Dynamic IPs, power outages etc. Solving them reliably comes at a cosr and also not so easy.
GordonS 6y ago
Some of the big benefits of major cloud providers (AWS, Azure, GCP): - the sheer number of services available - managed services - high availability, within and/or across regions - APIs/SDKs for many of their services, for many languages - monitoring and alerting built-in - RBAC - means of grouping and organising resources and subscriptions I work as an architect in the enterprise space, where those last 2 are pretty important because a lot of different teams are building/using a lot of different systems, and often multiple organisations are involved. High availability of some services is vital, because downtime can affect the bottom line, or mean an entire workforce has to down their tools. Considering only the first 3 items, most of the non-huge systems I design - and, I would imagine, most non-enterprise/FAANG-scale systems - have only modest availability requirements, very rarely needing cross-region availability. And most of those systems only really need web apps and APIs, proxies/gateways, a database, a message queue, blob storage, and often some form of background processing (VMs, containers, serverless). OK, that's still a fairly long list, but it's a tiny fraction of what the big cloud providers have on offer, and the database is probably the only tricky thing on the list.
cft 6y ago
I host a web app that has 200KB pages and reaches 3.5Gbps peak bandwidth in two racks in an expensive datacenter. The lease plus bandwidth is about 1/5 to 1/7 the cost of AWS.
CSDude 6y ago
I'm probably sure this is response to the AWS Reinvent announcments on the front page. Not that I work on AWS, but we heavily use and rely on it. The ones comparing AWS to VPSs even colocation think AWS is just short sighted. If VMs and bandwidth is your only case, yes it is expensive. Skipping the marketing talk, AWS has great services like SQS, S3, Dynamo, SNS, Kinesis, Firehose and hosting open-soure or paid alternatives of them requires a lot of engineering power, effort and extra monitoring, restless nights. I prefer our engineers to work on features, not infrastructure and only deal with it when it is required financially. Most organizations are fine with vendor lock-in when vendor is stable (not deprecating, not increasing costs). Money is always an issue but its relative, there are many factors in real business. Some can afford cloud, some cannot. I also agree that you would be fine if you did not use brand new X service that does AutoML, any shiny new over marketed features for you etc. but comparing AWS to Colocation is just one dimensional thought.
saberience 6y ago
So, as someone that works for an enterprise that uses AWS... if we went colocation, how would we replicate: SQS/SNS, DynamoDB, ECS (EC2 and Fargate), Lambda, Kinesis, ECR, ALB, Aurora, S3, Glacier, EMR, EKS, multiple AZs, multiple regions, etc? For us to get the same functionality as all of these services in colocated data centers would be an INSANE amount of work, we'd have to hire so many IT/hardware specialists for the networking, let alone hiring data storage tech specialists for stuff like Spark/Hadoop. Looking only at costs for instances is only 10% of the story here.
- nucleardog 6y ago
  Yeah, at least for us the value of AWS is not in "is it cheaper to run a boring old server here". It's in "how easily can we script and template all of our services and associated resources". It's in "when we need a highly durable and available queue, how much work/infra do we need". When you factor all that stuff in, no way it comes out cheaper to self host here. Just the couple full time employees we need to setup and maintain all the ancillary stuff would cost us more than our AWS bill.
- Daishiman 6y ago
  For real. I think people don't understand that the software that cloud providers bring us just a _tremendous_ force multiplier. How many different vendors and licensing schemes would do you need to bring to your own data center to get an equivalent amount of functionality going? (I ask rhetorically; this is the equivalent of a few FTEs at the least).
- pwinnski 6y ago
  100% agreed. Every one of those AWS services can be reproduced in a colo, but it takes time and effort to set up a queueing solution which just exists already in AWS. It takes time and effort to set up notifications for storage and database events which are easy-as-anything in AWS. Even without getting into multiple availability zones, things like SQS/SNS, Dynamo, S3, and Lamda combine to make things very easy on day one with AWS. Could it be cheaper? Well, yeah, I'm sure. And if all you're doing is running EC2 instances, go somewhere else. But also, maybe look into some of the other stuff AWS provides!
code4tee 6y ago
The fact that static old school servers in the cloud are more expensive than hosting it yourself is news to precisely nobody. That’s not the business case for the cloud. In 2020 if your tech stack is still a monolith of giant servers on a rack somewhere you’ve got far bigger problems in your future.
jasode 6y ago
If I'm reading the source html pages of "servethehome.com" correctly, it's a Wordpress site and that's what Patrick is basing hosting cost comparisons on. I've mentioned before[0] that AWS too expensive and overkill for Wordpress sites (especially simpler non-ecommerce ones). I don't think this conclusion is controversial. Simpler hosting requirements is why Patrick only has to spend 4 hours of labor in 1 year to upgrade some hardware. It's when you need the higher-level value-added services of AWS services (Dynamo, Redshift, region failover, etc) that the comparison becomes more complicated. E.g. Companies with mission-critical transactional websites or mobile backends are more complex and they need agility to add/change the infrastructure landscape in response to unknown workloads. They don't have the money (or expertise) to code an in-house version of AWS services portfolio. E.g.[1] [0] https://news.ycombinator.com/item?id=10797166 https://news.ycombinator.com/item?id=10797166 [1] https://www.cbronline.com/news/guardian-aws-migration https://www.cbronline.com/news/guardian-aws-migration older archive: https://web.archive.org/web/20160319022029/https://www.computerworlduk.com/cloud-computing/guardian-goes-all-in-on-aws-public-cloud-after-openstack-disaster-3629790/ https://web.archive.org/web/20160319022029/https://www.compu...
- Patrick-STH 6y ago
  Totally correct. I mention that a bit in the video as well. Also - I do not view this as an AWS v. Colo. We use AWS for some services as well so it is a specific part of the workload we run (not just WP) that is in our hosting cluster. And again, this is a fraction of what we have in data cetners due to the labs and such.
- rsanheim 6y ago
  Exactly this. A lot of the conversation here is missing a key point of these hosting debates: context is _everything_.
- sofixa 6y ago
  Wordpress hosting is expensive and overkill to do on AWS if you do it traditionalally. Add simply static or a similar plugin, whip up a script to upload the static HTML to S3, set up Cloudfront with good cache, and you're done. Your site is faster, more secure, cheaper and can easily run on a t3.micro for peanuts. Only gotcha is comments, but it's a solved problem ( disqus or any of the alternatives) and similar.
mark242 6y ago
I think the moral of the story is: if you have a predictable workload that fits within a smallish, stable infrastructure and you can make on-prem work, that's great, more power to you. If you are treating cloud services as purely an apples-to-apples cost comparison then you've missed the point of deploying to (insert your favorite cloud provider here). What you get with cloud services is flexibility and speed. If the OP wants to undertake a new project that they have no idea of the resulting workload or popularity, they need to guess about the required underlying infrastructure. If you miss on your guess, you can either kill momentum for that new project, or you can wind up overprovisioning and paying way way too much on hardware. I'm not sure how quickly their colo provider could spin up new hardware for this hypothetical new project, but you can assume it's not the dynamism that you get from deploying to an elastic cloud environment. Again, you're paying a premium on day-to-day costs in order to have this freedom to create and deploy. That's what cloud infrastructure is about. To give an example -- I have a very-CPU-heavy workload that started its life out on prem. As more and more customers signed up, I would go through tiers of adding more hardware to a rack, where my overall margins looked like a sawtooth waveform when I would provision more hardware. I started out moving to EC2 and then finally to ECS/Fargate. If I want to spin up a new piece of functionality for my users, I don't need to provision any new hardware, set out any real new infrastructure, or do anything that you would consider prework in order to get that new functionality deployed. My margins are much much more predictable and I get faster time-to-market on feature development. That's what you're paying the premium for, that ability to just move faster.
iptrans 6y ago
I find it interesting that these comparisons always pit AWS againsta colocation. It would be much more interesting if the comparison was between AWS and dedicated servers. Dedicated servers have all the benefits of colocation, only you don't need to deal with the hardware. The DC takes care of the hardware, all you have to do it administer the server. I find dedicated servers to be very cost effective. You can get off the shelf servers in minutes and if you need custom hardware that's only a quote away. Most providers will offer month-to-month billing, so there is no lock-in. Often dedicated servers are even cheaper than colocation when you add up all the costs of hardware, racking, sparing, financing, etc.
- deleted 6y ago
  [deleted]
- therealx 6y ago
  Yesh, I'm so tired of the idea that AWS doesn't involve the same management overhead as any other server. It's so far removed from the truth that I wonder if they are shills.
  - apple4ever 6y ago
    It's just a different set of problems to deal with not less problems. Not sure why they feel the need to hide it.
renewiltord 6y ago
Makes sense. Good reasoning for the use case you have in mind. Cloud stuff makes sense when you have some combo of the following imho: * Need for scalability - at one place I worked high-load was 3x low-load so going up and down saved a lot of time * Rare need for powerful resources (sort of the same as above) - I provisioned a real fatboi on AWS the other day to do some file processing. Worked like a charm and gone in the hour * Ability to use higher-level primitives - lambda vs. EC2 instances, databricks vs. EMR, IAM/Kinesis all that * Need for exploration - I could not afford the peak ElasticSearch + Redis + Postgresql that I had but I can trial it * Need for resilience - If you need to preserve the data/compute for whatever reason, AWS is way easier to get to a good place * Lack of knowledge on top-tier ops - I've run long-term dedicated servers for almost two decades now, with top uptime in a decade. I have fuck-all knowledge of doing smart colo ops. If you want to keep your team lean and you don't have this to lean on, it's going to sink you. If you're using hardware as hardware and it isn't changing that much and you have the skillset, you can get a long way with a colo even today.
marianov 6y ago
Do people deploy using kubernetes or has it passed as a fad? Self-hosting k8s looks like a nightmare
holoduke 6y ago
My advise as a CTO is to avoid AWS or GCS in all stages of a healthy company. The danger of getting locked-in. The aggressive sales team bugging you every week. The impossible to comprehend pricing model. Hire yourself a decent set of says engineers and go for VMs in an early stage of the company. A switch to bare metal could be taken in a later stage. No always needed.
yotamoron 6y ago
The cloud enables much faster iterations and experimentstions, I rarely see this factored into the cost of colo/on-prem.
NovemberWhiskey 6y ago
I tried to back into the estimates here, and I can't quite get my head around them. $737.19 of transfer out is 8TB per month at standard EC2->internet prices, which is fine; something like 25 Mb/s on average. $3558.56 for EC2 is something like 20 x m5a.2xlarge ... and that makes no sense to me. That's 160 vCPUs and 640GB of RAM. How does a WP site like that possibly need that much compute? The whole thing feels like it ought to be a pair of instances plus $20 a month for Cloudflare.
- trynumber9 6y ago
  Yeah, I'm rather confused too. I run a site with over 10TB transfer per month on a $15 / month Digital Ocean virtual machine. It's fast enough and Cloudflare caches over 80% so I seldom go over the included 2TB of monthly bandwidth. I'm sure STH has more visitors though.
segmondy 6y ago
AWS gains is like a bell curve, the value is highest in the middle. If you're very small, you probably don't need it, and DigitalOcean without surprise costs should do. When you start getting large enough, AWS starts paying off, as you get really huge, then it might be cheaper to colocate and hire your own admins. However, I think Netflix shows you can't be too big tho for AWS. Their hybrid approach is probably best. Run on AWS/GCP for things that you just can't afford to fail, then move your non critical workload to colocation.
- mekster 6y ago
  Places like netflix wants to put liability off its shoulders and wants someone to blame at. Very different demand than most.
  - segmondy 6y ago
    I don't buy that, liability to who? If Netflix goes down, I'm gonna cry and tweet about it, I'm not going to cancel my account. Their catalog is unmatched. colocating across the world is not easy, they also have scale that goes up and down based on time of day, and say holidays where they really need to scale up. Cloud is great for that, the elastic part is great, place the core on there and scale up and down as needed. What they did with active-active resiliency and being able to shift traffic across regions is the entire point of cloud, as we some a bunch of services where down last week with AWS having issue but netflix never skips a beat on such issues.
    - mekster 6y ago
      I don't mean they'll make excuses toward customers but say they can simply blame AWS toward content providers when something happens to the infrastructure instead of begging them to wait while they prepare for a damage report on their custom infrastructure.
KaiserPro 6y ago
I'm a bit conflicted about this. Yes, AWS is expensive. And depending on your workload you can certainly save a fair wedge of cash by colo-ing things. For wordpress, (the site that is running serverthehome) its perfectly possible to run it in a lambda, with aggressive caching and save a whole bunch of cash. depending on plugins, load and a number of other things, it could be a potential saving of 90%. (plus a massive reduction in attack surface) Before you ask, yes I do know from bitter experience, parts of the financial times had WP wedged into them. making them fast and secure was an interesting experience. AWS is only cost effective when you are using ec2 with a duty cycle of less than 50%. This means that you are not using AWS for hosting 24/7 compute. In terms of storage, There are two compelling offerings, S3 and EFS/lustre. however if you're a large scale EFS user, its better to run your own GPFS system. The hardest part of "scaling" is orchestration. If you're using K8s, there really isn't much difference between running it on virtual vs real steel, barring bringup scripts(don't get me started on networking in K8s, its totally warped.) TLDR: IF you need 100+ machines on 24/7, AWS is going to be more expensive. If you have transient loads, and the average time on for a machine is 4 hours or less, then AWS is for you. However, you'd better use all the other bits that come with AWS to make it cheaper, like fronting it with fastly/otherCDN
- cinquemb 6y ago
  Yeah, last place I worked had transient loads we pretty much did everything with just ec2 t3a.medium spot instances (minus s3 and rds [which kind sucks with using postgresql if you want to resize down or load from a pg_dump since you cant copy the sql to the rds instance and forced to backup over the network], I wanted to move to ec2 instances and from s3 to b2 since its cheaper for bandwidth with [we had cdn77 in front of cloudfront to lower costs outside of US/EU]). Using docker or k8s just doubled the costs over just using disk images and custom load-balancing daemon with nginx (with 3rd party modules for changing upstreams without restarts/reloads)/python, on top of adding more networking complexity (like to caching/search/rabbitmq instances). Probably around 20+ machines at peak when i left, but kept them all below 20% cpu. Sad to see so many places go in the docker/k8s direction.
kev009 6y ago
I have my own 1/2 cab, AS number, ARIN allocated IP4 and IP6 space. It costs about $500/mo. It took maybe 100 hours to set up the basics and $6000 for the way I built the hw (which is similar to servethehome's rack). I run everything on FreeBSD. I get to directly talk BGP and get bandwidth at .12c/gbit. Nothing else can come close to this level of low cost, control, learning, and autonomy.
- CKN23-ARIN 6y ago
  .12c/gbit?? I need to renegotiate my transit bill...
  - kev009 6y ago
    he.net is .09!
  - kev009 6y ago
    d'oh, I meant mbps commit not gbit :S
partiallypro 6y ago
We used to use an independent host (essentially self hosting) but they went under due to COVID. We moved nearly 1000 customers over to Azure and in the end we are saving money due to simple things like reservations and being cost conscious. We also took over a company that used AWS and they were over spending on their hosting by almost 10-20x what they needed. Not because AWS is inherently expensive but because it was so mismanaged. I don't think self hosting is worth it, when I can freely scale up or down machines in seconds on Azure and my price goes down, we have no fixed costs or maintenance worries. I think it's a fair trade off.
chmod775 6y ago
I mean yeah, AWS is a 10-100x markup over running your own stuff. I run a file hosting site that serves >500TB of data to about 2 million monthly users on a bunch of dedicated servers that cost me about $500 a month. If I ran this on AWS it would cost me more than $20,000 every month. I could hire four system administrators to do nothing but look after each of my four dedicated servers for that money. It's completely ridiculous.
- leesalminen 6y ago
  Outbound data transfer pricing is definitely _the_ gotcha for the big clouds.
- trianglem 6y ago
  Same. I’ve been in places that self host and it’s not nearly enough to overcome what AWS (in my case Azure) charge. 99.5% uptime in contracts lets you realistically have a lot of downtime to fix your problems. That’s more than 400 hours of acceptable downtime. We’re in the process of making the switch and we project to pay 30% of what we’re currently paying.
  - rubatuga 6y ago
    There's not even 44 hours of downtime in a year for 99.5% SLA, so I have no idea what you are talking about.
- MattGaiser 6y ago
  Isn't transfer a huge cost for AWS? I know of some people who have gone with Oracle because it supposedly has much lower pricing on that.
xfitm3 6y ago
Having previously performed an extensive TCO calculation (2010) it was substantially cheaper for on-prem vs AWS. We had <10000 machines, mostly cheap/supermicro. We had our own parts inventory and staff dedicated to maintaining equipment. I also ran facility operations for things like the generator, fuel testing, UPS PMs, air filters, professional cleaning, etc. You have to know your business and you need people with facilities skills. Investing in cold aisle containment reduced our PUE quite a bit. You also have to become an expert in logistics to run your own facility, where just almost anyone can click away in AWS and spend money.
joana035 6y ago
That's is great, not to mention you get real CPUs, no throttling inside your LAN, real storage. The cloud makes you addicted to the "but you have to scale" by selling 2 to 4 vCPUs plus 8 to 16GB of memory for the same price you can get a 12 vCPU with 64 or 128GB somewhere else.
ericpauley 6y ago
I have a hard time believing the setup pictured offers equivalent reliability/disaster recovery to "cloud" hosting. As far as I can tell, all traffic is being routed through two servers. These servers are running Linux? What happens when a routine software update bricks your routing (it could happen to anyone). Without remote access you need someone to go in and fix this. Do you have after hours access? Is someone nearby on call to respond to complex issues in person? Not all physical problems can be solved by a random colo tech. Colocation (as pictured) lacks management of a lot of variables that could lead to big problems. When your downtime targets are in minutes per year any incident requiring physical response is unacceptable. A simple Google search reveals a service philosophy compatible with this hosting[1]: > As with all web systems, at some point, downtime is required. For the rest of us there's "cloud" hosting. [1] https://www.servethehome.com/pardon-dust-upgrades-progress/ https://www.servethehome.com/pardon-dust-upgrades-progress/
mattbeckman 6y ago
Our small team spends roughly $40-50k/mo at AWS. If you're just hosting a website with a SQL database (glorified LAMP stack), you're probably in it for the wrong reasons. Stick to a dedicated box or DigitalOcean etc. AWS is like co-locating your hardware, and then the data center having 1000s of employees offering highly reliable services you can access from your infrastructure that lets you move WAY faster for things that are hard to do. e.g. Discussion today was increasing log retention. 10 years ago I would have run the numbers, extended some SAN volumes, considered procuring more NetApp shelves, etc. Today it's simply a cost question: is it worth it to us to store those logs for 10x longer? Sure? Ok, done.
- busterarm 6y ago
  I think the point that you're making here is subtle and that people are missing it. Your small team can _comfortably manage_ 50k/mo worth of AWS resources. That's _INCREDIBLE_. I think people mistakenly think that cloud costs only go up exponentially and that costs are an unmanageable mess. I've worked with teams with hundreds of engineers serving a major enterprise and only an AWS bill 3-5x yours. And they only had a small team managing it all. Comfortably. To do similar with physical servers requires a massive stack of people and salaries. The ongoing recruiting costs to maintain staffing would dwarf what the AWS bill is.
  - apple4ever 6y ago
    > To do similar with physical servers requires a massive stack of people and salaries. The ongoing recruiting costs to maintain staffing would dwarf what the AWS bill is. No no they wouldn't. Number of people is equal for both, its just different tasks they have to do.
    - busterarm 6y ago
      I do both at roughly 10^5 scale and I'm telling you that operating in the cloud lets us operate with an order of magnitude less people.
ngcc_hk 6y ago
Only use cloud provider like digital ocean with a limit on financial outlay. You prepay. Not for large corporations. But for micro firm ops, the worry of a big bill is there.
dragonsh 6y ago
The biggest advantage of using self-hosted infrastructure is real freedom from the whims of large cloud providers and the governments who indirectly exercise control over which company, country and individual such cloud providers can do business with. On a large cloud provider a government notice is enough to remove access to compute resources and data hosted in it. This in essence means company or individual using this cloud services neither own the compute resources and data hosted in it. On the other hand in self-hosted infrastructure government notice will not block access to resources owned by you and data in those resources. You can respond to notice and continue using those resources until court rules. So it is more freedom than a cloud can offer. Obviously given the large marketing budget and convenience cloud provider flourish at the expense of freedom. Self-hosted infrastructure needs a new renaissance given whats happening around the world and how government and large corporation taking away freedom one bite at a time. So it is very essential for development and freedom for humanity to have self-hosted compute, networking and data infrastructure.
- lmz 6y ago
  Do you really think your colo provider is going to ignore government notices to disconnect you?
waheoo 6y ago
Yet another article comparing pricing of self hosting vs self hosting in the cloud. I'm so shocked. /s Let me know when you compare the cost of running all of AWS service offerings self hosted.
AnthOlei 6y ago
For those of you who have a modern serverless architecture (lambda, dynamo, firebase etc.), how do the costs compare there? I’ve always thought serverless is cheaper, but I’ve never ran anything at scale.
xupybd 6y ago
I just had a simple PHP application move from simple VPS hosting to full blown AWS with load balancers, RDS databases and EFS storage. It's still the same app the load balancer can only send the traffic to one machine. RDS now means DB connections are TCP not pipes. The EFS is slower than an SSD. So it's slower and costs more. Why did this happen? Because an external consultant said AWS is what the big players use. I argued that our app needs to be engineered to take advantage of AWS or this is an expensive waste. But what would I know, we went with the consultants and payed them tens of thousands to set up a bunch of AWS infrastructure I could have run on a $10 a month VPS...
twothamendment 6y ago
It all depends on the scale of not only the app, but the organization too. We have enough people to click buttons in RDS. We don't have enough to mess around with replication, backups, etc. The cost isn't just for the hardware, but the services that we'd otherwise have to hire someone to do. I used to work at a place that would borrow hardware from one client in order to handle the load on another. It was for seasonal stuff, a few days a year when we knew someone would melt down if they didn't have double or triple the capacity, but they didn't need it the rest of the year. Now I deal in auto scale groups and don't care how many servers we use or need as long as it isn't wasteful. "Hardware" that isn't ours is so easy to swap out, try a different size or scale that is hate to go back to metal.

382 points by emperor_ 6y ago

deleted 6y ago

[deleted]

Wait - this article appears to compare hardware costs of self-hosting against the total cost of using AWS. It's missing the most expensive component (labour) which just so happens to be extremely expensive to scale.

shiftpgdn 6y ago
AWS doesn't manage itself. You still need someone to control your AWS instances.
- yowlingcat 6y ago
  The whole point of using the AWS ecosystem is generally to use as many of the managed services as possible so as to minimize the latter. If you're using it to be multi-cloud portable, then yes, your point stands -- but then it brings into question whether you're exploiting its capabilities to their full potential.
- threatofrain 6y ago
  AWS just needs to shave off a few engineers, and at least it has shaved off the need for hardware competency.
- TomBombadildoze 6y ago
  Of course. However, you don't need someone to provision and monitor power, connectivity, datacenter security, and all the responsibilities AWS absolutely prices in for you. You also substantially reduce your effort and cost toward capacity planning and managing depreciation. You pay for consumption of _all the things_ required to operate compute infrastructure, plus their margin. There are certainly companies and problems for which colocating still makes more sense than the cloud but 9 out of 10 of these articles completely gloss over most of the costs. This article is one of them.
  - riku_iki 6y ago
    > you don't need someone to provision and monitor power, connectivity, datacenter security this is all included into colocation service?
- Rantenki 6y ago
  That's false equivalence. If your instance dies, you don't need to drive over to the colo, plug into a KVM, diagnose, and then pull and re-rack a new server. They do all that stuff, and it's worth the $$$. Also, you'll never get into a situation on AWS where you have a 3 day lead-time for a replacement GBIC, or NVMe drive, or seomthing, while your customers scream and bail out for a competitor that isn't down. Yes, you need to manage the infrastructure, but now it's a software configuration task instead of a physical maintenance one.
  - shiftpgdn 6y ago
    99.9% of colo facilities offer 24/7 remote hands. With the extreme cost savings of colo you could build in very significant redundancy, which is to say nothing of how infrequently modern server hardware fails.
  - anonfornoreason 6y ago
    I manage a rack of 12 servers, I have 110 hard drives in that rack. I've been managing this rack for a decade. I occasionally go to the datacenter to swap out a machine that is end of life, and I go every 3-6 months to swap out a failing drive or two. I'd estimate I spend < 20 hours a year at the datacenter. Everything else is almost 1:1 with what it would take to manage that using AWS. Obvious anecdata here, but it's not hard to run a redundant physical infrastructure that is low maintenance. Obviously bad luck is a thing, but with the correct planning and setup, physical infrastructure isn't that big of a deal for most web businesses.
    - munk-a 6y ago
      That's true - I was comparing managing a server rack in the office building against AWS - not comparing the relatively light hardware interaction needs when it comes to co-location.
      - addicted 6y ago
        I'm pretty sure the original article is talking about Colo.
02020202 6y ago
that it the most used argument these days...but it's bs. if i use bare metal, the purchase is equivalent of 2-3 months of cloud pricing and the rest 9-10 months can be spent on admin/tech...the catch is that if you start scaling this up(ie. you get 10 powerful servers), bare metal becomes cheaper by thousands of percent yoy. the compounded savings are through the roof. cloud just has better PR, that's all. cloud is great for development, tinkering and figuring out the scale. after that, bare metal is king. and with docker, ansible/salt, nomad... there is zero need for over-engineered and over-complicated and over-paid cloud crap that sales people force down your throat day after day.
Patrick-STH 6y ago
Labor is included and discussed both for my time as well as remote hands we pay for. It is very low at this point.
- ericpauley 6y ago
  Are you including having someone on call who can drive to the colo on a moment's notice? Unless your hardware is redundant and remotely swappable then you need to be ready for physical disaster response. Cloud providers abstract this away for a price.
  - kayson 6y ago
    That's what he means by "remote hands"
    - ericpauley 6y ago
      Not true. In the article remote hands refers to actual hours of maintenance performed ($640 for 4 hours). The cost of having a human on call to solve hardware problems is far higher than that.
      - welterde 6y ago
        There is no need to have anyone on call yourself to drive to the colo, since any decent colocation provider will have staff on-site 24/7 to perform any physical intervention required. Part of that you pay through your colocation bill and part you pay by the hour (depending on your contract/provider).
        ericpauley 6y ago
        They're doing their own routing. Will the colo tech fix their iptables if they mess them up and can't ssh?
        6y ago

shiftpgdn 6y ago

You don't even need to buy a new EPYC or Xeon Scalable system. A 20 core Xeon v3/v4 system with 128gb of ram can be had off lease for $1000. Psychz.net provides 1u colocation with 1/Gbs link (10/Gbs burst) for $80/mo. Equivalent EC2 is $300-600/month with no better reliability. Plus you have to deal with noisy neighbors and bandwidth overages. Build your baseline usage in colo and burst into AWS unless you like buying Bezos more houses.

KingMachiavelli 6y ago
> No better reliability? Pretty sure EC2 instances and EBS volumes have a lot more redundency than a single server. You really need two colocated servers to replace a single EC2 instance. Still probably cheaper but also a larger time investment. If the difference between AWS vs colocation is an additional FTE then AWS is cheaper.
- WhatIsDukkha 6y ago
  I'm always nonplussed by the "additional FTE" argument, its obscenely overestimated - """We are going to say we used four hours of labor. This includes drive time to the primary data center. Since it is far away (18-20 minute drive), we actually did not go there for several quarters. So over 32 months, we had budgeted $640 for remote hands. We effectively either paid $160/ hr or paid less than that even rounding up to four hours. """ Software and systems configuration wise you aren't really going to be doing much different timewise then you would by doing loop de loops with AWS configuration stuff anyway. Tftp etc is just not that tough.
  - unethical_ban 6y ago
    About 7 years ago we brought in the ELK stack for security log ingestion and basic analytics. It took at least three FTEs to maintain the cluster, along with constant issues of queries crashing clusters, or provisioning/building/repairing the dozens of servers and storage units to handle the several terabytes per day of ingest. To be able to throw that behind an infinite, horizontally scalable mechanism would have saved us a lot of pain and troubleshooting.

yowlingcat 6y ago

I wondered whether this comparison (like many others like it) would /finally/ bake cluster operational costs into the calculus. Not this time, it looks like. To raise a more constructive point. Looking at [1] (an earlier post in the series), it looks like their workload is WordPress + VBulletin forums, and "pets" not "cattle" -- I wonder how much more they'd be able to get from separating their stateful and stateless layers more cleanly, and using some of the more powerful (and cheaper) AWS primitives for serving traffic that involves the latter. Why do they need such beefy instances for essentially serving content? The truth is, it's because the platforms they're using (vBulletin + WP) exist, they're very powerful and they get the job done. This leads me to believe that the operational UX for running these kinds of common applications (which compose most of the internet) on AWS without highly technical supervision is not great. [1] https://www.servethehome.com/falling-sky-part-3-evaluating-amazon-ec2-vps-dedicated-colocation-options/ https://www.servethehome.com/falling-sky-part-3-evaluating-a...

Zelphyr 6y ago

I see so often developers state something along the lines of, "We need to go with AWS!" (sometimes substituting AWS with GCP or Azure) and the reason they always give is "scalability". 99% of the time, what they're building not only doesn't need to scale to that degree but they somehow seem to think that it's always inherently cheaper. Meanwhile, when I build something for a client my first go-to is something like DigitalOcean or Amazon LightSail. I know, LightSail is still AWS. But it's not going to automatically increase the bill by 300% because someone accidentally toggled something in AWS's byzantine admin console. When ("if" is more often the case) that client starts to see the kinds of traffic that necessitates a full-scale AWS build out then I'll advise them of their options. By that point, they're actually generating real revenue and, as importantly, they didn't pay me to prematurely optimize a product that wasn't yet generating any revenue, which helped them succeed.

hangonhn 6y ago
I think your sort of thinking requires a certain level of maturity and experience. I see the opposite with with some engineers who always have this illusion that the first or even second version of their project will be the final one for all times so they build it to scale (which actually doesn't even really scale because they don't know where the real bottlenecks are) and highly configurable. I've seen way, way more projects bite the dust because of poor product design or product market fit than the inability to scale. I've definitely seen projects that can't scale but at least from an engineering point of view, those problems are way more tractable than people not wanting or needing what you've created. As I've gotten more experienced, I've realized one of the great skills a good engineer has is the ability to change/fix/scale a project once it is up and running and while people are using it.
- bob1029 6y ago
  > I've realized one of the great skills a good engineer has is the ability to change/fix/scale a project once it is up and running and while people are using it. This is by far the most important lesson I have learned this decade. Anyone can pull the rug out on iteration 1 and start a completely new attempt in a 2nd iteration bucket. It takes determined engineering efforts and talent to iterate on top of the existing code base while not impacting the ability to deliver ongoing feature support to production. We have historically tried to do complete rewrites with bombastic claims like "We can do it right this time". Well, we did it "right" 3 times until we decided that restarting from zero every time is a bad way to make forward progress. I will advocate for a clean restart if you are just beginning or it is clear you have totally missed the domain model target. If you have been in production for 3 years, maybe consider a more conservative iteration approach. The best analogy I can come up with is the Ship of Theseus thought experiment. Rebuilding the ship from the inside while you are sailing it to the new world will get you there much faster than if you have to restart in Spain every time someone is in a bad mood or doesn't like how the sails on the boat are arranged.

Rantenki 6y ago

TIL: AWS is only expensive if your time has no value. Article neglected to add the cost of labour for setup (edit: aside of racking), maintenance (security updates on infrastructure, etc), and disaster response. It also missed the cost of downtime based on projected MTTR for a failure. The word "fail" only appears once in the page, and not in regards to recovery, only preventative replacement. It's absolutely true that colo is cheaper than AWS for projects that can absorb the labour and failure related costs, but that usually is only true for hobby or very small business services.

riku_iki 6y ago
It also depends on how computationally intensive your application is, AWS premium can be higher than labor cost if you need large scale data crunching.
- Rantenki 6y ago
  That's definitely true. It still makes sense to run your own AI/ML servers on-prem if you're saturating some $20k/each GPU boxes, for example. But ... most web applications are still IO bound, and labour is surprisingly expensive.
- heipei 6y ago
  Or if you need to egress a lot of data to the Internet. Or send a lot of data between different AZs. Or between Regions. Or anywhere really. Plus now you have to get a PhD in AWS Billing first: https://twitter.com/QuinnyPig/status/1155838435065655296/photo/1 https://twitter.com/QuinnyPig/status/1155838435065655296/pho... and https://twitter.com/QuinnyPig/status/1172239124251709449/photo/1 https://twitter.com/QuinnyPig/status/1172239124251709449/pho...
shiftpgdn 6y ago
Do you actually use AWS or are you just parroting something you've read? Do you understand that with AWS you still have to setup instances, manage security updates, permission and deal with disaster response (US-East-1 outage?) If you've run anything at any scale for any length of time you'll know that EC2 hosts frequently get rebooted with short notice.

honkycat 6y ago

Good luck hiring the professionals needed to run a bunch of co-located servers in 2020.

nikisweeting 6y ago
Maybe this is my old-school sysadmin showing, but how is managing a rack server more difficult than managing an EC2 instance? The hardware part is not difficult, if you've built a PC you can set up a rack server (it's arguably easier because they're built to assemble without needing tools), and the software is all Linux on both, no? Or do you mean hiring on-site engineers within driving distance of your rack servers is difficult?
- yowlingcat 6y ago
  The pieces aren't difficult in isolation, but the combined logistical total cost of operational ownership is high and includes huge tail risks. When your metal fails, it stays down until physically fixed. Depending on your business, that's either acceptable or not.
  - nikisweeting 6y ago
    We mitigate that risk by having our backup servers and database replicas in the cloud, but the big beefy primary machines are bare metal run on cheap Quebec power w/ unlimited 1Gbps residential bandwidth.
- mekster 6y ago
  For those generations of people who grew up by being able to spin up servers with a few mouse clicks, doing that manually seems like a whole level of professional skills.
  - nikisweeting 6y ago
    I feel almost the opposite. The AWS console seems so chaotic and needlessly complex to me compared to the simple familiarity of plugging in a hardware server. Maybe I'm an old geezer already but I'm not even that old!

bpodgursky 6y ago

Even ignoring labor, none of these analyses ever consider BC/DR or backup costs. S3 for backups is the single biggest ease-of-mind feature AWS provides, IMO. I do not want all my data sitting in a single colocation facility without backups. If I run a database on RDS, maybe it costs more than a bare-metal server, but I can replicate the automatic hourly backups to as many regions as I want. Incredible peace of mind over a self-managed facility.

Patrick-STH 6y ago
Those are included in our colo costs with a second data center where we have a smaller DR cluster.
- teh_klev 6y ago
  It never fails to amaze me how many HN contributors don't seem to read the linked articles yet seem compelled to tell you off for things you've already accounted for, explained and justified.
  - bpodgursky 6y ago
    There's absolutely nothing in the article (which I did read) about it. If you yourself had read the article, you would know that it was not explained, accounted for, or justified. There is a single vague reference to a "primary DC". There is no discussion at all of backups or disaster recovery.
  - renewiltord 6y ago
    Mind quoting the section in the linked article that accounts for this, explains it, and justifies it?
  - mekster 6y ago
    It's because comments are usually far more intelligent and interesting to read and I set my app to open up comments than the story by default. I don't blame the others doing the same.

jedberg 6y ago

These calculations never account for capital overhead of having spares on hand. That cost is part of what you pay for on Amazon. If the hardware breaks on Amazon, you just switch hardware. If hardware breaks in your datacenter, it's an emergency and you better have spare capacity or extra hardware laying around.

Patrick-STH 6y ago
We include in our "Hardware Costs" having extra node and spares. For example, we keep a full spare node in the DC as well as spares.

sabalaba 6y ago

Always love Patrick's posts. Lambda has built a business on moving people from the cloud to on-prem and reducing the cost of cloud AI infrastructure. As some of the other comments have pointed out, for machine learning applications with GPUs, it almost always makes sense to run your workloads on prem, STH is showing that it can also true for traditional CPU workloads. For example, we're able to provide GPU instances (https://lambdalabs.com/service/gpu-cloud https://lambdalabs.com/service/gpu-cloud) that are half the hourly cost of AWS hourly on-demand pricing. How? Because there are huge markups on clouds services. We've done such extensive benchmarking and TCO analysis and the jury is out: it's simply less expensive to run on-prem. You're just paying for convenience when using a cloud. GPU or otherwise. Sources: - https://lambdalabs.com/gpu-benchmarks https://lambdalabs.com/gpu-benchmarks - https://lambdalabs.com/blog/hyperplane-16-infiniband-cluster-total-cost-of-ownership/ https://lambdalabs.com/blog/hyperplane-16-infiniband-cluster...

jeffbee 6y ago

I have a feeling you could host that site on a PC at home for even less. It looks like their forum gets < 100 posts per day. I can't imagine they're hitting much more than 1 QPS.

anaganisk 6y ago
Its not always an option, because of Dynamic IPs, power outages etc. Solving them reliably comes at a cosr and also not so easy.

GordonS 6y ago

Some of the big benefits of major cloud providers (AWS, Azure, GCP): - the sheer number of services available - managed services - high availability, within and/or across regions - APIs/SDKs for many of their services, for many languages - monitoring and alerting built-in - RBAC - means of grouping and organising resources and subscriptions I work as an architect in the enterprise space, where those last 2 are pretty important because a lot of different teams are building/using a lot of different systems, and often multiple organisations are involved. High availability of some services is vital, because downtime can affect the bottom line, or mean an entire workforce has to down their tools. Considering only the first 3 items, most of the non-huge systems I design - and, I would imagine, most non-enterprise/FAANG-scale systems - have only modest availability requirements, very rarely needing cross-region availability. And most of those systems only really need web apps and APIs, proxies/gateways, a database, a message queue, blob storage, and often some form of background processing (VMs, containers, serverless). OK, that's still a fairly long list, but it's a tiny fraction of what the big cloud providers have on offer, and the database is probably the only tricky thing on the list.

cft 6y ago

I host a web app that has 200KB pages and reaches 3.5Gbps peak bandwidth in two racks in an expensive datacenter. The lease plus bandwidth is about 1/5 to 1/7 the cost of AWS.

CSDude 6y ago

I'm probably sure this is response to the AWS Reinvent announcments on the front page. Not that I work on AWS, but we heavily use and rely on it. The ones comparing AWS to VPSs even colocation think AWS is just short sighted. If VMs and bandwidth is your only case, yes it is expensive. Skipping the marketing talk, AWS has great services like SQS, S3, Dynamo, SNS, Kinesis, Firehose and hosting open-soure or paid alternatives of them requires a lot of engineering power, effort and extra monitoring, restless nights. I prefer our engineers to work on features, not infrastructure and only deal with it when it is required financially. Most organizations are fine with vendor lock-in when vendor is stable (not deprecating, not increasing costs). Money is always an issue but its relative, there are many factors in real business. Some can afford cloud, some cannot. I also agree that you would be fine if you did not use brand new X service that does AutoML, any shiny new over marketed features for you etc. but comparing AWS to Colocation is just one dimensional thought.

saberience 6y ago

So, as someone that works for an enterprise that uses AWS... if we went colocation, how would we replicate: SQS/SNS, DynamoDB, ECS (EC2 and Fargate), Lambda, Kinesis, ECR, ALB, Aurora, S3, Glacier, EMR, EKS, multiple AZs, multiple regions, etc? For us to get the same functionality as all of these services in colocated data centers would be an INSANE amount of work, we'd have to hire so many IT/hardware specialists for the networking, let alone hiring data storage tech specialists for stuff like Spark/Hadoop. Looking only at costs for instances is only 10% of the story here.

nucleardog 6y ago
Yeah, at least for us the value of AWS is not in "is it cheaper to run a boring old server here". It's in "how easily can we script and template all of our services and associated resources". It's in "when we need a highly durable and available queue, how much work/infra do we need". When you factor all that stuff in, no way it comes out cheaper to self host here. Just the couple full time employees we need to setup and maintain all the ancillary stuff would cost us more than our AWS bill.
Daishiman 6y ago
For real. I think people don't understand that the software that cloud providers bring us just a _tremendous_ force multiplier. How many different vendors and licensing schemes would do you need to bring to your own data center to get an equivalent amount of functionality going? (I ask rhetorically; this is the equivalent of a few FTEs at the least).
pwinnski 6y ago
100% agreed. Every one of those AWS services can be reproduced in a colo, but it takes time and effort to set up a queueing solution which just exists already in AWS. It takes time and effort to set up notifications for storage and database events which are easy-as-anything in AWS. Even without getting into multiple availability zones, things like SQS/SNS, Dynamo, S3, and Lamda combine to make things very easy on day one with AWS. Could it be cheaper? Well, yeah, I'm sure. And if all you're doing is running EC2 instances, go somewhere else. But also, maybe look into some of the other stuff AWS provides!

code4tee 6y ago

The fact that static old school servers in the cloud are more expensive than hosting it yourself is news to precisely nobody. That’s not the business case for the cloud. In 2020 if your tech stack is still a monolith of giant servers on a rack somewhere you’ve got far bigger problems in your future.

jasode 6y ago

If I'm reading the source html pages of "servethehome.com" correctly, it's a Wordpress site and that's what Patrick is basing hosting cost comparisons on. I've mentioned before[0] that AWS too expensive and overkill for Wordpress sites (especially simpler non-ecommerce ones). I don't think this conclusion is controversial. Simpler hosting requirements is why Patrick only has to spend 4 hours of labor in 1 year to upgrade some hardware. It's when you need the higher-level value-added services of AWS services (Dynamo, Redshift, region failover, etc) that the comparison becomes more complicated. E.g. Companies with mission-critical transactional websites or mobile backends are more complex and they need agility to add/change the infrastructure landscape in response to unknown workloads. They don't have the money (or expertise) to code an in-house version of AWS services portfolio. E.g.[1] [0] https://news.ycombinator.com/item?id=10797166 https://news.ycombinator.com/item?id=10797166 [1] https://www.cbronline.com/news/guardian-aws-migration https://www.cbronline.com/news/guardian-aws-migration older archive: https://web.archive.org/web/20160319022029/https://www.computerworlduk.com/cloud-computing/guardian-goes-all-in-on-aws-public-cloud-after-openstack-disaster-3629790/ https://web.archive.org/web/20160319022029/https://www.compu...

Patrick-STH 6y ago
Totally correct. I mention that a bit in the video as well. Also - I do not view this as an AWS v. Colo. We use AWS for some services as well so it is a specific part of the workload we run (not just WP) that is in our hosting cluster. And again, this is a fraction of what we have in data cetners due to the labs and such.
rsanheim 6y ago
Exactly this. A lot of the conversation here is missing a key point of these hosting debates: context is _everything_.
sofixa 6y ago
Wordpress hosting is expensive and overkill to do on AWS if you do it traditionalally. Add simply static or a similar plugin, whip up a script to upload the static HTML to S3, set up Cloudfront with good cache, and you're done. Your site is faster, more secure, cheaper and can easily run on a t3.micro for peanuts. Only gotcha is comments, but it's a solved problem ( disqus or any of the alternatives) and similar.

mark242 6y ago

I think the moral of the story is: if you have a predictable workload that fits within a smallish, stable infrastructure and you can make on-prem work, that's great, more power to you. If you are treating cloud services as purely an apples-to-apples cost comparison then you've missed the point of deploying to (insert your favorite cloud provider here). What you get with cloud services is flexibility and speed. If the OP wants to undertake a new project that they have no idea of the resulting workload or popularity, they need to guess about the required underlying infrastructure. If you miss on your guess, you can either kill momentum for that new project, or you can wind up overprovisioning and paying way way too much on hardware. I'm not sure how quickly their colo provider could spin up new hardware for this hypothetical new project, but you can assume it's not the dynamism that you get from deploying to an elastic cloud environment. Again, you're paying a premium on day-to-day costs in order to have this freedom to create and deploy. That's what cloud infrastructure is about. To give an example -- I have a very-CPU-heavy workload that started its life out on prem. As more and more customers signed up, I would go through tiers of adding more hardware to a rack, where my overall margins looked like a sawtooth waveform when I would provision more hardware. I started out moving to EC2 and then finally to ECS/Fargate. If I want to spin up a new piece of functionality for my users, I don't need to provision any new hardware, set out any real new infrastructure, or do anything that you would consider prework in order to get that new functionality deployed. My margins are much much more predictable and I get faster time-to-market on feature development. That's what you're paying the premium for, that ability to just move faster.

iptrans 6y ago

I find it interesting that these comparisons always pit AWS againsta colocation. It would be much more interesting if the comparison was between AWS and dedicated servers. Dedicated servers have all the benefits of colocation, only you don't need to deal with the hardware. The DC takes care of the hardware, all you have to do it administer the server. I find dedicated servers to be very cost effective. You can get off the shelf servers in minutes and if you need custom hardware that's only a quote away. Most providers will offer month-to-month billing, so there is no lock-in. Often dedicated servers are even cheaper than colocation when you add up all the costs of hardware, racking, sparing, financing, etc.

deleted 6y ago
[deleted]
therealx 6y ago
Yesh, I'm so tired of the idea that AWS doesn't involve the same management overhead as any other server. It's so far removed from the truth that I wonder if they are shills.
- apple4ever 6y ago
  It's just a different set of problems to deal with not less problems. Not sure why they feel the need to hide it.

renewiltord 6y ago

Makes sense. Good reasoning for the use case you have in mind. Cloud stuff makes sense when you have some combo of the following imho: * Need for scalability - at one place I worked high-load was 3x low-load so going up and down saved a lot of time * Rare need for powerful resources (sort of the same as above) - I provisioned a real fatboi on AWS the other day to do some file processing. Worked like a charm and gone in the hour * Ability to use higher-level primitives - lambda vs. EC2 instances, databricks vs. EMR, IAM/Kinesis all that * Need for exploration - I could not afford the peak ElasticSearch + Redis + Postgresql that I had but I can trial it * Need for resilience - If you need to preserve the data/compute for whatever reason, AWS is way easier to get to a good place * Lack of knowledge on top-tier ops - I've run long-term dedicated servers for almost two decades now, with top uptime in a decade. I have fuck-all knowledge of doing smart colo ops. If you want to keep your team lean and you don't have this to lean on, it's going to sink you. If you're using hardware as hardware and it isn't changing that much and you have the skillset, you can get a long way with a colo even today.

marianov 6y ago

Do people deploy using kubernetes or has it passed as a fad? Self-hosting k8s looks like a nightmare

holoduke 6y ago

My advise as a CTO is to avoid AWS or GCS in all stages of a healthy company. The danger of getting locked-in. The aggressive sales team bugging you every week. The impossible to comprehend pricing model. Hire yourself a decent set of says engineers and go for VMs in an early stage of the company. A switch to bare metal could be taken in a later stage. No always needed.

yotamoron 6y ago

The cloud enables much faster iterations and experimentstions, I rarely see this factored into the cost of colo/on-prem.

NovemberWhiskey 6y ago

I tried to back into the estimates here, and I can't quite get my head around them. $737.19 of transfer out is 8TB per month at standard EC2->internet prices, which is fine; something like 25 Mb/s on average. $3558.56 for EC2 is something like 20 x m5a.2xlarge ... and that makes no sense to me. That's 160 vCPUs and 640GB of RAM. How does a WP site like that possibly need that much compute? The whole thing feels like it ought to be a pair of instances plus $20 a month for Cloudflare.

trynumber9 6y ago
Yeah, I'm rather confused too. I run a site with over 10TB transfer per month on a $15 / month Digital Ocean virtual machine. It's fast enough and Cloudflare caches over 80% so I seldom go over the included 2TB of monthly bandwidth. I'm sure STH has more visitors though.

segmondy 6y ago

AWS gains is like a bell curve, the value is highest in the middle. If you're very small, you probably don't need it, and DigitalOcean without surprise costs should do. When you start getting large enough, AWS starts paying off, as you get really huge, then it might be cheaper to colocate and hire your own admins. However, I think Netflix shows you can't be too big tho for AWS. Their hybrid approach is probably best. Run on AWS/GCP for things that you just can't afford to fail, then move your non critical workload to colocation.

mekster 6y ago
Places like netflix wants to put liability off its shoulders and wants someone to blame at. Very different demand than most.
- segmondy 6y ago
  I don't buy that, liability to who? If Netflix goes down, I'm gonna cry and tweet about it, I'm not going to cancel my account. Their catalog is unmatched. colocating across the world is not easy, they also have scale that goes up and down based on time of day, and say holidays where they really need to scale up. Cloud is great for that, the elastic part is great, place the core on there and scale up and down as needed. What they did with active-active resiliency and being able to shift traffic across regions is the entire point of cloud, as we some a bunch of services where down last week with AWS having issue but netflix never skips a beat on such issues.
  - mekster 6y ago
    I don't mean they'll make excuses toward customers but say they can simply blame AWS toward content providers when something happens to the infrastructure instead of begging them to wait while they prepare for a damage report on their custom infrastructure.

KaiserPro 6y ago

I'm a bit conflicted about this. Yes, AWS is expensive. And depending on your workload you can certainly save a fair wedge of cash by colo-ing things. For wordpress, (the site that is running serverthehome) its perfectly possible to run it in a lambda, with aggressive caching and save a whole bunch of cash. depending on plugins, load and a number of other things, it could be a potential saving of 90%. (plus a massive reduction in attack surface) Before you ask, yes I do know from bitter experience, parts of the financial times had WP wedged into them. making them fast and secure was an interesting experience. AWS is only cost effective when you are using ec2 with a duty cycle of less than 50%. This means that you are not using AWS for hosting 24/7 compute. In terms of storage, There are two compelling offerings, S3 and EFS/lustre. however if you're a large scale EFS user, its better to run your own GPFS system. The hardest part of "scaling" is orchestration. If you're using K8s, there really isn't much difference between running it on virtual vs real steel, barring bringup scripts(don't get me started on networking in K8s, its totally warped.) TLDR: IF you need 100+ machines on 24/7, AWS is going to be more expensive. If you have transient loads, and the average time on for a machine is 4 hours or less, then AWS is for you. However, you'd better use all the other bits that come with AWS to make it cheaper, like fronting it with fastly/otherCDN

cinquemb 6y ago
Yeah, last place I worked had transient loads we pretty much did everything with just ec2 t3a.medium spot instances (minus s3 and rds [which kind sucks with using postgresql if you want to resize down or load from a pg_dump since you cant copy the sql to the rds instance and forced to backup over the network], I wanted to move to ec2 instances and from s3 to b2 since its cheaper for bandwidth with [we had cdn77 in front of cloudfront to lower costs outside of US/EU]). Using docker or k8s just doubled the costs over just using disk images and custom load-balancing daemon with nginx (with 3rd party modules for changing upstreams without restarts/reloads)/python, on top of adding more networking complexity (like to caching/search/rabbitmq instances). Probably around 20+ machines at peak when i left, but kept them all below 20% cpu. Sad to see so many places go in the docker/k8s direction.

kev009 6y ago

I have my own 1/2 cab, AS number, ARIN allocated IP4 and IP6 space. It costs about $500/mo. It took maybe 100 hours to set up the basics and $6000 for the way I built the hw (which is similar to servethehome's rack). I run everything on FreeBSD. I get to directly talk BGP and get bandwidth at .12c/gbit. Nothing else can come close to this level of low cost, control, learning, and autonomy.

CKN23-ARIN 6y ago
.12c/gbit?? I need to renegotiate my transit bill...
- kev009 6y ago
  he.net is .09!
- kev009 6y ago
  d'oh, I meant mbps commit not gbit :S

partiallypro 6y ago

We used to use an independent host (essentially self hosting) but they went under due to COVID. We moved nearly 1000 customers over to Azure and in the end we are saving money due to simple things like reservations and being cost conscious. We also took over a company that used AWS and they were over spending on their hosting by almost 10-20x what they needed. Not because AWS is inherently expensive but because it was so mismanaged. I don't think self hosting is worth it, when I can freely scale up or down machines in seconds on Azure and my price goes down, we have no fixed costs or maintenance worries. I think it's a fair trade off.

chmod775 6y ago

I mean yeah, AWS is a 10-100x markup over running your own stuff. I run a file hosting site that serves >500TB of data to about 2 million monthly users on a bunch of dedicated servers that cost me about $500 a month. If I ran this on AWS it would cost me more than $20,000 every month. I could hire four system administrators to do nothing but look after each of my four dedicated servers for that money. It's completely ridiculous.

leesalminen 6y ago
Outbound data transfer pricing is definitely _the_ gotcha for the big clouds.
trianglem 6y ago
Same. I’ve been in places that self host and it’s not nearly enough to overcome what AWS (in my case Azure) charge. 99.5% uptime in contracts lets you realistically have a lot of downtime to fix your problems. That’s more than 400 hours of acceptable downtime. We’re in the process of making the switch and we project to pay 30% of what we’re currently paying.
- rubatuga 6y ago
  There's not even 44 hours of downtime in a year for 99.5% SLA, so I have no idea what you are talking about.
MattGaiser 6y ago
Isn't transfer a huge cost for AWS? I know of some people who have gone with Oracle because it supposedly has much lower pricing on that.

xfitm3 6y ago

Having previously performed an extensive TCO calculation (2010) it was substantially cheaper for on-prem vs AWS. We had <10000 machines, mostly cheap/supermicro. We had our own parts inventory and staff dedicated to maintaining equipment. I also ran facility operations for things like the generator, fuel testing, UPS PMs, air filters, professional cleaning, etc. You have to know your business and you need people with facilities skills. Investing in cold aisle containment reduced our PUE quite a bit. You also have to become an expert in logistics to run your own facility, where just almost anyone can click away in AWS and spend money.

joana035 6y ago

That's is great, not to mention you get real CPUs, no throttling inside your LAN, real storage. The cloud makes you addicted to the "but you have to scale" by selling 2 to 4 vCPUs plus 8 to 16GB of memory for the same price you can get a 12 vCPU with 64 or 128GB somewhere else.

ericpauley 6y ago

I have a hard time believing the setup pictured offers equivalent reliability/disaster recovery to "cloud" hosting. As far as I can tell, all traffic is being routed through two servers. These servers are running Linux? What happens when a routine software update bricks your routing (it could happen to anyone). Without remote access you need someone to go in and fix this. Do you have after hours access? Is someone nearby on call to respond to complex issues in person? Not all physical problems can be solved by a random colo tech. Colocation (as pictured) lacks management of a lot of variables that could lead to big problems. When your downtime targets are in minutes per year any incident requiring physical response is unacceptable. A simple Google search reveals a service philosophy compatible with this hosting[1]: > As with all web systems, at some point, downtime is required. For the rest of us there's "cloud" hosting. [1] https://www.servethehome.com/pardon-dust-upgrades-progress/ https://www.servethehome.com/pardon-dust-upgrades-progress/

mattbeckman 6y ago

Our small team spends roughly $40-50k/mo at AWS. If you're just hosting a website with a SQL database (glorified LAMP stack), you're probably in it for the wrong reasons. Stick to a dedicated box or DigitalOcean etc. AWS is like co-locating your hardware, and then the data center having 1000s of employees offering highly reliable services you can access from your infrastructure that lets you move WAY faster for things that are hard to do. e.g. Discussion today was increasing log retention. 10 years ago I would have run the numbers, extended some SAN volumes, considered procuring more NetApp shelves, etc. Today it's simply a cost question: is it worth it to us to store those logs for 10x longer? Sure? Ok, done.

busterarm 6y ago
I think the point that you're making here is subtle and that people are missing it. Your small team can _comfortably manage_ 50k/mo worth of AWS resources. That's _INCREDIBLE_. I think people mistakenly think that cloud costs only go up exponentially and that costs are an unmanageable mess. I've worked with teams with hundreds of engineers serving a major enterprise and only an AWS bill 3-5x yours. And they only had a small team managing it all. Comfortably. To do similar with physical servers requires a massive stack of people and salaries. The ongoing recruiting costs to maintain staffing would dwarf what the AWS bill is.
- apple4ever 6y ago
  > To do similar with physical servers requires a massive stack of people and salaries. The ongoing recruiting costs to maintain staffing would dwarf what the AWS bill is. No no they wouldn't. Number of people is equal for both, its just different tasks they have to do.
  - busterarm 6y ago
    I do both at roughly 10^5 scale and I'm telling you that operating in the cloud lets us operate with an order of magnitude less people.

ngcc_hk 6y ago

Only use cloud provider like digital ocean with a limit on financial outlay. You prepay. Not for large corporations. But for micro firm ops, the worry of a big bill is there.

dragonsh 6y ago

The biggest advantage of using self-hosted infrastructure is real freedom from the whims of large cloud providers and the governments who indirectly exercise control over which company, country and individual such cloud providers can do business with. On a large cloud provider a government notice is enough to remove access to compute resources and data hosted in it. This in essence means company or individual using this cloud services neither own the compute resources and data hosted in it. On the other hand in self-hosted infrastructure government notice will not block access to resources owned by you and data in those resources. You can respond to notice and continue using those resources until court rules. So it is more freedom than a cloud can offer. Obviously given the large marketing budget and convenience cloud provider flourish at the expense of freedom. Self-hosted infrastructure needs a new renaissance given whats happening around the world and how government and large corporation taking away freedom one bite at a time. So it is very essential for development and freedom for humanity to have self-hosted compute, networking and data infrastructure.

lmz 6y ago
Do you really think your colo provider is going to ignore government notices to disconnect you?

waheoo 6y ago

Yet another article comparing pricing of self hosting vs self hosting in the cloud. I'm so shocked. /s Let me know when you compare the cost of running all of AWS service offerings self hosted.

AnthOlei 6y ago

For those of you who have a modern serverless architecture (lambda, dynamo, firebase etc.), how do the costs compare there? I’ve always thought serverless is cheaper, but I’ve never ran anything at scale.

xupybd 6y ago

I just had a simple PHP application move from simple VPS hosting to full blown AWS with load balancers, RDS databases and EFS storage. It's still the same app the load balancer can only send the traffic to one machine. RDS now means DB connections are TCP not pipes. The EFS is slower than an SSD. So it's slower and costs more. Why did this happen? Because an external consultant said AWS is what the big players use. I argued that our app needs to be engineered to take advantage of AWS or this is an expensive waste. But what would I know, we went with the consultants and payed them tens of thousands to set up a bunch of AWS infrastructure I could have run on a $10 a month VPS...

twothamendment 6y ago

It all depends on the scale of not only the app, but the organization too. We have enough people to click buttons in RDS. We don't have enough to mess around with replication, backups, etc. The cost isn't just for the hardware, but the services that we'd otherwise have to hire someone to do. I used to work at a place that would borrow hardware from one client in order to handle the load on another. It was for seasonal stuff, a few days a year when we knew someone would melt down if they didn't have double or triple the capacity, but they didn't need it the rest of the year. Now I deal in auto scale groups and don't care how many servers we use or need as long as it isn't wasteful. "Hardware" that isn't ours is so easy to swap out, try a different size or scale that is hate to go back to metal.