28 ms·

We reduced our server costs by moving away from AWS

440 points by caberus 4y ago

P5fRxh5kUvp2th 4y ago
I'm glad to see more of these types of articles, but at the same time I'm a bit flabbergasted that this isn't obvious for so many people. These cloud providers are, by definition, charging you more than it would cost you to run it yourself. What you get in return is a guarantee of expertise and an ecosystem.
- HWR_14 4y ago
  You also get a lot of finely incremental costs and the ability to grow. Like, can you really buy 0.1% more IT people? Well, yeah if you already have a thousand. Otherwise you have to buy in bigger chunks. So you save money because you can timeshare talent and hardware. And the ability to write a larger check to scale is valuable for consumer facing applications you're hoping by to grow.
  - LanceH 4y ago
    And, importantly, you can turn off your commitment to spending in a second.
    - whatever1 4y ago
      Making IT an OPEX problem is the main reason that C-suites love cloud providers
    - HWR_14 4y ago
      I thought that was less true when you started signing deals for discounts. Commit to a spend of X and they'll reduce your bill. That kind of stuff
  - zo1 4y ago
    Sure you can. Ask one of your devs to spend X amount of time on something. Might not be an exact 1 for 1 or 0.1% but it still beats the many many hours you'd waste architecting around cloud-specific issues and nuances. Or the cost of dealing with the friction that comes when you want to do anything with cloud in a manner that isn't cloud-supported.
    - HWR_14 4y ago
      Devs love being told to set up servers! And always do so well. After all, computer person is a fungible skill set. I mean, it is fungible enough that they can learn, but they'll probably make a lot of mistakes along the way and be less efficient.
      - P5fRxh5kUvp2th 4y ago
        Any developer that can't setup a server has fundamental gaps in their skillset.
        alex_suzuki 4y ago
        Someone missed the DevOps revolution…
        unregistereddev 4y ago
        On a trivial level I agree, but only on a trivial level. Most developers will not set up a server in the most secure, maintainable manner. They're going to miss important things: - Disable password-based auth for ssh (require key-based auth) - Enable fail2ban or similar to slow down brute-force login attempts - Configure firewall - Install monitoring tools, antivirus, possibly backup daemons, etc - Setup a sane swapfile for your use case, and configure monitoring tools to alert when memory pressure gets too high - Setup disk mounts, configure monitoring tools to alert when disk space is low, and consider a cron job to automatically clean up tempfiles - Either set up automated updates (typically excluding kernel upgrades), or have a standard schedule for manually applying updates ...and probably other things that I'm forgetting because I'm a developer, and it has been years since I've been a sysadmin.
- hotpotamus 4y ago
  I mean, in theory all services are sold to you with the goal of making a profit, so there must be some margin baked in, but AWS has enough margins to run the Amazon retail store as basically a non-profit while also making enough money to finance Bezos's penis rocket rides.
- rossdavidh 4y ago
  I think some of this is due to the presence (until just recently) of abundant VC money, encouraging tech companies to trade of money for time by outsourcing their server admin (or part of it, anyway). It's one less thing you have to hire for and spend time getting good corporate policies and procedures on. But, now that the emphasis seems to have shifted from growth-at-all-costs to where-can-we-cut-costs, I think there may be more organizations with a large enough server load to realize that they can do it cheaper (or even just get a different cloud provider to do it cheaper).
  - dkarl 4y ago
    It would be shocking if, in the long term, there were not cloud providers with lower prices than any but the biggest companies could achieve by themselves. AWS will have to work hard to keep giving customers a reason to pay the premium.
    - nithril 4y ago
      Still waiting for OVH or Scaleway to be really multi AZ (>2) with a network that cannot be cut by anyone...
  - P5fRxh5kUvp2th 4y ago
    imo this is the right takeaway. You see a lot of posters commenting that it isn't obvious that it's more expensive, but that's a large part of why I said I'm flabbergasted at how many people don't find it obvious. It is obvious, someone may still choose to pay it because there's a natural tradeoff, but it's absolutely obvious. I suspect most developers don't find it obvious because they've never dealt with actual servers themselves. It's akin to arguing changing the oil on your vehicle is more expensive than paying someone to do it. No, it's more convenient, and that may make it worth the price.
    - elcomet 4y ago
      Again, you seem to not take into account the developer/admin time. In your example with oil, if you track your time, it's gonna be certainly more expensive than going to the garage. And also of course, it allows you to go faster, which can have a huge value
- Octoth0rpe 4y ago
  > These cloud providers are, by definition, charging you more than it would cost you to run it yourself. That is _not_ a given. They're charging you more than it costs _them_ to run it. They get: - much lower hardware prices than you - lower bandwidth prices than you - likely lower electricity costs than you
  - Marazan 4y ago
    Aws bandwidth costs charged to consumers are a notorious ripoff.
  - AtlasBarfed 4y ago
    AWS's bandwidth charges are highway robbery. It's my impression that AWS competes with the EC2 instance cost (as it that's what new customers look at), and the bandwidth/storage only becomes apparent when you are locked in. Really what AWS should be is one or two phases of service maturity: dev infrastructure and experimentation is phase one, phase two would be the "its a couple servers" production, BUT: with a scale plan for phase 3 being not-AWS. Having a mature/battle tested phase 2 --> phase 3 should be a market advantage in the modern business landscape, but that's also a post-acquisition/exit phase, so none of the presumed target HN crowd care about it. There are extensive articles and public knowledgebases on various technologies and architectures, but there is effectively nothing on AWS independence out there, even though almost every mature organization will need to face it.
    - hotpotamus 4y ago
      I'm also a bit blown away that anyone is surprised that AWS is expensive. Sure you can express a lot of the costs in small numbers per hour or $0.08 per gigabyte of bandwidth, but those things all add up quickly at scale. I assumed phase 3 was where AWS started giving you massive discounts because you represent so much business for them, but I've never been at a place big enough to swing that stick at them.
      - tablespoon 4y ago
        > I'm also a bit blown away that anyone is surprised that AWS is expensive. Sure you can express a lot of the costs in small numbers per hour or $0.08 per gigabyte of bandwidth, but those things all add up quickly at scale. Could it be magical thinking based on marketing hype and autoscaling fantasies?
- ehutch79 4y ago
  When you're a small team, possibly solo, having someone else deal with hardware is key. Focusing just on what you're working on is great.
  - P5fRxh5kUvp2th 4y ago
    assuming the money isn't an existential risk, that's absolutely true. I was commenting on the expense rather than why you would (or wouldn't) want to use cloud.
- elcomet 4y ago
  > These cloud providers are by definition, charging you more than it would cost you to run it yourself That's totally non-obvious and certainly false in most cases. Cloud providers have huge economies of scale, which certainly makes it less expensive to use cloud if you factor in all your costs related to running it yourself. This is like saying buying furniture is more expensive than making your own furniture because they have a profit margin. Even if you don't account for your own time, it will certainly be false as you'll pay your raw materials much more.
  - thehappypm 4y ago
    The furniture example is great. If I wanted to build an Ikea table from scratch, I could not do it for less than the $20 they charge, given the cost of materials available to me, and the fact that I can’t buy a cup of paint for $2.
  - unity1001 4y ago
    > which certainly makes it less expensive to use cloud AWS would pass that difference to Amazon shareholders than passing them to you.
- goodpoint 4y ago
  > These cloud providers are, by definition, charging you more than it would cost you to run it yourself That's pretty obvious but years ago you would have been downvoted into oblivion for writing that.
maxfurman 4y ago
I feel like there's a middle step missing from this article (or I just missed it reading quickly) - did they build their own data center? Where are these new non-AWS servers physically located?
- hotpotamus 4y ago
  Colo would also be an option. I run a mix of colo and cloud currently.
- fabian2k 4y ago
  You can easily rent space in a data center and put your own servers in there or just rent dedicated servers. The cheapest provider there is probably Hetzner, but then your servers are in Germany. This gets you a real, full server for something between 30-200 EUR per month. There are many other hosting providers that offer dedicated servers also in the US, e.g. OVH. This works the same way, they are usually more expensive than Hetzner. Renting a dedicated server is very similar to renting a VM. It has a much higher provisioning time, usually hours to 1-2 days. And it takes some upfront cost or a time commitment. But it also has a lot more raw power. So if you don't require the flexibility VMs and the cloud provide they can be a very effective alternative.
  - unity1001 4y ago
    > The cheapest provider there is probably Hetzner, but then your servers are in Germany They have a US DC now. Though it only gives cloud servers. However even those are still way cheaper than any alternative.
  - systemvoltage 4y ago
    If you want rocksolid dedicated servers of utmost HA and reliability, look no further than Equinix Metal: https://metal.equinix.com/product/servers/ https://metal.equinix.com/product/servers/ They run datacenters for other datacenter companies. Most people have never heard of them because they're a $6B/year revenue beast that runs behind the scenes; but they also offer direct metal servers.
- 4y ago
hnrodey 4y ago
I find this interesting, if nothing else. My first question is what was the opportunity cost of focusing manpower to setting up on-prem infrastructure that now needs maintained? What on the product roadmap was sacrificed/delayed in exchange for the time on this project? What are the projected future hiring costs to maintain these servers (and applications like Cassandra!) going forward? Nothing is free, and at just 4-5 additional hires they will be giving back a large chunk of that $800k to employees. IDK - maybe that's a fare trade-off to pump up the common man with money instead of the establishment.
- brodouevencode 4y ago
  This is what a lot of people miss when they talk about moving to/from cloud providers. The marginal cost to add X more servers in the cloud is basically nothing, whereas to set up a new rack for on-prem requires requirements gathering, purchase orders, finance approvals, someone being at the dock when the UPS truck arrives, rack and stacks, etc. Those are one-time, yet very real costs. These fall under cap-ex which accounting-wise is treated very differently than op-ex. Now the cost in the cloud bakes all that in, and is distributed around with other users of the provider. Your accounting models are also easier ("pay for what you use"). Couple that with the very well known fact that AWS has outrageous data egress charges and there are patterns that can emerge where you're still in cloud but not racking up massive outbound data charges.
  - fabian2k 4y ago
    You can rent servers if you don't want to bother with this. The choice is not only between doing everything yourself and the cloud, there are a lot of options in between.
    - brodouevencode 4y ago
      Yes - that's a very fair point. It should still be calculated in the cost, and I don't think the article does a very good job of identifying the tradeoffs to OPs point.
taldo 4y ago
When the cost of delivering your product/service is mostly compute or traffic, sure, migrating off of AWS is a must once you reach a certain scale. But for the other 99% [0], where infrastructure is but a small cost, then think really hard if you're willing to trade the engineering effort for the convenience of managed cloud services. [0]: or 90%, or 80% or who cares, but a majority of software services seem to NOT be compute- or traffic- heavy.
hazmazlaz 4y ago
There is no way this figure is accurate. The annual spend cited of $1,000,000 is purely hypothetical, as admitted here: "However, all this data and processes need to happen on a server and, of course, we used AWS for it. A few years of growth later, we’re handling over 70,000 pages per minute, storing around 560 million pages, and paying well over $1,000,000 per year. Or at least we would be paying that much if we stayed with AWS. Instead, we were able to cut costs by 80% in a little over three months with some out-of-the-box thinking and a clear plan."
- pessimizer 4y ago
  > There is no way this figure is accurate. You've got the FUD covered, but you also need to add at least some substance to your claim. How do you know this figure would not be accurate? Why is your (hypothetical, not offered) estimate better than the author's?
  - merb 4y ago
    well the problem with the article is basically that they left out a lot of important detail, like which bare metal servers, how many, where do they host now, did they use cloudfront what about cloudflare did they use an edge cache? what about reducing costs by killing stopping unneeded resources? a lot of their workload looks dynamic. it's also fishy what they wrote here: > After testing whether Prerender pages could be cached in both S3 and minio, we slowly diverted traffic away from AWS S3 and towards minio. if they served directly from s3 that would be, stupid? > In the last four weeks, we moved most of the cache workload from AWS S3 to our own Cassandra cluster. is also strange. it misses a lot of detail but it does not look like they just migrated away from s3... (looks like their new hoster is hetzner, from service.prerender.io )
    - pessimizer 4y ago
      > well the problem with the article is basically that they left out a lot of important detail, The problem with your comment is that you insisted that they gave you enough detail to definitively determine that they were either lying or mistaken. > There is no way this figure is accurate.
alberth 4y ago
Dedicated hosting providers. I'm so amazed that somehow people completely forget that for literally decades, web host provided dedicated hosting options at fantastic prices. Yes, loooong time ago - to get your dedicated server might have taken a few hours to provision and the instant server access that AWS brought should not be discredited. But large numbers of web host today allow you to programmatically spin up a dedicated web host instantaneously and at a fraction of the cost.
- that_guy_iain 4y ago
  > Yes, loooong time ago - to get your dedicated server might have taken a few hours to provision and the instant server access that AWS brought should not be discredited. At one point it took often days for a dedicated server to be set up. We also didn't have such nice provisioning tools. Now it just seems like cargo cult to use cloud providers as the only option. People just completely discount dedicated servers.
  - frognumber 4y ago
    In most businesses I've worked in: developer costs >> infrastructure costs An AWS large server is around $500/year, which is about 1-2 developer hours (with taxes, overhead, etc) at the cost scales last time I priced this out. That's crazy expensive in the absolute, but if it saves a couple of hours, it makes sense. PaaS providers cost even more. I've gone with those in the past, since it basically eliminated dev-ops. The major downside wasn't cost, so much as flexibility. Dedicated servers start to make sense for: - The very low end (e.g. personal use, or hosting something long-running for a small business) - The very high end (e.g. once cloud costs start to hit hundred of thousands of dollars per year) On the very high end, cloud providers will often cut a deal, though. My problem with AWS, recently, has been reliability. Servers crash or have performance degradation a bit too often. That leads to developer costs, and might be what pushes me back to dedicated.
    - isoprophlex 4y ago
      1-2 developer hours?! Anyone looking to hire an ML guy for 250-500$/hr flat fee, hourly rate, no taxes, no healthcare, cancel any time... get in touch! For that money I'll gladly bark like a dog, walk on all fours and fetch your slippers.
TheGuyWhoCodes 4y ago
"We used Apache Cassandra nodes on each of the servers that were compatible with AWS S3". What does this even mean? Regardless, starting a new Cassandra cluster in late 2022?! I bet they can save even more by just going with scylladb
- joshstrange 4y ago
  That was also confusing to me as Cassandra is a NoSQL DB last I checked. I found this [0] online that indicates with some extra software you can talk to it like S3 but yeah... [0] https://dzone.com/articles/s3-compatible-storage-with-cassandra https://dzone.com/articles/s3-compatible-storage-with-cassan...
deleted 4y ago
[deleted]
epberry 4y ago
I believe this is the use case Cloudflare is really targeting with R2. They recently connected Cache Reserve to R2 to make this even easier. We wrote up a breakdown for S3 vs R2 and found that R2 would be significantly cheaper when the majority of traffic is cached data, https://www.vantage.sh/blog/cloudflare-r2-aws-s3-comparison https://www.vantage.sh/blog/cloudflare-r2-aws-s3-comparison
floatinglotus 4y ago
This is the Trillion Dollar Paradox described by Martin Casado. You’re crazy if you don’t start your business in the cloud, you’re crazy if you stay there. My new startup is focused on helping application owners repatriate their workloads into their own infrastructure. Our goal is to solve the network complexity challenges with a fully open network stack (open source software with all of the hardware options you would expect, and some you wouldn’t). The solution is designed to be turnkey and require very little network experience. It will use standard DevOps tools that you’re already using. We’re announcing it in two weeks and will be posting info here on HN!
- user3939382 4y ago
  I've had tons of greenfield web apps with thousands of users (not millions) and setting up a server was a problem I didn't need help solving. I'm versed in AWS now but, I don't see any benefit. The only big headache I had with server admin was running email services/deliverability.
- JohnHaugeland 4y ago
  > This is the Trillion Dollar Paradox described by Martin Casado. You’re crazy if you don’t start your business in the cloud, you’re crazy if you stay there. That's not a paradox. You're not crazy in either case. Starting in the cloud reduces costs for some strategies by removing necessary engineering. Starting in the cloud increases costs for other strategies by charging too much for commodity offerings. It's relatively straightforward to make the choice as soon as you put a specific business in the crosshairs. Edit: why is this downvoted?
  - gadflyinyoureye 4y ago
    You’re going against the party line.
    - JohnHaugeland 4y ago
      It's getting to the point where even if I just say "that isn't what I meant" or "that's a paraphrase, not a quote," I get flagged for it
joshstrange 4y ago
They don't mention at all what services they were using (other than slight mention of S3) which makes it very hard to respond to this. If you are running everything on EC2 then you are going to have a bad time (especially if you aren't using reserved instances). AWS (IMHO) shines with the various services they provide (S3, Lambda, CloudFront, API Gateway, SQS< SES, to name a few). AWS is a game of trying to reduce your bill and often that means using AWS-specific services. If you want to stay completely "cloud agnostic" you are going to paying more than buying into a "cloud", in that scenario then you absolutely should be looking at dedicated servers. AWS is great because you can bring existing software and just run it in EC2 (or their container stuff if your software is containerized) but the AWS magic comes from using their managed services (also spinning up/down EC2 instances as needed, but if you are running them 24/7 then consider alternatives or at least pay for reserved).
- taylodl 4y ago
  Yes. If you use and manage AWS resources as you would your own on-prem resources then you're not going to have a good time. As soon as you think you need EC2 instances you need to re-think your architecture. You're probably not using AWS most effectively.
  - shudza 4y ago
    So what should you use for let say, a backend API service? Don't tell me beanstalk/fargate/etc, because they're actually more expensive.
    - joshstrange 4y ago
      Lambda is an option, it's been a very attractive one for me.
      - jjav 4y ago
        AWS lambda is super cheap at tiny scale. But if you get into any real constant load, it is way more expensive than a VM. The other drawback of lambda is the flip side of having the server opaquely managed by AWS. It is so opaque you can't debug anything. At one startup we had a weird connectivity issue from lambda to RDS but it was impossible to diagnose given the lack of access so it went on for months. Had it been running in a VM, I could've diagnosed that within an hour with tcpdump and bpf et.al.
deleted 4y ago
[deleted]
m0llusk 4y ago
There is not quite enough information here to be sure, but this article highlights transmission costs. This particular business model involves throwing around big chunks of data just in case they end up being needed and then handing them back out in response to potentially large numbers of requests. That would make this particular usage pattern fit to exactly what AWS is charging the most for. Also many alternative AWS services that can be used to speed up or simplify services are not really going to help with this case. So an alternative way of interpreting this is more along the lines of: We may have saved up to 80% of server costs by moving from AWS, but you almost certainly won't save that much even if a bunch gets spent on developing operations and tools.
- varsketiz 4y ago
  You can save even more if your app uses richer formats than images. Also, if you are bigger and can start really negotiating with hardware providers.
wglass 4y ago
I found the headline to be misleading. The article is mostly about the migration process (which is interesting), but very little about the details of the cost savings. What does it cost to run their data center? What are the salaries they are paying for internal IT efforts to administer it? Is it an apples-to-apples comparison, e.g. are they load balancing across multiple datacenters in case of an outage? It sounds like this was a good move for Prerender but it's hard to generalize the cost claims to other situations without details.
0xbadcafebee 4y ago
They saved $800K on their AWS bill, but - may spend $250K on servers, replaced after 3 years becomes $83k/yr - may spend $120-250K on extra staff to maintain the infrastructure - may spend $15K for a cage in a DC They still save $452K/yr overall (actual savings 1st year only $285K). It's still a savings for sure, but always keep TCO in mind. The real fun comes later when you outgrow your cage and there's not enough space left in that DC, or they just have shitty service constantly knocking out your racks, and you have to consider splitting your infra between DCs (a huge rewrite) or moving DCs (a huge literal lift and shift). Have been part of both, it's... definitely a learning experience.
shrubble 4y ago
I expect to see a great deal more of the "cheap and cheerful" AWS migration stories in the future. With the tanking of the market and (apparent) limits to growth being in the forefront, reducing expenses will become more important. Before, it was easy to justify almost any expense with the "we just need to get 1% of this $100 billion market" and now it is "hunker down and do everything you can to be ramen-profitable, in order to survive and thrive".
Joel_Mckay 4y ago
In general, last time I looked at AWS it made sense from 2TB to 30TB a month, and under 400k connections a day. If either range was exceeded, than the service ceased to be the economical choice when compared with CDN providers, and colo/self-managed unlimited-traffic options. For example, if you primarily serve large media or many tiny files to clients that don't support http Multipart Types, than AWS can cost a lot more than the alternatives. However, AWS is generally an economical cloud provider, and a good option for those who outsourced most of their IT infrastructure. The article would be better if it cited where the variable costs arose.
maerF0x0 4y ago
I've said this a hundred times and it seems not loud enough. AWS is not cheap because of your server costs. AWS is cheap because of elasticity, velocity (opportunity cost of next feature), and reduced maintenance hours. "The cloud" was never (afaik) was about getting a cheaper VPS. It was about being able to get them on demand, give them back on demand, and generally not have to maintain anything besides your code (and maybe apply updates to datastores / AMIs) Now, if those premises are not true for your startup/business, then AWS is not the tool for you. I didnt see any analysis of ongoing maintenance costs in the 800k saved, but will it take 1-2 FTE engineers to now be more oncall, more server upgrades, more security patches etc? That's easily 1/2 that savings gone already. Edit: for the most part these attributes apply to GCP, Azure, Heroku etc as well, its not just about AWS
- draw_down 4y ago
- shudza 4y ago
  This is one of the few accurate comments in this thread.
- chucky_z 4y ago
  I think there's some "cheat codes" now involving some of these things though. For instance, you can use some of AWS but keep your actual compute/networking out and have things like security patching and server inventory be completely solved by using AWS SSM. There's also options like EKS Anywhere to have managed on-prem k8s for way, way cheaper than running it in AWS proper. These kinds of services I think are the future for hybrid/on-prem folks.
- candiddevmike 4y ago
  Elasticity is a gamble. You're betting that you can save more money investing in an elastic/on demand stack than what you'd pay for static resources. Judging by how much the cloud providers push this, the unknown cost to create/maintain auto scaling infra/apps, and how intricate the pricing gets with elastic resources/spots/faas, I still think the cloud providers are coming out ahead vs folks using static compute.
jaclaz 4y ago
As a side note, I find this: >Do you have any advice for software engineers who are just starting out? >Don’t be afraid to talk with the customers. Throughout my career, the best software engineers were the ones who worked with the customer to solve their problems. Sometimes you can sack a half year of development time just by learning that you can solve the customer’s issue with a single line of code. I think the best engineers are creating solutions for real world problems. to be very good generic advice.
mabbo 4y ago
I'll always celebrate stories like this, but I also don't take some kind of anti-AWS lesson from it. This company saved $800k/year. Perfect time to go in-house with this solution. But when they were 1/10th this size, they'd only have saved $80k/year. Does that cover the cost of the engineering to build and maintain this system? Maybe not. And when they were 1/100th the size, it would have been laughable to go in-house. At the right time, you make the right transitions.
- lbriner 4y ago
  Yes, exactly this. At what point do you have the time/money/confidence to invest goodness knows how much in a data centre with space to grow, to purchase an enormous amount of capital to have it all installed etc. the building alone could eat that first years saving easily. How many people are now needed to fault-find bad hardware/software/networks, to be on call for any problems? How many calls out to the Electrician to fix some power issue? How much to setup and run a large air-con system for the data centre. Maybe not much in the US where aircon is common but much more expensive in Europe. The fact they could afford to do this over such a short time period speaks to having a decent amount of cash on-hand.
  - hamandcheese 4y ago
    > At what point do you have the time/money/confidence to invest goodness knows how much in a data centre with space to grow, to purchase an enormous amount of capital to have it all installed etc. the building alone could eat that first years saving easily. Co-locating has no capital investment other than hardware, and is pretty cheap. A 40U rack of compute charged as equivalent ec2 instances has a retail price easily of hundreds of thousands, if not a million+ USD per year. Suppose each U has a $10k capital cost to make the numbers round, that is $400k in capital. All this to say is that I don’t think capital is as big a factor as you might think.
    - boltzmann-brain 4y ago
      In my experience doing detailed projections of exactly something like this - with racks full of GPU compute power - the infrastructure has paid itself after one quarter, maybe two, depending on what volume you're at. There very rarely is a reason to use GPU compute on the cloud - and advantages start as quickly as with just one single GPU.
Gregioei 4y ago
So little real information... So the team is now responsible for backups, hardware ordering,.forecast etc? How big is the team now compared to before? Does it scale? If you price it correctly and keep the free tier small, I would either talked to AWS for better pricing or moved to another cloud Provider. S3 on AWS is a total no-brainer, minio on bare metal might mean much more work and a bigger infra team than business actually wants. I would also love to know what optimizations are already in place. Does cloudflare caching work? Are the results compressed on rest? Is geolocation latency relevant? Why even Cassandra? Are websites not unique? Wouldn't a nginx and a few big servers not work? But who knows? The article doesn't tell me that :-(
gibsonf1 4y ago
We've just finished moving servers from AWS to https://hetzner.com https://hetzner.com - and saved 10X with servers of double the capability. A great experience so far.
- jacooper 4y ago
  How did you get them to approve a large amount of Cloud instances / dedicated servers? I heard they are very stubborn to increase the per user limit of cloud instances. Also how did you deal with S3? Did you switch to another provider ? Like B2?
  - gibsonf1 4y ago
    We haven't run into the number of servers issue. For S3, we've switched to Wasabi which very nicely uses the identical API.
    - jacooper 4y ago
      Honestly Wasabi's advantage is free egress, but with b2 + cloudflare gives a better billing system without the three month object retention thing and a CDN with no egress limit.
  - CoolCold 4y ago
    May be it was not large. Say for one project DB server on Hetzner with 160GB RAM, 32cores/64theads,2x3.84TB NVMe + 2x512GB SSD costs ~ 240$, hosting mid-sized ~ 1.6TB MySQL DB. On managed RDS it was around ~ $2200/month when I checked. Even if you have 2 such boxes for master/replica, it's close to 5x savings.
theptip 4y ago
Perhaps I'm missing it in the OP -- I don't see any mention of what they actually moved to. CoLo? VPS? On-prem? This seems like a key detail when telling people about your migration off AWS.
- jacooper 4y ago
  On-prem
deleted 4y ago
[deleted]
zc2 4y ago
Then you need to spend 2M to hire engineers helping you to maintain the in premise infra
otabdeveloper4 4y ago
No kidding? You reduced your server costs by moving away from the most expensive hoster on the planet? Good for you, I guess :)
jacooper 4y ago
Some S3 providers can give you free egress when using a CDN. For example backblaze B2 offers free egress through Cloudflare, Fastly, BunnyCDN. https://help.backblaze.com/hc/en-us/articles/217666928-Using-Backblaze-B2-with-the-Cloudflare-CDN https://help.backblaze.com/hc/en-us/articles/217666928-Using...
throwaway20221 4y ago
Throwaway here. I work for a startup which runs compute-bound scientific simulations. We are considering migrating from our single self-managed HPC, to AWS EC2. We anticipate using the largest standard instances, with highly spikey usage: frequently none, at times up to 10 concurrent instances, perhaps more later. AWS seems ideal to me because it would let us easily scale up and down as our usage varies. But some of the anti-AWS sentiment in this article has given me pause. Any reason not to do this? Any alternatives I've missed? Our storage and transfer usage will be negligible; it's all compute.
thoop 4y ago
Hi! I’m Todd, the solopreneur founder of Prerender.io and I created that $1,000,000/year AWS bill. I sold Prerender.io to Saas.group in 2020 and the new team has done an incredible job growing and changing Prerender since I left. $1M per year bill is a lot, but the Prerender back end is extremely write-heavy. It’s constantly loading URLs in Chrome in order to update the cached HTML so that the HTML is ready for sub-second serving to Google and other crawlers. Being a solo founder with a profitable product that was growing organically every month, I really didn’t have the time to personally embark on a big server migration with a bunch of unknown risks (since I had never run any bare metal servers before). So the architecture was set early on and AWS allowed me the flexibility to continue to scale while I focused on the rest of business. Just for a little more context on what was part of that $1M bill, I was running 1,000+ ec2 spot instances running Chrome browsers (phantomjs in the early days). I forget which instance type but I generally tried to scale horizontally with more smaller instance sizes for a few different reasons. Those servers, the rest of the infrastructure around rendering and saving all the HTML, and some data costs ended up being a little more than 50% the bill. Running websites through Chrome at scale is not cheap! I had something like 20 Postgres databases on RDS used for different shards containing URL metadata, like last recache date. It was so write heavy that I had to really shard the databases. For a while I had one single shard and I eventually ran into the postgres transaction ID wraparound failure. That was not fun so I definitely over provisioned RDS shards in the future to prevent that from happening again. I think RDS costs were like 10%. All of the HTML was stored in s3 and the number of GET requests wasn’t too crazy but being so write heavy on PUT requests for recaching HTML, with a decent sized chunk of data, the servers to serve customer requests, and data-our from our public endpoint, that was probably 30%. There were a few other things like SQS for populating recache queues, elasticache, etc. I never bought reserved instances and I figured the new team would go down that route but they blew me away with what they were able to do with bare metal servers. So kudos to the current Prerender team for doing such great work! Maybe that helps provide a little more context for the great comments I’m seeing here.
seattle_spring 4y ago
The company I used to work for. They successfully did cut server costs! ...at the expense of 40 eng-years (20 eng over 2 years) spent on the migration.
jmyeet 4y ago
This is unsurprising. The point of AWS is to be flexible. You’re paying for that. It’s easy to start. It’s easy to stop. It’s easy to change capacity. Running your own servers is none of these things. But it is cheaper at sufficient scale. You can’t ignore the labor cost (particularly engineering) however. Where AWS shines is with highly volatile workloads. With your own servers you have to provision for peak capacity. That’s less the case with AWS. No shade on the author of course. It’s great to read things like this.
registeredcorn 4y ago
(Note: I have never done any professional work in cloud. I could be completely mistaken. Feel free to correct me if I'm completely off-base.) It's a fascinating article, for sure. I would have been interested to hear what their backup strategy looked like though. One of the big benefits of cloud services, that I am aware of, is the assurance that if natural disaster strikes, you don't lose all of your data. I kind of got the impression that, more than anything else, that is what you are paying for. Data protection and uptime. I suppose big enough bills could lead a company to make the kinds of changes that Prerender did, but when that disaster does strike, and it is time to try and recover from a fire, flood, earthquake, etc. the responsibility and speed of getting your customers back online is reliant completely upon your staff - a staff who might be extremely shaken up, hurt, or pre-occupied in taking care of their own affairs. I'm not saying it's not possible, but there is a kind of cost that comes in the form of responsibility. It's a trade off that I would not fault many people from avoiding.
henning 4y ago
OK, so they're now stuck maintaining their own Cassandra cluster. How much does that cost? If it costs you $1,000,000 a year to serve 1166 requests a second, maybe you fucked up.
lakomen 4y ago
In other news, we got wet in the rain SCNR. Are you really saying that AWS and other clouds are expensive? Say it ain't so :)
alexchantavy 4y ago
I wish the article went into detail about what hardware they used for each server, what was their disaster mitigation plan, and other considerations that you don't need to worry about with paying for a cloud provider.
xyzabc098123 4y ago
Want to add, cloud vendors' proposition is not about the capability; it's a basic requirement and no longer an USP for something to 'work with the cloud'. It's about security. It's about outsourcing the liability so that the manager has less responsibility.

440 points by caberus 4y ago

I'm glad to see more of these types of articles, but at the same time I'm a bit flabbergasted that this isn't obvious for so many people. These cloud providers are, by definition, charging you more than it would cost you to run it yourself. What you get in return is a guarantee of expertise and an ecosystem.

HWR_14 4y ago
You also get a lot of finely incremental costs and the ability to grow. Like, can you really buy 0.1% more IT people? Well, yeah if you already have a thousand. Otherwise you have to buy in bigger chunks. So you save money because you can timeshare talent and hardware. And the ability to write a larger check to scale is valuable for consumer facing applications you're hoping by to grow.
- LanceH 4y ago
  And, importantly, you can turn off your commitment to spending in a second.
  - whatever1 4y ago
    Making IT an OPEX problem is the main reason that C-suites love cloud providers
  - HWR_14 4y ago
    I thought that was less true when you started signing deals for discounts. Commit to a spend of X and they'll reduce your bill. That kind of stuff
- zo1 4y ago
  Sure you can. Ask one of your devs to spend X amount of time on something. Might not be an exact 1 for 1 or 0.1% but it still beats the many many hours you'd waste architecting around cloud-specific issues and nuances. Or the cost of dealing with the friction that comes when you want to do anything with cloud in a manner that isn't cloud-supported.
  - HWR_14 4y ago
    Devs love being told to set up servers! And always do so well. After all, computer person is a fungible skill set. I mean, it is fungible enough that they can learn, but they'll probably make a lot of mistakes along the way and be less efficient.
    - P5fRxh5kUvp2th 4y ago
      Any developer that can't setup a server has fundamental gaps in their skillset.
      - alex_suzuki 4y ago
        Someone missed the DevOps revolution…
      - unregistereddev 4y ago
        On a trivial level I agree, but only on a trivial level. Most developers will not set up a server in the most secure, maintainable manner. They're going to miss important things: - Disable password-based auth for ssh (require key-based auth) - Enable fail2ban or similar to slow down brute-force login attempts - Configure firewall - Install monitoring tools, antivirus, possibly backup daemons, etc - Setup a sane swapfile for your use case, and configure monitoring tools to alert when memory pressure gets too high - Setup disk mounts, configure monitoring tools to alert when disk space is low, and consider a cron job to automatically clean up tempfiles - Either set up automated updates (typically excluding kernel upgrades), or have a standard schedule for manually applying updates ...and probably other things that I'm forgetting because I'm a developer, and it has been years since I've been a sysadmin.
hotpotamus 4y ago
I mean, in theory all services are sold to you with the goal of making a profit, so there must be some margin baked in, but AWS has enough margins to run the Amazon retail store as basically a non-profit while also making enough money to finance Bezos's penis rocket rides.
rossdavidh 4y ago
I think some of this is due to the presence (until just recently) of abundant VC money, encouraging tech companies to trade of money for time by outsourcing their server admin (or part of it, anyway). It's one less thing you have to hire for and spend time getting good corporate policies and procedures on. But, now that the emphasis seems to have shifted from growth-at-all-costs to where-can-we-cut-costs, I think there may be more organizations with a large enough server load to realize that they can do it cheaper (or even just get a different cloud provider to do it cheaper).
- dkarl 4y ago
  It would be shocking if, in the long term, there were not cloud providers with lower prices than any but the biggest companies could achieve by themselves. AWS will have to work hard to keep giving customers a reason to pay the premium.
  - nithril 4y ago
    Still waiting for OVH or Scaleway to be really multi AZ (>2) with a network that cannot be cut by anyone...
- P5fRxh5kUvp2th 4y ago
  imo this is the right takeaway. You see a lot of posters commenting that it isn't obvious that it's more expensive, but that's a large part of why I said I'm flabbergasted at how many people don't find it obvious. It is obvious, someone may still choose to pay it because there's a natural tradeoff, but it's absolutely obvious. I suspect most developers don't find it obvious because they've never dealt with actual servers themselves. It's akin to arguing changing the oil on your vehicle is more expensive than paying someone to do it. No, it's more convenient, and that may make it worth the price.
  - elcomet 4y ago
    Again, you seem to not take into account the developer/admin time. In your example with oil, if you track your time, it's gonna be certainly more expensive than going to the garage. And also of course, it allows you to go faster, which can have a huge value
Octoth0rpe 4y ago
> These cloud providers are, by definition, charging you more than it would cost you to run it yourself. That is _not_ a given. They're charging you more than it costs _them_ to run it. They get: - much lower hardware prices than you - lower bandwidth prices than you - likely lower electricity costs than you
- Marazan 4y ago
  Aws bandwidth costs charged to consumers are a notorious ripoff.
- AtlasBarfed 4y ago
  AWS's bandwidth charges are highway robbery. It's my impression that AWS competes with the EC2 instance cost (as it that's what new customers look at), and the bandwidth/storage only becomes apparent when you are locked in. Really what AWS should be is one or two phases of service maturity: dev infrastructure and experimentation is phase one, phase two would be the "its a couple servers" production, BUT: with a scale plan for phase 3 being not-AWS. Having a mature/battle tested phase 2 --> phase 3 should be a market advantage in the modern business landscape, but that's also a post-acquisition/exit phase, so none of the presumed target HN crowd care about it. There are extensive articles and public knowledgebases on various technologies and architectures, but there is effectively nothing on AWS independence out there, even though almost every mature organization will need to face it.
  - hotpotamus 4y ago
    I'm also a bit blown away that anyone is surprised that AWS is expensive. Sure you can express a lot of the costs in small numbers per hour or $0.08 per gigabyte of bandwidth, but those things all add up quickly at scale. I assumed phase 3 was where AWS started giving you massive discounts because you represent so much business for them, but I've never been at a place big enough to swing that stick at them.
    - tablespoon 4y ago
      > I'm also a bit blown away that anyone is surprised that AWS is expensive. Sure you can express a lot of the costs in small numbers per hour or $0.08 per gigabyte of bandwidth, but those things all add up quickly at scale. Could it be magical thinking based on marketing hype and autoscaling fantasies?
ehutch79 4y ago
When you're a small team, possibly solo, having someone else deal with hardware is key. Focusing just on what you're working on is great.
- P5fRxh5kUvp2th 4y ago
  assuming the money isn't an existential risk, that's absolutely true. I was commenting on the expense rather than why you would (or wouldn't) want to use cloud.
elcomet 4y ago
> These cloud providers are by definition, charging you more than it would cost you to run it yourself That's totally non-obvious and certainly false in most cases. Cloud providers have huge economies of scale, which certainly makes it less expensive to use cloud if you factor in all your costs related to running it yourself. This is like saying buying furniture is more expensive than making your own furniture because they have a profit margin. Even if you don't account for your own time, it will certainly be false as you'll pay your raw materials much more.
- thehappypm 4y ago
  The furniture example is great. If I wanted to build an Ikea table from scratch, I could not do it for less than the $20 they charge, given the cost of materials available to me, and the fact that I can’t buy a cup of paint for $2.
- unity1001 4y ago
  > which certainly makes it less expensive to use cloud AWS would pass that difference to Amazon shareholders than passing them to you.
goodpoint 4y ago
> These cloud providers are, by definition, charging you more than it would cost you to run it yourself That's pretty obvious but years ago you would have been downvoted into oblivion for writing that.

maxfurman 4y ago

I feel like there's a middle step missing from this article (or I just missed it reading quickly) - did they build their own data center? Where are these new non-AWS servers physically located?

hotpotamus 4y ago
Colo would also be an option. I run a mix of colo and cloud currently.
fabian2k 4y ago
You can easily rent space in a data center and put your own servers in there or just rent dedicated servers. The cheapest provider there is probably Hetzner, but then your servers are in Germany. This gets you a real, full server for something between 30-200 EUR per month. There are many other hosting providers that offer dedicated servers also in the US, e.g. OVH. This works the same way, they are usually more expensive than Hetzner. Renting a dedicated server is very similar to renting a VM. It has a much higher provisioning time, usually hours to 1-2 days. And it takes some upfront cost or a time commitment. But it also has a lot more raw power. So if you don't require the flexibility VMs and the cloud provide they can be a very effective alternative.
- unity1001 4y ago
  > The cheapest provider there is probably Hetzner, but then your servers are in Germany They have a US DC now. Though it only gives cloud servers. However even those are still way cheaper than any alternative.
- systemvoltage 4y ago
  If you want rocksolid dedicated servers of utmost HA and reliability, look no further than Equinix Metal: https://metal.equinix.com/product/servers/ https://metal.equinix.com/product/servers/ They run datacenters for other datacenter companies. Most people have never heard of them because they're a $6B/year revenue beast that runs behind the scenes; but they also offer direct metal servers.
4y ago

hnrodey 4y ago

I find this interesting, if nothing else. My first question is what was the opportunity cost of focusing manpower to setting up on-prem infrastructure that now needs maintained? What on the product roadmap was sacrificed/delayed in exchange for the time on this project? What are the projected future hiring costs to maintain these servers (and applications like Cassandra!) going forward? Nothing is free, and at just 4-5 additional hires they will be giving back a large chunk of that $800k to employees. IDK - maybe that's a fare trade-off to pump up the common man with money instead of the establishment.

brodouevencode 4y ago
This is what a lot of people miss when they talk about moving to/from cloud providers. The marginal cost to add X more servers in the cloud is basically nothing, whereas to set up a new rack for on-prem requires requirements gathering, purchase orders, finance approvals, someone being at the dock when the UPS truck arrives, rack and stacks, etc. Those are one-time, yet very real costs. These fall under cap-ex which accounting-wise is treated very differently than op-ex. Now the cost in the cloud bakes all that in, and is distributed around with other users of the provider. Your accounting models are also easier ("pay for what you use"). Couple that with the very well known fact that AWS has outrageous data egress charges and there are patterns that can emerge where you're still in cloud but not racking up massive outbound data charges.
- fabian2k 4y ago
  You can rent servers if you don't want to bother with this. The choice is not only between doing everything yourself and the cloud, there are a lot of options in between.
  - brodouevencode 4y ago
    Yes - that's a very fair point. It should still be calculated in the cost, and I don't think the article does a very good job of identifying the tradeoffs to OPs point.

taldo 4y ago

When the cost of delivering your product/service is mostly compute or traffic, sure, migrating off of AWS is a must once you reach a certain scale. But for the other 99% [0], where infrastructure is but a small cost, then think really hard if you're willing to trade the engineering effort for the convenience of managed cloud services. [0]: or 90%, or 80% or who cares, but a majority of software services seem to NOT be compute- or traffic- heavy.

hazmazlaz 4y ago

There is no way this figure is accurate. The annual spend cited of $1,000,000 is purely hypothetical, as admitted here: "However, all this data and processes need to happen on a server and, of course, we used AWS for it. A few years of growth later, we’re handling over 70,000 pages per minute, storing around 560 million pages, and paying well over $1,000,000 per year. Or at least we would be paying that much if we stayed with AWS. Instead, we were able to cut costs by 80% in a little over three months with some out-of-the-box thinking and a clear plan."

pessimizer 4y ago
> There is no way this figure is accurate. You've got the FUD covered, but you also need to add at least some substance to your claim. How do you know this figure would not be accurate? Why is your (hypothetical, not offered) estimate better than the author's?
- merb 4y ago
  well the problem with the article is basically that they left out a lot of important detail, like which bare metal servers, how many, where do they host now, did they use cloudfront what about cloudflare did they use an edge cache? what about reducing costs by killing stopping unneeded resources? a lot of their workload looks dynamic. it's also fishy what they wrote here: > After testing whether Prerender pages could be cached in both S3 and minio, we slowly diverted traffic away from AWS S3 and towards minio. if they served directly from s3 that would be, stupid? > In the last four weeks, we moved most of the cache workload from AWS S3 to our own Cassandra cluster. is also strange. it misses a lot of detail but it does not look like they just migrated away from s3... (looks like their new hoster is hetzner, from service.prerender.io )
  - pessimizer 4y ago
    > well the problem with the article is basically that they left out a lot of important detail, The problem with your comment is that you insisted that they gave you enough detail to definitively determine that they were either lying or mistaken. > There is no way this figure is accurate.

alberth 4y ago

Dedicated hosting providers. I'm so amazed that somehow people completely forget that for literally decades, web host provided dedicated hosting options at fantastic prices. Yes, loooong time ago - to get your dedicated server might have taken a few hours to provision and the instant server access that AWS brought should not be discredited. But large numbers of web host today allow you to programmatically spin up a dedicated web host instantaneously and at a fraction of the cost.

that_guy_iain 4y ago
> Yes, loooong time ago - to get your dedicated server might have taken a few hours to provision and the instant server access that AWS brought should not be discredited. At one point it took often days for a dedicated server to be set up. We also didn't have such nice provisioning tools. Now it just seems like cargo cult to use cloud providers as the only option. People just completely discount dedicated servers.
- frognumber 4y ago
  In most businesses I've worked in: developer costs >> infrastructure costs An AWS large server is around $500/year, which is about 1-2 developer hours (with taxes, overhead, etc) at the cost scales last time I priced this out. That's crazy expensive in the absolute, but if it saves a couple of hours, it makes sense. PaaS providers cost even more. I've gone with those in the past, since it basically eliminated dev-ops. The major downside wasn't cost, so much as flexibility. Dedicated servers start to make sense for: - The very low end (e.g. personal use, or hosting something long-running for a small business) - The very high end (e.g. once cloud costs start to hit hundred of thousands of dollars per year) On the very high end, cloud providers will often cut a deal, though. My problem with AWS, recently, has been reliability. Servers crash or have performance degradation a bit too often. That leads to developer costs, and might be what pushes me back to dedicated.
  - isoprophlex 4y ago
    1-2 developer hours?! Anyone looking to hire an ML guy for 250-500$/hr flat fee, hourly rate, no taxes, no healthcare, cancel any time... get in touch! For that money I'll gladly bark like a dog, walk on all fours and fetch your slippers.

TheGuyWhoCodes 4y ago

"We used Apache Cassandra nodes on each of the servers that were compatible with AWS S3". What does this even mean? Regardless, starting a new Cassandra cluster in late 2022?! I bet they can save even more by just going with scylladb

joshstrange 4y ago
That was also confusing to me as Cassandra is a NoSQL DB last I checked. I found this [0] online that indicates with some extra software you can talk to it like S3 but yeah... [0] https://dzone.com/articles/s3-compatible-storage-with-cassandra https://dzone.com/articles/s3-compatible-storage-with-cassan...

deleted 4y ago

[deleted]

epberry 4y ago

I believe this is the use case Cloudflare is really targeting with R2. They recently connected Cache Reserve to R2 to make this even easier. We wrote up a breakdown for S3 vs R2 and found that R2 would be significantly cheaper when the majority of traffic is cached data, https://www.vantage.sh/blog/cloudflare-r2-aws-s3-comparison https://www.vantage.sh/blog/cloudflare-r2-aws-s3-comparison

floatinglotus 4y ago

This is the Trillion Dollar Paradox described by Martin Casado. You’re crazy if you don’t start your business in the cloud, you’re crazy if you stay there. My new startup is focused on helping application owners repatriate their workloads into their own infrastructure. Our goal is to solve the network complexity challenges with a fully open network stack (open source software with all of the hardware options you would expect, and some you wouldn’t). The solution is designed to be turnkey and require very little network experience. It will use standard DevOps tools that you’re already using. We’re announcing it in two weeks and will be posting info here on HN!

user3939382 4y ago
I've had tons of greenfield web apps with thousands of users (not millions) and setting up a server was a problem I didn't need help solving. I'm versed in AWS now but, I don't see any benefit. The only big headache I had with server admin was running email services/deliverability.
JohnHaugeland 4y ago
> This is the Trillion Dollar Paradox described by Martin Casado. You’re crazy if you don’t start your business in the cloud, you’re crazy if you stay there. That's not a paradox. You're not crazy in either case. Starting in the cloud reduces costs for some strategies by removing necessary engineering. Starting in the cloud increases costs for other strategies by charging too much for commodity offerings. It's relatively straightforward to make the choice as soon as you put a specific business in the crosshairs. Edit: why is this downvoted?
- gadflyinyoureye 4y ago
  You’re going against the party line.
  - JohnHaugeland 4y ago
    It's getting to the point where even if I just say "that isn't what I meant" or "that's a paraphrase, not a quote," I get flagged for it

joshstrange 4y ago

They don't mention at all what services they were using (other than slight mention of S3) which makes it very hard to respond to this. If you are running everything on EC2 then you are going to have a bad time (especially if you aren't using reserved instances). AWS (IMHO) shines with the various services they provide (S3, Lambda, CloudFront, API Gateway, SQS< SES, to name a few). AWS is a game of trying to reduce your bill and often that means using AWS-specific services. If you want to stay completely "cloud agnostic" you are going to paying more than buying into a "cloud", in that scenario then you absolutely should be looking at dedicated servers. AWS is great because you can bring existing software and just run it in EC2 (or their container stuff if your software is containerized) but the AWS magic comes from using their managed services (also spinning up/down EC2 instances as needed, but if you are running them 24/7 then consider alternatives or at least pay for reserved).

taylodl 4y ago
Yes. If you use and manage AWS resources as you would your own on-prem resources then you're not going to have a good time. As soon as you think you need EC2 instances you need to re-think your architecture. You're probably not using AWS most effectively.
- shudza 4y ago
  So what should you use for let say, a backend API service? Don't tell me beanstalk/fargate/etc, because they're actually more expensive.
  - joshstrange 4y ago
    Lambda is an option, it's been a very attractive one for me.
    - jjav 4y ago
      AWS lambda is super cheap at tiny scale. But if you get into any real constant load, it is way more expensive than a VM. The other drawback of lambda is the flip side of having the server opaquely managed by AWS. It is so opaque you can't debug anything. At one startup we had a weird connectivity issue from lambda to RDS but it was impossible to diagnose given the lack of access so it went on for months. Had it been running in a VM, I could've diagnosed that within an hour with tcpdump and bpf et.al.

deleted 4y ago

[deleted]

m0llusk 4y ago

There is not quite enough information here to be sure, but this article highlights transmission costs. This particular business model involves throwing around big chunks of data just in case they end up being needed and then handing them back out in response to potentially large numbers of requests. That would make this particular usage pattern fit to exactly what AWS is charging the most for. Also many alternative AWS services that can be used to speed up or simplify services are not really going to help with this case. So an alternative way of interpreting this is more along the lines of: We may have saved up to 80% of server costs by moving from AWS, but you almost certainly won't save that much even if a bunch gets spent on developing operations and tools.

varsketiz 4y ago
You can save even more if your app uses richer formats than images. Also, if you are bigger and can start really negotiating with hardware providers.

wglass 4y ago

I found the headline to be misleading. The article is mostly about the migration process (which is interesting), but very little about the details of the cost savings. What does it cost to run their data center? What are the salaries they are paying for internal IT efforts to administer it? Is it an apples-to-apples comparison, e.g. are they load balancing across multiple datacenters in case of an outage? It sounds like this was a good move for Prerender but it's hard to generalize the cost claims to other situations without details.

0xbadcafebee 4y ago

They saved $800K on their AWS bill, but - may spend $250K on servers, replaced after 3 years becomes $83k/yr - may spend $120-250K on extra staff to maintain the infrastructure - may spend $15K for a cage in a DC They still save $452K/yr overall (actual savings 1st year only $285K). It's still a savings for sure, but always keep TCO in mind. The real fun comes later when you outgrow your cage and there's not enough space left in that DC, or they just have shitty service constantly knocking out your racks, and you have to consider splitting your infra between DCs (a huge rewrite) or moving DCs (a huge literal lift and shift). Have been part of both, it's... definitely a learning experience.

shrubble 4y ago

I expect to see a great deal more of the "cheap and cheerful" AWS migration stories in the future. With the tanking of the market and (apparent) limits to growth being in the forefront, reducing expenses will become more important. Before, it was easy to justify almost any expense with the "we just need to get 1% of this $100 billion market" and now it is "hunker down and do everything you can to be ramen-profitable, in order to survive and thrive".

Joel_Mckay 4y ago

In general, last time I looked at AWS it made sense from 2TB to 30TB a month, and under 400k connections a day. If either range was exceeded, than the service ceased to be the economical choice when compared with CDN providers, and colo/self-managed unlimited-traffic options. For example, if you primarily serve large media or many tiny files to clients that don't support http Multipart Types, than AWS can cost a lot more than the alternatives. However, AWS is generally an economical cloud provider, and a good option for those who outsourced most of their IT infrastructure. The article would be better if it cited where the variable costs arose.

maerF0x0 4y ago

I've said this a hundred times and it seems not loud enough. AWS is not cheap because of your server costs. AWS is cheap because of elasticity, velocity (opportunity cost of next feature), and reduced maintenance hours. "The cloud" was never (afaik) was about getting a cheaper VPS. It was about being able to get them on demand, give them back on demand, and generally not have to maintain anything besides your code (and maybe apply updates to datastores / AMIs) Now, if those premises are not true for your startup/business, then AWS is not the tool for you. I didnt see any analysis of ongoing maintenance costs in the 800k saved, but will it take 1-2 FTE engineers to now be more oncall, more server upgrades, more security patches etc? That's easily 1/2 that savings gone already. Edit: for the most part these attributes apply to GCP, Azure, Heroku etc as well, its not just about AWS

draw_down 4y ago
shudza 4y ago
This is one of the few accurate comments in this thread.
chucky_z 4y ago
I think there's some "cheat codes" now involving some of these things though. For instance, you can use some of AWS but keep your actual compute/networking out and have things like security patching and server inventory be completely solved by using AWS SSM. There's also options like EKS Anywhere to have managed on-prem k8s for way, way cheaper than running it in AWS proper. These kinds of services I think are the future for hybrid/on-prem folks.
candiddevmike 4y ago
Elasticity is a gamble. You're betting that you can save more money investing in an elastic/on demand stack than what you'd pay for static resources. Judging by how much the cloud providers push this, the unknown cost to create/maintain auto scaling infra/apps, and how intricate the pricing gets with elastic resources/spots/faas, I still think the cloud providers are coming out ahead vs folks using static compute.

jaclaz 4y ago

As a side note, I find this: >Do you have any advice for software engineers who are just starting out? >Don’t be afraid to talk with the customers. Throughout my career, the best software engineers were the ones who worked with the customer to solve their problems. Sometimes you can sack a half year of development time just by learning that you can solve the customer’s issue with a single line of code. I think the best engineers are creating solutions for real world problems. to be very good generic advice.

mabbo 4y ago

I'll always celebrate stories like this, but I also don't take some kind of anti-AWS lesson from it. This company saved $800k/year. Perfect time to go in-house with this solution. But when they were 1/10th this size, they'd only have saved $80k/year. Does that cover the cost of the engineering to build and maintain this system? Maybe not. And when they were 1/100th the size, it would have been laughable to go in-house. At the right time, you make the right transitions.

lbriner 4y ago
Yes, exactly this. At what point do you have the time/money/confidence to invest goodness knows how much in a data centre with space to grow, to purchase an enormous amount of capital to have it all installed etc. the building alone could eat that first years saving easily. How many people are now needed to fault-find bad hardware/software/networks, to be on call for any problems? How many calls out to the Electrician to fix some power issue? How much to setup and run a large air-con system for the data centre. Maybe not much in the US where aircon is common but much more expensive in Europe. The fact they could afford to do this over such a short time period speaks to having a decent amount of cash on-hand.
- hamandcheese 4y ago
  > At what point do you have the time/money/confidence to invest goodness knows how much in a data centre with space to grow, to purchase an enormous amount of capital to have it all installed etc. the building alone could eat that first years saving easily. Co-locating has no capital investment other than hardware, and is pretty cheap. A 40U rack of compute charged as equivalent ec2 instances has a retail price easily of hundreds of thousands, if not a million+ USD per year. Suppose each U has a $10k capital cost to make the numbers round, that is $400k in capital. All this to say is that I don’t think capital is as big a factor as you might think.
  - boltzmann-brain 4y ago
    In my experience doing detailed projections of exactly something like this - with racks full of GPU compute power - the infrastructure has paid itself after one quarter, maybe two, depending on what volume you're at. There very rarely is a reason to use GPU compute on the cloud - and advantages start as quickly as with just one single GPU.

Gregioei 4y ago

So little real information... So the team is now responsible for backups, hardware ordering,.forecast etc? How big is the team now compared to before? Does it scale? If you price it correctly and keep the free tier small, I would either talked to AWS for better pricing or moved to another cloud Provider. S3 on AWS is a total no-brainer, minio on bare metal might mean much more work and a bigger infra team than business actually wants. I would also love to know what optimizations are already in place. Does cloudflare caching work? Are the results compressed on rest? Is geolocation latency relevant? Why even Cassandra? Are websites not unique? Wouldn't a nginx and a few big servers not work? But who knows? The article doesn't tell me that :-(

gibsonf1 4y ago

We've just finished moving servers from AWS to https://hetzner.com https://hetzner.com - and saved 10X with servers of double the capability. A great experience so far.

jacooper 4y ago
How did you get them to approve a large amount of Cloud instances / dedicated servers? I heard they are very stubborn to increase the per user limit of cloud instances. Also how did you deal with S3? Did you switch to another provider ? Like B2?
- gibsonf1 4y ago
  We haven't run into the number of servers issue. For S3, we've switched to Wasabi which very nicely uses the identical API.
  - jacooper 4y ago
    Honestly Wasabi's advantage is free egress, but with b2 + cloudflare gives a better billing system without the three month object retention thing and a CDN with no egress limit.
- CoolCold 4y ago
  May be it was not large. Say for one project DB server on Hetzner with 160GB RAM, 32cores/64theads,2x3.84TB NVMe + 2x512GB SSD costs ~ 240$, hosting mid-sized ~ 1.6TB MySQL DB. On managed RDS it was around ~ $2200/month when I checked. Even if you have 2 such boxes for master/replica, it's close to 5x savings.

theptip 4y ago

Perhaps I'm missing it in the OP -- I don't see any mention of what they actually moved to. CoLo? VPS? On-prem? This seems like a key detail when telling people about your migration off AWS.

jacooper 4y ago
On-prem

deleted 4y ago

[deleted]

zc2 4y ago

Then you need to spend 2M to hire engineers helping you to maintain the in premise infra

otabdeveloper4 4y ago

No kidding? You reduced your server costs by moving away from the most expensive hoster on the planet? Good for you, I guess :)

jacooper 4y ago

Some S3 providers can give you free egress when using a CDN. For example backblaze B2 offers free egress through Cloudflare, Fastly, BunnyCDN. https://help.backblaze.com/hc/en-us/articles/217666928-Using-Backblaze-B2-with-the-Cloudflare-CDN https://help.backblaze.com/hc/en-us/articles/217666928-Using...

throwaway20221 4y ago

Throwaway here. I work for a startup which runs compute-bound scientific simulations. We are considering migrating from our single self-managed HPC, to AWS EC2. We anticipate using the largest standard instances, with highly spikey usage: frequently none, at times up to 10 concurrent instances, perhaps more later. AWS seems ideal to me because it would let us easily scale up and down as our usage varies. But some of the anti-AWS sentiment in this article has given me pause. Any reason not to do this? Any alternatives I've missed? Our storage and transfer usage will be negligible; it's all compute.

thoop 4y ago

Hi! I’m Todd, the solopreneur founder of Prerender.io and I created that $1,000,000/year AWS bill. I sold Prerender.io to Saas.group in 2020 and the new team has done an incredible job growing and changing Prerender since I left. $1M per year bill is a lot, but the Prerender back end is extremely write-heavy. It’s constantly loading URLs in Chrome in order to update the cached HTML so that the HTML is ready for sub-second serving to Google and other crawlers. Being a solo founder with a profitable product that was growing organically every month, I really didn’t have the time to personally embark on a big server migration with a bunch of unknown risks (since I had never run any bare metal servers before). So the architecture was set early on and AWS allowed me the flexibility to continue to scale while I focused on the rest of business. Just for a little more context on what was part of that $1M bill, I was running 1,000+ ec2 spot instances running Chrome browsers (phantomjs in the early days). I forget which instance type but I generally tried to scale horizontally with more smaller instance sizes for a few different reasons. Those servers, the rest of the infrastructure around rendering and saving all the HTML, and some data costs ended up being a little more than 50% the bill. Running websites through Chrome at scale is not cheap! I had something like 20 Postgres databases on RDS used for different shards containing URL metadata, like last recache date. It was so write heavy that I had to really shard the databases. For a while I had one single shard and I eventually ran into the postgres transaction ID wraparound failure. That was not fun so I definitely over provisioned RDS shards in the future to prevent that from happening again. I think RDS costs were like 10%. All of the HTML was stored in s3 and the number of GET requests wasn’t too crazy but being so write heavy on PUT requests for recaching HTML, with a decent sized chunk of data, the servers to serve customer requests, and data-our from our public endpoint, that was probably 30%. There were a few other things like SQS for populating recache queues, elasticache, etc. I never bought reserved instances and I figured the new team would go down that route but they blew me away with what they were able to do with bare metal servers. So kudos to the current Prerender team for doing such great work! Maybe that helps provide a little more context for the great comments I’m seeing here.

seattle_spring 4y ago

The company I used to work for. They successfully did cut server costs! ...at the expense of 40 eng-years (20 eng over 2 years) spent on the migration.

jmyeet 4y ago

This is unsurprising. The point of AWS is to be flexible. You’re paying for that. It’s easy to start. It’s easy to stop. It’s easy to change capacity. Running your own servers is none of these things. But it is cheaper at sufficient scale. You can’t ignore the labor cost (particularly engineering) however. Where AWS shines is with highly volatile workloads. With your own servers you have to provision for peak capacity. That’s less the case with AWS. No shade on the author of course. It’s great to read things like this.

registeredcorn 4y ago

(Note: I have never done any professional work in cloud. I could be completely mistaken. Feel free to correct me if I'm completely off-base.) It's a fascinating article, for sure. I would have been interested to hear what their backup strategy looked like though. One of the big benefits of cloud services, that I am aware of, is the assurance that if natural disaster strikes, you don't lose all of your data. I kind of got the impression that, more than anything else, that is what you are paying for. Data protection and uptime. I suppose big enough bills could lead a company to make the kinds of changes that Prerender did, but when that disaster does strike, and it is time to try and recover from a fire, flood, earthquake, etc. the responsibility and speed of getting your customers back online is reliant completely upon your staff - a staff who might be extremely shaken up, hurt, or pre-occupied in taking care of their own affairs. I'm not saying it's not possible, but there is a kind of cost that comes in the form of responsibility. It's a trade off that I would not fault many people from avoiding.

henning 4y ago

OK, so they're now stuck maintaining their own Cassandra cluster. How much does that cost? If it costs you $1,000,000 a year to serve 1166 requests a second, maybe you fucked up.

lakomen 4y ago

In other news, we got wet in the rain SCNR. Are you really saying that AWS and other clouds are expensive? Say it ain't so :)

alexchantavy 4y ago

I wish the article went into detail about what hardware they used for each server, what was their disaster mitigation plan, and other considerations that you don't need to worry about with paying for a cloud provider.

xyzabc098123 4y ago

Want to add, cloud vendors' proposition is not about the capability; it's a basic requirement and no longer an USP for something to 'work with the cloud'. It's about security. It's about outsourcing the liability so that the manager has less responsibility.