13 ms·

"Our backend system is built on AWS. Today I’m going to tell you how we had cut costs" This is a recurring topic here on HN and it boggles me and makes me wond

"Our backend system is built on AWS. Today I’m going to tell you how we had cut costs" This is a recurring topic here on HN and it boggles me and makes me wonder if people know that there are other platforms than aws, azure and google cloud out there that are very capable and much much cheaper. Unless any of the big 3 has a feature or certification you need I don't see any reason to use them at all due to the insane complexity and cost. So why do you or your company who uses any of the big 3 use them if you had to cut cost at some point?

jorams 6y ago
I can give you the reason we moved from Digital Ocean to Google Cloud: Spaces (object storage) was ridiculously slow and unreliable. There were several incidents per week where Spaces had bad connectivity or simply seemed to have crashed entirely, and that's not something we can deal with for production traffic. We now run a cheap CDN in front of it to cut down on the ridiculous bandwidth costs, but Google Cloud Storage has been very reliable and fast.
- saddlerustle 6y ago
  This. There is no alternative for high performance object storage outside of AWS S3 and Google Cloud Storage. Even Azure's offering is wonky. Lots of providers claim to offer object storage, but try hitting them from couple thousand cores and they all tend to immediately fall over.
  - speedgoose 6y ago
    That's not a very common use case though. Most companies don't need to DDOS their cloud provider.
    - jrockway 6y ago
      25Gbps is 250 cable Internet users downloading the release of your new software. What you call a DDOS is actually a woefully underwhelming instantaneous transfer rate.
      - speedgoose 6y ago
        You should put your objects storage behind a CDN if you expect that many downloads. That's much cheaper and faster.
      - Epixors 6y ago
        If you're distributing releases (or anything, really) users should not have to go directly to your object store...
  - dijit 6y ago
    AWS S3 buckets have a limit of 5gb/s access to a single bucket. Unless that limitation has changed in the last couple of years; I can easily make a system beat that, if that's the requirement.. and ultimately it does come down to understanding requirements. :\ I think generally people forget that cloud is just computers too, it's really not anything special, and amazon/google/microsoft are solving the general case (and, doing so well, actually) but this comes at a high premium.
    - duggan 6y ago
      It's 25Gbps now[1]. I have a rule of thumb: if I haven't refreshed something I think I know about AWS for more than half a year, it's probably out of date. [1]: https://aws.amazon.com/premiumsupport/knowledge-center/s3-maximum-transfer-speed-ec2/ https://aws.amazon.com/premiumsupport/knowledge-center/s3-ma...
      - saddlerustle 6y ago
        That the throughput available to a single EC2 instance. S3 as a whole will do hundreds of Gbps no problem.
  - fulafel 6y ago
    There's options outside SaaS object storage though, eg running your own MinIO or Ceph, + DAS (or SAN) at your colo. Yes it's some hassle but if you have significant expenses from SaaS it might be worth checking out.
- csunbird 6y ago
  is it that bad? we are thinking about bootstrapping a project, which will use spaces for file storage and we will be using them frequently.
  - jonatron 6y ago
    I tried it for a small personal project, just for somewhere to put some small images. On pages of about 30 images, usually one of them just wouldn't load. It didn't look reliable to me either.
  - barake 6y ago
    Provisioning of new Spaces was recently disabled in some regions while they add capacity. Vaguely concerning IMO and has caused me to seriously reconsider using DO Spaces for a new product I'm building. https://www.digitalocean.com/docs/release-notes/upcoming/spaces-fra1-nyc3/ https://www.digitalocean.com/docs/release-notes/upcoming/spa...
    - csunbird 6y ago
      I do know that they have limited the creation and looking at the docs for limitation, it seems like they have a 750 requests per second limit per space, which might not scale well. My idea was to create multiple spaces to distribute a load, e.g. sharding per customer, but if the availability is not that good, I might have to re-consider our choice. The main reason we wanted to work with DO is that the egress traffic cost is reasonable, unlike AWS or Google Cloud where it is questionably high.
  - jorams 6y ago
    It was that bad 2 years ago, in the AMS3 region. I don't know how it is these days, but I expect after 2 years it has matured at least a little bit. They also now include a CDN, which might improve things a bit if it reduces the load on the buckets behind it. Honestly it soured my opinion on DO a bit. I'd previously had a droplet there for several years, which never seemed to experience any issues at all.
    - 6y ago
- jrockway 6y ago
  I run my own Docker registry for my personal projects backed by DO Spaces. It was so slow that I couldn't actually believe it, and had to instrument the registry code extensively to account for all the time spent in the storage API, to see what the actual problem was. Turns out Spaces is really, really slow.
PetahNZ 6y ago
Tell me who/what I should migrate my Elastic Beanstalk/Aurora Serverless/S3 stack to to save significant costs?
- dijit 6y ago
  The solution to saving costs is to go for cheaper cloud providers or run k8s on VPS's/colocated physical hardware. I'm not sure where the idea came from that this is hard to do. To answer your questions in order: 1) People warned aggressively about lock-in of cloud providers with proprietary extensions and you must have chosen not to listen; so, I have little sympathy. I'm not saying it's black and white; but I hope that you got your velocity required to hit market faster and have made more than you spent because this is the price you paid; and now to get out you'll have to invest a little time on cleaning house, that's the reality of lock-in. 2) S3 is pretty easy as there are "s3 compatible" FOSS projects; min.io comes to mind, or ceph with a RADOS plugin, or Riak with the s3 plugin... there's also s3proxy with a multitude of backends.. 3) Aurora serverless is replaced by knative 4) Beanstalk is just classical servers with an auto-scaler component, auto-scaling depends greatly on your provider, so I can't say how easy or hard it will be, if you're using kubernetes then understanding your load should be easy at least.
  - marcinzm 6y ago
    Now you have five more things you need to research, learn, keep track of, debug, etc. Plus, in my experience, anyone who says kubernetes is easy to run (cloud or native) has never had to run kubernetes in production.
    - dijit 6y ago
      Sure, then pay? I don't understand the argument: "I want to outsource understanding but I want to save costs"; You can think of things as a spider-graph of three points: Quality -- Low-Cost -- Knowledge-Required. No solution can score high points on all three reasonably. Anyway, things are a bit skewed because I'm an infrastructure type, and people in my profession really do think of systems administration tasks as being "very easy" and if done right soak up nearly no time at all, but developers don't like hearing that because sysadmins are "old world". I don't really care if you're paying someone elses sysadmins or not, the fact remains that you're going to be spending something in that area, and if you balk at the cost of cloud then maybe taking ownership of what they do can help optimise costs. Obviously they put a premium on their own time in these areas.
kristianpaul 6y ago
What are those other platforms (for people awareness)? Also probably as you said due complexity you can infer and expend more not knowing what you’re doing but that complexity also give you many options to design your systems architecture as well and its you duty to grow it to what is more convenient. You cant have all and those kind of compromises are everywhere.
CSDude 6y ago
Always this poor argument pops up. 1st AWS and big 3 is much more reliable. 2nd AWS is not only a VPS provider. Changing vendors is not the answer to all cost related problems.
- gridlockd 6y ago
  > 1st AWS and big 3 is much more reliable. I think that's a myth. People assume it's true because it should be true. I don't think it is true. > 2nd AWS is not only a VPS provider. It's a glorified VPS provider. Most of the stuff doesn't matter to most of the people using AWS, but they go for it so they can put it on their resume and because they don't want to get fired for choosing something that's not a big name.
  - jungturk 6y ago
    EC2 might be a glorified VPS provider, but you seem to be ignoring the vast array of managed services in modern cloud providers (or unaware of their utility beyond padding resumes). Load balancing, fault tolerance, high availability, arbitrary scale, messaging, orchestration, autoscaling, warehousing, big data processing, identity management, desktop management, secrets management, container registries, source code management, build tools, hardware test suites, gpu hardware, observability tools... Those of use that use cloud providers know full well why we use them (and certainly know when not to).
    - deleted 6y ago
      [deleted]
    - gridlockd 6y ago
      If you really need any of that stuff you probably shouldn't use Amazon's managed version of it.
      - rospaya 6y ago
        Why? I run both on prem and cloud workloads (on more than one provider) so I'm wondering what's wrong with Amazon's managed services?
secondcoming 6y ago
It's probably because when devs jump into the cloud they can be distracted by Shiny New Toys and cost is only considered once the first bill comes in.
marcinzm 6y ago
>So why do you or your company who uses any of the big 3 use them if you had to cut cost at some point? Because it takes X hours of effort to cut costs and using another provider would have required Y hours of effort where Y >> X. Y may be greater due to reliability issues, missing features that you need to build yourself, training costs of new employees, etc. edit: Also X is paid once you're succeeding, Y has to be paid before you're succeeding which makes Y even more expensive in opportunity cost.
- yellow_mixer 6y ago
  It’s also much easier to hire engineers to develop and maintain theses platforms.
l33tman 6y ago
I'm sure there are other services that are cheaper in some of the areas, but at least the argument about "insane complexity" is moot; if you've used AWS before it's no problem to set something up from scratch. I've done it numerous times and it's a one day job to setup most of the infrastructure required with VPCs, IAM users, policies, databases, scaling groups, storage configs etc.. That part of it is not really an argument to switch at least.
pm90 6y ago
I don't think moving cloud providers is an effective way to solve for cost problems; especially considering the engineering cost involved in making these migrations, which reduce the marginal utility of such a migration to 0, possibly even <0. If you're really looking to save on costs, hosting solutions (i.e. hosting racks or some managed solution) are probably what you would need to look at. But then there are other costs involved there as well (infrastructure team, upfront capex costs etc.), but it might still be worth it if you run e.g. a ton of batch processing over a ton of data.

jorams 6y ago

I can give you the reason we moved from Digital Ocean to Google Cloud: Spaces (object storage) was ridiculously slow and unreliable. There were several incidents per week where Spaces had bad connectivity or simply seemed to have crashed entirely, and that's not something we can deal with for production traffic. We now run a cheap CDN in front of it to cut down on the ridiculous bandwidth costs, but Google Cloud Storage has been very reliable and fast.

saddlerustle 6y ago
This. There is no alternative for high performance object storage outside of AWS S3 and Google Cloud Storage. Even Azure's offering is wonky. Lots of providers claim to offer object storage, but try hitting them from couple thousand cores and they all tend to immediately fall over.
- speedgoose 6y ago
  That's not a very common use case though. Most companies don't need to DDOS their cloud provider.
  - jrockway 6y ago
    25Gbps is 250 cable Internet users downloading the release of your new software. What you call a DDOS is actually a woefully underwhelming instantaneous transfer rate.
    - speedgoose 6y ago
      You should put your objects storage behind a CDN if you expect that many downloads. That's much cheaper and faster.
    - Epixors 6y ago
      If you're distributing releases (or anything, really) users should not have to go directly to your object store...
- dijit 6y ago
  AWS S3 buckets have a limit of 5gb/s access to a single bucket. Unless that limitation has changed in the last couple of years; I can easily make a system beat that, if that's the requirement.. and ultimately it does come down to understanding requirements. :\ I think generally people forget that cloud is just computers too, it's really not anything special, and amazon/google/microsoft are solving the general case (and, doing so well, actually) but this comes at a high premium.
  - duggan 6y ago
    It's 25Gbps now[1]. I have a rule of thumb: if I haven't refreshed something I think I know about AWS for more than half a year, it's probably out of date. [1]: https://aws.amazon.com/premiumsupport/knowledge-center/s3-maximum-transfer-speed-ec2/ https://aws.amazon.com/premiumsupport/knowledge-center/s3-ma...
    - saddlerustle 6y ago
      That the throughput available to a single EC2 instance. S3 as a whole will do hundreds of Gbps no problem.
- fulafel 6y ago
  There's options outside SaaS object storage though, eg running your own MinIO or Ceph, + DAS (or SAN) at your colo. Yes it's some hassle but if you have significant expenses from SaaS it might be worth checking out.
csunbird 6y ago
is it that bad? we are thinking about bootstrapping a project, which will use spaces for file storage and we will be using them frequently.
- jonatron 6y ago
  I tried it for a small personal project, just for somewhere to put some small images. On pages of about 30 images, usually one of them just wouldn't load. It didn't look reliable to me either.
- barake 6y ago
  Provisioning of new Spaces was recently disabled in some regions while they add capacity. Vaguely concerning IMO and has caused me to seriously reconsider using DO Spaces for a new product I'm building. https://www.digitalocean.com/docs/release-notes/upcoming/spaces-fra1-nyc3/ https://www.digitalocean.com/docs/release-notes/upcoming/spa...
  - csunbird 6y ago
    I do know that they have limited the creation and looking at the docs for limitation, it seems like they have a 750 requests per second limit per space, which might not scale well. My idea was to create multiple spaces to distribute a load, e.g. sharding per customer, but if the availability is not that good, I might have to re-consider our choice. The main reason we wanted to work with DO is that the egress traffic cost is reasonable, unlike AWS or Google Cloud where it is questionably high.
- jorams 6y ago
  It was that bad 2 years ago, in the AMS3 region. I don't know how it is these days, but I expect after 2 years it has matured at least a little bit. They also now include a CDN, which might improve things a bit if it reduces the load on the buckets behind it. Honestly it soured my opinion on DO a bit. I'd previously had a droplet there for several years, which never seemed to experience any issues at all.
  - 6y ago
jrockway 6y ago
I run my own Docker registry for my personal projects backed by DO Spaces. It was so slow that I couldn't actually believe it, and had to instrument the registry code extensively to account for all the time spent in the storage API, to see what the actual problem was. Turns out Spaces is really, really slow.

PetahNZ 6y ago

Tell me who/what I should migrate my Elastic Beanstalk/Aurora Serverless/S3 stack to to save significant costs?

dijit 6y ago
The solution to saving costs is to go for cheaper cloud providers or run k8s on VPS's/colocated physical hardware. I'm not sure where the idea came from that this is hard to do. To answer your questions in order: 1) People warned aggressively about lock-in of cloud providers with proprietary extensions and you must have chosen not to listen; so, I have little sympathy. I'm not saying it's black and white; but I hope that you got your velocity required to hit market faster and have made more than you spent because this is the price you paid; and now to get out you'll have to invest a little time on cleaning house, that's the reality of lock-in. 2) S3 is pretty easy as there are "s3 compatible" FOSS projects; min.io comes to mind, or ceph with a RADOS plugin, or Riak with the s3 plugin... there's also s3proxy with a multitude of backends.. 3) Aurora serverless is replaced by knative 4) Beanstalk is just classical servers with an auto-scaler component, auto-scaling depends greatly on your provider, so I can't say how easy or hard it will be, if you're using kubernetes then understanding your load should be easy at least.
- marcinzm 6y ago
  Now you have five more things you need to research, learn, keep track of, debug, etc. Plus, in my experience, anyone who says kubernetes is easy to run (cloud or native) has never had to run kubernetes in production.
  - dijit 6y ago
    Sure, then pay? I don't understand the argument: "I want to outsource understanding but I want to save costs"; You can think of things as a spider-graph of three points: Quality -- Low-Cost -- Knowledge-Required. No solution can score high points on all three reasonably. Anyway, things are a bit skewed because I'm an infrastructure type, and people in my profession really do think of systems administration tasks as being "very easy" and if done right soak up nearly no time at all, but developers don't like hearing that because sysadmins are "old world". I don't really care if you're paying someone elses sysadmins or not, the fact remains that you're going to be spending something in that area, and if you balk at the cost of cloud then maybe taking ownership of what they do can help optimise costs. Obviously they put a premium on their own time in these areas.

kristianpaul 6y ago

What are those other platforms (for people awareness)? Also probably as you said due complexity you can infer and expend more not knowing what you’re doing but that complexity also give you many options to design your systems architecture as well and its you duty to grow it to what is more convenient. You cant have all and those kind of compromises are everywhere.

CSDude 6y ago

Always this poor argument pops up. 1st AWS and big 3 is much more reliable. 2nd AWS is not only a VPS provider. Changing vendors is not the answer to all cost related problems.

gridlockd 6y ago
> 1st AWS and big 3 is much more reliable. I think that's a myth. People assume it's true because it should be true. I don't think it is true. > 2nd AWS is not only a VPS provider. It's a glorified VPS provider. Most of the stuff doesn't matter to most of the people using AWS, but they go for it so they can put it on their resume and because they don't want to get fired for choosing something that's not a big name.
- jungturk 6y ago
  EC2 might be a glorified VPS provider, but you seem to be ignoring the vast array of managed services in modern cloud providers (or unaware of their utility beyond padding resumes). Load balancing, fault tolerance, high availability, arbitrary scale, messaging, orchestration, autoscaling, warehousing, big data processing, identity management, desktop management, secrets management, container registries, source code management, build tools, hardware test suites, gpu hardware, observability tools... Those of use that use cloud providers know full well why we use them (and certainly know when not to).
  - deleted 6y ago
    [deleted]
  - gridlockd 6y ago
    If you really need any of that stuff you probably shouldn't use Amazon's managed version of it.
    - rospaya 6y ago
      Why? I run both on prem and cloud workloads (on more than one provider) so I'm wondering what's wrong with Amazon's managed services?

secondcoming 6y ago

It's probably because when devs jump into the cloud they can be distracted by Shiny New Toys and cost is only considered once the first bill comes in.

marcinzm 6y ago

>So why do you or your company who uses any of the big 3 use them if you had to cut cost at some point? Because it takes X hours of effort to cut costs and using another provider would have required Y hours of effort where Y >> X. Y may be greater due to reliability issues, missing features that you need to build yourself, training costs of new employees, etc. edit: Also X is paid once you're succeeding, Y has to be paid before you're succeeding which makes Y even more expensive in opportunity cost.

yellow_mixer 6y ago
It’s also much easier to hire engineers to develop and maintain theses platforms.

l33tman 6y ago

I'm sure there are other services that are cheaper in some of the areas, but at least the argument about "insane complexity" is moot; if you've used AWS before it's no problem to set something up from scratch. I've done it numerous times and it's a one day job to setup most of the infrastructure required with VPCs, IAM users, policies, databases, scaling groups, storage configs etc.. That part of it is not really an argument to switch at least.

pm90 6y ago

I don't think moving cloud providers is an effective way to solve for cost problems; especially considering the engineering cost involved in making these migrations, which reduce the marginal utility of such a migration to 0, possibly even <0. If you're really looking to save on costs, hosting solutions (i.e. hosting racks or some managed solution) are probably what you would need to look at. But then there are other costs involved there as well (infrastructure team, upfront capex costs etc.), but it might still be worth it if you run e.g. a ton of batch processing over a ton of data.