5 ms·

We Are Saving Costs by Dumping AWS Cloud

20 points by ftreml 6y ago

Amazon is using the same trick with AWS pricing as cellphone providers: they rely on bad math capabilities of their clients. The costs for one single item, in the case of AWS this is one hour of computing, is incredibly low, parts of a cent - but most people out there are not able to do simple maths like multiplication (by 720, the number of hours in a month) or summation (with the costs of the other services you have to implicitly use). When hosting a simple one-node Kubernetes cluster the costs are easily at a three-figure number per month - the fees for EKS, the fee for the EC2 node, the fee for the inbound load balancer, the fee for storage, ... compared to a root server for $10 that easily outperforms a $60 EC2 instance that's really overpriced.

grisuBotium 6y ago
test comment
- ftreml 6y ago
  thanks for this valuable input
rvz 6y ago
It's how the whole 'cloud' operates; and it only adds up. The cloud is a complete scam in costs, especially if you're using it for very high compute tasks, deep learning, source control management or CI/CD. Another example of this scam is Firebase (aka Google Cloud). Popular with mobile devs who not only risk getting locked in, but are offered limited functionality (Firestore) that charges for reads which that optimises for a huge bill if one is not careful. For servers, as soon as you add Kubernetes, just remember to not show the 'operating costs' section of the balance sheet to the investors, unless you have a healthy recurring revenue stream.
smt88 6y ago
Not a big AWS fan or anything, but... > they rely on bad math capabilities of their clients No, they don't. Showing hourly prices makes sense for hourly services. They also provide a detailed cost estimator, because the arithmetic gets pretty detailed: https://calculator.aws https://calculator.aws > compared to a root server for $10 that easily outperforms a $60 EC2 instance that's really overpriced. Let's say a developer costs $75/hr. If AWS saves my dev team 10 hrs/mo., then I'm willing to pay a $750 premium for it. Developer costs are also unpredictable. Turnover will cause spikes in my costs, for example. Server costs are predictable. If I need to pay AWS more to save my developers time, I can always do that. I can't always just throw another developer onto my team when I need one. The only people who are optimizing for three-figure costs every month are either: 1) paying very, very low salaries to their developers, or 2) not thinking about their biggest cost, which is developer time. > When hosting a simple one-node Kubernetes cluster Who would do this and why? Why would you have a load balancer in front of a single server?
- ftreml 6y ago
  > The only people who are optimizing for three-figure costs every month are either: In our case, we are doing the same for around 50 clients, so it sums up :-) > Who would do this and why? Why would you have a load balancer in front of a single server? afaik, for publishing something to the outside world from EKS, that's the only way - even for single node clusters
  - smt88 6y ago
    > In our case, we are doing the same for around 50 clients, so it sums up :-) Wow, so AWS is actually a better value for you than for most people. Being able to easily script and deploy your infra is hugely valuable when you have so much overhead. We heavily use Elastic Beanstalk for dozens of running services, and it's amazing. We don't think about infra at all. > for publishing something to the outside world from EKS But why are you using Kubernetes at all? What problem is it solving for you? See also: https://endler.dev/2019/maybe-you-dont-need-kubernetes/ https://endler.dev/2019/maybe-you-dont-need-kubernetes/
    - ftreml 6y ago
      We built a software (Botium Box - https://botium.ai https://botium.ai) mainly for On-Premise use, and we delivered as Kubernetes, Openshift, Docker. We added a hosted plan later and thought it would be a good idea to just use managed Kubernetes for this offer as it didn't require much coding changes. We have to support multiple clouds (Azure and AWS), but with Rancher, it is really easy in usage - setting up new clusters, deploying new services, restarting, logging etc. But now that we built up container technology know-how we are transitioning every service where we don't need the scaling capabilities of Kubernetes to plain old docker-compose on baremetal. Thanks for the interesting article, didn't know about Nomad and will try it for sure.
toomuchtodo 6y ago
If you've got funding (or immediate profitability due to traction) and a hockey stick growth rate, the cloud makes sense. It's just the cost of business to support rapidly scaling and developer velocity to try to capture as much of the market as fast as possible. As long as your profit stays ahead of your cloud costs, mission accomplished. If the above does not apply, of course you're going to be better off using a combination of Cloudflare (CDN, networking), Backblaze (object store, also has S3 compatibility layer), and either dedicated service providers (OVH) or VPS providers (Digital Ocean). Perhaps even colocation if you've got the experience in house (Stackoverflow and Wikipedia, for example). AWS and other cloud providers are designed for the price insensitive, who prefer having a single vendor, more abstraction away from the metal, or require support for bursty workloads. Shop around and model your run rate based on expected workloads. There is no best or worst solution, only a scale of solutions for your use case(s), ranging from suboptimal to optimal.
- jermaustin1 6y ago
  Cloud venders also help by keeping all the network chatter internal and you not paying in/egress fees to connect to your external object store, to your external managed database, to your. The bandwidth costs can be a significant chunk of hosting. Thankfully, I've never had a popular enough service that made the bandwidth a meaningful cost.
  - toomuchtodo 6y ago
    I assume you're using Backblaze to host static assets you're rendering from your compute, and using Cloudflare to front and cache those static assets. You'd also use Backblaze for backups. Most network chatter would remain within your compute, and you'd want Redis, Mongo, Elastic, or Postgres as close to the compute (maybe k8s, maybe VMs, up to you) as possible, while shipping the snapshots to your object store. To your point about bandwidth costs, I recommend Backblaze and Cloudflare because they have an arrangement in that regard [1]. Reliably decouple whenever possible. [1] https://www.cloudflare.com/bandwidth-alliance/ https://www.cloudflare.com/bandwidth-alliance/ (Control-F "partners")
908B64B197 6y ago
Thank you for the twitter post.
thinkindie 6y ago
what i think most of people are not getting about cloud providers is that you can surely get better deal out of your money if you go to bare metal, but what you will miss out is a lot of services that come with AWS and that you would end up provision and maintain (e.g. S3, cloudfront). If you are a large business that can afford certain upfront costs to setup certain services cloud services might not be the best place, but for pretty much everyone I think you get a very simplified environment to start your business.
- ftreml 6y ago
  fully agree - that's how we started - getting some servers up and running is quick and easy, but as soon as you can foresee what computer power you will need in the next months and years, baremetal is surely the better choice. If you can deal with the technical stuff, of course.
guessmyname 6y ago
During the COVID-19 pandemic I found myself doing nothing interesting during the weekends, so I decided to offer my AWS knowledge to medium sized companies in regards to their monthly expenses in cloud compute. I thought I would help them save a couple of thousand dollars from their annual bills, but so far I already shaved more than $100k from three different companies. I am quite excited about this side gig, and I hope to be able to continue helping other companies this year too.
codingprograms 6y ago
Yeah but you can spin up everything you need in a day. Then you can shut it all down. If you’re a startup, it can make a ton of sense.
- ftreml 6y ago
  sure, that's a good point. and scaling a one-node-cluster to a multi-node-cluster is also nearly no effort on AWS. but for the topics where we can predict the computing power for the next months, we migrated everything to baremetal.
tonymet 6y ago
A more important figure to watch out for is iOPS and burst VCPU . Things are running fine, and when you get a small traffic spike, suddenly everything hangs (IO and cpu time). Then you reluctantly upgrade to solve the problem, leaving you over-resourced for most of the day. Next bump, next upgrade until you stair-step your spending into 6 figures
deleted 6y ago
[deleted]

20 points by ftreml 6y ago

grisuBotium 6y ago

test comment

ftreml 6y ago
thanks for this valuable input

rvz 6y ago

It's how the whole 'cloud' operates; and it only adds up. The cloud is a complete scam in costs, especially if you're using it for very high compute tasks, deep learning, source control management or CI/CD. Another example of this scam is Firebase (aka Google Cloud). Popular with mobile devs who not only risk getting locked in, but are offered limited functionality (Firestore) that charges for reads which that optimises for a huge bill if one is not careful. For servers, as soon as you add Kubernetes, just remember to not show the 'operating costs' section of the balance sheet to the investors, unless you have a healthy recurring revenue stream.

smt88 6y ago

Not a big AWS fan or anything, but... > they rely on bad math capabilities of their clients No, they don't. Showing hourly prices makes sense for hourly services. They also provide a detailed cost estimator, because the arithmetic gets pretty detailed: https://calculator.aws https://calculator.aws > compared to a root server for $10 that easily outperforms a $60 EC2 instance that's really overpriced. Let's say a developer costs $75/hr. If AWS saves my dev team 10 hrs/mo., then I'm willing to pay a $750 premium for it. Developer costs are also unpredictable. Turnover will cause spikes in my costs, for example. Server costs are predictable. If I need to pay AWS more to save my developers time, I can always do that. I can't always just throw another developer onto my team when I need one. The only people who are optimizing for three-figure costs every month are either: 1) paying very, very low salaries to their developers, or 2) not thinking about their biggest cost, which is developer time. > When hosting a simple one-node Kubernetes cluster Who would do this and why? Why would you have a load balancer in front of a single server?

ftreml 6y ago
> The only people who are optimizing for three-figure costs every month are either: In our case, we are doing the same for around 50 clients, so it sums up :-) > Who would do this and why? Why would you have a load balancer in front of a single server? afaik, for publishing something to the outside world from EKS, that's the only way - even for single node clusters
- smt88 6y ago
  > In our case, we are doing the same for around 50 clients, so it sums up :-) Wow, so AWS is actually a better value for you than for most people. Being able to easily script and deploy your infra is hugely valuable when you have so much overhead. We heavily use Elastic Beanstalk for dozens of running services, and it's amazing. We don't think about infra at all. > for publishing something to the outside world from EKS But why are you using Kubernetes at all? What problem is it solving for you? See also: https://endler.dev/2019/maybe-you-dont-need-kubernetes/ https://endler.dev/2019/maybe-you-dont-need-kubernetes/
  - ftreml 6y ago
    We built a software (Botium Box - https://botium.ai https://botium.ai) mainly for On-Premise use, and we delivered as Kubernetes, Openshift, Docker. We added a hosted plan later and thought it would be a good idea to just use managed Kubernetes for this offer as it didn't require much coding changes. We have to support multiple clouds (Azure and AWS), but with Rancher, it is really easy in usage - setting up new clusters, deploying new services, restarting, logging etc. But now that we built up container technology know-how we are transitioning every service where we don't need the scaling capabilities of Kubernetes to plain old docker-compose on baremetal. Thanks for the interesting article, didn't know about Nomad and will try it for sure.

toomuchtodo 6y ago

If you've got funding (or immediate profitability due to traction) and a hockey stick growth rate, the cloud makes sense. It's just the cost of business to support rapidly scaling and developer velocity to try to capture as much of the market as fast as possible. As long as your profit stays ahead of your cloud costs, mission accomplished. If the above does not apply, of course you're going to be better off using a combination of Cloudflare (CDN, networking), Backblaze (object store, also has S3 compatibility layer), and either dedicated service providers (OVH) or VPS providers (Digital Ocean). Perhaps even colocation if you've got the experience in house (Stackoverflow and Wikipedia, for example). AWS and other cloud providers are designed for the price insensitive, who prefer having a single vendor, more abstraction away from the metal, or require support for bursty workloads. Shop around and model your run rate based on expected workloads. There is no best or worst solution, only a scale of solutions for your use case(s), ranging from suboptimal to optimal.

jermaustin1 6y ago
Cloud venders also help by keeping all the network chatter internal and you not paying in/egress fees to connect to your external object store, to your external managed database, to your. The bandwidth costs can be a significant chunk of hosting. Thankfully, I've never had a popular enough service that made the bandwidth a meaningful cost.
- toomuchtodo 6y ago
  I assume you're using Backblaze to host static assets you're rendering from your compute, and using Cloudflare to front and cache those static assets. You'd also use Backblaze for backups. Most network chatter would remain within your compute, and you'd want Redis, Mongo, Elastic, or Postgres as close to the compute (maybe k8s, maybe VMs, up to you) as possible, while shipping the snapshots to your object store. To your point about bandwidth costs, I recommend Backblaze and Cloudflare because they have an arrangement in that regard [1]. Reliably decouple whenever possible. [1] https://www.cloudflare.com/bandwidth-alliance/ https://www.cloudflare.com/bandwidth-alliance/ (Control-F "partners")

908B64B197 6y ago

Thank you for the twitter post.

thinkindie 6y ago

what i think most of people are not getting about cloud providers is that you can surely get better deal out of your money if you go to bare metal, but what you will miss out is a lot of services that come with AWS and that you would end up provision and maintain (e.g. S3, cloudfront). If you are a large business that can afford certain upfront costs to setup certain services cloud services might not be the best place, but for pretty much everyone I think you get a very simplified environment to start your business.

ftreml 6y ago
fully agree - that's how we started - getting some servers up and running is quick and easy, but as soon as you can foresee what computer power you will need in the next months and years, baremetal is surely the better choice. If you can deal with the technical stuff, of course.

guessmyname 6y ago

During the COVID-19 pandemic I found myself doing nothing interesting during the weekends, so I decided to offer my AWS knowledge to medium sized companies in regards to their monthly expenses in cloud compute. I thought I would help them save a couple of thousand dollars from their annual bills, but so far I already shaved more than $100k from three different companies. I am quite excited about this side gig, and I hope to be able to continue helping other companies this year too.

codingprograms 6y ago

Yeah but you can spin up everything you need in a day. Then you can shut it all down. If you’re a startup, it can make a ton of sense.

ftreml 6y ago
sure, that's a good point. and scaling a one-node-cluster to a multi-node-cluster is also nearly no effort on AWS. but for the topics where we can predict the computing power for the next months, we migrated everything to baremetal.

tonymet 6y ago

A more important figure to watch out for is iOPS and burst VCPU . Things are running fine, and when you get a small traffic spike, suddenly everything hangs (IO and cpu time). Then you reluctantly upgrade to solve the problem, leaving you over-resourced for most of the day. Next bump, next upgrade until you stair-step your spending into 6 figures

deleted 6y ago

[deleted]