10 ms·

The AWS Spend of Our SaaS Side-Business

165 points by encoderer 9y ago

encoderer 9y ago
A few months ago my co-founder August did a great Indie Hackers interview https://www.indiehackers.com/businesses/cronitor https://www.indiehackers.com/businesses/cronitor and we heard a lot of positive feedback on sharing revenue and growth numbers. On that theme, we thought we'd write a blog post to give a little more color to our largest expense, AWS.
- aeorgnoieang 9y ago
  The post was great. The only thing I didn't like was "spend" in the title – "costs" or "expenses" would be less pretentious, to my ears and in my mind. This is a really trivial quibble! [And apparently this is a really old word in English anyways][1]. Now I feel like a grumpy old person. [1]: https://english.stackexchange.com/a/79055/42347 https://english.stackexchange.com/a/79055/42347
  - ci5er 9y ago
    Not to quibble with your quibble (but here I go anyway)... The term "Marketing Spend of X" or "Operating Spend of Y" is fairly common business-analysis-speak. At least in certain regions of the United States. IIRC, I also heard it used by Telstra and ANZ execs during some projects I had to take care of in Australia.
  - ljw1001 9y ago
    I think that's a reasonable "ask".
mi100hael 9y ago
> You pay by number of requests, not number of messages, so it reduces costs and makes it significantly faster to gulp down messages. Are there any other technical changes you've made to your application code specifically in response to AWS costs? Any other recommendations when designing a new application?
- encoderer 9y ago
  I'll think about this and reply this weekend. Check back.
- FigmentEngine 9y ago
  There is advice like this in the "Cost Optimization" whitepaper of Well-Architected aws.amazon.com/well-architected disclaimer: work for AWS and author
  - FigmentEngine 9y ago
    https://aws.amazon.com/well-architected https://aws.amazon.com/well-architected
thesandlord 9y ago
Disclaimer: I work for Google Cloud > Cloud solutions like elastic load balancing are generally engineered for the average use case. Think about the ways you are not like the average. This might be the case for some things, but many of Google Cloud's "elastic/serverless" products (Load Balancers, App Engine, Pub/Sub, BigQuery, Datastore, etc) can truly scale up from nothing to huge traffic spike without the need for pre-warming and things like that.
- encoderer 9y ago
  When I read that your load balancer is not individually provisioned the way Amazon's is I was envious. I couldn't imagine migrating to a new cloud provider now but I'm really impressed with what Google has put together!
  - shusson 9y ago
    > I couldn't imagine migrating to a new cloud provider now ugh vendor lock-in is the worst :)
    - cobookman 9y ago
      You should talk to a gcp sales rep. They can help you move to a multi Cloud deployment. As well as describe some migration options.
- nodesocket 9y ago
  Came here just to reiterate this. Google Cloud Load balancers don't need warming, manual creation of support tickets, etc. Also, networking on GCP is vastly superior to AWS. The VPC is setup with sane defaults and VM instances can communicate privately to other VM instances in any zone or region without needing complicated constructs such as VPN's or NAT instances. Lastly, Google Cloud is generally cheaper (sustained use-discounts), higher performance, and takes all the lessons learned from AWS and applies them. The one knock I have on GCP is billing. It is incredibly obtuse. Even the monthly billing invoices I still have a hard time digesting (especially if you have multiple projects under a single billing account).
douglasfshearer 9y ago
Great post. It's good to see an enterprise smaller than Segment [0] talk about how complex AWS pricing is. You mention you use SQS to queue incoming metrics. What server/framework/language are you using to do this? IME using SQS requires tools with good parallelism to deal with it's high-latency/high-throughput performance characteristics. [0] https://segment.com/blog/spotting-a-million-dollars-in-your-aws-account/ https://segment.com/blog/spotting-a-million-dollars-in-your-...
ecesena 9y ago
Surprised to not see RDS and ElastiCache. What do you use as "db"?
- encoderer 9y ago
  We run our own MySQL and Redis on an Ec2 instance. Originally it was due to cost -- you essentially are charged by instance when using these services and we could run them both from a single instance. Today, it's really due to not wanting or needing to do the migration. I will add, we've paid when we've had to do several manual db upgrades in that time: - Original m3.medium on non-provisioned iops - Upgrade m4.large - Upgrade to an EBS with provisioned iops - Upgrade to m4.xlarge These are easy enough, but not nearly as easy as an RDS upgrade is.
desdiv 9y ago
Dumb question: is there a way to show the total AWS costs you're incurring in the past second/minute/hour? The finest granularity I can find in the AWS web console is per day; many times I've butterfingered an input and only caught it a few days after the fact due to the unexpected bill increase.
- cyberferret 9y ago
  You may be able to set up a CloudWatch alarm and monitor the dashboard there? Haven't played with it for billing, but have done so for Lambda executions. I think their granularity can go down to around 5 minutes IIRC.
  - desdiv 9y ago
    Thanks for the tip. I enabled CloudWatch for billing, and I'm getting something, but it's not pretty: http://i.imgur.com/SO2dJzd.png http://i.imgur.com/SO2dJzd.png No matter what graph format I choose (between line and stacked area), all I'm getting is a single dot at 00:00 AM today for $0.35, which is the correct amount for cumulative spending this month. But it doesn't seem to be updating or graphing it correctly.
    - cyberferret 9y ago
      Yeah, I think even from my experience with their other services, that graph can be a bit misleading - even though it shows the timeline in 5 min increments, I think the actual refresh in the back end is a lot longer than that. No idea with their Billing system, but I think even Lambda doesn't get updated more than once every 15 minutes or so.
- cldellow 9y ago
  AWS supports publishing detailed billing information to an S3 bucket on an hourly frequency with excellent granularity as to which resource and operation is driving spend. Unfortunately, to make meaningful use of this you pretty much have to roll your own infra to download it, load it and analyze it. :(
cyberferret 9y ago
Nice post, and interesting to see your learning pretty much matches my own when building my SaaS business. Curious as to why you didn't opt for RDS for the database side of things? I actually have about 5 side projects as well as my main SaaS, and they all share the same RDS instance for the database which saves me cost while keeping performance high. Also, do you use Lambda much at all for periodic scripts that need to run? I've more recently been building more and more mini applets on Lambda for things like health checks or replicating data across Amazon services etc. with good results. Also, and this might be a good complimentary service to your monitoring service, but I've been using CloudWatch's timed tasks a lot recently to trigger some of those Lambda instances - very quick and easy. I've been using another service to monitor and report on missed triggers, but will look into Cronitor a bit more as a viable option.
- paulddraper 9y ago
  Failover time for RDS is high, like a minute or two. That's probably fine for lots of stuff, but not everything.
  - mrep 9y ago
    How do you get higher availability than that?
    - paulddraper 9y ago
      Use Virtual IP rather than DNS for failover. Minutes is a really long time to go down.
      - mrep 9y ago
        Have you tested that? If you actually have an in-house solution to providing better availability than amazon's RDS options, then I recommend you market your engineering solution to the major cloud providers as they will pay you very well to help them implement your solution to that problem.
dasil003 9y ago
I love AWS and for any startup that I expected to need to scale quickly I would choose it in a heartbeat. However when it comes to low-traffic side projects and experiments where costs of a few hundred dollars matter, I prefer Linode or DigitalOcean with ansible provisioning and B2 for block storage. It will cost you more time for sure, but it will give you better performance at the low-end which means you can break even sooner. If it takes off you can always migrate to AWS, GCP or Azure later.
- uji 9y ago
  GCloud recently updated their free-tier (https://cloud.google.com/free/ https://cloud.google.com/free/). They provide $300 credit which can be used for any product in GCP, and some services when used within quota are always free. So you might want to give GCP a try for your low-traffic side projects.
  - shusson 9y ago
    Just to clarify, the $300 is linked to your personal account and once it expires it's gone forever (at least for now).
    - nunez 9y ago
      can't you just create new accounts ad infintum?
      - icebraining 9y ago
        You need to add a CC or bank account.
deleted 9y ago
[deleted]
bluedino 9y ago
What are the reasons for three months being 3X the rest?
- encoderer 9y ago
  We purchased reserved instances. When I quote the 12.5% number it's amortizing the ri cost. Reserved instances have saved us a ton of money, even with my screw up last October.
zitterbewegung 9y ago
Reserved instances are never really good starting out. I think it's better used when you understand your setup and your setup has been working for years. Or if you are migrating a service that you understand the load that was on premesis.
paultopia 9y ago
How much of devops in AWS-land is really forensic accountancy? Moneymaking idea for someone: a devops person, a lawyer, and an accountant get together and charge companies to predict their cloud bill.
- samstave 9y ago
  Use cloudability to monitor cloud spend and prediction is pretty easy..
  - paultopia 9y ago
    Naturally someone has already started that business :-)
alberts00 9y ago
You should only use T2 instances within an auto-scaling group. Could someone please elaborate? I always thought this is a horrible idea because upon CPU credit exhaustion the AWS metrics show CPU utilisation including the burst. For example, if you run out of CPU credits for t2.micro it will show around 15% in AWS whilst on the instance itself you will see 100%.
- nunez 9y ago
  i think you can alarm on remaining CPU credits: https://serverfault.com/a/828193 https://serverfault.com/a/828193
- philliphaydon 9y ago
  Using t2 rds in production for 1 year for some stuff at work. We used cloudwatch to monitor the cpu credit balance. Moved from t2 small to t2 medium. Fixed some queries that used too much cpu. Still running strong. Never had down time.
danerov 9y ago
Really interesting to see the cost of other small companies.
gingerlime 9y ago
I'm a bit curious about the cost/revenue ratio. Is 12.5% considered a reasonable ratio? I just checked our spend, which includes not only hosting but other IT related spends (helpdesk, email, slack, email marketing service, etc). I think our spend is around the 3-4% mark compared to revenue. Now I'm not trying to show off or say we're doing things better. If anything, maybe this post can help me convince my cofounders that we should spend more on infrastructure hosting (there's lots of things we could improve). I'm mostly curious what's a typical ratio in other smallish startups.
- mwcampbell 9y ago
  Just curious, what kind of hosting are you using? Are you also on AWS, or something cheaper like DigitalOcean? Or bare metal?
  - gingerlime 9y ago
    We're using mostly DO, but to be honest, I'm not sure moving to AWS or Google Cloud would shift the ratio considerably. It might double our hosting costs, but hosting probably isn't our biggest spend when looking at overall IT. To give an example, I think we spend a similar amount or even more on customer.io (email marketing automation) than we do on our entire hosting. You could argue that this cost is a marketing cost rather than technology though. But currently we still attribute it to IT.
- meritt 9y ago
  3 person company with revenue in the low seven-figures. Our AWS + IT spend is about 3.5% of revenue
- augustflanagan 9y ago
  Cronitor co-founder here. This is a great question! I would love to see others post some numbers here. As Shane mentioned in the post 12.5% has been a consistent number for us as we have scaled up over the past couple years. That said, at this point I suspect we will see this percentage decrease a bit going forward. I'm basing this on intuiting that we're a bit over provisioned at the moment and won't have to scale our infrastructure linearly with user growth over the next year or two. Of course, that remains to be seen... To add some more numbers to the conversation my full time job is CTO at a consumer facing tech company (Babylist.com) I just looked at our spending on IT/infrastructure for the last couple of months and it's around 1.5-2% of revenue.
magacloud 9y ago
Nice post indeed. Few cents from our experience at ActOnCloud, Often Cloud-first and Cloud-only companies would face similar issues in future. Cloud is a one way trip. Unless one takes care of vendor lock-in and prepare the team for better governance and financial control, its going to be nightmare. With cloud providers pushing for Serverless, it will be even more darker, there is not a way to get control, Just hope that everything will be greener.

165 points by encoderer 9y ago

A few months ago my co-founder August did a great Indie Hackers interview https://www.indiehackers.com/businesses/cronitor https://www.indiehackers.com/businesses/cronitor and we heard a lot of positive feedback on sharing revenue and growth numbers. On that theme, we thought we'd write a blog post to give a little more color to our largest expense, AWS.

aeorgnoieang 9y ago
The post was great. The only thing I didn't like was "spend" in the title – "costs" or "expenses" would be less pretentious, to my ears and in my mind. This is a really trivial quibble! [And apparently this is a really old word in English anyways][1]. Now I feel like a grumpy old person. [1]: https://english.stackexchange.com/a/79055/42347 https://english.stackexchange.com/a/79055/42347
- ci5er 9y ago
  Not to quibble with your quibble (but here I go anyway)... The term "Marketing Spend of X" or "Operating Spend of Y" is fairly common business-analysis-speak. At least in certain regions of the United States. IIRC, I also heard it used by Telstra and ANZ execs during some projects I had to take care of in Australia.
- ljw1001 9y ago
  I think that's a reasonable "ask".

mi100hael 9y ago

> You pay by number of requests, not number of messages, so it reduces costs and makes it significantly faster to gulp down messages. Are there any other technical changes you've made to your application code specifically in response to AWS costs? Any other recommendations when designing a new application?

encoderer 9y ago
I'll think about this and reply this weekend. Check back.
FigmentEngine 9y ago
There is advice like this in the "Cost Optimization" whitepaper of Well-Architected aws.amazon.com/well-architected disclaimer: work for AWS and author
- FigmentEngine 9y ago
  https://aws.amazon.com/well-architected https://aws.amazon.com/well-architected

thesandlord 9y ago

Disclaimer: I work for Google Cloud > Cloud solutions like elastic load balancing are generally engineered for the average use case. Think about the ways you are not like the average. This might be the case for some things, but many of Google Cloud's "elastic/serverless" products (Load Balancers, App Engine, Pub/Sub, BigQuery, Datastore, etc) can truly scale up from nothing to huge traffic spike without the need for pre-warming and things like that.

encoderer 9y ago
When I read that your load balancer is not individually provisioned the way Amazon's is I was envious. I couldn't imagine migrating to a new cloud provider now but I'm really impressed with what Google has put together!
- shusson 9y ago
  > I couldn't imagine migrating to a new cloud provider now ugh vendor lock-in is the worst :)
  - cobookman 9y ago
    You should talk to a gcp sales rep. They can help you move to a multi Cloud deployment. As well as describe some migration options.
nodesocket 9y ago
Came here just to reiterate this. Google Cloud Load balancers don't need warming, manual creation of support tickets, etc. Also, networking on GCP is vastly superior to AWS. The VPC is setup with sane defaults and VM instances can communicate privately to other VM instances in any zone or region without needing complicated constructs such as VPN's or NAT instances. Lastly, Google Cloud is generally cheaper (sustained use-discounts), higher performance, and takes all the lessons learned from AWS and applies them. The one knock I have on GCP is billing. It is incredibly obtuse. Even the monthly billing invoices I still have a hard time digesting (especially if you have multiple projects under a single billing account).

douglasfshearer 9y ago

Great post. It's good to see an enterprise smaller than Segment [0] talk about how complex AWS pricing is. You mention you use SQS to queue incoming metrics. What server/framework/language are you using to do this? IME using SQS requires tools with good parallelism to deal with it's high-latency/high-throughput performance characteristics. [0] https://segment.com/blog/spotting-a-million-dollars-in-your-aws-account/ https://segment.com/blog/spotting-a-million-dollars-in-your-...

ecesena 9y ago

Surprised to not see RDS and ElastiCache. What do you use as "db"?

encoderer 9y ago
We run our own MySQL and Redis on an Ec2 instance. Originally it was due to cost -- you essentially are charged by instance when using these services and we could run them both from a single instance. Today, it's really due to not wanting or needing to do the migration. I will add, we've paid when we've had to do several manual db upgrades in that time: - Original m3.medium on non-provisioned iops - Upgrade m4.large - Upgrade to an EBS with provisioned iops - Upgrade to m4.xlarge These are easy enough, but not nearly as easy as an RDS upgrade is.

desdiv 9y ago

Dumb question: is there a way to show the total AWS costs you're incurring in the past second/minute/hour? The finest granularity I can find in the AWS web console is per day; many times I've butterfingered an input and only caught it a few days after the fact due to the unexpected bill increase.

cyberferret 9y ago
You may be able to set up a CloudWatch alarm and monitor the dashboard there? Haven't played with it for billing, but have done so for Lambda executions. I think their granularity can go down to around 5 minutes IIRC.
- desdiv 9y ago
  Thanks for the tip. I enabled CloudWatch for billing, and I'm getting something, but it's not pretty: http://i.imgur.com/SO2dJzd.png http://i.imgur.com/SO2dJzd.png No matter what graph format I choose (between line and stacked area), all I'm getting is a single dot at 00:00 AM today for $0.35, which is the correct amount for cumulative spending this month. But it doesn't seem to be updating or graphing it correctly.
  - cyberferret 9y ago
    Yeah, I think even from my experience with their other services, that graph can be a bit misleading - even though it shows the timeline in 5 min increments, I think the actual refresh in the back end is a lot longer than that. No idea with their Billing system, but I think even Lambda doesn't get updated more than once every 15 minutes or so.
cldellow 9y ago
AWS supports publishing detailed billing information to an S3 bucket on an hourly frequency with excellent granularity as to which resource and operation is driving spend. Unfortunately, to make meaningful use of this you pretty much have to roll your own infra to download it, load it and analyze it. :(

cyberferret 9y ago

Nice post, and interesting to see your learning pretty much matches my own when building my SaaS business. Curious as to why you didn't opt for RDS for the database side of things? I actually have about 5 side projects as well as my main SaaS, and they all share the same RDS instance for the database which saves me cost while keeping performance high. Also, do you use Lambda much at all for periodic scripts that need to run? I've more recently been building more and more mini applets on Lambda for things like health checks or replicating data across Amazon services etc. with good results. Also, and this might be a good complimentary service to your monitoring service, but I've been using CloudWatch's timed tasks a lot recently to trigger some of those Lambda instances - very quick and easy. I've been using another service to monitor and report on missed triggers, but will look into Cronitor a bit more as a viable option.

paulddraper 9y ago
Failover time for RDS is high, like a minute or two. That's probably fine for lots of stuff, but not everything.
- mrep 9y ago
  How do you get higher availability than that?
  - paulddraper 9y ago
    Use Virtual IP rather than DNS for failover. Minutes is a really long time to go down.
    - mrep 9y ago
      Have you tested that? If you actually have an in-house solution to providing better availability than amazon's RDS options, then I recommend you market your engineering solution to the major cloud providers as they will pay you very well to help them implement your solution to that problem.

dasil003 9y ago

I love AWS and for any startup that I expected to need to scale quickly I would choose it in a heartbeat. However when it comes to low-traffic side projects and experiments where costs of a few hundred dollars matter, I prefer Linode or DigitalOcean with ansible provisioning and B2 for block storage. It will cost you more time for sure, but it will give you better performance at the low-end which means you can break even sooner. If it takes off you can always migrate to AWS, GCP or Azure later.

uji 9y ago
GCloud recently updated their free-tier (https://cloud.google.com/free/ https://cloud.google.com/free/). They provide $300 credit which can be used for any product in GCP, and some services when used within quota are always free. So you might want to give GCP a try for your low-traffic side projects.
- shusson 9y ago
  Just to clarify, the $300 is linked to your personal account and once it expires it's gone forever (at least for now).
  - nunez 9y ago
    can't you just create new accounts ad infintum?
    - icebraining 9y ago
      You need to add a CC or bank account.

deleted 9y ago

[deleted]

bluedino 9y ago

What are the reasons for three months being 3X the rest?

encoderer 9y ago
We purchased reserved instances. When I quote the 12.5% number it's amortizing the ri cost. Reserved instances have saved us a ton of money, even with my screw up last October.

zitterbewegung 9y ago

Reserved instances are never really good starting out. I think it's better used when you understand your setup and your setup has been working for years. Or if you are migrating a service that you understand the load that was on premesis.

paultopia 9y ago

How much of devops in AWS-land is really forensic accountancy? Moneymaking idea for someone: a devops person, a lawyer, and an accountant get together and charge companies to predict their cloud bill.

samstave 9y ago
Use cloudability to monitor cloud spend and prediction is pretty easy..
- paultopia 9y ago
  Naturally someone has already started that business :-)

alberts00 9y ago

You should only use T2 instances within an auto-scaling group. Could someone please elaborate? I always thought this is a horrible idea because upon CPU credit exhaustion the AWS metrics show CPU utilisation including the burst. For example, if you run out of CPU credits for t2.micro it will show around 15% in AWS whilst on the instance itself you will see 100%.

nunez 9y ago
i think you can alarm on remaining CPU credits: https://serverfault.com/a/828193 https://serverfault.com/a/828193
philliphaydon 9y ago
Using t2 rds in production for 1 year for some stuff at work. We used cloudwatch to monitor the cpu credit balance. Moved from t2 small to t2 medium. Fixed some queries that used too much cpu. Still running strong. Never had down time.

danerov 9y ago

Really interesting to see the cost of other small companies.

gingerlime 9y ago

I'm a bit curious about the cost/revenue ratio. Is 12.5% considered a reasonable ratio? I just checked our spend, which includes not only hosting but other IT related spends (helpdesk, email, slack, email marketing service, etc). I think our spend is around the 3-4% mark compared to revenue. Now I'm not trying to show off or say we're doing things better. If anything, maybe this post can help me convince my cofounders that we should spend more on infrastructure hosting (there's lots of things we could improve). I'm mostly curious what's a typical ratio in other smallish startups.

mwcampbell 9y ago
Just curious, what kind of hosting are you using? Are you also on AWS, or something cheaper like DigitalOcean? Or bare metal?
- gingerlime 9y ago
  We're using mostly DO, but to be honest, I'm not sure moving to AWS or Google Cloud would shift the ratio considerably. It might double our hosting costs, but hosting probably isn't our biggest spend when looking at overall IT. To give an example, I think we spend a similar amount or even more on customer.io (email marketing automation) than we do on our entire hosting. You could argue that this cost is a marketing cost rather than technology though. But currently we still attribute it to IT.
meritt 9y ago
3 person company with revenue in the low seven-figures. Our AWS + IT spend is about 3.5% of revenue
augustflanagan 9y ago
Cronitor co-founder here. This is a great question! I would love to see others post some numbers here. As Shane mentioned in the post 12.5% has been a consistent number for us as we have scaled up over the past couple years. That said, at this point I suspect we will see this percentage decrease a bit going forward. I'm basing this on intuiting that we're a bit over provisioned at the moment and won't have to scale our infrastructure linearly with user growth over the next year or two. Of course, that remains to be seen... To add some more numbers to the conversation my full time job is CTO at a consumer facing tech company (Babylist.com) I just looked at our spending on IT/infrastructure for the last couple of months and it's around 1.5-2% of revenue.

magacloud 9y ago

Nice post indeed. Few cents from our experience at ActOnCloud, Often Cloud-first and Cloud-only companies would face similar issues in future. Cloud is a one way trip. Unless one takes care of vendor lock-in and prepare the team for better governance and financial control, its going to be nightmare. With cloud providers pushing for Serverless, it will be even more darker, there is not a way to get control, Just hope that everything will be greener.