8 ms·

How are they managing to spend 7.5k a month in server expenses? Are they using AWS?

by Laaas 2y ago

re-thc 2y ago
> Are they using AWS? Yes!
- Laaas 2y ago
  You’d think they’d use a cheaper hosting service if that’s literally the only thing preventing them from having positive gross profit.
  - fendy3002 2y ago
    honest questions, is there any ECS-like hosting that's much cheaper?
    - re-thc 2y ago
      > is there any ECS-like hosting that's much cheaper? Most have adopted EKS-like services i.e. kubernetes. There is fly.io that's closer. Hope they improve on the reliability aspect.
    - jfdjkfdhjds 2y ago
      for compute of containerized payloads, in house servers is a no brainer for cost. almos zero sysadmin troubles. might even reduce the troubles of working with eks/ecs. now for storage and db, that's a different story.
    - evantbyrne 2y ago
      Docker Swarm. After building a whole CD PaaS for ECS, I came to believe that swarm is a much more reasonable place for teams to start, so I built a free tool for deploying single-machine swarms called Rove. Do your own research though, and remember you don't have to use one tool or platform for everything.
  - happyopossum 2y ago
    > if that’s literally the only thing preventing them from having positive gross profit It’s clearly not if the founder isn’t taking a salary and they just had to lay off their only employee..
- andrewstuart 2y ago
  $50/month gets you a 12 core 24GB VPS with unlimited traffic on a 1Gbps link on Ionos. https://www.ionos.com/servers/vps https://www.ionos.com/servers/vps $7500/month would get them 150 such servers. Maybe they should cut the AWS costs and hire their developer back. Id be really interested to hear the breakdown of their AWS bill. It would be a crime if they were blowing what money they have giving Amazon 9 cents per gigabyte egress.
  - dspillett 2y ago
    If they are self-managing all the extras you get with a decent cloud setup (backups, node failover, load distribution and auto-scaling, multi-region or at least multi-DC for availability beyond single node failures, …), they are going to need an infrastructure person as well as that developer. Preferably two so the one isn't effectively on-call 24/7. And for that multi-DC for availability thing: you might need someone (or assign time from existing people) to manage the accounts with your various providers, you won't want tens+ of VPSs from just one provider like that. Of and on backups & failover, you need person-time (and other resources, but the people are probably the expensive part from the business PoV) to regularly test and adjust all of that, so you can be reasonably sure it all works when actually needed. And you need to manage replacing those people when/if they decide to move on to something new, etc… Also note that a lot of the things you are paying for (CPU cores, traffic, network throughput) in those nodes are shared resources (that Gbit link especially) and/or have “fair use” policies attached to them, and while the same might be true of cloud providers those policies are often either more generous or (perhaps more important from the business stability PoV) at least better defined. “Cloud” is still expensive compared to buying and managing individual nodes, even if you add in all the above and the things I no doubt forgot to mention, but it does give a lot more than the same cost in individual nodes than this sort of comparison suggests. And sometimes just not having to deal with all that, keeping the business more focused on its core competencies, is worth the extra expense. In DayJob we use Azure a lot, and sometimes I see the costs of certain things¹² and balk, and we do still have infrastructure people to manage the platform, but overall it works better for us than managing our own resources more directly. We have an extra complication due to our client base (regulated companies like banks and insurers, who are storing PII of both their own people and their customers with us) in that we have to give a lot of assurances on security and such which would be more work (it is already a _lot_ of work as anyone else in that sort of B2B arena can attest) if we self-managed everything. ---- [1] $2,400/yr for SFTP access to a storage account if you need it available 24/7?! Especially given we have at least one such account per client as their requirements understandably require that level of separation. I think we'll keep using the relay & management dashboard I setup in a few cheap VMs, thanks… [2] and the performance given the costs: AzureSQL³ I'm looking at you! [3] though again, some of that cost is in things like the scaling flexibility and other infrastructure convenience, which the business finds worth paying for
j-a-a-p 2y ago
That is easy to turn around, if the company would do smarter and cheaper hosting we would ask why they didn't spend more time into finding a viable business model.
michaelt 2y ago
It sounds like a lot, but even if they dropped server expenses to zero they'd only have $90k/year to spend. That is not much, considering they need to cover all aspects of running a nonprofit; they need enough technical staff to maintain a 24/7 on call rota while letting people occasionally go on holiday; and they need to pay healthcare and pensions and suchlike on top of salaries. And it's not like they can offer a meagre salary now but promise vast wealth in the future, like a startup might.
swiftcoder 2y ago
That's only around $10/hour, which doesn't actually buy one a whole lot of servers/databases/bandwidth/monitoring/logs/etc.
- andrewstuart 2y ago
  >> which doesn't actually buy one a whole lot of servers/databases/bandwidth/monitoring/logs/etc. It buys you 150 machines for a month 12 core 24GB VPS with unlimited traffic on a 1Gbps link See my comment below.
  - swiftcoder 2y ago
    And then you get to stand up your own databases, load balancers, monitoring, logging, etc. For which you need a development team with significant operations experience to do correctly - who will surely cost you more than $90k/year I get it, AWS looks expensive, but a bunch of their foundational services are real force-multipliers if you don't have the cash to build out entire operational teams.
    - andrewstuart 2y ago
      >> if you don't have the cash to build out entire operational teams Its just not true that AWS doesn't need expensive experts to get stuff done - it really does. Anyone who is half decent on the command line in Linux can get all those servers installed and running without the spaghetti complexity of AWS. The cloud as a magical place of simplicity and ease of use and infinite scalability in every direction - I think its the opposite of that - AWS is a nightmarish tangle of complexity and hard to configure, understand, relate and maintain systems. Its MUCH easier just to load up a single powerful machine with everything you need. I'm not saying that works for all workloads but a single machine or a few machines can take you an awful long way.
      - swiftcoder 2y ago
        > Its MUCH easier just to load up a single powerful machine with everything you need. I'm not saying that works for all workloads but a single machine or a few machines can take you an awful long way. For the core service I tend to favour monoliths too, but I would say you are vastly underestimating the halo of other crap needed to operationalise a real website/SaaS. Where is your load balancer? Your database redundancy? Where are backups stored? Where are you streaming your logs for long-term retention? Where are you handling metrics/alarming? Bare metal is great, but you have to build a ton of shit to actually ship product.
ihid 2y ago
We're paying ~$1k/month for all our webservers (which is a dozen ECS instances). They're handling about 3,000 requests per second (but that does sometimes massively spike to tens of thousands if not more). We're paying ~$1k/month for all our tooling servers (so the 150 different test runners, representers, analyzers that are used to check peoples code. There's >=1 of those running every second). Bare in mind, we're running student's code in over 70 languages here. Each is a docker container (often many gb large) - so we pay for HDD too. The biggest actual cost is the database at $2k/month. We have about 600 queries per second, and around 10MB per second (spiking to 47MB per second) of read throughput. It's an autoscaling database, but AWS determines that it's at the level it needs to be, and if I turn that down, performance suffers (I've tried). Beyond that, all the other individual services are ~$300/m, so quite small amounts, but for things we rely on (e.g. caching servers, a shared filesystem amongst all those servers, and other things). $1.2k on tax is also fun.
- ahofmann 2y ago
  Thank you so much for sharing your costs! I was very curious to see what the real expenses for AWS services look like, as I've always assumed AWS is massively overpriced. From the numbers you've provided, it seems like your total cost is around $5.5k, so I assume the remaining $2k is attributed to traffic. Everything looks quite reasonable, except for the database and traffic costs. I've run MySQL servers handling 150k reads/sec and 50k updates/sec with no issues, even on very cheap machines (around €30 per month). Years ago, we were serving over 100 million pages (of heavy content) per month, and we didn’t even bother looking at traffic statistics because, here in Germany, it’s hard to hit the traffic limits that most hosting providers impose. That being said, AWS is less expensive than I initially thought. At the same time, I’m confident you could reduce your hosting bill by up to $2k without even leaving AWS by setting up your own database server. Moving away from AWS entirely might be challenging, as managing a fleet of about 30 servers would likely take one or two days of work per week (I'm managing a dozen mostly idling servers and I work one day per month on them). When your hosting bill reaches $30k, I'm very sure it would be cheaper to hire someone (hint, hint ;-)), that moves everything to dedicated servers and manages them.
- RobRivera 2y ago
evantbyrne 2y ago
Cloud providers have gotten really good at selling scale anxiety. Very easy to reach $10k/mo when building on a modern web stack using managed services. Kafka alone could get you there once you factor in multiple environments and add-ons.

How are they managing to spend 7.5k a month in server expenses? Are they using AWS?

by Laaas 2y ago

How are they managing to spend 7.5k a month in server expenses? Are they using AWS?

re-thc 2y ago

> Are they using AWS? Yes!

Laaas 2y ago
You’d think they’d use a cheaper hosting service if that’s literally the only thing preventing them from having positive gross profit.
- fendy3002 2y ago
  honest questions, is there any ECS-like hosting that's much cheaper?
  - re-thc 2y ago
    > is there any ECS-like hosting that's much cheaper? Most have adopted EKS-like services i.e. kubernetes. There is fly.io that's closer. Hope they improve on the reliability aspect.
  - jfdjkfdhjds 2y ago
    for compute of containerized payloads, in house servers is a no brainer for cost. almos zero sysadmin troubles. might even reduce the troubles of working with eks/ecs. now for storage and db, that's a different story.
  - evantbyrne 2y ago
    Docker Swarm. After building a whole CD PaaS for ECS, I came to believe that swarm is a much more reasonable place for teams to start, so I built a free tool for deploying single-machine swarms called Rove. Do your own research though, and remember you don't have to use one tool or platform for everything.
- happyopossum 2y ago
  > if that’s literally the only thing preventing them from having positive gross profit It’s clearly not if the founder isn’t taking a salary and they just had to lay off their only employee..
andrewstuart 2y ago
$50/month gets you a 12 core 24GB VPS with unlimited traffic on a 1Gbps link on Ionos. https://www.ionos.com/servers/vps https://www.ionos.com/servers/vps $7500/month would get them 150 such servers. Maybe they should cut the AWS costs and hire their developer back. Id be really interested to hear the breakdown of their AWS bill. It would be a crime if they were blowing what money they have giving Amazon 9 cents per gigabyte egress.
- dspillett 2y ago
  If they are self-managing all the extras you get with a decent cloud setup (backups, node failover, load distribution and auto-scaling, multi-region or at least multi-DC for availability beyond single node failures, …), they are going to need an infrastructure person as well as that developer. Preferably two so the one isn't effectively on-call 24/7. And for that multi-DC for availability thing: you might need someone (or assign time from existing people) to manage the accounts with your various providers, you won't want tens+ of VPSs from just one provider like that. Of and on backups & failover, you need person-time (and other resources, but the people are probably the expensive part from the business PoV) to regularly test and adjust all of that, so you can be reasonably sure it all works when actually needed. And you need to manage replacing those people when/if they decide to move on to something new, etc… Also note that a lot of the things you are paying for (CPU cores, traffic, network throughput) in those nodes are shared resources (that Gbit link especially) and/or have “fair use” policies attached to them, and while the same might be true of cloud providers those policies are often either more generous or (perhaps more important from the business stability PoV) at least better defined. “Cloud” is still expensive compared to buying and managing individual nodes, even if you add in all the above and the things I no doubt forgot to mention, but it does give a lot more than the same cost in individual nodes than this sort of comparison suggests. And sometimes just not having to deal with all that, keeping the business more focused on its core competencies, is worth the extra expense. In DayJob we use Azure a lot, and sometimes I see the costs of certain things¹² and balk, and we do still have infrastructure people to manage the platform, but overall it works better for us than managing our own resources more directly. We have an extra complication due to our client base (regulated companies like banks and insurers, who are storing PII of both their own people and their customers with us) in that we have to give a lot of assurances on security and such which would be more work (it is already a _lot_ of work as anyone else in that sort of B2B arena can attest) if we self-managed everything. ---- [1] $2,400/yr for SFTP access to a storage account if you need it available 24/7?! Especially given we have at least one such account per client as their requirements understandably require that level of separation. I think we'll keep using the relay & management dashboard I setup in a few cheap VMs, thanks… [2] and the performance given the costs: AzureSQL³ I'm looking at you! [3] though again, some of that cost is in things like the scaling flexibility and other infrastructure convenience, which the business finds worth paying for

j-a-a-p 2y ago

That is easy to turn around, if the company would do smarter and cheaper hosting we would ask why they didn't spend more time into finding a viable business model.

michaelt 2y ago

It sounds like a lot, but even if they dropped server expenses to zero they'd only have $90k/year to spend. That is not much, considering they need to cover all aspects of running a nonprofit; they need enough technical staff to maintain a 24/7 on call rota while letting people occasionally go on holiday; and they need to pay healthcare and pensions and suchlike on top of salaries. And it's not like they can offer a meagre salary now but promise vast wealth in the future, like a startup might.

swiftcoder 2y ago

That's only around $10/hour, which doesn't actually buy one a whole lot of servers/databases/bandwidth/monitoring/logs/etc.

andrewstuart 2y ago
>> which doesn't actually buy one a whole lot of servers/databases/bandwidth/monitoring/logs/etc. It buys you 150 machines for a month 12 core 24GB VPS with unlimited traffic on a 1Gbps link See my comment below.
- swiftcoder 2y ago
  And then you get to stand up your own databases, load balancers, monitoring, logging, etc. For which you need a development team with significant operations experience to do correctly - who will surely cost you more than $90k/year I get it, AWS looks expensive, but a bunch of their foundational services are real force-multipliers if you don't have the cash to build out entire operational teams.
  - andrewstuart 2y ago
    >> if you don't have the cash to build out entire operational teams Its just not true that AWS doesn't need expensive experts to get stuff done - it really does. Anyone who is half decent on the command line in Linux can get all those servers installed and running without the spaghetti complexity of AWS. The cloud as a magical place of simplicity and ease of use and infinite scalability in every direction - I think its the opposite of that - AWS is a nightmarish tangle of complexity and hard to configure, understand, relate and maintain systems. Its MUCH easier just to load up a single powerful machine with everything you need. I'm not saying that works for all workloads but a single machine or a few machines can take you an awful long way.
    - swiftcoder 2y ago
      > Its MUCH easier just to load up a single powerful machine with everything you need. I'm not saying that works for all workloads but a single machine or a few machines can take you an awful long way. For the core service I tend to favour monoliths too, but I would say you are vastly underestimating the halo of other crap needed to operationalise a real website/SaaS. Where is your load balancer? Your database redundancy? Where are backups stored? Where are you streaming your logs for long-term retention? Where are you handling metrics/alarming? Bare metal is great, but you have to build a ton of shit to actually ship product.

ihid 2y ago

We're paying ~$1k/month for all our webservers (which is a dozen ECS instances). They're handling about 3,000 requests per second (but that does sometimes massively spike to tens of thousands if not more). We're paying ~$1k/month for all our tooling servers (so the 150 different test runners, representers, analyzers that are used to check peoples code. There's >=1 of those running every second). Bare in mind, we're running student's code in over 70 languages here. Each is a docker container (often many gb large) - so we pay for HDD too. The biggest actual cost is the database at $2k/month. We have about 600 queries per second, and around 10MB per second (spiking to 47MB per second) of read throughput. It's an autoscaling database, but AWS determines that it's at the level it needs to be, and if I turn that down, performance suffers (I've tried). Beyond that, all the other individual services are ~$300/m, so quite small amounts, but for things we rely on (e.g. caching servers, a shared filesystem amongst all those servers, and other things). $1.2k on tax is also fun.

ahofmann 2y ago
Thank you so much for sharing your costs! I was very curious to see what the real expenses for AWS services look like, as I've always assumed AWS is massively overpriced. From the numbers you've provided, it seems like your total cost is around $5.5k, so I assume the remaining $2k is attributed to traffic. Everything looks quite reasonable, except for the database and traffic costs. I've run MySQL servers handling 150k reads/sec and 50k updates/sec with no issues, even on very cheap machines (around €30 per month). Years ago, we were serving over 100 million pages (of heavy content) per month, and we didn’t even bother looking at traffic statistics because, here in Germany, it’s hard to hit the traffic limits that most hosting providers impose. That being said, AWS is less expensive than I initially thought. At the same time, I’m confident you could reduce your hosting bill by up to $2k without even leaving AWS by setting up your own database server. Moving away from AWS entirely might be challenging, as managing a fleet of about 30 servers would likely take one or two days of work per week (I'm managing a dozen mostly idling servers and I work one day per month on them). When your hosting bill reaches $30k, I'm very sure it would be cheaper to hire someone (hint, hint ;-)), that moves everything to dedicated servers and manages them.
RobRivera 2y ago

evantbyrne 2y ago

Cloud providers have gotten really good at selling scale anxiety. Very easy to reach $10k/mo when building on a modern web stack using managed services. Kafka alone could get you there once you factor in multiple environments and add-ons.