5 ms·

Off-topic maybe, but I'm very curious as it's hard to find hard numbers: > my usual bill is $200/month How many req/second are you serving? What kind of thing

Off-topic maybe, but I'm very curious as it's hard to find hard numbers: > my usual bill is $200/month How many req/second are you serving? What kind of things are happening? It seems like the bills are outrageously expensive when it comes to various cloud services, as I'm personally hosting a service that does between 10-100 req/second on average during a month, and my monthly bill end up being closer to $40/month, including traffic and everything. I'm running a database on the same server, and 20% of the requests writes stuff both to disk and to the database, while the rest just reads from DB or disk. The whole setup took around 5 hours to setup on one day, and have been running flawlessly from day one, we haven't had to migrate servers yet after ~6 months of production usage. Probably one day we're gonna have to upgrade the server to a $80/month one, but that's a one-time thing and our revenue easily covers that.

rozenmd 4y ago
I run 1.2 million uptime checks per week, my total AWS bill was $150/mo before I migrated to permanently running VMs - it's definitely doable without trying too hard.
- logifail 4y ago
  > my total AWS bill was $150/mo before I migrated to permanently running VMs I know I keep harping on about this, but [if you have a vague idea what you're doing] you can squeeze an awful lot out of a cheap VM. I'm currently working on a project that we're deliberately prototyping using ultra-cheap VMs. If it takes off we know how to scale it up, if it doesn't the costs stay very very low.
  - rozenmd 4y ago
    Definitely - with startups offering 3x free firecracker VMs (of 256MB RAM) these days, there's almost no reason to start ideas with serverless.
- huksley 4y ago
  Yes, moving to VM is definitely doable, but now, being a 1 person dev team, it is challenging in maintenance. My fear is, being on a vacation, and suddenly this VM dies. It might take too much time to bring it back online, and I might be out of good network coverage.
  - rozenmd 4y ago
    Self healing VMs - look into fly.io, they just restart when out of memory etc took about a day to rewrite and a week of 30 mins per day to figure out how to optimise my code for self healing. knowing my bill is capped by number of VMs * memory, saves a lot of stress
    - rozenmd 4y ago
      In case you're curious, I wrote about my experience here: https://onlineornot.com/on-moving-million-uptime-checks-onto-fly-io https://onlineornot.com/on-moving-million-uptime-checks-onto...
  - amerkhalid 4y ago
    Back in days when our services ran on metal or VM, if we got paged during off-hours and had something else going on, a simple reboot almost always fixed the issue. We, developers, never liked the reboots though, always wanted to find out root cause, so that we won't be paged again. So, I guess, we moved to the cloud. Now we don't get paged in the middle of night. But yeah if it was my own company and I was on vacation, I rather take 30 seconds to reboot the server instead of worry about paying thousands of dollars. As for lack of network coverage, you could do scheduled daily/hourly reboots while on vacation, if it makes sense for your service. Or if your an outage will cause massive disruption to your users, then perhaps hire a part-time sysadmin while on a vacation.
steve_adams_86 4y ago
In my consulting days I helped people with issues like this quite often, and what I found tended to come down to severe inefficiency caused by people not fully understanding how services need to perform at scale and what causes them to perform poorly. You hear a lot of “oh, it seemed to work really well on my machine”, which doesn’t necessarily translate well to performing or scaling smoothly in the wild. With a bit of refactoring and simplifying it tended to cut out a lot of issues. I suppose the issue is that a lot of people don’t really know what to look for or how to anticipate issues in complex infrastructure (which is totally fair, I only learned via trial by fire and have done some really stupid stuff). I don’t think I ever encountered a case where the code was correct and bills were too high. Some AWS/GCP configurations would be pretty bad, but the code would also tend to be incredibly inefficient. I always encourage people to respect people who are great at dev ops and to either hire them if they’re big enough or just consult them if they’re uncertain about things. Throwing a day of consulting rate at someone smarter than you is a great way to learn and it could easily save you money in the not-so-long term. I’d say that because they’re crazy to rely on me for infrastructure, but I’d make things better if I was around already and they asked me to help out. But I’m no substitute for someone who actually knows what they’re doing.
- deleted 4y ago
  [deleted]
ryandvm 4y ago
It depends on how efficient the dev is. For comparison, I've been running a paid Slack app (a few $K per year) that manages to run entirely within the free tier on Google Cloud Platform.
- deleted 4y ago
  [deleted]
- capableweb 4y ago
  Paid Slack app just receives webhooks from a Slack organization right? So the amount of traffic it handles amount at max to the amount of messages that gets sent around in Slack, which tends to be very low (in comparison to other applications where user interactions can trigger many requests for example)
  - ryandvm 4y ago
    True, but in my app's case each webhook ends up making a few dozen Slack API calls. In any case the free tier will definitely allow you to handle enough traffic to figure out if you've got a viable business model, which I guess is the point.
huksley 4y ago
It is depends on the user activity but typically it is not so many requests. Around 20 req/second with a reply in 100-400 ms. These recursive requests were going at the rate 17k/second with 30s timeout each. The benefits of the current approach is I don't need to manage any servers, and I have different environments for free. Also sizing is not an issue, I just tune AWS Lambda limits to be able to serve one single request. I will need to invest some time to understand how big the instance should be (how much memory) because I struggle to measure how big it should be without going out of memory or CPU.

rozenmd 4y ago

I run 1.2 million uptime checks per week, my total AWS bill was $150/mo before I migrated to permanently running VMs - it's definitely doable without trying too hard.

logifail 4y ago
> my total AWS bill was $150/mo before I migrated to permanently running VMs I know I keep harping on about this, but [if you have a vague idea what you're doing] you can squeeze an awful lot out of a cheap VM. I'm currently working on a project that we're deliberately prototyping using ultra-cheap VMs. If it takes off we know how to scale it up, if it doesn't the costs stay very very low.
- rozenmd 4y ago
  Definitely - with startups offering 3x free firecracker VMs (of 256MB RAM) these days, there's almost no reason to start ideas with serverless.
huksley 4y ago
Yes, moving to VM is definitely doable, but now, being a 1 person dev team, it is challenging in maintenance. My fear is, being on a vacation, and suddenly this VM dies. It might take too much time to bring it back online, and I might be out of good network coverage.
- rozenmd 4y ago
  Self healing VMs - look into fly.io, they just restart when out of memory etc took about a day to rewrite and a week of 30 mins per day to figure out how to optimise my code for self healing. knowing my bill is capped by number of VMs * memory, saves a lot of stress
  - rozenmd 4y ago
    In case you're curious, I wrote about my experience here: https://onlineornot.com/on-moving-million-uptime-checks-onto-fly-io https://onlineornot.com/on-moving-million-uptime-checks-onto...
- amerkhalid 4y ago
  Back in days when our services ran on metal or VM, if we got paged during off-hours and had something else going on, a simple reboot almost always fixed the issue. We, developers, never liked the reboots though, always wanted to find out root cause, so that we won't be paged again. So, I guess, we moved to the cloud. Now we don't get paged in the middle of night. But yeah if it was my own company and I was on vacation, I rather take 30 seconds to reboot the server instead of worry about paying thousands of dollars. As for lack of network coverage, you could do scheduled daily/hourly reboots while on vacation, if it makes sense for your service. Or if your an outage will cause massive disruption to your users, then perhaps hire a part-time sysadmin while on a vacation.

steve_adams_86 4y ago

In my consulting days I helped people with issues like this quite often, and what I found tended to come down to severe inefficiency caused by people not fully understanding how services need to perform at scale and what causes them to perform poorly. You hear a lot of “oh, it seemed to work really well on my machine”, which doesn’t necessarily translate well to performing or scaling smoothly in the wild. With a bit of refactoring and simplifying it tended to cut out a lot of issues. I suppose the issue is that a lot of people don’t really know what to look for or how to anticipate issues in complex infrastructure (which is totally fair, I only learned via trial by fire and have done some really stupid stuff). I don’t think I ever encountered a case where the code was correct and bills were too high. Some AWS/GCP configurations would be pretty bad, but the code would also tend to be incredibly inefficient. I always encourage people to respect people who are great at dev ops and to either hire them if they’re big enough or just consult them if they’re uncertain about things. Throwing a day of consulting rate at someone smarter than you is a great way to learn and it could easily save you money in the not-so-long term. I’d say that because they’re crazy to rely on me for infrastructure, but I’d make things better if I was around already and they asked me to help out. But I’m no substitute for someone who actually knows what they’re doing.

deleted 4y ago
[deleted]

ryandvm 4y ago

It depends on how efficient the dev is. For comparison, I've been running a paid Slack app (a few $K per year) that manages to run entirely within the free tier on Google Cloud Platform.

deleted 4y ago
[deleted]
capableweb 4y ago
Paid Slack app just receives webhooks from a Slack organization right? So the amount of traffic it handles amount at max to the amount of messages that gets sent around in Slack, which tends to be very low (in comparison to other applications where user interactions can trigger many requests for example)
- ryandvm 4y ago
  True, but in my app's case each webhook ends up making a few dozen Slack API calls. In any case the free tier will definitely allow you to handle enough traffic to figure out if you've got a viable business model, which I guess is the point.

huksley 4y ago

It is depends on the user activity but typically it is not so many requests. Around 20 req/second with a reply in 100-400 ms. These recursive requests were going at the rate 17k/second with 30s timeout each. The benefits of the current approach is I don't need to manage any servers, and I have different environments for free. Also sizing is not an issue, I just tune AWS Lambda limits to be able to serve one single request. I will need to invest some time to understand how big the instance should be (how much memory) because I struggle to measure how big it should be without going out of memory or CPU.