4 ms·

> And if it never takes off, that's fine too because you didn't spend a ton of time investing in making it work. But if it “takes off” while you are caught wit

> And if it never takes off, that's fine too because you didn't spend a ton of time investing in making it work. But if it “takes off” while you are caught with your pants down and you have no clue, you are stuffed with a friggin $60k bill and potentially have to remortgage your house.

minimaxir 6y ago
That's why businesses use expected cost to make business decisions. The odds of it taking off are very, very low. And in the rare case it does, the infra is relatively portable (Docker container) and can be moved to a more-suitable VMs in less than the full month.
- ALittleLight 6y ago
  But if it doesn't really take off - why do you need it? The main benefit of easy scaling strikes me as the scaling, but if you don't want to scale too much for cost reasons...
- bushbaba 6y ago
  In that case why not use digital ocean or similar cheap VM offering. Cheaper, scales in near term. And provides a path forward to a more enterprise stack if necessary. I think people underestimate the serving capabilities of a few VMs coupled with caching/cdn.
- jillesvangurp 6y ago
  Exactly, you can repurpose much of your build pipeline and simply boot the same docker image using a vm or scaling group of vms with a single command. The cost of switching to something else if you actually need to is minimal. I improvised it a few weeks ago when I realized that doing background jobs wasn't something I should be doing on cloud run (it gets throttled). I added an endpoint that processed a data import in the background and that just wouldn't work in cloudrun. So, I wrote a script that runs the same container on a vm and then shuts it down. Took me about 10 minutes to write and it zipped through the import in a few minutes. For a proper setup, you'd want to use terraform or similar to provision the vms but that's not particularly challenging. Basically you point your provisioned infrastructure at your docker container and it will run it. Very few businesses will ever see a quarter million concurrent users doing anything on their services. If you have that challenge, you can afford a few full time devops people to get you whatever infrastructure you need to handle this. We've been running on cloud run since August. Standout features for me: - it created a working CI/CD pipeline for me from my github repository with minimal effort and sane defaults. I got the service up and running within 20 minutes of me deciding to give it a try. I was so impressed with this that I decided on the spot to continue using it. That 20 minutes of devops time is technically the largest expense we've had so far on this setup as we mostly stay below the freemium threshold and it's otherwise a zero maintenance thing for us. Merge to master ... code deploys & runs. We've done this hundreds of times in the last months. - comes with proper logging, monitoring, etc. tooling with absolutely no additional effort. I don't consider running blind an option so this stuff is usually not optional for me. - we have no cloud run specifics in our github repository beyond the cloud build yml file. There is no vendor lock in. If you need a cheap way to run your docker container; this is it. Infrastructure is a blackbox that you get as a service. That's how infrastructure should be these days. - once we got in to the test program; web sockets have been great as well. We just launched that feature a few days ago. - low cost, low hassle, yet ticks all the boxes on the operational front. This is important to me as we don't have 24x7 people on staff and I hate having to deal with outages on a weekend or at night (as the CTO, I'm basically the person that gets to deal with this). - I've been involved with devops on other projects. It can be a serious time sink. It certainly has sucked up months of my time. Many projects end up having several full time people dealing with that at typically premium consulting rates. Days of tinkering with terraform, chasing stackoverflow posts, reading piles and piles of documentation, etc. It just never ends. That stuff costs magnitudes more than you are likely to spend on hosting in your first year. Even the most expensive fancy AWS setup is a rounding error compared to that. Cloud run on the other hand is dead easy to setup and dirt cheap for small setups. It's great way to defer devops cost; potentially indefinitely.
  - plesiv 6y ago
    Thank you for an informative comment. One question: > I realized that doing background jobs wasn't something I should be doing on cloud run (it gets throttled) What do you mean by "it gets throttled". Do you mean that you exceed the CPU/memory limit in a container. Maybe you just have to plan the capacity for the background jobs: connections vs CPU vs memory. We use Cloud Run for background jobs with 0 issues.
    - jillesvangurp 6y ago
      We ran with 0 hosts so in between requests, we had a lot of things failing or getting killed. The pattern of firing up a background thread from a request, returning immediately, and processing in the background simply did not work for us. The same logic worked fine elsewhere so it really was the environment. You can't really use it like a stateful environment. In some cases we saw no evidence of the job even starting because the whole thing died right after the request returned. In other cases it would start but never finish. The actual import jobs weren't that big even. It was just super flaky. The whole problem went away as soon as we gave it a dedicated vm. All I did was run the same container and then trigger the same request via a curl command sshed into the vm. Ugly but it got the job done.
asdfsfkwqe 6y ago
If you can constantly have 250k concurrent users. I don't think money would be an issue at that time.
deleted 6y ago
[deleted]
thesandlord 6y ago
1) you can cap the max instances in GCR. You can put a limit of like 2 instances and ramp it up gradually. 2) it's 60k for a full month running at 100% capacity. Chances are you will notice this. This blog post is showing the upper limits of GCR, not the typical use case.
0xy 6y ago
If it takes off and you're supporting millions of customers, a $60k infra bill is an amazing problem to have. "Oh no, we're too popular!". Who says this?
etxm 6y ago
We had a logging service that cost us 30k / mo. $60/k for your breadwinner seems like a deal.
- layoric 6y ago
  I personally see the flip side, the original logging service at 30k/mo is totally wild. How much data was being logged/searched if you don't mind me asking?
  - etxm 6y ago
    About 300-600GB / hr. Data retention was based on a few different rules but could be stored for up to a month. 30k was ingress and ETL, that amount didn’t include storage. Not sure what storage costs were off-hand.
    - layoric 6y ago
      Thanks for sharing, that's a lot of logs but another good example of how linear scaling costs hurts. A lot cheaper than standard AWS CloudWatch which reckons it would be $200k+ a month.. but though 600GB an hr is not an easy problem to solve but that is a hell of a premium on both.

minimaxir 6y ago

That's why businesses use expected cost to make business decisions. The odds of it taking off are very, very low. And in the rare case it does, the infra is relatively portable (Docker container) and can be moved to a more-suitable VMs in less than the full month.

ALittleLight 6y ago
But if it doesn't really take off - why do you need it? The main benefit of easy scaling strikes me as the scaling, but if you don't want to scale too much for cost reasons...
bushbaba 6y ago
In that case why not use digital ocean or similar cheap VM offering. Cheaper, scales in near term. And provides a path forward to a more enterprise stack if necessary. I think people underestimate the serving capabilities of a few VMs coupled with caching/cdn.
jillesvangurp 6y ago
Exactly, you can repurpose much of your build pipeline and simply boot the same docker image using a vm or scaling group of vms with a single command. The cost of switching to something else if you actually need to is minimal. I improvised it a few weeks ago when I realized that doing background jobs wasn't something I should be doing on cloud run (it gets throttled). I added an endpoint that processed a data import in the background and that just wouldn't work in cloudrun. So, I wrote a script that runs the same container on a vm and then shuts it down. Took me about 10 minutes to write and it zipped through the import in a few minutes. For a proper setup, you'd want to use terraform or similar to provision the vms but that's not particularly challenging. Basically you point your provisioned infrastructure at your docker container and it will run it. Very few businesses will ever see a quarter million concurrent users doing anything on their services. If you have that challenge, you can afford a few full time devops people to get you whatever infrastructure you need to handle this. We've been running on cloud run since August. Standout features for me: - it created a working CI/CD pipeline for me from my github repository with minimal effort and sane defaults. I got the service up and running within 20 minutes of me deciding to give it a try. I was so impressed with this that I decided on the spot to continue using it. That 20 minutes of devops time is technically the largest expense we've had so far on this setup as we mostly stay below the freemium threshold and it's otherwise a zero maintenance thing for us. Merge to master ... code deploys & runs. We've done this hundreds of times in the last months. - comes with proper logging, monitoring, etc. tooling with absolutely no additional effort. I don't consider running blind an option so this stuff is usually not optional for me. - we have no cloud run specifics in our github repository beyond the cloud build yml file. There is no vendor lock in. If you need a cheap way to run your docker container; this is it. Infrastructure is a blackbox that you get as a service. That's how infrastructure should be these days. - once we got in to the test program; web sockets have been great as well. We just launched that feature a few days ago. - low cost, low hassle, yet ticks all the boxes on the operational front. This is important to me as we don't have 24x7 people on staff and I hate having to deal with outages on a weekend or at night (as the CTO, I'm basically the person that gets to deal with this). - I've been involved with devops on other projects. It can be a serious time sink. It certainly has sucked up months of my time. Many projects end up having several full time people dealing with that at typically premium consulting rates. Days of tinkering with terraform, chasing stackoverflow posts, reading piles and piles of documentation, etc. It just never ends. That stuff costs magnitudes more than you are likely to spend on hosting in your first year. Even the most expensive fancy AWS setup is a rounding error compared to that. Cloud run on the other hand is dead easy to setup and dirt cheap for small setups. It's great way to defer devops cost; potentially indefinitely.
- plesiv 6y ago
  Thank you for an informative comment. One question: > I realized that doing background jobs wasn't something I should be doing on cloud run (it gets throttled) What do you mean by "it gets throttled". Do you mean that you exceed the CPU/memory limit in a container. Maybe you just have to plan the capacity for the background jobs: connections vs CPU vs memory. We use Cloud Run for background jobs with 0 issues.
  - jillesvangurp 6y ago
    We ran with 0 hosts so in between requests, we had a lot of things failing or getting killed. The pattern of firing up a background thread from a request, returning immediately, and processing in the background simply did not work for us. The same logic worked fine elsewhere so it really was the environment. You can't really use it like a stateful environment. In some cases we saw no evidence of the job even starting because the whole thing died right after the request returned. In other cases it would start but never finish. The actual import jobs weren't that big even. It was just super flaky. The whole problem went away as soon as we gave it a dedicated vm. All I did was run the same container and then trigger the same request via a curl command sshed into the vm. Ugly but it got the job done.

asdfsfkwqe 6y ago

If you can constantly have 250k concurrent users. I don't think money would be an issue at that time.

deleted 6y ago

[deleted]

thesandlord 6y ago

1) you can cap the max instances in GCR. You can put a limit of like 2 instances and ramp it up gradually. 2) it's 60k for a full month running at 100% capacity. Chances are you will notice this. This blog post is showing the upper limits of GCR, not the typical use case.

0xy 6y ago

If it takes off and you're supporting millions of customers, a $60k infra bill is an amazing problem to have. "Oh no, we're too popular!". Who says this?

etxm 6y ago

We had a logging service that cost us 30k / mo. $60/k for your breadwinner seems like a deal.

layoric 6y ago
I personally see the flip side, the original logging service at 30k/mo is totally wild. How much data was being logged/searched if you don't mind me asking?
- etxm 6y ago
  About 300-600GB / hr. Data retention was based on a few different rules but could be stored for up to a month. 30k was ingress and ETL, that amount didn’t include storage. Not sure what storage costs were off-hand.
  - layoric 6y ago
    Thanks for sharing, that's a lot of logs but another good example of how linear scaling costs hurts. A lot cheaper than standard AWS CloudWatch which reckons it would be $200k+ a month.. but though 600GB an hr is not an easy problem to solve but that is a hell of a premium on both.