11 ms·

Building a high-scale chat server on Cloud Run

148 points by alpb 6y ago

minimaxir 6y ago
Granted, the pricing assumptions assume 100% utilization at all times and not taking advantage of scaling-to-zero, which is where Cloud Run excels. Given that, it seems like WebSockets/persistent connections is a weird use case.
- cavisne 6y ago
  It demonstrates the combination of concurrency and scale to 0 which I havent seen in other public clouds.
rad_gruchalski 6y ago
Oh dear. “$87/hour, which is $62.6K/month” for only 250k concurrent connections? Around €1500 will get you a second hand Dell R720 with 190+ GB RAM and 48 cores. It would be cheaper to handle such scaling with a vm instead of functions. Any kernel with a bit of tweaking will easily handle that number of connections. One just needs RAM and sufficient CPU. 16GB RAM and 8 vCPUs on aws will be enough for 500k. Sleeping would be very uncomfortable if my service using such approach started being popular and was running at increasing capacity for a week because my team needs time to move back to vms.
- jacques_chester 6y ago
  > It would be cheaper to handle such scaling with a vm instead of functions. Ahmet makes that exact point: > In the long term, as your load becomes more predictable, it makes more sense to move to VM-based compute (such as GCE or GKE) as several mid-size virtual machines can handle the same load, potentially 50x cheaper. The shiny spot for something like Cloud Run -- or Knative -- is varying workloads. I described this as "workloads with unpredictable, latency-insensitive demand" in my book on the subject[0]. [0] https://livebook.manning.com/book/knative-in-action/chapter-1/section-1-3 https://livebook.manning.com/book/knative-in-action/chapter-...
  - rad_gruchalski 6y ago
    What I never get with these is: one has to really go out of their way to run the same code on a vm and serverless. So why even bother building the thing twice? Do people really consider putting a server app in a function?
    - jacques_chester 6y ago
      > So why even bother building the thing twice? Time elapses. What makes sense today may not tomorrow. It's fine to adapt designs when the economics point that way. Also fine to leave things as they are when the economics point that way.
    - leetrout 6y ago
      Yep we’re running our API at work in a lambda behind API gateway. It’s “interesting”
      - rad_gruchalski 6y ago
        Sounds fun! Could you elaborate on the “interesting” part a bit more?
        leetrout 6y ago
        On one hand it’s very easy to run the code locally or in lambda because we use Go along with a library from AWS that translates the gateway proxy request event to a normal http request. So the request handling is all the same in code. To prevent 404s hitting your function you have to define your routes in both your code and in api gateway. This starts to marry your code to your deployment. Observability is now harder and you need yet another thing. We’re using X-ray with datadog. To “safely” talk to Postgres you should use an rds proxy. It’s still unclear how much is different from pgbouncer or similar but in any case it’s pretty easy to knock over your db with runaway lambda invocations. I could go on about the mistake we made of using SAM and how it ends up not supporting everything you need but I guess my main point is it’s easy to get sucked in by the idea of it being easy but you are trading that for service / operational complexity with a healthy dose of platform lock-in.
- durkie 6y ago
  where can you find servers like that for that price?
  - rad_gruchalski 6y ago
    servershop24.de; not affiliated, just a happy client.
    - gallexme 6y ago
      And as hosting Hetzner.de Also happy client im easily handling hundred thousand connections for less than 80€/mo
      - layoric 6y ago
        Love to read more about these stories rather than the same crazy cloud costs that get normalized all the time. Just by looking at Hetzner and others, I can see this kind of thing would be extremely doable but concrete stories of those doing it would be something I think a lot of people would enjoy reading about trade offs etc.
- pier25 6y ago
  Serverless is for uses cases with unpredictable load. Bare metal is cheaper, but you're paying 100% of the time for your peak traffic. If you only have a couple of peaks with 250k concurrent connections, with serverless you'd pay much less than €1500 but still be able to satisfy that demand.
  - layoric 6y ago
    Taking steps a bit further, it would be interesting to know what variation saves what money. Eg if your load is 80% of the time between 80-100k connections but the 20% at 250k, you save $500 a month using serverless or vice versa. Is it as simple as $60k/number of connections? If so the cross over point would be just 6250 connections as your base load. IMO, that $1500 server starts to look very appealing very quickly but I imagine it isn't as simple as that.
- csomar 6y ago
  This is useful for services that have short surges but very little traffic around the month. If you only need a surge for 1 hour per month, then it should cost around $100/month. > Any kernel with a bit of tweaking will easily handle that number of connections. I don't think tweaking a kernel and "easily" fit together. Unless you are kernel developer.
  - lelanthran 6y ago
    >> Any kernel with a bit of tweaking will easily handle that number of connections. > I don't think tweaking a kernel and "easily" fit together. Unless you are kernel developer. I think he means twiddling the kernel knobs in /proc to raise the limit of file descriptors. Last I checked, most distros default to 100k maxmimum file descriptors, with a different (10k) max per process. Changing the max by 'tweaking' involved echoing a new number to a file in /proc.
  - rad_gruchalski 6y ago
    It's called a C1M problem. From 2013: https://www.metabrew.com/article/a-million-user-comet-application-with-mochiweb-part-1 https://www.metabrew.com/article/a-million-user-comet-applic.... It's pretty easy. Been working on a product doing this for 500k connections per machine (MQTT) back in 2014. EC2, m3.xlarge instances back then.
- bullen 6y ago
  Adding to this, how many watts is this consuming? epoll allows high concurrent client count, but how many requests per second and more importantly how many requests per watt!? Also what is the response payload size? I can do 1000 HTTP req/res (4K) / watt on my new atom 8-core server with my own HTTP app server passively cooled and tiny mini-itx size!
toolslive 6y ago
compare that to 400k connections on a single server using erlang early this Millenium. Wirth's law in action. https://en.wikipedia.org/wiki/Wirth%27s_law https://en.wikipedia.org/wiki/Wirth%27s_law
- jacques_chester 6y ago
  You wouldn't get 400k connections on Cloud Run using erlang, because it limits concurrent requests per container to 250.
  - justinsaccount 6y ago
    but.. why does it do that?
    - 0xy 6y ago
      $$$
    - jacques_chester 6y ago
      Most software becomes unhappy if you send unlimited concurrent requests. It's better to have a hard limit that can be raised than to find out under fire what your software tolerates.
      - justinsaccount 6y ago
        but can it be raised?
        jacques_chester 6y ago
        No idea. Probably, but it depends on whether that limit is hard coded or is a config flag that an account team can get changed.
londons_explore 6y ago
WhatsApp had 2 Million simultaneous chat TCP connections per server back in 2014... Why do we now need thousands of instances to achieve the same?
- asdfsfkwqe 6y ago
  Stackoverflow uses two 768g RAM servers as databases servers (one as backup). Why would we have distributed databases anyway?
- abrookewood 6y ago
  From memory, they also had really big servers and did a lot of work tuning the OS. Pretty sure it was BSD initially. The suggested approach by the author is for people who don't have (1) the money to buy big servers or (2) the expertise to tune the OS appropriately. Not saying I agree with the approach, but I can see where it would be helpful.
- bastawhiz 6y ago
  This isn't a valid comparison. You don't know how many (and the hardware of) instances WhatsApp was running to support those 2M connections. And you're not comparing apples-to-apples: WhatsApp is not the same software that the OP is running. Moreover, the 250K limit is an artificial limit imposed by Google for a generic cloud service. If you controlled _that_ software and tuned it for your specific application, you probably could squeeze an order of magnitude (or two or three) more connections out of it. Whether that's a worthwhile tradeoff is a function of whether the time required to rewrite the functionality of Cloud Run yourself (and run it, and maintain it) is lower than the cost of simply spinning up more boxes. If you're building a product and have the cash, it really doesn't matter whether you can run 250k connections or 2M connections if one could exist this week and suit your needs and the other could exist in 6-8 months when you build/scale/deploy it.
  - Tepix 6y ago
    To me it seems like it's directly comparable and the efficiency demonstrated here seems truly abysmal.
davidbanham 6y ago
I use Cloud Run for all my server rendered web apps and I _love_ it. That’s where the pricing model makes sense since as soon as the request is served I stop paying. The actual cash price is less than what I was paying to run on GKE and the total cost of ownership is far lower. Web sockets are a handy thing to have access to but the app described is a pretty pathological case for the Cloud Run pricing model. I think people knocking the article on pricing alone are missing the point.
jacques_chester 6y ago
I think Ahmet's arithmetic is incorrect. The limits that he says lead to the 250k figure are: 1. 250 connections per container, and 2. 1,000 containers. However the 250 concurrency limit does not refer to connections, it refers to requests. 250 concurrent requests can actually represent thousands of clients, depending on their think time.
- tln 6y ago
  I think it's correct... https://cloud.google.com/run/docs/triggering/websockets https://cloud.google.com/run/docs/triggering/websockets says: "Since Cloud Run supports concurrent connections (up to 250 per container)..." "WebSockets requests are treated as long-running HTTP requests in Cloud Run."
  - jacques_chester 6y ago
    I stand corrected.
thesandlord 6y ago
I think some people are missing the point when comparing this to a traditional VM setup. Yes it is way more expensive, but it lets you deploy something that works in 10 minutes vs messing with VMs and auto-scaling groups and all that jazz. If you are a GCP or AWS or bare metal expert that can set this thing up in their sleep, that's great but the majority of people can really benefit from a PaaS like GCR. Because Cloud Run uses vanilla Docker containers, once you have validated the idea you can move to GKE or VMs or a server under your desk or whatever. And if it never takes off, that's fine too because you didn't spend a ton of time investing in making it work. Ahmet if you are reading this (hi) it would be REALLY cool to see something like using GKE for the base load and dynamically bursting to GCR to fill in the gaps. Not sure if this possible with GCLB today, but would be super cool. (Disclaimer: I used to work on this team at Google.)
- rad_gruchalski 6y ago
  > And if it never takes off, that's fine too because you didn't spend a ton of time investing in making it work. But if it “takes off” while you are caught with your pants down and you have no clue, you are stuffed with a friggin $60k bill and potentially have to remortgage your house.
  - minimaxir 6y ago
    That's why businesses use expected cost to make business decisions. The odds of it taking off are very, very low. And in the rare case it does, the infra is relatively portable (Docker container) and can be moved to a more-suitable VMs in less than the full month.
    - ALittleLight 6y ago
      But if it doesn't really take off - why do you need it? The main benefit of easy scaling strikes me as the scaling, but if you don't want to scale too much for cost reasons...
    - bushbaba 6y ago
      In that case why not use digital ocean or similar cheap VM offering. Cheaper, scales in near term. And provides a path forward to a more enterprise stack if necessary. I think people underestimate the serving capabilities of a few VMs coupled with caching/cdn.
syspec 6y ago
> If you deploy this app with 128MB RAM and 1 vCPU today, it will cost (0.00002400 + (0.00000250/8)) * 60 * 60 = $0.0875 per hour per instance. 1 This means if you have 1,000 instances actively running and serving 250K clients, it will cost $87/hour, which is $62.6K/month. > Any Cloud Run service, by default, can scale up to 1,000 instances. (However, by opening a support ticket, you can get this number elevated.) This means we can support 250,000 clients simultaneously without having to worry about infrastructure and scaling! Aren't these statements at odds with one another?
- monsieurbanana 6y ago
  I don't see why, maybe you can elaborate?
  - neolog 6y ago
    Because that's a lot of money.
    - monsieurbanana 6y ago
      Is it the "without having to worry about infrastructure and scaling" that you disagree? I don't personally find the two statements at odds. If you have 250k connections 24/24 7/7 for a whole month, that should mean your business is generating more than enough money to cover the cost. Or, at the very least, that the losses by not being able to scale are higher than the cost of the infrastructure. If this is false and you'll be bankrupt by those 250k connections, you should obviously not let your instances to scale that much.
      - neolog 6y ago
        It's still a lot of money even if it won't bankrupt the business.
  - reilly3000 6y ago
    Scaling limits put in place by cloud providers are there in part to make sure their customers don’t accidentally incur runaway costs due to a configuration mistake. It’s not uncommon that a cloud provider would cover the costs if a customer make such an error (first time, ask nicely, etc). They are covering their own bases from having to do that on a massive scale.
emilsedgh 6y ago
Cloud Run has been fantastic. I had a worker service running on Heroku. Very CPU intensive. The traffic pattern was extremely low throughout the day, but had completely unexpected surges. On Heroku, my choices were: Paying $3k (basically paying for peak surge throughout the month) or having a lot of slow/failed responses during the surge. Moved to Cloud Run very easily. Just a normal dockerized 12 factor app. Now I pay ~$50/m and it automatically scales up when I need more workers. If you want a managed app platform, it couldn't be more simple and cheap. The scaling model of Cloud Run has been great. Now that they support websockets, I will be moving all my apps slowly to it. My only gripe: I wish they had something for worker processes. I know that the rest of Google Cloud has solutions for it but being able to just spawn worker processes as part of the same deployment would be fantastic.
- rattray 6y ago
  How quick does it spin up?
  - emilsedgh 6y ago
    In my case since its background workers, I don't mind the slow startup and I never measured it. Sorry couldn't be more helpful.
  - CaptainJustin 6y ago
    My Go-based containers start in much less than a second. I believe its dozens to maybe low hundreds of milliseconds
    - gcbirzan 6y ago
      Depends on the size of the docker image too, we have one with libreoffice (for generating PDFs, invoices, contracts), it takes almost 3 seconds to start at 550MB
- gt50201 6y ago
  How do you differentiate background and worker processes? I’m currently running VMs for background processes processing pub sub topics. Is that not something that could go on cloud run?
intrasight 6y ago
I'm trying to figure out a good architecture for a chat back end so I found this interesting. Anyone have some good pointers to other such articles?
- tempest_ 6y ago
  Discord's blog has some interesting articles. https://blog.discord.com/engineering-posts/home https://blog.discord.com/engineering-posts/home
cunac 6y ago
Why you wouldn't use AWS http events for this. Especially because connections are decoupled from processing
itsthecourier 6y ago
Guys last time I checked in our team, we decided serverless is good for requests under 800ms. After that, specially with permanent connections like web socket, consider using aws iot core mqtt over we socket. $0.08 per million minute of connections. In this case: 60 * 24 * 30 * 250000 * $0.08 / 1000000=usd864 per month
synthmeat 6y ago
It's a fun example in how easy it is to setup, and I don't want to poke too hard on a hypotheticals, but I don't like two things: 1) Egress is hand-waved, when in reality the free GBs would basically exhaust in a month on just pongs and reconnection requests, so doing nothing at all. That amount of people just sending single emojis as messages for just one day (to play along "short marketing event") is already matching monthly napkin estimate stated. 2) Isn't there's a good chance Memorystore wouldn't be able to handle this at fairly charitable load interpretations under that many CCUs? 1‰ of those 250k CCUs sending a message every second means Redis needs to publish 250k messages as well. Again though, I'm aware it's only a hypothetical scenario demonstration, and it really does look cute in how it's simple to deploy. Fun stuff.
- alpb 6y ago
  OP here. 1) You’re right, 1 GiB free tier will exhaust fast. Beyond that standard GCP-wide egress rates apply. So if you were using VMs to serve this chatroom, you'd be paying the same https://cloud.google.com/network-tiers/pricing#premium-pricing https://cloud.google.com/network-tiers/pricing#premium-prici... 2) I think you’re missing out the part that in this setup only 1000 instances connect to Redis (using 1000 connections). We don’t initiate a per-user connection from the backend to Redis. Once a message is pushed to an app instance, the sample app pushes it to all its connected users.
dealforager 6y ago
One thing I haven't seen mentioned in the comments is how easy it is to quickly iterate on Cloud Run. I built a web app that uses Cloud Run for 90% of my server needs (Cloud Functions for the rest). One of my favorite features is that I can make some local changes, upload the container, and click a button to deploy the latest version or A/B test. Then if I notice anything weird I can just as easily revert the changes, which has saved me a few times. Admittedly, I haven't used VMs in years and it might be just as easy there these days with containerized VMs, but I don't know. Cloud Run was so trivial to set up that I didn't want to waste time on alternatives. Another really nice thing is that it integrates nicely with Firebase, which I use for hosting. As my site matures, I might move some predictable parts to containerized VMs to save on costs. I have a crawler-type service that has extremely consistent traffic that probably shouldn't be on Cloud Run, but I did it anyway due to being able to iterate so quickly (see revisions here: https://imgur.com/fkFtIRM https://imgur.com/fkFtIRM). Cloud Run also has a free tier which was enough when I was starting out and prototyping. It's also nice since my site gets very little traffic at night so at the moment the cost is not too bad. This is what my billing looks like: https://imgur.com/gIo3IGJ https://imgur.com/gIo3IGJ If I was a big company or my site got super popular I might do things differently. But for a side project it has made me enjoy programming more than anything I've used before because I can focus 95% on code.
- dealforager 6y ago
  I wanted to edit my post to add more details, but since I can't do that here are some more screenshots: This is the (very consistent) traffic for my scraper that should probably not be on Cloud Run but it's too convenient for me to switch at the moment: https://imgur.com/RYLdp2t https://imgur.com/RYLdp2t This is the traffic for my web server, but since it's 30 days it's hard to see that within each day I get very large (relative) spikes in the middle of the day. I also had some huge spikes earlier due to some Reddit posts I made that are not shown in the graph. Being able to automatically scale to any load has been a life saver as I've tried to do marketing on social media. Otherwise I'd need to provision to handle as much as the maximum load I expect: https://imgur.com/RL534de https://imgur.com/RL534de If you're a startup or need to handle random spikes from social media or other unpredictable sources, I think Cloud Run is worth a look. Another great use if for random jobs you might need to run every now and then. Web servers also seem like a good fit due to the fluctuations depending on the time of day.
kondro 6y ago
A little off-topic, but AWS has a service (actually 2) specifically for managing websocket communications. API Gateway supports an unlimited (you need to ask for an increase from the 500 new connections per second rate) number of connections and is only around $0.25 per million minutes + $1 per million messages. So just having 250,000 connections open would cost only around $270 per month (and scales up and down as you please).
- seanieb 6y ago
  That’s only part of the API gateway pricing, but it is still much much cheaper.
didibus 6y ago
I might be missing the point of this article? Is it very difficult when using Cloud Run to figure out how to achieve a high number of concurrent connections? Is that what the article is about? Or like what is otherwise noteworthy here?
valzam 6y ago
I don't really get what's so special about this. But I am always surprised by how extremely scalable Redis is, scaling the stateless chat servers is trivial, scaling state is hard.
- tuyguntn 6y ago
  I feel exactly same, other than ease of scaling to 1000 instances I don't see anything special, everything is offloaded to Redis. Maybe example use case should have been different to showcase scaling ease
deleted 6y ago
[deleted]
fxtentacle 6y ago
Use a dedicated server and IRC. 250k users isn't that scary anymore. I remember that in 2012, the Rizon IRC network maxed out at 80,000 concurrent users on an AMD Bulldozer CPU.
- pelorat 6y ago
  Exactly. Numbers like these are actually quite pathetic when we look back 10 and even 20 years to IRC.
_jezell_ 6y ago
This is a really neat demo, but there's a few limitations that are glaring issues for a chat server or most scenarios I can think of that I would want to use websockets. As many have mentioned, the 250 connection limit per instance is crazy. You can get 200x that many easily on a single 1 VCPU box. The second problem is that really the demo has just kicked your scaling problem over to redis. The demo isn't doing anything actually interesting, redis is doing all the work. The reality is that one of the best parts of Cloud Run over something like serverless functions is that you can have state in your server. You don't need redis at all if you are doing the same demo in Kubernetes without Cloud Run, since it's pretty easy to use channels in golang to do the same thing. However, in order for you to really make that work, you need some way to at the very least route the traffic consistently so that people watching the same resource get routed to the same server, or to be able to have cross node communication. So this is a great start, but a little work to go before many people would be able to switch over to something like this due.

148 points by alpb 6y ago

Granted, the pricing assumptions assume 100% utilization at all times and not taking advantage of scaling-to-zero, which is where Cloud Run excels. Given that, it seems like WebSockets/persistent connections is a weird use case.

cavisne 6y ago
It demonstrates the combination of concurrency and scale to 0 which I havent seen in other public clouds.

rad_gruchalski 6y ago

Oh dear. “$87/hour, which is $62.6K/month” for only 250k concurrent connections? Around €1500 will get you a second hand Dell R720 with 190+ GB RAM and 48 cores. It would be cheaper to handle such scaling with a vm instead of functions. Any kernel with a bit of tweaking will easily handle that number of connections. One just needs RAM and sufficient CPU. 16GB RAM and 8 vCPUs on aws will be enough for 500k. Sleeping would be very uncomfortable if my service using such approach started being popular and was running at increasing capacity for a week because my team needs time to move back to vms.

jacques_chester 6y ago
> It would be cheaper to handle such scaling with a vm instead of functions. Ahmet makes that exact point: > In the long term, as your load becomes more predictable, it makes more sense to move to VM-based compute (such as GCE or GKE) as several mid-size virtual machines can handle the same load, potentially 50x cheaper. The shiny spot for something like Cloud Run -- or Knative -- is varying workloads. I described this as "workloads with unpredictable, latency-insensitive demand" in my book on the subject[0]. [0] https://livebook.manning.com/book/knative-in-action/chapter-1/section-1-3 https://livebook.manning.com/book/knative-in-action/chapter-...
- rad_gruchalski 6y ago
  What I never get with these is: one has to really go out of their way to run the same code on a vm and serverless. So why even bother building the thing twice? Do people really consider putting a server app in a function?
  - jacques_chester 6y ago
    > So why even bother building the thing twice? Time elapses. What makes sense today may not tomorrow. It's fine to adapt designs when the economics point that way. Also fine to leave things as they are when the economics point that way.
  - leetrout 6y ago
    Yep we’re running our API at work in a lambda behind API gateway. It’s “interesting”
    - rad_gruchalski 6y ago
      Sounds fun! Could you elaborate on the “interesting” part a bit more?
      - leetrout 6y ago
        On one hand it’s very easy to run the code locally or in lambda because we use Go along with a library from AWS that translates the gateway proxy request event to a normal http request. So the request handling is all the same in code. To prevent 404s hitting your function you have to define your routes in both your code and in api gateway. This starts to marry your code to your deployment. Observability is now harder and you need yet another thing. We’re using X-ray with datadog. To “safely” talk to Postgres you should use an rds proxy. It’s still unclear how much is different from pgbouncer or similar but in any case it’s pretty easy to knock over your db with runaway lambda invocations. I could go on about the mistake we made of using SAM and how it ends up not supporting everything you need but I guess my main point is it’s easy to get sucked in by the idea of it being easy but you are trading that for service / operational complexity with a healthy dose of platform lock-in.
durkie 6y ago
where can you find servers like that for that price?
- rad_gruchalski 6y ago
  servershop24.de; not affiliated, just a happy client.
  - gallexme 6y ago
    And as hosting Hetzner.de Also happy client im easily handling hundred thousand connections for less than 80€/mo
    - layoric 6y ago
      Love to read more about these stories rather than the same crazy cloud costs that get normalized all the time. Just by looking at Hetzner and others, I can see this kind of thing would be extremely doable but concrete stories of those doing it would be something I think a lot of people would enjoy reading about trade offs etc.
pier25 6y ago
Serverless is for uses cases with unpredictable load. Bare metal is cheaper, but you're paying 100% of the time for your peak traffic. If you only have a couple of peaks with 250k concurrent connections, with serverless you'd pay much less than €1500 but still be able to satisfy that demand.
- layoric 6y ago
  Taking steps a bit further, it would be interesting to know what variation saves what money. Eg if your load is 80% of the time between 80-100k connections but the 20% at 250k, you save $500 a month using serverless or vice versa. Is it as simple as $60k/number of connections? If so the cross over point would be just 6250 connections as your base load. IMO, that $1500 server starts to look very appealing very quickly but I imagine it isn't as simple as that.
csomar 6y ago
This is useful for services that have short surges but very little traffic around the month. If you only need a surge for 1 hour per month, then it should cost around $100/month. > Any kernel with a bit of tweaking will easily handle that number of connections. I don't think tweaking a kernel and "easily" fit together. Unless you are kernel developer.
- lelanthran 6y ago
  >> Any kernel with a bit of tweaking will easily handle that number of connections. > I don't think tweaking a kernel and "easily" fit together. Unless you are kernel developer. I think he means twiddling the kernel knobs in /proc to raise the limit of file descriptors. Last I checked, most distros default to 100k maxmimum file descriptors, with a different (10k) max per process. Changing the max by 'tweaking' involved echoing a new number to a file in /proc.
- rad_gruchalski 6y ago
  It's called a C1M problem. From 2013: https://www.metabrew.com/article/a-million-user-comet-application-with-mochiweb-part-1 https://www.metabrew.com/article/a-million-user-comet-applic.... It's pretty easy. Been working on a product doing this for 500k connections per machine (MQTT) back in 2014. EC2, m3.xlarge instances back then.
bullen 6y ago
Adding to this, how many watts is this consuming? epoll allows high concurrent client count, but how many requests per second and more importantly how many requests per watt!? Also what is the response payload size? I can do 1000 HTTP req/res (4K) / watt on my new atom 8-core server with my own HTTP app server passively cooled and tiny mini-itx size!

toolslive 6y ago

compare that to 400k connections on a single server using erlang early this Millenium. Wirth's law in action. https://en.wikipedia.org/wiki/Wirth%27s_law https://en.wikipedia.org/wiki/Wirth%27s_law

jacques_chester 6y ago
You wouldn't get 400k connections on Cloud Run using erlang, because it limits concurrent requests per container to 250.
- justinsaccount 6y ago
  but.. why does it do that?
  - 0xy 6y ago
    $$$
  - jacques_chester 6y ago
    Most software becomes unhappy if you send unlimited concurrent requests. It's better to have a hard limit that can be raised than to find out under fire what your software tolerates.
    - justinsaccount 6y ago
      but can it be raised?
      - jacques_chester 6y ago
        No idea. Probably, but it depends on whether that limit is hard coded or is a config flag that an account team can get changed.

londons_explore 6y ago

WhatsApp had 2 Million simultaneous chat TCP connections per server back in 2014... Why do we now need thousands of instances to achieve the same?

asdfsfkwqe 6y ago
Stackoverflow uses two 768g RAM servers as databases servers (one as backup). Why would we have distributed databases anyway?
abrookewood 6y ago
From memory, they also had really big servers and did a lot of work tuning the OS. Pretty sure it was BSD initially. The suggested approach by the author is for people who don't have (1) the money to buy big servers or (2) the expertise to tune the OS appropriately. Not saying I agree with the approach, but I can see where it would be helpful.
bastawhiz 6y ago
This isn't a valid comparison. You don't know how many (and the hardware of) instances WhatsApp was running to support those 2M connections. And you're not comparing apples-to-apples: WhatsApp is not the same software that the OP is running. Moreover, the 250K limit is an artificial limit imposed by Google for a generic cloud service. If you controlled _that_ software and tuned it for your specific application, you probably could squeeze an order of magnitude (or two or three) more connections out of it. Whether that's a worthwhile tradeoff is a function of whether the time required to rewrite the functionality of Cloud Run yourself (and run it, and maintain it) is lower than the cost of simply spinning up more boxes. If you're building a product and have the cash, it really doesn't matter whether you can run 250k connections or 2M connections if one could exist this week and suit your needs and the other could exist in 6-8 months when you build/scale/deploy it.
- Tepix 6y ago
  To me it seems like it's directly comparable and the efficiency demonstrated here seems truly abysmal.

davidbanham 6y ago

I use Cloud Run for all my server rendered web apps and I _love_ it. That’s where the pricing model makes sense since as soon as the request is served I stop paying. The actual cash price is less than what I was paying to run on GKE and the total cost of ownership is far lower. Web sockets are a handy thing to have access to but the app described is a pretty pathological case for the Cloud Run pricing model. I think people knocking the article on pricing alone are missing the point.

jacques_chester 6y ago

I think Ahmet's arithmetic is incorrect. The limits that he says lead to the 250k figure are: 1. 250 connections per container, and 2. 1,000 containers. However the 250 concurrency limit does not refer to connections, it refers to requests. 250 concurrent requests can actually represent thousands of clients, depending on their think time.

tln 6y ago
I think it's correct... https://cloud.google.com/run/docs/triggering/websockets https://cloud.google.com/run/docs/triggering/websockets says: "Since Cloud Run supports concurrent connections (up to 250 per container)..." "WebSockets requests are treated as long-running HTTP requests in Cloud Run."
- jacques_chester 6y ago
  I stand corrected.

thesandlord 6y ago

I think some people are missing the point when comparing this to a traditional VM setup. Yes it is way more expensive, but it lets you deploy something that works in 10 minutes vs messing with VMs and auto-scaling groups and all that jazz. If you are a GCP or AWS or bare metal expert that can set this thing up in their sleep, that's great but the majority of people can really benefit from a PaaS like GCR. Because Cloud Run uses vanilla Docker containers, once you have validated the idea you can move to GKE or VMs or a server under your desk or whatever. And if it never takes off, that's fine too because you didn't spend a ton of time investing in making it work. Ahmet if you are reading this (hi) it would be REALLY cool to see something like using GKE for the base load and dynamically bursting to GCR to fill in the gaps. Not sure if this possible with GCLB today, but would be super cool. (Disclaimer: I used to work on this team at Google.)

rad_gruchalski 6y ago
> And if it never takes off, that's fine too because you didn't spend a ton of time investing in making it work. But if it “takes off” while you are caught with your pants down and you have no clue, you are stuffed with a friggin $60k bill and potentially have to remortgage your house.
- minimaxir 6y ago
  That's why businesses use expected cost to make business decisions. The odds of it taking off are very, very low. And in the rare case it does, the infra is relatively portable (Docker container) and can be moved to a more-suitable VMs in less than the full month.
  - ALittleLight 6y ago
    But if it doesn't really take off - why do you need it? The main benefit of easy scaling strikes me as the scaling, but if you don't want to scale too much for cost reasons...
  - bushbaba 6y ago
    In that case why not use digital ocean or similar cheap VM offering. Cheaper, scales in near term. And provides a path forward to a more enterprise stack if necessary. I think people underestimate the serving capabilities of a few VMs coupled with caching/cdn.

syspec 6y ago

> If you deploy this app with 128MB RAM and 1 vCPU today, it will cost (0.00002400 + (0.00000250/8)) * 60 * 60 = $0.0875 per hour per instance. 1 This means if you have 1,000 instances actively running and serving 250K clients, it will cost $87/hour, which is $62.6K/month. > Any Cloud Run service, by default, can scale up to 1,000 instances. (However, by opening a support ticket, you can get this number elevated.) This means we can support 250,000 clients simultaneously without having to worry about infrastructure and scaling! Aren't these statements at odds with one another?

monsieurbanana 6y ago
I don't see why, maybe you can elaborate?
- neolog 6y ago
  Because that's a lot of money.
  - monsieurbanana 6y ago
    Is it the "without having to worry about infrastructure and scaling" that you disagree? I don't personally find the two statements at odds. If you have 250k connections 24/24 7/7 for a whole month, that should mean your business is generating more than enough money to cover the cost. Or, at the very least, that the losses by not being able to scale are higher than the cost of the infrastructure. If this is false and you'll be bankrupt by those 250k connections, you should obviously not let your instances to scale that much.
    - neolog 6y ago
      It's still a lot of money even if it won't bankrupt the business.
- reilly3000 6y ago
  Scaling limits put in place by cloud providers are there in part to make sure their customers don’t accidentally incur runaway costs due to a configuration mistake. It’s not uncommon that a cloud provider would cover the costs if a customer make such an error (first time, ask nicely, etc). They are covering their own bases from having to do that on a massive scale.

emilsedgh 6y ago

Cloud Run has been fantastic. I had a worker service running on Heroku. Very CPU intensive. The traffic pattern was extremely low throughout the day, but had completely unexpected surges. On Heroku, my choices were: Paying $3k (basically paying for peak surge throughout the month) or having a lot of slow/failed responses during the surge. Moved to Cloud Run very easily. Just a normal dockerized 12 factor app. Now I pay ~$50/m and it automatically scales up when I need more workers. If you want a managed app platform, it couldn't be more simple and cheap. The scaling model of Cloud Run has been great. Now that they support websockets, I will be moving all my apps slowly to it. My only gripe: I wish they had something for worker processes. I know that the rest of Google Cloud has solutions for it but being able to just spawn worker processes as part of the same deployment would be fantastic.

rattray 6y ago
How quick does it spin up?
- emilsedgh 6y ago
  In my case since its background workers, I don't mind the slow startup and I never measured it. Sorry couldn't be more helpful.
- CaptainJustin 6y ago
  My Go-based containers start in much less than a second. I believe its dozens to maybe low hundreds of milliseconds
  - gcbirzan 6y ago
    Depends on the size of the docker image too, we have one with libreoffice (for generating PDFs, invoices, contracts), it takes almost 3 seconds to start at 550MB
gt50201 6y ago
How do you differentiate background and worker processes? I’m currently running VMs for background processes processing pub sub topics. Is that not something that could go on cloud run?

intrasight 6y ago

I'm trying to figure out a good architecture for a chat back end so I found this interesting. Anyone have some good pointers to other such articles?

tempest_ 6y ago
Discord's blog has some interesting articles. https://blog.discord.com/engineering-posts/home https://blog.discord.com/engineering-posts/home

cunac 6y ago

Why you wouldn't use AWS http events for this. Especially because connections are decoupled from processing

itsthecourier 6y ago

Guys last time I checked in our team, we decided serverless is good for requests under 800ms. After that, specially with permanent connections like web socket, consider using aws iot core mqtt over we socket. $0.08 per million minute of connections. In this case: 60 * 24 * 30 * 250000 * $0.08 / 1000000=usd864 per month

synthmeat 6y ago

It's a fun example in how easy it is to setup, and I don't want to poke too hard on a hypotheticals, but I don't like two things: 1) Egress is hand-waved, when in reality the free GBs would basically exhaust in a month on just pongs and reconnection requests, so doing nothing at all. That amount of people just sending single emojis as messages for just one day (to play along "short marketing event") is already matching monthly napkin estimate stated. 2) Isn't there's a good chance Memorystore wouldn't be able to handle this at fairly charitable load interpretations under that many CCUs? 1‰ of those 250k CCUs sending a message every second means Redis needs to publish 250k messages as well. Again though, I'm aware it's only a hypothetical scenario demonstration, and it really does look cute in how it's simple to deploy. Fun stuff.

alpb 6y ago
OP here. 1) You’re right, 1 GiB free tier will exhaust fast. Beyond that standard GCP-wide egress rates apply. So if you were using VMs to serve this chatroom, you'd be paying the same https://cloud.google.com/network-tiers/pricing#premium-pricing https://cloud.google.com/network-tiers/pricing#premium-prici... 2) I think you’re missing out the part that in this setup only 1000 instances connect to Redis (using 1000 connections). We don’t initiate a per-user connection from the backend to Redis. Once a message is pushed to an app instance, the sample app pushes it to all its connected users.

dealforager 6y ago

One thing I haven't seen mentioned in the comments is how easy it is to quickly iterate on Cloud Run. I built a web app that uses Cloud Run for 90% of my server needs (Cloud Functions for the rest). One of my favorite features is that I can make some local changes, upload the container, and click a button to deploy the latest version or A/B test. Then if I notice anything weird I can just as easily revert the changes, which has saved me a few times. Admittedly, I haven't used VMs in years and it might be just as easy there these days with containerized VMs, but I don't know. Cloud Run was so trivial to set up that I didn't want to waste time on alternatives. Another really nice thing is that it integrates nicely with Firebase, which I use for hosting. As my site matures, I might move some predictable parts to containerized VMs to save on costs. I have a crawler-type service that has extremely consistent traffic that probably shouldn't be on Cloud Run, but I did it anyway due to being able to iterate so quickly (see revisions here: https://imgur.com/fkFtIRM https://imgur.com/fkFtIRM). Cloud Run also has a free tier which was enough when I was starting out and prototyping. It's also nice since my site gets very little traffic at night so at the moment the cost is not too bad. This is what my billing looks like: https://imgur.com/gIo3IGJ https://imgur.com/gIo3IGJ If I was a big company or my site got super popular I might do things differently. But for a side project it has made me enjoy programming more than anything I've used before because I can focus 95% on code.

dealforager 6y ago
I wanted to edit my post to add more details, but since I can't do that here are some more screenshots: This is the (very consistent) traffic for my scraper that should probably not be on Cloud Run but it's too convenient for me to switch at the moment: https://imgur.com/RYLdp2t https://imgur.com/RYLdp2t This is the traffic for my web server, but since it's 30 days it's hard to see that within each day I get very large (relative) spikes in the middle of the day. I also had some huge spikes earlier due to some Reddit posts I made that are not shown in the graph. Being able to automatically scale to any load has been a life saver as I've tried to do marketing on social media. Otherwise I'd need to provision to handle as much as the maximum load I expect: https://imgur.com/RL534de https://imgur.com/RL534de If you're a startup or need to handle random spikes from social media or other unpredictable sources, I think Cloud Run is worth a look. Another great use if for random jobs you might need to run every now and then. Web servers also seem like a good fit due to the fluctuations depending on the time of day.

kondro 6y ago

A little off-topic, but AWS has a service (actually 2) specifically for managing websocket communications. API Gateway supports an unlimited (you need to ask for an increase from the 500 new connections per second rate) number of connections and is only around $0.25 per million minutes + $1 per million messages. So just having 250,000 connections open would cost only around $270 per month (and scales up and down as you please).

seanieb 6y ago
That’s only part of the API gateway pricing, but it is still much much cheaper.

didibus 6y ago

I might be missing the point of this article? Is it very difficult when using Cloud Run to figure out how to achieve a high number of concurrent connections? Is that what the article is about? Or like what is otherwise noteworthy here?

valzam 6y ago

I don't really get what's so special about this. But I am always surprised by how extremely scalable Redis is, scaling the stateless chat servers is trivial, scaling state is hard.

tuyguntn 6y ago
I feel exactly same, other than ease of scaling to 1000 instances I don't see anything special, everything is offloaded to Redis. Maybe example use case should have been different to showcase scaling ease

deleted 6y ago

[deleted]

fxtentacle 6y ago

Use a dedicated server and IRC. 250k users isn't that scary anymore. I remember that in 2012, the Rizon IRC network maxed out at 80,000 concurrent users on an AMD Bulldozer CPU.

pelorat 6y ago
Exactly. Numbers like these are actually quite pathetic when we look back 10 and even 20 years to IRC.

_jezell_ 6y ago

This is a really neat demo, but there's a few limitations that are glaring issues for a chat server or most scenarios I can think of that I would want to use websockets. As many have mentioned, the 250 connection limit per instance is crazy. You can get 200x that many easily on a single 1 VCPU box. The second problem is that really the demo has just kicked your scaling problem over to redis. The demo isn't doing anything actually interesting, redis is doing all the work. The reality is that one of the best parts of Cloud Run over something like serverless functions is that you can have state in your server. You don't need redis at all if you are doing the same demo in Kubernetes without Cloud Run, since it's pretty easy to use channels in golang to do the same thing. However, in order for you to really make that work, you need some way to at the very least route the traffic consistently so that people watching the same resource get routed to the same server, or to be able to have cross node communication. So this is a great start, but a little work to go before many people would be able to switch over to something like this due.