4 ms·
If it is just a backend, why not port it over to one of the myriad of cloud autoscaling solutions that are out there? The opportunity cost of spending time fig
by latchkey 5y ago
If it is just a backend, why not port it over to one of the myriad of cloud autoscaling solutions that are out there?
The opportunity cost of spending time figuring out why only 29 workers are receiving requests over adding new features that generate more revenue, seems like a quick decision.
Personally, I just start off with that now in the first place, the development load isn't any greater and the solutions that are out there are quite good.
- _tom_ 5y agoThey are on a back-end that does auto-scaling. They stated that they had problems when scaling up past 1000 nodes. Now, maybe they could have fixed that issue instead, but going from 29 to 58 workers is easy, it's not the same going to 29,000 to 58,000. And 1000 hosts vs 500 is a non-trivial cost.
- PaywallBuster 5y agocontainers would've solved it one process per container, easy peasy
- motoboi 5y agoyou now containers are just processes, right? This is what they did, but because they didn't need to schedule other jobs on the same machine, kubernetes or even docker would be overkill. In this case, simple VM orchestration seems like a fine solution.
- PaywallBuster 5y agoIndeed, but you wouldn't be thinking about instance sizes, how many processes per instance and wondering if you're hitting kernel limits with all the issues coming up
- motoboi 5y agoWhy not?
- zbentley 5y agoYou'd probably be worrying more about instance sizes if you ran a single executor per container; the memory overhead of your app would become a problem very quickly unless it's startup footprint was quite small.
- PaywallBuster 5y agoThat's what they're doing now. One app pool with one worker x number of cores
- zbentley 5y agoI assumed they're managing all those workers under one parent process which compiled their codebase on start. Perhaps that assumption was in error.
- Spivak 5y agoThis doesn’t work so easily with architectures with process pools for workers. So now your app server needs to speak docker (or whatever control plane) to spawn new workers and deal with more complicated IPC. Also the startup time is brutal. One process per container and multiprocessing is a huge lift most of the time. I’ve done it but it can be a mess because you don’t really have as much a handle on containers than subprocesses because you can only poke them at a distance through the control plane.
- zbentley 5y ago> One process per container and multiprocessing Do you mean multiprocessing inside the containers? Or are you managing multiprocessing child procs by forking into a container somehow? If the latter, I'd be really interested to learn how to do that; I didn't think it was possible, and it would be super useful for some of what I work on.
- PaywallBuster 5y agoThat's what they're doing now. One app pool with one worker x number of cores Wrapping it around a container makes no difference
- lddemi 5y agoAuthor here. We do and did use autoscaling heavily but at a certain scale we just ran out of headroom on the smaller instance types we were using. Jumping to a much larger instance types meant that we will likely never run into those headroom issues again, plus solves other problems like faster spin up, better sidecar connection pooling and allows for a much higher hit rate on per instance caching.
- TekMol 5y agoDid you consider switching form CPython to Pypy?
- latchkey 5y agoYou were autoscaling a single threaded process. You had 1000 connections coming in and scaling 1000 workers for those connections. Everything was filtered through gunicorn and nginx, which just adds additional latencies and complexity, for no real benefit. What I'm talking about is just pointing at something like AppEngine, Cloud Functions, etc... (or whatever solution AWS has that is similar) and being done with it. I'm talking about not running your own infrastructure, at all. Let AWS and Google be your devops so that you can focus on building features.
- ddorian43 5y agoNow you just 5x their costs.
- latchkey 5y agoNot if you do it right. a) you get to fire the devops person, which saves $150k+ a year. b) you add appropriate caching layers in front of everything. c) you spend time adding features, which generate revenue. I've done all of this before at scale. This whole case study was written about work I did [1]. Two devs, 3 months to release, first year was $80m gross revenue on $500/month cloud bills. Infinite scalability, zero devops. [1] https://cloud.google.com/customers/gearlaunch https://cloud.google.com/customers/gearlaunch