3 ms·

Total noob question here. I am not from an ops background... To paraphrase the docs, it's possible to manage a process and specify how many of that process I w

by acron0 13y ago

Total noob question here. I am not from an ops background... To paraphrase the docs, it's possible to manage a process and specify how many of that process I want to spawn (i.e. numprocesses = 5). I can also manage sockets and pass the managed sockets' fd to the managed processes. First question, what is a typical use case for wanting multiple instances of a specific process? Second question, if I am running multiple managed processes, how can they all bind to the same managed socket? I guess there are some fundamental gaps in my understanding.

tarekmoz 13y ago
> First question, what is a typical use case for wanting multiple instances of a specific process? distribute the load. For example I can run several "redis workers" that gets jobs to do in a redis queue. > Second question, if I am running multiple managed processes, how can they all bind to the same managed socket? if you bind a socket and then forks 10 child processes, they can all accept connections on that socket and let the operating system do the load-balancing. This is exactly how the Apache pre-fork model works.
- acron0 13y ago
  Awesome, thanks for answering my questions.
SEJeff 13y ago
Think of circus as a more modern and intelligent version of xinetd. It binds to the socket and then hands off requests to the workers. Here is why this is awesome... So we use circus to maintain about a dozen apps and a few of them are django apps running under Mozilla's chaussette wsgi server. When we do an autodeploy via jenkins, jenkins will run circusctl to stop that worker, replace the code, futz a symlink, and then start the worker again. We front the apps with apache 2.4 (because it is as fast as nginx) + mod_proxy. That gives us ldap access control through apache and shiney LTSV formatted logs that tools like logstash or splunk can easily consume. We then front apache with a virtual ip provided by a tool such as keepalived for moar web scales. For all in-flight requests, they will just hang as circus proper will hold onto the connection keeping the socket open. It will pass the socket to the first worker started once it is restarted. So during a deploy, a request might hang for a few milliseconds, but it never blows up. It is awesome in that it allows you 0 downtime deploys (at least for our django web stack it does). Circus is a super elegant piece of software. Kudos to Mozilla for finally making a release. I'd been bugging them for awhile: https://github.com/mozilla-services/circus/issues/561 https://github.com/mozilla-services/circus/issues/561 Edit: Example of a 0 downtime restart config using uwsgi (instead of chaussette): https://github.com/scottkmaxwell/circus/commit/ef0000fe4ef3c09553dd8069e459754839b3673a https://github.com/scottkmaxwell/circus/commit/ef0000fe4ef3c...

by acron0 13y ago

tarekmoz 13y ago

> First question, what is a typical use case for wanting multiple instances of a specific process? distribute the load. For example I can run several "redis workers" that gets jobs to do in a redis queue. > Second question, if I am running multiple managed processes, how can they all bind to the same managed socket? if you bind a socket and then forks 10 child processes, they can all accept connections on that socket and let the operating system do the load-balancing. This is exactly how the Apache pre-fork model works.

acron0 13y ago
Awesome, thanks for answering my questions.

SEJeff 13y ago

Think of circus as a more modern and intelligent version of xinetd. It binds to the socket and then hands off requests to the workers. Here is why this is awesome... So we use circus to maintain about a dozen apps and a few of them are django apps running under Mozilla's chaussette wsgi server. When we do an autodeploy via jenkins, jenkins will run circusctl to stop that worker, replace the code, futz a symlink, and then start the worker again. We front the apps with apache 2.4 (because it is as fast as nginx) + mod_proxy. That gives us ldap access control through apache and shiney LTSV formatted logs that tools like logstash or splunk can easily consume. We then front apache with a virtual ip provided by a tool such as keepalived for moar web scales. For all in-flight requests, they will just hang as circus proper will hold onto the connection keeping the socket open. It will pass the socket to the first worker started once it is restarted. So during a deploy, a request might hang for a few milliseconds, but it never blows up. It is awesome in that it allows you 0 downtime deploys (at least for our django web stack it does). Circus is a super elegant piece of software. Kudos to Mozilla for finally making a release. I'd been bugging them for awhile: https://github.com/mozilla-services/circus/issues/561 https://github.com/mozilla-services/circus/issues/561 Edit: Example of a 0 downtime restart config using uwsgi (instead of chaussette): https://github.com/scottkmaxwell/circus/commit/ef0000fe4ef3c09553dd8069e459754839b3673a https://github.com/scottkmaxwell/circus/commit/ef0000fe4ef3c...