4 ms·

Could you elaborate, e.g. link to documentation, subprojects or example code? I'd love to get rid of Celery because of how difficult it is for me to tweak it fo

by d33 6y ago

Could you elaborate, e.g. link to documentation, subprojects or example code? I'd love to get rid of Celery because of how difficult it is for me to tweak it for good performance.

dzonga 6y ago
you can also use dramatiq
darkerside 6y ago
Would love to see more detail as well because I'm a bit skeptical. From uWSGI documentation, it looks like to coordinate cron across anything more than a single server setup, you'd need to configure a Legion, which means you're then integrating uWSGI's orchestration framework with whatever you're already using (k8, ECS, etc). I like minimalism, but sometimes batteries are included for a reason.
- GlennS 6y ago
  I suspect they are using a single server. I found uwsgi's mules and cache2 very useful in that situation. If someone finds that Redis and Celery are more complication than they need for a given task, then I think they're probably not using an orchestration framework.
GlennS 6y ago
This is what I have used in the past. I think they're very convenient if you are just running one instance of uwsgi and want to share some state quickly without going through a persistent database. If you're worrying about tweaking Celery for performance, then I suspect your uses may be a bit more complex than uwsgi's mules are designed for though. https://uwsgi-docs.readthedocs.io/en/latest/Caching.html https://uwsgi-docs.readthedocs.io/en/latest/Caching.html https://uwsgi-docs.readthedocs.io/en/latest/Mules.html https://uwsgi-docs.readthedocs.io/en/latest/Mules.html The cache is a simple key-value store. Works well. I later swapped it out for Redis, because I needed to shared my cache between multiple machines. Switching to Redis was a very quick and easy replacement, so you don't need to worry about lock in. I used mules in a couple of ways. I had some background task mules which mostly just ran in a loop with a `sleep()` call. An example is deleting old records from a database once a day. Another example is listening to an Amazon SQS queue for files being uploaded to an S3 bucket. I also had some mules which were triggered by web requests. These are normally what you would use "farms" for. The web request sends a farm message, and a mule picks it up and acts on it. For example, I used this for sending webhook callbacks in response to certain web requests. You could probably also combine this with uwsgi's async mode. That would be useful if you needed a web request to wait for a long running task to finish before sending a response back. I handled that kind of situation with the aforementioned webhook callbacks instead. A sibling comment has mentioned Legion. I've never used that, so can't comment on whether or not the caching and messaging works together with that.

by d33 6y ago

Could you elaborate, e.g. link to documentation, subprojects or example code? I'd love to get rid of Celery because of how difficult it is for me to tweak it for good performance.

dzonga 6y ago

you can also use dramatiq

darkerside 6y ago

Would love to see more detail as well because I'm a bit skeptical. From uWSGI documentation, it looks like to coordinate cron across anything more than a single server setup, you'd need to configure a Legion, which means you're then integrating uWSGI's orchestration framework with whatever you're already using (k8, ECS, etc). I like minimalism, but sometimes batteries are included for a reason.

GlennS 6y ago
I suspect they are using a single server. I found uwsgi's mules and cache2 very useful in that situation. If someone finds that Redis and Celery are more complication than they need for a given task, then I think they're probably not using an orchestration framework.

GlennS 6y ago

This is what I have used in the past. I think they're very convenient if you are just running one instance of uwsgi and want to share some state quickly without going through a persistent database. If you're worrying about tweaking Celery for performance, then I suspect your uses may be a bit more complex than uwsgi's mules are designed for though. https://uwsgi-docs.readthedocs.io/en/latest/Caching.html https://uwsgi-docs.readthedocs.io/en/latest/Caching.html https://uwsgi-docs.readthedocs.io/en/latest/Mules.html https://uwsgi-docs.readthedocs.io/en/latest/Mules.html The cache is a simple key-value store. Works well. I later swapped it out for Redis, because I needed to shared my cache between multiple machines. Switching to Redis was a very quick and easy replacement, so you don't need to worry about lock in. I used mules in a couple of ways. I had some background task mules which mostly just ran in a loop with a `sleep()` call. An example is deleting old records from a database once a day. Another example is listening to an Amazon SQS queue for files being uploaded to an S3 bucket. I also had some mules which were triggered by web requests. These are normally what you would use "farms" for. The web request sends a farm message, and a mule picks it up and acts on it. For example, I used this for sending webhook callbacks in response to certain web requests. You could probably also combine this with uwsgi's async mode. That would be useful if you needed a web request to wait for a long running task to finish before sending a response back. I handled that kind of situation with the aforementioned webhook callbacks instead. A sibling comment has mentioned Legion. I've never used that, so can't comment on whether or not the caching and messaging works together with that.