3 ms·
The most interesting part of this to me was something tangential: they use Redis Queues. Anyone have experience with this? Good or bad impressions? The documen
by sillysaurusx 6y ago
The most interesting part of this to me was something tangential: they use Redis Queues. Anyone have experience with this? Good or bad impressions?
The documentation is tantalizing, but hilariously short: https://devcenter.heroku.com/articles/python-rq https://devcenter.heroku.com/articles/python-rq
Very "And then draw the rest of the owl." Oh really, you can just do `from utils import count_words_at_url;
q.enqueue(count_words_at_url, 'http://heroku.com')` http://heroku.com')` and presto, your blocking function -- whose source code exists locally -- is run successfully at the other end?
I'll have to set aside some time to try this out. Python does have introspection facilities that could make that possible. I could imagine that since the code is executed on the same box, it's relatively simple to send a request like "here's which module the function was loaded from; here's the order all modules were loaded in; load those modules and call this function." But it leaves so many questions: serialization, performance, scaling, and all the tiny bugs that inevitably come up.
I guess I was hoping someone could give me a quick gut check of positive/negative reactions. The full RQ documentation is slightly better: https://python-rq.org/docs/ https://python-rq.org/docs/ but has some worrying signs:
Make sure that the function call does not depend on its context. In particular, global variables are evil (as always), but also any state that the function depends on (for example a “current” user or “current” web request) is not there when the worker will process it. If you want work done for the “current” user, you should resolve that user to a concrete instance and pass a reference to that user object to the job as an argument.
Yes, sure, global variables are the root of satan, but they're also a fact of life in many scenarios.
Interesting approach... I wonder how much of a nightmare it makes devops...
- ashrodan 6y agoI've used it for small projects and it's okay. Rq is quite simple and limited in a sense. I guess what I'd recommend to Mozilla, although fantastic work, is that's it worth trying to compute certain features (data points) upfront to simplify the main model. These calculated features can be used as inputs to the main model possibly reducing the response time.
- kkaranth 6y agoI've used it a bit in production. Our use case avoided a lot of the potential issues you mentioned, so it may not be entirely helpful: * serialization: the input was passed a json string argument. the output was a file uploaded to S3, so just the URL was returned, again in a JSON string * global variables: the program was quite self-contained: there was an initial state setup that was not mutated afterwards. So RQ's fork-exec model(the default) worked well enough Sorry I don't have much to say about performance and scaling. It was quite fine for our needs, and we could scale horizontally upto a certain point by just starting extra processes, and beyond that with more VMs. Since they all listened on the same RQ, it worked fine. (the number of items in our queue never really hit any of Redis' limitations either) RQ lets you customize the worker model: so you could for instance use threads instead of processes. Regarding monitoring: there's RQ dashboard[1] which gives a nice web interface to view jobs, failures, and restart them. 1: https://python-rq.org/docs/monitoring/ https://python-rq.org/docs/monitoring/
- sillysaurusx 6y agoThank you very much for the detailed response :) I really appreciate the thoroughness; it convinced me to use RQ.
- sillysaurusx 6y ago+1 datapoint in favor of rq: https://twitter.com/sdan_io/status/1285687026386444288 https://twitter.com/sdan_io/status/1285687026386444288 was built with it.