3 ms·

As a proof of concept, this is pretty neat! This bit of code in worker.py makes me uncomfortable though: # Deserialize the task d_fun, d_args = dill.l

As a proof of concept, this is pretty neat! This bit of code in worker.py makes me uncomfortable though: # Deserialize the task d_fun, d_args = dill.loads(data) # Run the task d_fun(*d_args) Basically the function (as in the Python object that holds the function's code, etc) to run gets pulled out of redis along with its arguments, then it gets executed. If the entire system is trusted, this is a really cool way to let workers run code without needing to deploy it to them. Likewise, you wouldn't even need to restart the workers for a new version of the function to run - just start feeding the new code into the queue and the workers will run it. The more I think about this, the cooler it seems. On the other hand, from the worker's perspective, this is basically a wide-open "feed me any code and I'll run it" thing.

aidos 9y ago
That sort of behaviour definitely comes with some caveats, but probably works pretty well in a closed ecosystem. A few years back we were using a service called PiCloud that effectively allowed you to do that. It went a whole lever deeper and actually bundled and shipped your local code to the remote runner so you could actually run a single function from your codebase remotely. It was a one of the best products I've used but a) their service was unreliable and b) the got acqui-hired (by Dropbox, I think). It was something simple like: import cloud def my_long_fun(): ..... cloud.run(my_long_fun) and all the deps and everything would be shipped. Next time you called a function, if it already had everything it would be able to run remotely without shipping again.
- hcrisp 9y ago
  This still exists as cloudpickle, I believe? import cloudpickle pkl = cloudpickle.dumps(my_long_fun) # Ship it somewhere new_fun = cloudpickle.loads(pkl) new_fun(...) https://github.com/cloudpipe/cloudpickle https://github.com/cloudpipe/cloudpickle
  - xapata 9y ago
    Holy Moly, thanks for pointing that out!
- xapata 9y ago
  > PiCloud ... one of the best products I've used It was so wonderful. And I was so disappointed that Multyvac (http://www.multyvac.com/ http://www.multyvac.com/) dropped the ball. I thought I'd be able to continue using the service ... BTW, Multyvac has one of the more hilarious "trusted by" sections I've seen on the internet.
  - aidos 9y ago
    Aaron told me they were shutting down the service shortly before I was due to go on holiday. I looked at the multivac thing and decided it was just to risky to assume they would actually pick it up (especially given their Trusted By section!) I spent a couple of weeks replacing it with celery, and then later RQ. It was the correct call, but you never know with these things. As far as it goes it was a pretty cheap lesson in not relying on external providers. Having said that, I've spent a fortune on EC2 since because I can't have micro billing like I could with picloud!
omegote 9y ago
Yep, you definitely need to secure your system in order to avoid the worker running malicious code. Now that you mention it, I've left a test vps running this on the wide and now I'm laying on the bed about to fall asleep, too tired to shut it down. Hope noone detects the redis server running and sends some simple task with os.system('rm -rf /') :D

aidos 9y ago

That sort of behaviour definitely comes with some caveats, but probably works pretty well in a closed ecosystem. A few years back we were using a service called PiCloud that effectively allowed you to do that. It went a whole lever deeper and actually bundled and shipped your local code to the remote runner so you could actually run a single function from your codebase remotely. It was a one of the best products I've used but a) their service was unreliable and b) the got acqui-hired (by Dropbox, I think). It was something simple like: import cloud def my_long_fun(): ..... cloud.run(my_long_fun) and all the deps and everything would be shipped. Next time you called a function, if it already had everything it would be able to run remotely without shipping again.

hcrisp 9y ago
This still exists as cloudpickle, I believe? import cloudpickle pkl = cloudpickle.dumps(my_long_fun) # Ship it somewhere new_fun = cloudpickle.loads(pkl) new_fun(...) https://github.com/cloudpipe/cloudpickle https://github.com/cloudpipe/cloudpickle
- xapata 9y ago
  Holy Moly, thanks for pointing that out!
xapata 9y ago
> PiCloud ... one of the best products I've used It was so wonderful. And I was so disappointed that Multyvac (http://www.multyvac.com/ http://www.multyvac.com/) dropped the ball. I thought I'd be able to continue using the service ... BTW, Multyvac has one of the more hilarious "trusted by" sections I've seen on the internet.
- aidos 9y ago
  Aaron told me they were shutting down the service shortly before I was due to go on holiday. I looked at the multivac thing and decided it was just to risky to assume they would actually pick it up (especially given their Trusted By section!) I spent a couple of weeks replacing it with celery, and then later RQ. It was the correct call, but you never know with these things. As far as it goes it was a pretty cheap lesson in not relying on external providers. Having said that, I've spent a fortune on EC2 since because I can't have micro billing like I could with picloud!

omegote 9y ago

Yep, you definitely need to secure your system in order to avoid the worker running malicious code. Now that you mention it, I've left a test vps running this on the wide and now I'm laying on the bed about to fall asleep, too tired to shut it down. Hope noone detects the redis server running and sends some simple task with os.system('rm -rf /') :D