4 ms·
That sort of behaviour definitely comes with some caveats, but probably works pretty well in a closed ecosystem. A few years back we were using a service calle
by aidos 9y ago
That sort of behaviour definitely comes with some caveats, but probably works pretty well in a closed ecosystem.
A few years back we were using a service called PiCloud that effectively allowed you to do that. It went a whole lever deeper and actually bundled and shipped your local code to the remote runner so you could actually run a single function from your codebase remotely. It was a one of the best products I've used but a) their service was unreliable and b) the got acqui-hired (by Dropbox, I think).
It was something simple like:
import cloud
def my_long_fun():
.....
cloud.run(my_long_fun)
and all the deps and everything would be shipped. Next time you called a function, if it already had everything it would be able to run remotely without shipping again.
- hcrisp 9y agoThis still exists as cloudpickle, I believe? import cloudpickle pkl = cloudpickle.dumps(my_long_fun) # Ship it somewhere new_fun = cloudpickle.loads(pkl) new_fun(...) https://github.com/cloudpipe/cloudpickle https://github.com/cloudpipe/cloudpickle
- xapata 9y agoHoly Moly, thanks for pointing that out!
- xapata 9y ago> PiCloud ... one of the best products I've used It was so wonderful. And I was so disappointed that Multyvac (http://www.multyvac.com/ http://www.multyvac.com/) dropped the ball. I thought I'd be able to continue using the service ... BTW, Multyvac has one of the more hilarious "trusted by" sections I've seen on the internet.
- aidos 9y agoAaron told me they were shutting down the service shortly before I was due to go on holiday. I looked at the multivac thing and decided it was just to risky to assume they would actually pick it up (especially given their Trusted By section!) I spent a couple of weeks replacing it with celery, and then later RQ. It was the correct call, but you never know with these things. As far as it goes it was a pretty cheap lesson in not relying on external providers. Having said that, I've spent a fortune on EC2 since because I can't have micro billing like I could with picloud!