3 ms·
Looks nice, but I would rather have the processing power local to me. I have used the python framework, octopy. It is a mapreduce framework for python. Caveat:
by brianobush 17y ago
Looks nice, but I would rather have the processing power local to me. I have used the python framework, octopy. It is a mapreduce framework for python. Caveat: I have only used it for trivial problems thus far.
http://code.google.com/p/octopy/ http://code.google.com/p/octopy/
- mbreese 17y agoOctopy looks far too simplistic for anything too large. For example, it pushes the entire python source file to the client, including all of the data. It then assigns a client data to process? I've been reading the code for a few minutes and can't imagine using it for anything substantial... I too prefer to have local processing, but there are times where I also like to farm out jobs to a larger HPC cluster (I'm at a university with 2 large clusters). The downside to this is that I usually have to wait in a job queue. I guess what I really want is a hybrid approach where I can run jobs locally, but if there are too many, spin up either a loadleveller / pbs job on a university cluster or spin up a few EC2 instances.