3 ms·

What are the differences between this and Disco?

by IndianAstronaut 12y ago

detaro 12y ago
Disco seems to be purely a MapReduce framework?
- vtuulos 12y ago
  You can use Disco to execute arbitrary Python tasks or workflows - it is not limited to MapReduce. Disco was designed for data-intensive workloads. It wouldn't make much sense to use it to supervise Python servers/services, for instance.
- rcarmo 12y ago
  Not just that. Disco also provides a distributed storage service for locality of reference when doing Map-Reduce. This, in turn, might be something you'd use to build a tailored distributed system where you'd call out to Python (and its ecosystem, like Pandas, etc.) to do your own data processing. My personal interest in it is that it uses LISP Flavored Erlang (lfe.io) and might therefore be nice to use with Hy (hylang.org)
  - oubiwann 12y ago
    Nice that you mention Pandas ... it's on the list for being supported by lsci (which in turn will be using LFE py). As for Hy: LFE py comes with custom ErlPort encoders/decoders, and I came this close to writing those in Hy. However, I did want anyone with a Python background to be able to easily read and contribute, not just Lispers ;-) Regardless, it would be nice to use with Hy (and probably trivial: a requirements.txt update, and then adding some Hy modules in ./python/lfe, at which point you should be able to access them from LFE).
- oubiwann 12y ago
  Disco has gone beyond MR with pipelines, too. This allows for more generalized workflows. Great stuff.
oubiwann 12y ago
The points made below about Disco a great. Similar confusion came up on Twitter with regard to CloudI. As a result of that, I updated the README with clarification on what LFE py is not for, and what you might use in those cases. Disco is brilliant stuff (I greatly prefer to to Hadoop), and the fact that it has a distributed file system should really clarify its use cases (any why one would choose to use it over something life LFE py which doesn't have one!). (As you can see, I can't help myself from saying great things about Disco :-))

What are the differences between this and Disco?

detaro 12y ago

Disco seems to be purely a MapReduce framework?

vtuulos 12y ago
You can use Disco to execute arbitrary Python tasks or workflows - it is not limited to MapReduce. Disco was designed for data-intensive workloads. It wouldn't make much sense to use it to supervise Python servers/services, for instance.
rcarmo 12y ago
Not just that. Disco also provides a distributed storage service for locality of reference when doing Map-Reduce. This, in turn, might be something you'd use to build a tailored distributed system where you'd call out to Python (and its ecosystem, like Pandas, etc.) to do your own data processing. My personal interest in it is that it uses LISP Flavored Erlang (lfe.io) and might therefore be nice to use with Hy (hylang.org)
- oubiwann 12y ago
  Nice that you mention Pandas ... it's on the list for being supported by lsci (which in turn will be using LFE py). As for Hy: LFE py comes with custom ErlPort encoders/decoders, and I came this close to writing those in Hy. However, I did want anyone with a Python background to be able to easily read and contribute, not just Lispers ;-) Regardless, it would be nice to use with Hy (and probably trivial: a requirements.txt update, and then adding some Hy modules in ./python/lfe, at which point you should be able to access them from LFE).
oubiwann 12y ago
Disco has gone beyond MR with pipelines, too. This allows for more generalized workflows. Great stuff.

oubiwann 12y ago

The points made below about Disco a great. Similar confusion came up on Twitter with regard to CloudI. As a result of that, I updated the README with clarification on what LFE py is not for, and what you might use in those cases. Disco is brilliant stuff (I greatly prefer to to Hadoop), and the fact that it has a distributed file system should really clarify its use cases (any why one would choose to use it over something life LFE py which doesn't have one!). (As you can see, I can't help myself from saying great things about Disco :-))