7 ms·
Wouldn't it make a lot of sense to just use Pyspark with RDDs? Latency would be relatively high, but it'd also bypass the GIL while also being more modern.
by rev_d 10y ago
Wouldn't it make a lot of sense to just use Pyspark with RDDs? Latency would be relatively high, but it'd also bypass the GIL while also being more modern.
- mangecoeur 10y agoIn my experience pyspark is much more flaky and annoying that doing parallel computing with more 'python native' tools. It only really makes sense when you outgrown small clusters and really need huge infrastructure.
- splike 10y agoWhat python tools do you use for small clusters?
- elyase 10y agoDask would be an option.
- mangecoeur 10y agoWas going to say that. Or ipython parallel if you want to go lower level