4 ms·
A quick summary, from the slides: The mechanism PyParallel uses is that it still has the one, main thread with GIL. It also has parallel threads with their own
by ghusbands 12y ago
A quick summary, from the slides:
The mechanism PyParallel uses is that it still has the one, main thread with GIL. It also has parallel threads with their own heap and a trivial pointer-incrementing allocator in a thread-local pool that deallocates on completion by dropping the whole pool. Either the main thread is running, in typical Python fashion, or some parallel threads are running. When parallel threads are running, the main Python heap is all marked read-only and all reference counting is disabled. I believe communication from parallel threads to the main thread is done by sending serialised objects.
- dorfsmay 12y agoI tend to do that with multiprocessing queues, but at some point the serialisation becomes the bottleneck. Another way is to use a faster serialiser (marshal, ujson) and write them to files on SSDs with flock, but again a trick that's slower than real datasharing like java and go.
- sriku 12y agoGood summary ... which highlights Greenspun's tenth rule as applied to Erlang.
- trentnelson 12y agoInfoQ did a surprisingly accurate summation, too: http://www.infoq.com/articles/PyParallel http://www.infoq.com/articles/PyParallel > I believe communication from parallel threads to the main thread is done by sending serialised objects. That's one of the open areas of exploration... how do you share things in a shared-nothing approach? This isn't as important for stateless I/O-driven applications (like an HTTP server), but it becomes very important when trying to leverage PyParallel for, well, exactly that, parallel computation.