7 ms·

Pulsar: Concurrent framework for Python

95 points by bsg75 11y ago

calebm 11y ago
How does it compare/contrast with Tornado?
- jonesetc 11y ago
  Looks like an actor framework, built directly on top of stdlib asyncio, and is designed to use multiprocessing first instead of tornado's cooperative multitasking.
dikaiosune 11y ago
Is this related to the fibers/channels/actor framework by the same name for Clojure? https://github.com/puniverse/pulsar https://github.com/puniverse/pulsar
- klibertp 11y ago
  Good question. The name is the same and goals of both projects are similar. Implementation is very different, though.
omginternets 11y ago
I really want to use Python in a distributed computation project of mine, but I've been running into serious performance bottlenecks with regards to object serialization. Like many concurrent Python frameworks, Pulsar uses `pickle` to serialize data across processes, and for all but the smallest data structures, workers end up spending 95+% of their time (de)serializing data. I work with video data, so I just ran these rough benchmarks on my MacBook Pro: import cPickle as pickle # yes, I'm on Python 2.7. Sue me. import numpy as np a = np.random.rand(800, 600, 3) # 800x600 px, RGB channel %timeit pickle.dumps(a) // %timeit magic from IPython 1 loops, best of 3: 855 ms per loop It takes nearly a second to serialize a not-so-huge `numpy` array, which makes it very difficult to do any sort of soft real-time analysis. This is a huge pain, and (very sadly for this Python aficionado) suggests that Python might be the wrong language for this kind of work. Any suggestions?
- aidos 11y ago
  For comparison, I get the same sort of speed on my system: %timeit p = pickle.dumps(a) 1 loops, best of 3: 1.14 s per loop But tostring and fromstring run in a fraction of the time. %timeit s = a.tostring() 1000 loops, best of 3: 1.13 ms per loop %timeit b = np.fromstring(s).reshape(800, 600, 3) 1000 loops, best of 3: 1.57 ms per loop np.all(a == b) True
- hcrisp 11y ago
  Pass the data by compressed, binary file using something like bcolz or h5py?
- Cieplak 11y ago
  Check out the performance of these serializers: https://github.com/eishay/jvm-serializers/wiki https://github.com/eishay/jvm-serializers/wiki Although they're mostly limited to Java, I know at least one (Avro) can be used in Python.
  - omginternets 11y ago
    Arvo seems pretty nifty, indeed. I'll see how far I get using higher protocol versions with pickle, but I'm bookmarking this for a later date. Thanks!
- sepeth 11y ago
  You should use highest protocol of pickle, here are the numbers in my machine: In [5]: %timeit pickle.dumps(a) 1 loops, best of 3: 724 ms per loop In [6]: %timeit pickle.dumps(a, protocol=pickle.HIGHEST_PROTOCOL) 100 loops, best of 3: 12.4 ms per loop
  - omginternets 11y ago
    Oh wow! This is more than I dared hope for, many thanks!
  - darkxanthos 11y ago
    What's the trade off here?
sciurus 11y ago
Previously: https://news.ycombinator.com/item?id=6543266 https://news.ycombinator.com/item?id=6543266
ignoramous 11y ago
I am not sure whether Pulsar would anywhere achieve the speed / throughout of a truly concurrent system like JVM, Go, Haskell, and Erlang. I don't see benchmarks posted on Pulsar's website, but I came across this discussion [1] where aphyr and KiranDave discuss about why Clojure/Erlang's model of explicit concurrency (which enabled parallelism) is superior to NodeJs' (implicit concurrency via libuv, and multi-process parallelism via clusters). From reading the "design" section on the website, to me it looks like Pulsar is an attempt to replicate NodeJs (?) and by extension cannot compete with languages with truly concurrent runtimes? [1] https://news.ycombinator.com/item?id=4306241 https://news.ycombinator.com/item?id=4306241
- scribu 11y ago
  Great link - thanks for posting! Pulsar can be configured to use either processes or threads. If you use processes, you pay the IPC penalty, just like NodeJS does. If you use threads, you pay the GIL penalty. So yeah, it won't be as fast as BEAM or the JVM either way.
ivoras 11y ago
Oh, another one. Doesn't the fact that there is a new concurrent framework for Python nearly every single month point to the existence of a rather large pachyderm in the room?
lsbardel 11y ago
Pulsar is 100% written on top of asyncio. In this respect is not dissimilar to Tornado or Twisted, which also use an event loop for their asynchronous implementation (with their own event loop and Future classes). However, pulsar is built on top of the standard lib with all the benefits that that brings, especially in view of the changes in python 3.5 (https://www.python.org/dev/peps/pep-0492/ https://www.python.org/dev/peps/pep-0492/). The actor model in pulsar refers to the parallel side of the asynchronous framework. This is where pulsar differs from twisted for example. In pulsar each actor (think of a specialised thread or process) has its own event loop. In this way any actor can run its own asynchronous server for example. Tornado is an asynchronous web framework, pulsar is not, you can use any web framework and run it on pulsar wsgi application. You can use pulsar to create any other socket application, not just HTTP.

95 points by bsg75 11y ago

calebm 11y ago

How does it compare/contrast with Tornado?

jonesetc 11y ago
Looks like an actor framework, built directly on top of stdlib asyncio, and is designed to use multiprocessing first instead of tornado's cooperative multitasking.

dikaiosune 11y ago

Is this related to the fibers/channels/actor framework by the same name for Clojure? https://github.com/puniverse/pulsar https://github.com/puniverse/pulsar

klibertp 11y ago
Good question. The name is the same and goals of both projects are similar. Implementation is very different, though.

omginternets 11y ago

I really want to use Python in a distributed computation project of mine, but I've been running into serious performance bottlenecks with regards to object serialization. Like many concurrent Python frameworks, Pulsar uses `pickle` to serialize data across processes, and for all but the smallest data structures, workers end up spending 95+% of their time (de)serializing data. I work with video data, so I just ran these rough benchmarks on my MacBook Pro: import cPickle as pickle # yes, I'm on Python 2.7. Sue me. import numpy as np a = np.random.rand(800, 600, 3) # 800x600 px, RGB channel %timeit pickle.dumps(a) // %timeit magic from IPython 1 loops, best of 3: 855 ms per loop It takes nearly a second to serialize a not-so-huge `numpy` array, which makes it very difficult to do any sort of soft real-time analysis. This is a huge pain, and (very sadly for this Python aficionado) suggests that Python might be the wrong language for this kind of work. Any suggestions?

aidos 11y ago
For comparison, I get the same sort of speed on my system: %timeit p = pickle.dumps(a) 1 loops, best of 3: 1.14 s per loop But tostring and fromstring run in a fraction of the time. %timeit s = a.tostring() 1000 loops, best of 3: 1.13 ms per loop %timeit b = np.fromstring(s).reshape(800, 600, 3) 1000 loops, best of 3: 1.57 ms per loop np.all(a == b) True
hcrisp 11y ago
Pass the data by compressed, binary file using something like bcolz or h5py?
Cieplak 11y ago
Check out the performance of these serializers: https://github.com/eishay/jvm-serializers/wiki https://github.com/eishay/jvm-serializers/wiki Although they're mostly limited to Java, I know at least one (Avro) can be used in Python.
- omginternets 11y ago
  Arvo seems pretty nifty, indeed. I'll see how far I get using higher protocol versions with pickle, but I'm bookmarking this for a later date. Thanks!
sepeth 11y ago
You should use highest protocol of pickle, here are the numbers in my machine: In [5]: %timeit pickle.dumps(a) 1 loops, best of 3: 724 ms per loop In [6]: %timeit pickle.dumps(a, protocol=pickle.HIGHEST_PROTOCOL) 100 loops, best of 3: 12.4 ms per loop
- omginternets 11y ago
  Oh wow! This is more than I dared hope for, many thanks!
- darkxanthos 11y ago
  What's the trade off here?

sciurus 11y ago

Previously: https://news.ycombinator.com/item?id=6543266 https://news.ycombinator.com/item?id=6543266

ignoramous 11y ago

I am not sure whether Pulsar would anywhere achieve the speed / throughout of a truly concurrent system like JVM, Go, Haskell, and Erlang. I don't see benchmarks posted on Pulsar's website, but I came across this discussion [1] where aphyr and KiranDave discuss about why Clojure/Erlang's model of explicit concurrency (which enabled parallelism) is superior to NodeJs' (implicit concurrency via libuv, and multi-process parallelism via clusters). From reading the "design" section on the website, to me it looks like Pulsar is an attempt to replicate NodeJs (?) and by extension cannot compete with languages with truly concurrent runtimes? [1] https://news.ycombinator.com/item?id=4306241 https://news.ycombinator.com/item?id=4306241

scribu 11y ago
Great link - thanks for posting! Pulsar can be configured to use either processes or threads. If you use processes, you pay the IPC penalty, just like NodeJS does. If you use threads, you pay the GIL penalty. So yeah, it won't be as fast as BEAM or the JVM either way.

ivoras 11y ago

Oh, another one. Doesn't the fact that there is a new concurrent framework for Python nearly every single month point to the existence of a rather large pachyderm in the room?

lsbardel 11y ago

Pulsar is 100% written on top of asyncio. In this respect is not dissimilar to Tornado or Twisted, which also use an event loop for their asynchronous implementation (with their own event loop and Future classes). However, pulsar is built on top of the standard lib with all the benefits that that brings, especially in view of the changes in python 3.5 (https://www.python.org/dev/peps/pep-0492/ https://www.python.org/dev/peps/pep-0492/). The actor model in pulsar refers to the parallel side of the asynchronous framework. This is where pulsar differs from twisted for example. In pulsar each actor (think of a specialised thread or process) has its own event loop. In this way any actor can run its own asynchronous server for example. Tornado is an asynchronous web framework, pulsar is not, you can use any web framework and run it on pulsar wsgi application. You can use pulsar to create any other socket application, not just HTTP.