6 ms·
I really want to use Python in a distributed computation project of mine, but I've been running into serious performance bottlenecks with regards to object seri
by omginternets 11y ago
I really want to use Python in a distributed computation project of mine, but I've been running into serious performance bottlenecks with regards to object serialization. Like many concurrent Python frameworks, Pulsar uses `pickle` to serialize data across processes, and for all but the smallest data structures, workers end up spending 95+% of their time (de)serializing data.
I work with video data, so I just ran these rough benchmarks on my MacBook Pro:
import cPickle as pickle # yes, I'm on Python 2.7. Sue me.
import numpy as np
a = np.random.rand(800, 600, 3) # 800x600 px, RGB channel
%timeit pickle.dumps(a) // %timeit magic from IPython
1 loops, best of 3: 855 ms per loop
It takes nearly a second to serialize a not-so-huge `numpy` array, which makes it very difficult to do any sort of soft real-time analysis.
This is a huge pain, and (very sadly for this Python aficionado) suggests that Python might be the wrong language for this kind of work.
Any suggestions?
- aidos 11y agoFor comparison, I get the same sort of speed on my system: %timeit p = pickle.dumps(a) 1 loops, best of 3: 1.14 s per loop But tostring and fromstring run in a fraction of the time. %timeit s = a.tostring() 1000 loops, best of 3: 1.13 ms per loop %timeit b = np.fromstring(s).reshape(800, 600, 3) 1000 loops, best of 3: 1.57 ms per loop np.all(a == b) True
- hcrisp 11y agoPass the data by compressed, binary file using something like bcolz or h5py?
- Cieplak 11y agoCheck out the performance of these serializers: https://github.com/eishay/jvm-serializers/wiki https://github.com/eishay/jvm-serializers/wiki Although they're mostly limited to Java, I know at least one (Avro) can be used in Python.
- omginternets 11y agoArvo seems pretty nifty, indeed. I'll see how far I get using higher protocol versions with pickle, but I'm bookmarking this for a later date. Thanks!
- sepeth 11y agoYou should use highest protocol of pickle, here are the numbers in my machine: In [5]: %timeit pickle.dumps(a) 1 loops, best of 3: 724 ms per loop In [6]: %timeit pickle.dumps(a, protocol=pickle.HIGHEST_PROTOCOL) 100 loops, best of 3: 12.4 ms per loop
- omginternets 11y agoOh wow! This is more than I dared hope for, many thanks!
- darkxanthos 11y agoWhat's the trade off here?
- AnkhMorporkian 11y agoBackwards compatibility. Anything pickled with HIGHEST_PROTOCOL will be unreadable in anything below Python 2.3.
- robzyb 11y agoSo my Ford Model T won't be able to read it? (That is my humorous way of saying that Python 2.3 is rather old and therefore HIGHEST_PROTOCOL seems like it would be a desirable trade off for most.)
- StavrosK 11y agoYeah, what the hell? Even 2.5 is way too old, I generally only support 2.6+ nowadays. If something breaks compatibility with 2.5, no problem there. Hell, Python 3 has been out for years!
- AnkhMorporkian 11y agoHey, I'm with you. I use 3.4 on almost everything, and 2.7 at worst anywhere else. I weep for those stuck using Python 2.2.
- stuaxo 11y agoThis interesting, I'm also looking for an Actor framework for python to do work with video. It would be good if this could use a nanomsg and its zero-copy mechanism instead of just raw sockets.
- Wicher 11y agoMaybe you can use shared memory. You could use raw mmap.mmap(), or use the sharedctypes multiprocessing submodule (https://docs.python.org/2/library/multiprocessing.html#module-multiprocessing.sharedctypes https://docs.python.org/2/library/multiprocessing.html#modul...) . You'd allocate memory in the parent process and then you'd write to that memory from within the child processes. The Python function you're "multiprocessing" won't necessarily need to return anything.