5 ms·
> The solution to Python’s GIL bottleneck is not some trick, it is to stop using Python for data-path code. What about using pytorch multiprocessing[1]? [1] h
by O5vYtytb 6y ago
> The solution to Python’s GIL bottleneck is not some trick, it is to stop using Python for data-path code.
What about using pytorch multiprocessing[1]?
[1] https://pytorch.org/docs/stable/notes/multiprocessing.html https://pytorch.org/docs/stable/notes/multiprocessing.html
- amelius 6y agoI can't attest to the usefulness of pytorch's multiprocessing module, but using python's multiprocessing module feels like low-level programming (serializing, packing and unpacking data-structures, etc. where you'd hope the environment would handle it for you).
- modeless 6y agoI found python multiprocessing to work well to parallelize deep learning data loading and preprocessing, because all I needed to communicate was a couple of tensors which are easy to allocate in shared memory. I didn't need complex data structures or synchronization.
- threatripper 6y agoProcessing separate video streams works well with separate processes. There is some cost related to starting the other processes and sometimes libraries may stumble (e.g. several instances of ML libraries allocating all the GPU memory) but once it's running it's literally two separate processes that can do their work independently. Multiprocessing could be a pain if you need to pass frames of a single video stream. Traditionally you'd need to pickle/unpickle them to pass them between processes.