3 ms·
I wish machine learning would have become mainstream on a language with more competent multithreading capabilities than python. During my machine learning cours
by bno1 6y ago
I wish machine learning would have become mainstream on a language with more competent multithreading capabilities than python. During my machine learning course I knew I could squeeze more performance out of my code by parallelising data preprocessing and training (pytorch), but python cannot do proper multithreading. The multiprocessing module requires you to move data between processes, which is slow.
- cosmodisk 6y agoWould porting certain libraries be a huge undertaking?
- bno1 6y agoWell the core of those libraries is already written in C or C++, so it's a matter of writing bindings to interface with the core. I think tensorflow and pytorch today have some C/C++ bindings, but I don't know how good they are.
- rpedela 6y agoShouldn't be slow if shared memory is used. https://docs.python.org/3/library/multiprocessing.shared_memory.html https://docs.python.org/3/library/multiprocessing.shared_mem...
- bno1 6y agoFirst time I've seen this, looks like its brand new (python 3.8). The problem now is that you have to serialize/deserialize your data and modify your numpy/pytorch code to use the shared memory. It's an improvement in performance, true, but not as fast and easy as just sharing variables between threads.
- deleted 6y ago[deleted]
- deleted 6y ago[deleted]