4 ms·
> Also, Python is not great for parallelism. I agree that cpython threads are not parallelism, but python still comes with built-in support for multiprocessing
by nurettin 7y ago
> Also, Python is not great for parallelism.
I agree that cpython threads are not parallelism, but python still comes with built-in support for multiprocessing and I've been earning my bread using that within the past two years, so unless you have to use in-process parallelism for some weird reason, python and your OS scheduler of choice has you covered there.
- logicchains 7y ago> have to use in-process parallelism for some weird reason Needing shared memory paralellism is a weird reason now? Pretty much any parallel algorithm that's not embarassingly parallel is going to perform better with threads able to share memory than with message passing between processes.
- nurettin 7y agoExactly, pretty much any parallel algorithm is embarrassingly parallel (gather or calculate a bunch of data, process it and merge it together) so I question the need for continuously needing in-process parallelism instead of solving it trivially.
- lmeyerov 7y agoPython is not great for fine-grained data parallelism (SIMD, GPU), which is increasingly the lion's share: non-starter for direct inline and pretty bad for DSLs. The result is runtime heroics for embedded ~dataframe DSLs (pyspark, rapids.ai) with high overhead. OTOH, those heroics do happen, and been OK so far. Accelerating differentiable programming is basically an extra transform layer on accelerating data parallel programming. Thankfully, our team writes zero raw OpenCL/CUDA nowadays and instead fairly dense dataframes code. Similar to async/await being added to doing a lot for web programming on Python, curious what it'll take for data parallel fragments (incl. differentiable.) If it wasn't for language resistance for UDF + overhead, and legacy libs around blocking, we'd be happy.
- mratsim 7y agoI'm not sure what's your domain but for machine learning that will be slow: - Parallel matrix multiplication may be embarassingly parallel, you have a reduction step that is not trivial to parallelize across processes. Also you need to take care of register tiling, L1 cache tiling and L2 cache tiling. It is way easier to do this in OpenMP - Parallel Monte Carlo Tree-Search: it's much easier and more efficient to spawn/collect trees with a proper spawn/sync librairie.
- 72deluxe 7y agoIsn't being interpreted in multiple threads incredibly slow?? Like single thread python, just multiplied. Slow. Is it because it's easy to write? Honestly I am curious as an outsider to the Python world.
- nurettin 7y agoComing from a complicated background (pascal, c++, ruby, typescript, delphi, kotlin) I'm pretty new to python myself. Python is dead in water for trivial things like parsing datetimes and floats, or writing complex algorithms. C++ pretty easily beats it by orders of magnitude. However, it is pretty good at other things such as (off the top of my mind) running a regex over a bunch of text to find subgroups, serializing/deserializing data, scipy's curve fitting, etc. So as long as you are using some module that calls natively compiled code, you are pretty much golden.
- 72deluxe 7y agoLooks like I'll stick to the wonderful world of C++ then! It seems that when a language has to resort to calling another language's compiled modules for speed, it has lost the fight for efficiency.