3 ms·
Dask looks really cool, I hope I remember it exists next time I need it. I've been pretty baffled, and disappointed, by how bad Python is at parallel processin
by mastax 4y ago
Dask looks really cool, I hope I remember it exists next time I need it.
I've been pretty baffled, and disappointed, by how bad Python is at parallel processing. Yeah, yeah, I know: The GIL. But so much time and effort has been spent engineering around every other flaw in Python and yet this part is still so bad. I've tried every "easy to use" parallelism library that gets recommended and none of them has satisfied. Always: "couldn't pickle this function" or spawning loads of processes that use up all my RAM for no visible reason but don't use any CPU or make any indication of progress. I'm sure I'm missing something, I'm not a Python guy. But every other language I've used has an easy to use stateless parallel map that hasn't given me any trouble.
- isoprophlex 4y agoI've been seeing python at least once every week for a looooong time. Years. A decade maybe. You are not missing anything. It's a big steamy pile of horse manure.
- xk3 4y agoThreadPoolExecutor if IO-bound ProcessPoolExecutor if CPU-bound for example with ThreadPoolExecutor(max_workers=4) as e: e.submit(shutil.copy, 'src1.txt', 'dest1.txt') e.submit(shutil.copy, 'src2.txt', 'dest2.txt') but yeah if you're truly CPU bound then move to something lower level like C or Rust
- dermesser 4y agoI can recommend Julia for easier parallelization while being reasonably Python-like. It's compiled, too, which helps even with single-threaded throughput.