9 ms·
Did they ever fix the global interpreter lock? Sort of a show stopper with doing stuff concurrently in python. I've done a bit of batch processing using the mul
by jillesvangurp 8y ago
Did they ever fix the global interpreter lock? Sort of a show stopper with doing stuff concurrently in python. I've done a bit of batch processing using the multi process module; which uses processes instead of threads. This works but it is a bit of a kludge if you are used to languages that support concurrency properly.
- armitron 8y agoThey did not, which is why this "course" illustrates taking advantage of multiple cores via multiprocessing without mentioning the GIL at all. Which is a little misleading if you think about it. Also, by having the introductory chapter be about "functional programming" (which incidentally Python does not do well), he completely bypasses the serious issue of shared state. Which goes to show that parallelism in Python is more like a gimmick than a real-world solution since it doesn't let you do in-process shared-memory processing via threads in parallel which is so important for many applications. In my case, the vast majority of the time I do not want to farm workers out to different operating system processes and deal with serialization and communication, but this is the only way for Python code to take advantage of multiple cores [1]. [1] Another way is to write a module in C and have Python code call into it on a new thread and release the GIL while doing so, but of course this is even worse pain-wise than doing it with multiprocessing and you end up writing/compiling C.
- devxpy 8y ago> deal with serialization and communication I thought a lot about this problem, for over 2 years, and came up with zproc https://github.com/pycampers/zproc https://github.com/pycampers/zproc Basically, > It lets you do message passing parallelism without the effort of tedious wiring. You'll be doing message passing without ever dealing with sockets! Also, Shared memory parallelism is hard to get right irregardless of which language you use. I would recommend strongly against it, unless you're writing some really really really niche thing where message passing is a bottleneck (it isn't most of the time)
- TickleSteve 8y agoyou claim "To make utterly perfect MT programs (and I mean that literally)". you've rediscovered message-passing... please take an elementary CS course on parallel systems. That claim is naive in the extreme.
- devxpy 8y agoThat's not my claim man, its written in the zguide http://zguide.zeromq.org/page:all#Multithreading-with-ZeroMQ http://zguide.zeromq.org/page:all#Multithreading-with-ZeroMQ Maybe I should've just linked it there,sorry! Okay, I will take that course and get back, thanks for the suggestion. P.S. You just implied Pieter Hintjens is naive. You have to live with that now :(
- TickleSteve 8y agoI think you took that claim out of context: "By "perfect MT programs", I mean code that's easy to write and understand, that works with the same design approach in any programming language, and on any operating system, and that scales across any number of CPUs with zero wait states and no point of diminishing returns." That doesn't mean to say its "perfect" or "solves" multithreading, just that its easy to write and understand and portable across architectures. That says nothing of how optimal it is for concurrency or parallelism ease-of-use wise or performance-wise, just that its 'easy'.
- devxpy 8y ago> That doesn't mean to say its "perfect" or "solves" multithreading, just that its easy to write and understand Try saying that out loud?
- TickleSteve 8y agoyes. That makes perfect sense... easy to write and understand is something completely different to correctness, robustness, scalability, etc. All those must be considered if you think you have 'solved' parallelism, but they are orthogonal to 'easy to understand'.
- dragonwriter 8y ago> Did they ever fix the global interpreter lock? Sort of a show stopper with doing stuff concurrently in python. It means threas-based parallelism of pure-python code is unavailable; concurrency is just fine on Python.
- skrause 8y agoI have to work with Python on Windows and believe me, concurrency is not just fine in Python when you can't use fork().
- zbentley 8y agoObligatory "concurrency != parallelism" statement; concurrency is fine on both platforms with Python threading in a single process with a GIL; parallelism is less of a done deal. While it's a very big hammer, consider experimenting with Celery for your parallelism needs on Windows. I've had good results using per-script Celery "clusters" with either a filesystem (on a ramdisk for extra speed) or an embedded Redis backend to accomplish pretty nice bidirectional RPC-ish parallelism. The initial setup is much more complicated than something like goroutines, but once you get it working you can boilerplate it onto other tasks without much trouble. It still won't save you from memory constraints imposed by the lack of good fork() emulation, though. Hopefully the WSL stuff will either bring better fork() emulation, or allow support for shared memory objects (e.g. multiprocessing.Value) in order to ease some of that pain.
- another-cuppa 8y agoConcurrency and parallelism are two different things. Python is fine for concurrency.
- devxpy 8y agoI believe that since the Advent of zeromq, parallelism is possible in almost any language, including python. My library lets you do parallelism in a unique way, where you do message passing parallelism without being explicit about it. https://github.com/pycampers/zproc/ https://github.com/pycampers/zproc/
- heavenlyblue 8y ago>> Zproc uses a Server, which is responsible for storing and communicating the state. >> >> This isolates our resource (state), eliminating the need for locks. So you've just invented a new name for a coordinator process and called it a new fashion in computation?
- TickleSteve 8y agoNo, he's reinvented multiprocessing... pickling data structures across multiple processes. Just without the 'niceties'.
- zbentley 8y agoYou're probably right, but see my comment above: not only is MP possibly superior at being a picking/arbitrating server, but it also supports taking advantage of copy-on-write semantics on Unix-ish systems to transfer memory to children at startup in constant time with no pickling/unpickling necessary.
- TickleSteve 8y agoI agree, multiprocessing will be more performant than ZProc, much more thought has gone into it than the simple 0mq wrapper that is ZProc.