4 ms·
Why are real threads so important? Does anyone have an example where threads would be much better than using the multiprocessing module?
by zepolen 17y ago
Why are real threads so important? Does anyone have an example where threads would be much better than using the multiprocessing module?
- heresy 17y agoBlocking operating system calls? This has always been the problem I've run into with green threads.
- parbo 17y agoI think the GIL is released during most of those, so that should already work with real threads.
- j_baker 17y agoNote that Python doesn't use green threads. It uses real, native threads. The thing is that the interpreter will prevent any other threads from accessing it until the GIL is released.
- nas 17y agoThe problem with the GIL is nearly the opposite. The lock is dropped for system calls but kept when Python code is running. That's not a problem for IO bound workloads but does not scale well over multiple CPUs for CPU bound tasks that do most of the work in Python-land. I not sure dropping the GIL is the best solution. Something like Erlang message passing sounds like a better model. I'm not sure if that's a good fit either though since it works best with functional style programming.
- mahmud 17y agoWith native threads, all your threads have their own identity and they're known to the OS task scheduler. So they can all block or run independently. But with green threads, the OS doesn't know about your "threads"; when the parent process is blocked, so are all the threads.
- stonemetal 17y agoNote GP said multiprocessing (that is a python library for multiple processes with a thread like interface, not green threads.) As for why some see it as not enough well it is kind of like CGI vs FCGI spining up a whole new process with and VM is not a cheap operation compared to starting up a thread. Sure you can work around it with design changes pools and what not, but some would rather see the problem fixed correctly rather than worked around.
- barrkel 17y agoOne of the basic assumptions behind green threads is that you aren't going to make blocking OS calls from your green threads - the interpreter should intercept those, run them in a different thread (or ideally, run with async I/O) so that it can schedule a different thread until the call returns.
- zweiterlinde 17y agoIn the long run, it seems like approaches like ConCRT and User Mode Scheduling (Windows only, sadly) could address this. The OS can provide support for detecting any blocking operations (even page faults, for example) and doing task-level switching for you.
- fauigerzigerk 17y agoComplex in memory data structures are the main use case that is not well supported by multi process architectures. There's a trend towards in memory databases and using disk for archival only, so this issue is becoming critical. Processes don't share pointers, they only share BLOBs. Threads do share pointers and that's why they are more suitable to in memory data analysis/manipulation.
- krakensden 17y agoDon't the in-memory databases generally run in another process (memcached) anyway? (Not that your general point is wrong)
- old-gregg 17y agoBut don't you want your in-memory database to be non-locking, i.e. serving many clients simultaneously and fast?
- krakensden 17y agoThe point was your in-memory database is an external, non-python program, and you run lots of python clients. The way most web applications are structured, for instance.
- fauigerzigerk 17y agoYes but if I'm the one writing in memory database/analytics stuff I might want to use a Python/C combination to do that. Right now that's not an option because of the GIL issue.
- deleted 17y ago[deleted]