4 ms·

"Oh, and for people saying "use multiple processes" have never really had a problem where the process has multiple gigabytes of state that it needs to do its wo

"Oh, and for people saying "use multiple processes" have never really had a problem where the process has multiple gigabytes of state that it needs to do its work. The state that can be easily shared in multithreaded mode, but can be only shared with a great deal of manual work in multiprocess situation." Doesn't python have fork with copy-on-write semantics?

unshift 16y ago
copy-on-write doesn't write back to the original process, which may be desired. there are situations where a single process space are preferred (multiple threads sharing a file descriptor, etc) so "just use processes" isn't always the right answer. it can be done, but it's not always the nicest solution.
thwarted 16y ago
The copy-on-write semantics are a function of fork, an OS call (which is why it's available in python as os.fork), not a function of python. So if your OS supports COW for forked processes, python will have it. Check the man pages for fork(2) and vfork(2) for fork related COW information. There's some interesting stuff in there. As a side note, there's an interesting write-up on the portability of fork between UNIX and win32 in the perlfork man page (forgive me for mentioning perl in a python related thread), and the hoops that needed to be jumped through to emulate fork on win32 and keep it compatible with real fork from the visibility of the perl script itself.
andraz 16y ago
You do know that every temporary storage of a an object in python causes an increase of refcount which is a write? Each of those writes causes a 4kb block to be copied, and they are eventually scattered around enough to matter. What we've done is mmaping multi-gb files into memory by separate processes. It works, but it's a cumbersome workaround which makes you do way more work than you would ever wish to be able to use multiple cores.

unshift 16y ago

copy-on-write doesn't write back to the original process, which may be desired. there are situations where a single process space are preferred (multiple threads sharing a file descriptor, etc) so "just use processes" isn't always the right answer. it can be done, but it's not always the nicest solution.

thwarted 16y ago

The copy-on-write semantics are a function of fork, an OS call (which is why it's available in python as os.fork), not a function of python. So if your OS supports COW for forked processes, python will have it. Check the man pages for fork(2) and vfork(2) for fork related COW information. There's some interesting stuff in there. As a side note, there's an interesting write-up on the portability of fork between UNIX and win32 in the perlfork man page (forgive me for mentioning perl in a python related thread), and the hoops that needed to be jumped through to emulate fork on win32 and keep it compatible with real fork from the visibility of the perl script itself.

andraz 16y ago

You do know that every temporary storage of a an object in python causes an increase of refcount which is a write? Each of those writes causes a 4kb block to be copied, and they are eventually scattered around enough to matter. What we've done is mmaping multi-gb files into memory by separate processes. It works, but it's a cumbersome workaround which makes you do way more work than you would ever wish to be able to use multiple cores.