3 ms·
You do know that every temporary storage of a an object in python causes an increase of refcount which is a write? Each of those writes causes a 4kb block to be
by andraz 16y ago
You do know that every temporary storage of a an object in python causes an increase of refcount which is a write? Each of those writes causes a 4kb block to be copied, and they are eventually scattered around enough to matter.
What we've done is mmaping multi-gb files into memory by separate processes. It works, but it's a cumbersome workaround which makes you do way more work than you would ever wish to be able to use multiple cores.