3 ms·
One annoying part with multiprocessing in Python is that you could abuse the COW mechanism to save on loading time when forking. But Python stores ref counters
by kroolik 3y ago
One annoying part with multiprocessing in Python is that you could abuse the COW mechanism to save on loading time when forking. But Python stores ref counters together with objects so every single read will bust your COW cache.
Now, you wanted it simple, but got to fight with the memory model of a language that wasn't designed with performance in mind, for programs whose focus wasn't performance.
- viraptor 3y agoThere's gc.freeze for that now https://docs.python.org/3/library/gc.html#gc.freeze https://docs.python.org/3/library/gc.html#gc.freeze If you load something big before forking workers, there's no CoW issue with that big structure anymore.
- justinc-md 3y agogc.freeze prevents considering the objects in gc, but doesn’t disable reference counting so you’ll still have CoW issues. PEP 683 introduces a way to make an object immortal which disables reference counting, which will address that issue.