4 ms·
I recently had an adventure with Python subprocesses and memory. This isn't just a Python thing, but it's more of an issue because Python lacks proper multitre
by dehrmann 5y ago
I recently had an adventure with Python subprocesses and memory. This isn't just a Python thing, but it's more of an issue because Python lacks proper multitreading: processes don't necessarily return memory to the OS after free(), and Python doesn't necessarily free() unused memory after a GC run. If you're using a process pool where some work can be memory-intensive, this can massively increase total memory use as the job bounces between subprocesses.
- viraptor 5y agoIf you're using subprocesses, you may also run into the GC duplicating your heap with unnecessary collection. Check https://docs.python.org/3/library/gc.html#gc.freeze https://docs.python.org/3/library/gc.html#gc.freeze and the note about forking.