11 ms·
I don't fully understand the post, but I don't think the author found a bug anywhere. The main discovery the author made was that CPython uses its own allocator
by malisper 10y ago
I don't fully understand the post, but I don't think the author found a bug anywhere. The main discovery the author made was that CPython uses its own allocator on top of memory allocated by mmap. This leads to a difference between the reported allocated memory by malloc and the memory used by the program. The author still hasn't discovered a root cause for the large amounts of memory allocated.
- glandium 10y agoAuthor here. The last graph actually shows that the problem still happens with CPython's allocator disabled. Also, to clarify, the amount of memory allocated is actually decreasing (in-use bytes), while the memory allocator requests more and more memory from the system (system bytes), presumably because of fragmentation.
- malisper 10y agoI see. I thought system bytes included the mmaped block of memory (the docs for malloc_stats explicitly say it does not) and I thought the last graph was a demonstration of why the Python memory allocator was necessary.
- Kenji 10y ago>The last graph actually shows that the problem still happens with CPython's allocator disabled. Excuse my ignorance, but couldn't it still be a problem of the python runtime? I.e. they use malloc wrong, etc?