4 ms·
When I use Python sometimes I run into a MemoryError when working with large datasets. How does the Python runtime know I am out of memory if the kernel won't t
by Paul-ish 7y ago
When I use Python sometimes I run into a MemoryError when working with large datasets. How does the Python runtime know I am out of memory if the kernel won't tell it. Does it try a write and catch the signal?
- amelius 7y agoPerhaps any write can fail, and if one does then it triggers some code which frees some pre-reserved space which allows the interpreter to continue, and properly unwind the stack.
- lalaithion 7y agoI believe that the python interpreter creates a MemoryError at startup, and throws that when you run out of memory.
- amelius 7y agoOk, but while unwinding a try/except block, the interpreter might need some extra memory, I suppose, depending on the code in the except clause.
- cesarb 7y agoIf your large dataset leads to a single huge memory allocation, the kernel will fail the allocation even though overcommit is enabled. Early versions of libvpx had a similar issue, where it allocated a huge amount of memory at once (actually, told the dynamic linker to do so) which failed on low-memory machines (https://bugzilla.redhat.com/show_bug.cgi?id=600663 https://bugzilla.redhat.com/show_bug.cgi?id=600663 "libvpx.so.0 fails to load on low-memory machines due to ridiculously oversized .bss").