3 ms·

I've just put in production an app that intermittently can't allocate enough memory. Is this the best tool to debug it? I've never had to debug memory problems

by neves 5y ago

I've just put in production an app that intermittently can't allocate enough memory. Is this the best tool to debug it? I've never had to debug memory problems in Python.

orf 5y ago
If this isn't due to latent memory pressure, it's usually because _something_ in your code is trying to allocate a gigantic amount of memory due to a bug. Think a 1 billion x 1 billion numpy array, or a billion element list.
itamarst 5y ago
Fil can dump memory profiling reports on failed allocations; as other commenter said, Linux by default happily just gives you memory even if doesn't have any, so failed allocation implies giant allocation. https://pythonspeed.com/fil/docs/oom.html https://pythonspeed.com/fil/docs/oom.html
noitpmeder 5y ago
Is this due to memory constraints on your system? If so, then this would definitely help you profile _what_ in your code is trying to claim so much memory.
dbenamy 5y ago
Datadog's profiler [0] might help. The python profiler has a heap profile type which shows (the call site which allocates) data using memory [1]. Feedback welcome! Disclosure: I work on this :-) [0] https://app.datadoghq.com/profiling https://app.datadoghq.com/profiling [1] From https://docs.datadoghq.com/tracing/profiler/search_profiles/ https://docs.datadoghq.com/tracing/profiler/search_profiles/ > Heap Live Size > Shows the amount of heap memory allocated by each function that has not been garbage collected (yet). This is useful for investigating the overall memory usage of your service and identifying potential memory leaks.
vdfs 5y ago
Yes, I just give this tool a try and reduced our app initial memory allocation from 120MB to about 80MB in few minutes by moving imports around, ex: rarely called functions importing big 3rd party library. You can use this to figure out what need to be optimized, but in your case optimization might not need to be easy, in above example you would have to re-implement the 3rd party library in memory efficient way or find an other one

by neves 5y ago

I've just put in production an app that intermittently can't allocate enough memory. Is this the best tool to debug it? I've never had to debug memory problems in Python.

orf 5y ago

If this isn't due to latent memory pressure, it's usually because _something_ in your code is trying to allocate a gigantic amount of memory due to a bug. Think a 1 billion x 1 billion numpy array, or a billion element list.

itamarst 5y ago

Fil can dump memory profiling reports on failed allocations; as other commenter said, Linux by default happily just gives you memory even if doesn't have any, so failed allocation implies giant allocation. https://pythonspeed.com/fil/docs/oom.html https://pythonspeed.com/fil/docs/oom.html

noitpmeder 5y ago

Is this due to memory constraints on your system? If so, then this would definitely help you profile _what_ in your code is trying to claim so much memory.

dbenamy 5y ago

Datadog's profiler [0] might help. The python profiler has a heap profile type which shows (the call site which allocates) data using memory [1]. Feedback welcome! Disclosure: I work on this :-) [0] https://app.datadoghq.com/profiling https://app.datadoghq.com/profiling [1] From https://docs.datadoghq.com/tracing/profiler/search_profiles/ https://docs.datadoghq.com/tracing/profiler/search_profiles/ > Heap Live Size > Shows the amount of heap memory allocated by each function that has not been garbage collected (yet). This is useful for investigating the overall memory usage of your service and identifying potential memory leaks.

vdfs 5y ago

Yes, I just give this tool a try and reduced our app initial memory allocation from 120MB to about 80MB in few minutes by moving imports around, ex: rarely called functions importing big 3rd party library. You can use this to figure out what need to be optimized, but in your case optimization might not need to be easy, in above example you would have to re-implement the 3rd party library in memory efficient way or find an other one