3 ms·

This looks great. I think it would help to detect performance issues in running applications. By the ways is there some solution to profile memory? So basicall

This looks great. I think it would help to detect performance issues in running applications. By the ways is there some solution to profile memory? So basically find out which part of the code is accumulating memory (that may happen very very slow). That is often, at least in my perspective, much much harder than profile cpu usage.

petethepig 6y ago
Yes, this kind of data would definitely be great to have as well and we're planning to add that at some point. I think in Go there's at least a clear path to that, afaik pprof already has something for memory profiling, but with other languages it might be more complicated.
itamarst 6y ago
For memory that are actually two different domains, with different problems, different UX requirements, and different solutions: 1. For server workloads, the main issue is usually leaks. Lacking leaks you can just characterize the workload, give it enough memory (usually it's not very much per request), and call it a day. 2. For batch data processing, the main issue is ... using lots of memory to process the data :) Like, loading 4GB of data and doing stuff to it can easily use 20GB of RAM if you're not careful. For the latter case, and specifically for Python, I've written an open source memory profiler (https://pythonspeed.com/fil https://pythonspeed.com/fil) that helps you spot which code allocated the memory responsible for peak memory. You can also use it for leaks, but that's not the main use case. It has some performance overhead, so it's not usable in production, but I'm also working on version that will be fast enough to run in production, at the cost of slightly less accuracy. For batch processing workloads that loss of accuracy isn't meaningful, mostly, who cares if you're off by 1MB on a 20GB peak memory usage. For server workloads with memory leaks... might take a lot longer to catch the problem as a result of reduced accuracy, and a tool designed specifically for server workloads might work better by taking a different implementation approach.

petethepig 6y ago

Yes, this kind of data would definitely be great to have as well and we're planning to add that at some point. I think in Go there's at least a clear path to that, afaik pprof already has something for memory profiling, but with other languages it might be more complicated.

itamarst 6y ago

For memory that are actually two different domains, with different problems, different UX requirements, and different solutions: 1. For server workloads, the main issue is usually leaks. Lacking leaks you can just characterize the workload, give it enough memory (usually it's not very much per request), and call it a day. 2. For batch data processing, the main issue is ... using lots of memory to process the data :) Like, loading 4GB of data and doing stuff to it can easily use 20GB of RAM if you're not careful. For the latter case, and specifically for Python, I've written an open source memory profiler (https://pythonspeed.com/fil https://pythonspeed.com/fil) that helps you spot which code allocated the memory responsible for peak memory. You can also use it for leaks, but that's not the main use case. It has some performance overhead, so it's not usable in production, but I'm also working on version that will be fast enough to run in production, at the cost of slightly less accuracy. For batch processing workloads that loss of accuracy isn't meaningful, mostly, who cares if you're off by 1MB on a 20GB peak memory usage. For server workloads with memory leaks... might take a lot longer to catch the problem as a result of reduced accuracy, and a tool designed specifically for server workloads might work better by taking a different implementation approach.