2 ms·
The Spark community (LinkedIn) developed Dr Elephant to profile jobs and provide suggestions for reducing memory/cpu consumption. Metaflow would need something
by jamesblonde 6y ago
The Spark community (LinkedIn) developed Dr Elephant to profile jobs and provide suggestions for reducing memory/cpu consumption. Metaflow would need something similar:
https://github.com/linkedin/dr-elephant https://github.com/linkedin/dr-elephant
- itamarst 6y agoI've seen that, yeah. I've already implemented a memory profiler for Python batch jobs (https://pythonspeed.com/fil https://pythonspeed.com/fil), but starting to think about how to integrate it into specific pipeline frameworks.