3 ms·
There's "not for performance-critical use", and then there's "why is it gobbling up all the RAM (~10GB) and make it impossible for me to do anything, when the e
by yongjik 10y ago
There's "not for performance-critical use", and then there's "why is it gobbling up all the RAM (~10GB) and make it impossible for me to do anything, when the entire data I fed it is several hundred MB."
Python frequently saunters into the second territory. Well, I guess there are tools available to profile memory usage and stuff, but if I had to spend that much effort tracking down memory issues, I might as well rewrite it in C++. (It doesn't help (or it helps?) that I'm much more comfortable with C++ than Python. YMMV.)
- cookiecaper 10y agoMy experience is that Python is much better about this than many other languages in its class, Ruby in particular. It's going to be hard to beat manual pointer/memory management in terms of raw performance, but Python is pretty respectable for an untyped, memory-managed language.
- jacquesm 10y agoThere are some easy to avoid pathological cases such as concatenating numpy arrays in a loop. Take this gem from a well known course: np.concatenate([x.next() for i in range(x.nb)]) That looks pretty innocent but it can eat up your memory in an eyeblink if the input is large enough. That's the sort of pitfall that a lot of python code suffers from because the abstractions are just nice enough to make you believe this will work without penalty and without knowing how it is implemented under the hood you're suddenly out a few gigs of ram.