4 ms·

This is nice, how would you go about as a performance noob? I can't imagine there's a line in the docs saying "this is slow!".

by oblio 4y ago

akasakahakada 4y ago
LineProfiler is the best tool to learn how to write performant Python and do code optimization. https://github.com/pyutils/line_profiler https://github.com/pyutils/line_profiler You can literally see the hot spot of your code, then you can grind different algorithms or change the whole architecture to make it faster. For example replace short for loops to list comprehensions, vectorize all numpy operations (only vectorize partially do not help the issue), using 'not any()' instead or 'all()' for boolean, etc. Doing this for like 2 weeks, basically you can automatically recognize most bad code patterns at a glance.
- dralley 4y ago
  You should take a look at Scalene - it's even better. https://github.com/plasma-umass/scalene https://github.com/plasma-umass/scalene
  - akasakahakada 4y ago
    Wow this looks sick! Great thanks!
brahbrah 4y ago
So in addition to what akasaka said (another thumbs up for line profiler from me, great tool) this isn’t a problem with linalg.norm being slow. It’s plenty fast, but calling it thousands of separate times in a Python loop will be slow. This is more just about learning how to vectorize properly. If you’re working in numpy land and you’re calling a numpy function in a loop that’s iterating over more than a handful of items, chances are you’re not vectorizing properly
- akasakahakada 4y ago
  I realized that I should say more specifically about numpy usage. If you see a piece of code like this, it rings the bell that the person has no idea what he/she is doing: Bad Pattern: my_list = np.array(xxx) summed = [] for row in my_list: summed.append(np.sum(row)) Worst Pattern: my_list = np.array(xxx) def get_summed(arr): return np.sum(arr) summed = [] for i in range(len(my_list)): summed.append( get_summed(my_list[i]) ) Perfered: # np.array(xxx) is redundant summed = np.sum(xxx, axis=1) Same applies to all numpy operations.

This is nice, how would you go about as a performance noob? I can't imagine there's a line in the docs saying "this is slow!".

by oblio 4y ago

This is nice, how would you go about as a performance noob? I can't imagine there's a line in the docs saying "this is slow!".

akasakahakada 4y ago

LineProfiler is the best tool to learn how to write performant Python and do code optimization. https://github.com/pyutils/line_profiler https://github.com/pyutils/line_profiler You can literally see the hot spot of your code, then you can grind different algorithms or change the whole architecture to make it faster. For example replace short for loops to list comprehensions, vectorize all numpy operations (only vectorize partially do not help the issue), using 'not any()' instead or 'all()' for boolean, etc. Doing this for like 2 weeks, basically you can automatically recognize most bad code patterns at a glance.

dralley 4y ago
You should take a look at Scalene - it's even better. https://github.com/plasma-umass/scalene https://github.com/plasma-umass/scalene
- akasakahakada 4y ago
  Wow this looks sick! Great thanks!

brahbrah 4y ago

So in addition to what akasaka said (another thumbs up for line profiler from me, great tool) this isn’t a problem with linalg.norm being slow. It’s plenty fast, but calling it thousands of separate times in a Python loop will be slow. This is more just about learning how to vectorize properly. If you’re working in numpy land and you’re calling a numpy function in a loop that’s iterating over more than a handful of items, chances are you’re not vectorizing properly

akasakahakada 4y ago
I realized that I should say more specifically about numpy usage. If you see a piece of code like this, it rings the bell that the person has no idea what he/she is doing: Bad Pattern: my_list = np.array(xxx) summed = [] for row in my_list: summed.append(np.sum(row)) Worst Pattern: my_list = np.array(xxx) def get_summed(arr): return np.sum(arr) summed = [] for i in range(len(my_list)): summed.append( get_summed(my_list[i]) ) Perfered: # np.array(xxx) is redundant summed = np.sum(xxx, axis=1) Same applies to all numpy operations.