5 ms·

I also see this as a bit of a premature optimization. The triple memory usage one may be significantly faster than the in-place one -- and the memory usage is t

I also see this as a bit of a premature optimization. The triple memory usage one may be significantly faster than the in-place one -- and the memory usage is temporary so unless you're in a constrained environment that's not a concern. Avoiding unintentional or unexpected side effects is a good thing. Pure functions are great. Do whatever you can do keep your visualization helper thing pure and avoid bugs. Then, once it works, if you have performance issues, do some profiling. If the profiling shows this helper method as a hotspot, then it's worth optimizing. At that point, you'd want to follow standard Python, numpy, and pandas optimization strategies -- which I'm not familiar with. Aimless optimization like this is a waste of time. Good program structure (data is available when you need it without tonnes of calls), the right choice of data structures and algorithms (hashmap vs list), the right choice of underlying tools (Julia vs Python) and targeted optimization (profiling) are the ways to go.

kardos 7y ago
My experience finds that the in-place optimization can indeed speed up numpy code. Even if you're not memory constrained, fewer arrays in play means less memory bandwidth needed. >> Many NumPy APIs include an out keyword argument, allowing you to write the results to an existing array, often including the original one. I'd wager that saving memory or memory bandwidth is the impetus for expanding the API to include this option.
philipkglass 7y ago
The article starts with "You have a large chunk of data — a NumPy array, or a Pandas DataFrame — and you need to do a series of operations on it." So it's suggesting to use these techniques when you already know that you are working with a large chunk of data, either because it was obvious from the beginning or because profiling highlighted it. In scientific computing it is common to store gigabytes of data in a single array. In fact it is common that even the compromise-solution suggested in this article, which leads to 2x memory instead of 3x, would be too much extra overhead. But the compromise solution is nice if you can afford 2x memory overhead, so I upvoted the article.
kllrnohj 7y ago
The slowest thing CPUs do by far is access memory. I don't see any possibly way that, for the given example, using more memory would ever be faster. At best 3*N all fits in cache and it's not as bad, but even then it's still at least one more cache miss which means it will still be slower even if it's not by much. Temporary memory is the worst type of memory. It's cache misses & pipeline stalls for no point. "Premature optimization" is not an excuse to write obviously bad code. Fixing this simple thing now is much faster, and simpler, than digging a big hole, jumping in it, and then hoping you can find something to pull yourself back out later. In the same way it's important to scatter about asserts and verify parameter arguments do in fact conform to the documented contract, it's also important to always keep an eye out for openly wasteful code.
- davedx 7y ago
  > it's also important to always keep an eye out for openly wasteful code. No, that's the point. It's really not. If your "openly wasteful code" does its job well and is readable and maintainable, it DOES NOT MATTER that it is "wasteful". If, on the other hand, the wasteful code causes your program to execute in 60 minutes instead of 10 seconds every time you run it, then it does matter and you should optimize it. Why is this such a difficult concept?
  - kllrnohj 7y ago
    If you're trying to use Knuth's quote from 1974 as your justification for that concept then you've misunderstood his argument and should read the full thing, not the single phrase. This article is not advocating premature optimization as Knuth was discussing nor does the article's advice result in the evil that Knuth was warning against. If you have some other evidence or paper justifying why we can ignore obvious inefficiencies please provide a source. That's a rather radical claim to make, though, which may be why you find people struggle with the concept.
  - deleted 7y ago
    [deleted]

kardos 7y ago

My experience finds that the in-place optimization can indeed speed up numpy code. Even if you're not memory constrained, fewer arrays in play means less memory bandwidth needed. >> Many NumPy APIs include an out keyword argument, allowing you to write the results to an existing array, often including the original one. I'd wager that saving memory or memory bandwidth is the impetus for expanding the API to include this option.

philipkglass 7y ago

The article starts with "You have a large chunk of data — a NumPy array, or a Pandas DataFrame — and you need to do a series of operations on it." So it's suggesting to use these techniques when you already know that you are working with a large chunk of data, either because it was obvious from the beginning or because profiling highlighted it. In scientific computing it is common to store gigabytes of data in a single array. In fact it is common that even the compromise-solution suggested in this article, which leads to 2x memory instead of 3x, would be too much extra overhead. But the compromise solution is nice if you can afford 2x memory overhead, so I upvoted the article.

kllrnohj 7y ago

The slowest thing CPUs do by far is access memory. I don't see any possibly way that, for the given example, using more memory would ever be faster. At best 3*N all fits in cache and it's not as bad, but even then it's still at least one more cache miss which means it will still be slower even if it's not by much. Temporary memory is the worst type of memory. It's cache misses & pipeline stalls for no point. "Premature optimization" is not an excuse to write obviously bad code. Fixing this simple thing now is much faster, and simpler, than digging a big hole, jumping in it, and then hoping you can find something to pull yourself back out later. In the same way it's important to scatter about asserts and verify parameter arguments do in fact conform to the documented contract, it's also important to always keep an eye out for openly wasteful code.

davedx 7y ago
> it's also important to always keep an eye out for openly wasteful code. No, that's the point. It's really not. If your "openly wasteful code" does its job well and is readable and maintainable, it DOES NOT MATTER that it is "wasteful". If, on the other hand, the wasteful code causes your program to execute in 60 minutes instead of 10 seconds every time you run it, then it does matter and you should optimize it. Why is this such a difficult concept?
- kllrnohj 7y ago
  If you're trying to use Knuth's quote from 1974 as your justification for that concept then you've misunderstood his argument and should read the full thing, not the single phrase. This article is not advocating premature optimization as Knuth was discussing nor does the article's advice result in the evil that Knuth was warning against. If you have some other evidence or paper justifying why we can ignore obvious inefficiencies please provide a source. That's a rather radical claim to make, though, which may be why you find people struggle with the concept.
- deleted 7y ago
  [deleted]