2 ms·

and hold open a resource longer than necessary? or when looping over lines multiple times? it doesn't unambiguously belong in there. often, yes, but not alwa

by Groxx 6y ago

and hold open a resource longer than necessary? or when looping over lines multiple times? it doesn't unambiguously belong in there. often, yes, but not always.

hansvm 6y ago
As the caller you can always choose to write lines=list(process(f)) if your use case requires it. If process(f) returned a list though, you wouldn't be able to avoid the potentially large memory allocation if you only needed to iterate over the results.
- Groxx 6y ago
  Sure. It's entirely possible. Do you `list(...)` literally everywhere you want a list, whether or not it's currently a list? Extreme-defensive-programming like that tends to be extremely rare in my experience. Not nonexistent, but I'd be willing to bet that if you took a random stack overflow or github line of code, it wouldn't do this.
  - xapata 6y ago
    No. I put that in places where I receive a generator and want a list. But, there are plenty of instances of what you're talking about. Take a look at the standard library. Itertools often creates an iterator as the first operation, not knowing whether it received an iterator or some other kind of iterable. https://github.com/python/cpython/blob/384621c42f9102e31ba2c47feba144af09c989e5/Modules/itertoolsmodule.c#L965 https://github.com/python/cpython/blob/384621c42f9102e31ba2c... If you search for `list` in the CPython code, you'll find many examples of what you're asking about. Similarly, Pandas often makes a copy of a Series or DataFrame. It might be inefficient, but it keeps the API (more) simple.
  - hansvm 6y ago
    Defensive programming wasn't meant to be the takeaway. Rather, if you need a list and the otherwise perfect library method `foo` returns an iterator then you as the caller have almost no downside in writing `list(foo())` instead. On the flip-side, if you need an iterator (e.g. because of memory concerns) and the otherwise perfect library method `foo` returns a list then you as the caller have no options other than re-implementing `foo` or finding another tool to solve your problem. If your project usually requires lists rather than iterators then the syntactic burden of wrapping everything in `list(...)` could be annoying, and I wouldn't be at all surprised to find that performance-critical code couldn't tolerate the iterator overhead (though I'd posit that most of the time just using a list instead probably wouldn't fix it), but using iterators rather than lists seems like a good default if you don't have a good reason not to.
    - Groxx 6y ago
      Isn't that just the opposite tradeoff for code you call though? E.g. lists are strictly more flexible than iterators since they are multi-pass and can index. If you take a return from lib func A and pass it to lib funcs B and C (possibly in a different lib), if A returns an iterator then you need to know what B and C are going to do with it. If A returns a list, you do not - B and C cannot affect each other in this way unless they directly mutate the argument, which is relatively rare and usually has very strong documentation and/or naming patterns to make it not surprising. And B and C and all their calls can change without you needing to change your code between them and A. You can of course `list(A())` before passing it to B and C, but that's arguably defensive programming unless you know A returns an iterator. tbh I'd say I see multi-iteration and indexing several times more often than I see iterator-only use in most code, and almost never see memory issues except in large or embedded systems. Gigabytes of RAM are standard now, and dumping a few thousand files into memory before processing will quite often out-perform a more memory-efficient streaming approach. In the rare cases where you do exceed those memory bounds, you're likely in a niche area (gigabytes of files, tiny memory space, significant computation cost) and are likely using niche-oriented libraries that benefit more from coordinating tightly than from being perfectly general.
      - xapata 6y ago
        > never see memory issues In contrast, I see them all the time (damn you, Pandas). Beyond out-of-memory errors, generators can be vastly more compute-efficient because of the CPU cache.
        Groxx 6y ago
        Pandas I'd definitely categorize as being niche :) A moderately-sized one, but still a niche. And there you use whatever performs the way you need it, yeah - there are special constraints on the system, and pandas is built around dealing with them.
        hansvm 6y ago
        Haha, that one hits close to home. Iirc Pandas was the author's first big project, and memory use is one of the key things he wants to improve if he gives it another go. It's still an impressive piece of software.

by Groxx 6y ago

and hold open a resource longer than necessary? or when looping over lines multiple times? it doesn't unambiguously belong in there. often, yes, but not always.

hansvm 6y ago

As the caller you can always choose to write lines=list(process(f)) if your use case requires it. If process(f) returned a list though, you wouldn't be able to avoid the potentially large memory allocation if you only needed to iterate over the results.

Groxx 6y ago
Sure. It's entirely possible. Do you `list(...)` literally everywhere you want a list, whether or not it's currently a list? Extreme-defensive-programming like that tends to be extremely rare in my experience. Not nonexistent, but I'd be willing to bet that if you took a random stack overflow or github line of code, it wouldn't do this.
- xapata 6y ago
  No. I put that in places where I receive a generator and want a list. But, there are plenty of instances of what you're talking about. Take a look at the standard library. Itertools often creates an iterator as the first operation, not knowing whether it received an iterator or some other kind of iterable. https://github.com/python/cpython/blob/384621c42f9102e31ba2c47feba144af09c989e5/Modules/itertoolsmodule.c#L965 https://github.com/python/cpython/blob/384621c42f9102e31ba2c... If you search for `list` in the CPython code, you'll find many examples of what you're asking about. Similarly, Pandas often makes a copy of a Series or DataFrame. It might be inefficient, but it keeps the API (more) simple.
- hansvm 6y ago
  Defensive programming wasn't meant to be the takeaway. Rather, if you need a list and the otherwise perfect library method `foo` returns an iterator then you as the caller have almost no downside in writing `list(foo())` instead. On the flip-side, if you need an iterator (e.g. because of memory concerns) and the otherwise perfect library method `foo` returns a list then you as the caller have no options other than re-implementing `foo` or finding another tool to solve your problem. If your project usually requires lists rather than iterators then the syntactic burden of wrapping everything in `list(...)` could be annoying, and I wouldn't be at all surprised to find that performance-critical code couldn't tolerate the iterator overhead (though I'd posit that most of the time just using a list instead probably wouldn't fix it), but using iterators rather than lists seems like a good default if you don't have a good reason not to.
  - Groxx 6y ago
    Isn't that just the opposite tradeoff for code you call though? E.g. lists are strictly more flexible than iterators since they are multi-pass and can index. If you take a return from lib func A and pass it to lib funcs B and C (possibly in a different lib), if A returns an iterator then you need to know what B and C are going to do with it. If A returns a list, you do not - B and C cannot affect each other in this way unless they directly mutate the argument, which is relatively rare and usually has very strong documentation and/or naming patterns to make it not surprising. And B and C and all their calls can change without you needing to change your code between them and A. You can of course `list(A())` before passing it to B and C, but that's arguably defensive programming unless you know A returns an iterator. tbh I'd say I see multi-iteration and indexing several times more often than I see iterator-only use in most code, and almost never see memory issues except in large or embedded systems. Gigabytes of RAM are standard now, and dumping a few thousand files into memory before processing will quite often out-perform a more memory-efficient streaming approach. In the rare cases where you do exceed those memory bounds, you're likely in a niche area (gigabytes of files, tiny memory space, significant computation cost) and are likely using niche-oriented libraries that benefit more from coordinating tightly than from being perfectly general.
    - xapata 6y ago
      > never see memory issues In contrast, I see them all the time (damn you, Pandas). Beyond out-of-memory errors, generators can be vastly more compute-efficient because of the CPU cache.
      - Groxx 6y ago
        Pandas I'd definitely categorize as being niche :) A moderately-sized one, but still a niche. And there you use whatever performs the way you need it, yeah - there are special constraints on the system, and pandas is built around dealing with them.
      - hansvm 6y ago
        Haha, that one hits close to home. Iirc Pandas was the author's first big project, and memory use is one of the key things he wants to improve if he gives it another go. It's still an impressive piece of software.