4 ms·

> Then the caller must know they are receiving a generator dependent on the files lifetime How would they not? The caller provides the file as an argument to t

• Groxx 6y ago
  it's not too hard to write code like: with open(...) as f: lines = process(f) # do stuff with lines ^ if `process` returns a generator, the file will be closed without having done anything.
  • xapata 6y ago
    Put `do stuff with lines` inside the context. Where it belongs.
    • Groxx 6y ago
      and hold open a resource longer than necessary? or when looping over lines multiple times? it doesn't unambiguously belong in there. often, yes, but not always.
      • hansvm 6y ago
        As the caller you can always choose to write lines=list(process(f)) if your use case requires it. If process(f) returned a list though, you wouldn't be able to avoid the potentially large memory allocation if you only needed to iterate over the results.
        • Groxx 6y ago
          Sure. It's entirely possible. Do you `list(...)` literally everywhere you want a list, whether or not it's currently a list? Extreme-defensive-programming like that tends to be extremely rare in my experience. Not nonexistent, but I'd be willing to bet that if you took a random stack overflow or github line of code, it wouldn't do this.
          • xapata 6y ago
            No. I put that in places where I receive a generator and want a list. But, there are plenty of instances of what you're talking about. Take a look at the standard library. Itertools often creates an iterator as the first operation, not knowing whether it received an iterator or some other kind of iterable. https://github.com/python/cpython/blob/384621c42f9102e31ba2c47feba144af09c989e5/Modules/itertoolsmodule.c#L965 https://github.com/python/cpython/blob/384621c42f9102e31ba2c... If you search for `list` in the CPython code, you'll find many examples of what you're asking about. Similarly, Pandas often makes a copy of a Series or DataFrame. It might be inefficient, but it keeps the API (more) simple.
          • hansvm 6y ago
            Defensive programming wasn't meant to be the takeaway. Rather, if you need a list and the otherwise perfect library method `foo` returns an iterator then you as the caller have almost no downside in writing `list(foo())` instead. On the flip-side, if you need an iterator (e.g. because of memory concerns) and the otherwise perfect library method `foo` returns a list then you as the caller have no options other than re-implementing `foo` or finding another tool to solve your problem. If your project usually requires lists rather than iterators then the syntactic burden of wrapping everything in `list(...)` could be annoying, and I wouldn't be at all surprised to find that performance-critical code couldn't tolerate the iterator overhead (though I'd posit that most of the time just using a list instead probably wouldn't fix it), but using iterators rather than lists seems like a good default if you don't have a good reason not to.
            • Groxx 6y ago
              Isn't that just the opposite tradeoff for code you call though? E.g. lists are strictly more flexible than iterators since they are multi-pass and can index. If you take a return from lib func A and pass it to lib funcs B and C (possibly in a different lib), if A returns an iterator then you need to know what B and C are going to do with it. If A returns a list, you do not - B and C cannot affect each other in this way unless they directly mutate the argument, which is relatively rare and usually has very strong documentation and/or naming patterns to make it not surprising. And B and C and all their calls can change without you needing to change your code between them and A. You can of course `list(A())` before passing it to B and C, but that's arguably defensive programming unless you know A returns an iterator. tbh I'd say I see multi-iteration and indexing several times more often than I see iterator-only use in most code, and almost never see memory issues except in large or embedded systems. Gigabytes of RAM are standard now, and dumping a few thousand files into memory before processing will quite often out-perform a more memory-efficient streaming approach. In the rare cases where you do exceed those memory bounds, you're likely in a niche area (gigabytes of files, tiny memory space, significant computation cost) and are likely using niche-oriented libraries that benefit more from coordinating tightly than from being perfectly general.