6 ms·

If you want a lazy iterator, then you will need to keep the file handle open. I don’t see how this one use case invalidates the general guideline of not sitting

If you want a lazy iterator, then you will need to keep the file handle open. I don’t see how this one use case invalidates the general guideline of not sitting on file descriptors that you are finished with.

dmurray 6y ago
How you feel about this is key to the article's argument. It makes the point that you shouldn't have to know if the iterator is lazy or not, that's an implementation detail. That does seem...generally desirable? But given Python already gives you easy and different syntax to create lazy vs greedy iterators, and you often can't use them interchangeably, it's clearly not a top design goal of the language.
- joshuamorton 6y ago
  It sounds desirable, but isn't feasible unless the language is stateless. If you iterate over a thing that is stateful, you need to know whether the iterator is lazy or not to decide if you can modify the underlying thing.
  - dmurray 6y ago
    I don't think that's right. You can always turn it into a greedy iterable (e.g. convert it to a list) if you need the certainty that you can modify the underlying thing, while still reaping some performance benefits from the maybe-lazy iterator in the rest of your use cases. Assuming the iterator wasn't infinite, of course, but in that case you already knew it was lazy.
    - joshuamorton 6y ago
      Right but you have to know the iterator was lazy to know that you need to convert it.
      - hansvm 6y ago
        Is it really that arduous to have a high-level understanding of what the method you're calling does, at least at the level of knowing whether or not it returns the type of object you want and whether it takes ownership of its arguments? There's a separate question of whether the standard library should be inconsistent with itself, but it seems like a reasonable choice -- you can't efficiently use a json file without doing _something_ invasive to it like loading it all into ram, but a csv file isn't burdened the same way, and you probably don't want to be limited by available ram when all you need to do is enumerate a csv.
        joshuamorton 6y ago
        I'm not sure what you're arguing for. My point is that you cannot, in general, abstract over both lazy and eager iterators. There are irreconcilable differences in how they work (and how you have to treat them).
- dragonwriter 6y ago
  > It makes the point that you shouldn't have to know if the iterator is lazy or not, that's an implementation detail. That's both right and wrong. That it's a lazy iterator itself may be an implementation detail that doesn't effect the interface (if the interface doesn't assure reusability; a concrete collection and a lazy iterator do have different behavior in important ways) but leaving a dangling external resource is not. For certain resources, when you are given a reference rather than the opened resource, you can't build a lazy iterator without also leaving dangling external resource that the caller doesn't have the information to close. That is a significant difference.
- Skunkleton 6y ago
  > It makes the point that you shouldn't have to know if the iterator is lazy or not, that's an implementation detail. It certainly isn't an implementation detail. How can you write a correct program without knowing this behavior?
  - lixtra 6y ago
    The fact that python changed that behavior in the past (i.e. range) shows that writing correct programs usually works in practice.
    - Skunkleton 6y ago
      The fact that python has a lazy and non-lazy range is pretty much the point. Yes there are (common) trivial cases where you don't care if the range is lazy or not, but in general it is something you need to be aware of.

dmurray 6y ago

How you feel about this is key to the article's argument. It makes the point that you shouldn't have to know if the iterator is lazy or not, that's an implementation detail. That does seem...generally desirable? But given Python already gives you easy and different syntax to create lazy vs greedy iterators, and you often can't use them interchangeably, it's clearly not a top design goal of the language.

joshuamorton 6y ago
It sounds desirable, but isn't feasible unless the language is stateless. If you iterate over a thing that is stateful, you need to know whether the iterator is lazy or not to decide if you can modify the underlying thing.
- dmurray 6y ago
  I don't think that's right. You can always turn it into a greedy iterable (e.g. convert it to a list) if you need the certainty that you can modify the underlying thing, while still reaping some performance benefits from the maybe-lazy iterator in the rest of your use cases. Assuming the iterator wasn't infinite, of course, but in that case you already knew it was lazy.
  - joshuamorton 6y ago
    Right but you have to know the iterator was lazy to know that you need to convert it.
    - hansvm 6y ago
      Is it really that arduous to have a high-level understanding of what the method you're calling does, at least at the level of knowing whether or not it returns the type of object you want and whether it takes ownership of its arguments? There's a separate question of whether the standard library should be inconsistent with itself, but it seems like a reasonable choice -- you can't efficiently use a json file without doing _something_ invasive to it like loading it all into ram, but a csv file isn't burdened the same way, and you probably don't want to be limited by available ram when all you need to do is enumerate a csv.
      - joshuamorton 6y ago
        I'm not sure what you're arguing for. My point is that you cannot, in general, abstract over both lazy and eager iterators. There are irreconcilable differences in how they work (and how you have to treat them).
dragonwriter 6y ago
> It makes the point that you shouldn't have to know if the iterator is lazy or not, that's an implementation detail. That's both right and wrong. That it's a lazy iterator itself may be an implementation detail that doesn't effect the interface (if the interface doesn't assure reusability; a concrete collection and a lazy iterator do have different behavior in important ways) but leaving a dangling external resource is not. For certain resources, when you are given a reference rather than the opened resource, you can't build a lazy iterator without also leaving dangling external resource that the caller doesn't have the information to close. That is a significant difference.
Skunkleton 6y ago
> It makes the point that you shouldn't have to know if the iterator is lazy or not, that's an implementation detail. It certainly isn't an implementation detail. How can you write a correct program without knowing this behavior?
- lixtra 6y ago
  The fact that python changed that behavior in the past (i.e. range) shows that writing correct programs usually works in practice.
  - Skunkleton 6y ago
    The fact that python has a lazy and non-lazy range is pretty much the point. Yes there are (common) trivial cases where you don't care if the range is lazy or not, but in general it is something you need to be aware of.