4 ms·
No, that's not at all what's happening. f.readlines() creates and returns a full list of all lines, loaded into memory. But set(f) uses f as an iterator, whic
by jzwinck 12y ago
No, that's not at all what's happening. f.readlines() creates and returns a full list of all lines, loaded into memory. But set(f) uses f as an iterator, which reads chunks of the file and yields one line at a time, which can then be inserted into the set, de-duping on the fly. Your parent is correct (and clever).
- bdevine 12y agoI think you may have gotten ahead of yourself in your explanation, but to be clear, it's the "with" statement that causes the line iteration that then feeds into set(f). To say that "set(f) uses f as an iterator" might imply to some that set() is causing the iteration, which isn't true. ETA: See raymondh's post below.
- raymondh 12y agoSorry, this post is also filled with misinformation. * The with-statement only causes the file to be closed after use. It has nothing to do with iteration. * Files themselves are self-iterable (anything that loops over a file object uses a line-at-a-time iterator). Even writing "for line in f: ..." causes you to loop a line at a time without the whole file being in memory all at once. * Sets are just one of many objects that takes an iterable as an argument: min(f), max(f), list(f), tuple(f), etc.
- bdevine 12y agoAh! Thanks for the correction. Somewhere along the way I picked up the belief that in addition to the file-closing aspect, "with" ensured a line-at-a-time iterator, akin to adding on "for line in f:". I'm not sure I understand your third comment though. As I understand it, iterables have an __iter__ method. __iter__ methods return iterators. So an iterator for the iterable f traverses f and sends f's values to, in this case, set(). I believe we agree there. My initial concern was simply that the statement could be read "The iterator, set(), uses f.", and I don't see where the disagreement arises.
- meowface 12y agoIt was incorrect of me to say it was equivalent, but if all lines in the file are unique (ignoring the parent comment's assumption there will be lots of duplicates) then you're still creating a large object in memory. I'm not sure of the space of sets/hash tables vs. lists, but I imagine they'd be prety close. But of course that will be unavoidable if you want fast deduplication. You could do fully lazy deduplication using a generator, but you'd have to avoid using sets.
- DasIch 12y agoIf all lines in the file are unique, memory usage stays constant after the first line is read. A set by definition contains only unique items, that is all items in a set are different from each other.
- meowface 12y agoIf you're going from generator object (or any other lazy iterator, like the file object in this case) -> set object, there will be a memory usage increase with each additional line read. What I meant is you could in theory process a generator and omit duplicates without any real memory usage (even with a file of millions of lines) by chaining generators together. This would be slower than a set but much more memory efficient.