4 ms·

set() is a great way to deduplicate small lists, but it's important to note that it requires O(n) extra space (in-place sorting can avoid this overhead, but is

set() is a great way to deduplicate small lists, but it's important to note that it requires O(n) extra space (in-place sorting can avoid this overhead, but is more complex).

raymondh 12y ago
Sorting is almost always the wrong way to do it. (Wordy and slow). It is fragile design to write code that depends on 1) the data is so large that you don't have room for a set() BUT 2) it is small enough for an in-memory sort. (IOW, the almost-out-of-memory case invariably degrades over time to flat-out-of-memory). Another thought: people seem to place too much concern about about the size of various data structures rather than thinking about the data itself. Python containers don't contain anything, they just hold references. (Usually, the weight of a bucket of water is mostly the water, not the bucket itself). Finally, if your task is to dedup a lot of data, it doesn't make sense to read it all into memory in the first place (which you would need for a sorting approach). It is better dedup it using a set as you read in the data: # Only the unique lines are ever kept in memory with open('hugefile.txt') as f: uniq_lines = set(f) Dude, sorry to go off like this, but the advice you gave is almost always the wrong way to do it.
- meowface 12y ago
  You're still reading all of the lines into memory in that example as soon as you call `set(f)` (which is basically equivalent to set(f.readlines()), though, which may not necessarily be what you want.
  - jzwinck 12y ago
    No, that's not at all what's happening. f.readlines() creates and returns a full list of all lines, loaded into memory. But set(f) uses f as an iterator, which reads chunks of the file and yields one line at a time, which can then be inserted into the set, de-duping on the fly. Your parent is correct (and clever).
    - bdevine 12y ago
      I think you may have gotten ahead of yourself in your explanation, but to be clear, it's the "with" statement that causes the line iteration that then feeds into set(f). To say that "set(f) uses f as an iterator" might imply to some that set() is causing the iteration, which isn't true. ETA: See raymondh's post below.
      - raymondh 12y ago
        Sorry, this post is also filled with misinformation. * The with-statement only causes the file to be closed after use. It has nothing to do with iteration. * Files themselves are self-iterable (anything that loops over a file object uses a line-at-a-time iterator). Even writing "for line in f: ..." causes you to loop a line at a time without the whole file being in memory all at once. * Sets are just one of many objects that takes an iterable as an argument: min(f), max(f), list(f), tuple(f), etc.
        bdevine 12y ago
        Ah! Thanks for the correction. Somewhere along the way I picked up the belief that in addition to the file-closing aspect, "with" ensured a line-at-a-time iterator, akin to adding on "for line in f:". I'm not sure I understand your third comment though. As I understand it, iterables have an __iter__ method. __iter__ methods return iterators. So an iterator for the iterable f traverses f and sends f's values to, in this case, set(). I believe we agree there. My initial concern was simply that the statement could be read "The iterator, set(), uses f.", and I don't see where the disagreement arises.
    - meowface 12y ago
      It was incorrect of me to say it was equivalent, but if all lines in the file are unique (ignoring the parent comment's assumption there will be lots of duplicates) then you're still creating a large object in memory. I'm not sure of the space of sets/hash tables vs. lists, but I imagine they'd be prety close. But of course that will be unavoidable if you want fast deduplication. You could do fully lazy deduplication using a generator, but you'd have to avoid using sets.
  - beagle3 12y ago
    I'm not sure about the internals of set(f), but the logically equivalent s = set() for x in f: s.add(x) Does not read everything into memory - by using iterators, it only reads one line at a time.
  - deleted 12y ago
    [deleted]
- deathanatos 12y ago
  Just to note, but if it's possible that your worst case is that all lines are unique, then you've effectively read the entire file into memory. But this just gets back to your point: > rather than thinking about the data itself …i.e, in this case, we need to realize that the set will require on the order of the size of how many unique items there are, and that in the worst case, that's O(n). Of course, if you can guarantee that the size of unique set will fit into memory, then you're fine. But you might need a bit of knowledge about the data to do so.

set() is a great way to deduplicate small lists, but it's important to note that it requires O(n) extra space (in-place sorting can avoid this overhead, but is more complex).

raymondh 12y ago

Sorting is almost always the wrong way to do it. (Wordy and slow). It is fragile design to write code that depends on 1) the data is so large that you don't have room for a set() BUT 2) it is small enough for an in-memory sort. (IOW, the almost-out-of-memory case invariably degrades over time to flat-out-of-memory). Another thought: people seem to place too much concern about about the size of various data structures rather than thinking about the data itself. Python containers don't contain anything, they just hold references. (Usually, the weight of a bucket of water is mostly the water, not the bucket itself). Finally, if your task is to dedup a lot of data, it doesn't make sense to read it all into memory in the first place (which you would need for a sorting approach). It is better dedup it using a set as you read in the data: # Only the unique lines are ever kept in memory with open('hugefile.txt') as f: uniq_lines = set(f) Dude, sorry to go off like this, but the advice you gave is almost always the wrong way to do it.

meowface 12y ago
You're still reading all of the lines into memory in that example as soon as you call `set(f)` (which is basically equivalent to set(f.readlines()), though, which may not necessarily be what you want.
- jzwinck 12y ago
  No, that's not at all what's happening. f.readlines() creates and returns a full list of all lines, loaded into memory. But set(f) uses f as an iterator, which reads chunks of the file and yields one line at a time, which can then be inserted into the set, de-duping on the fly. Your parent is correct (and clever).
  - bdevine 12y ago
    I think you may have gotten ahead of yourself in your explanation, but to be clear, it's the "with" statement that causes the line iteration that then feeds into set(f). To say that "set(f) uses f as an iterator" might imply to some that set() is causing the iteration, which isn't true. ETA: See raymondh's post below.
    - raymondh 12y ago
      Sorry, this post is also filled with misinformation. * The with-statement only causes the file to be closed after use. It has nothing to do with iteration. * Files themselves are self-iterable (anything that loops over a file object uses a line-at-a-time iterator). Even writing "for line in f: ..." causes you to loop a line at a time without the whole file being in memory all at once. * Sets are just one of many objects that takes an iterable as an argument: min(f), max(f), list(f), tuple(f), etc.
      - bdevine 12y ago
        Ah! Thanks for the correction. Somewhere along the way I picked up the belief that in addition to the file-closing aspect, "with" ensured a line-at-a-time iterator, akin to adding on "for line in f:". I'm not sure I understand your third comment though. As I understand it, iterables have an __iter__ method. __iter__ methods return iterators. So an iterator for the iterable f traverses f and sends f's values to, in this case, set(). I believe we agree there. My initial concern was simply that the statement could be read "The iterator, set(), uses f.", and I don't see where the disagreement arises.
  - meowface 12y ago
    It was incorrect of me to say it was equivalent, but if all lines in the file are unique (ignoring the parent comment's assumption there will be lots of duplicates) then you're still creating a large object in memory. I'm not sure of the space of sets/hash tables vs. lists, but I imagine they'd be prety close. But of course that will be unavoidable if you want fast deduplication. You could do fully lazy deduplication using a generator, but you'd have to avoid using sets.
- beagle3 12y ago
  I'm not sure about the internals of set(f), but the logically equivalent s = set() for x in f: s.add(x) Does not read everything into memory - by using iterators, it only reads one line at a time.
- deleted 12y ago
  [deleted]
deathanatos 12y ago
Just to note, but if it's possible that your worst case is that all lines are unique, then you've effectively read the entire file into memory. But this just gets back to your point: > rather than thinking about the data itself …i.e, in this case, we need to realize that the set will require on the order of the size of how many unique items there are, and that in the worst case, that's O(n). Of course, if you can guarantee that the size of unique set will fit into memory, then you're fine. But you might need a bit of knowledge about the data to do so.