4 ms·
Simple scheme for deduplicating strings in Python: dedup = {} for s in strings: s = dedup.setdefault(s, s) # Do something with s It ha
by rav 5y ago
Simple scheme for deduplicating strings in Python:
dedup = {}
for s in strings:
s = dedup.setdefault(s, s)
# Do something with s
It hashes each string "s" and looks in the dictionary "dedup" to see if there's an existing string object previously in the list "strings", in which case it uses it, and otherwise it stores the current string object in "dedup".
- bombela 5y agodedup = set(strings)
- quietbritishjim 5y agoThen what do you replace this line with? s = dedup.setdefault(s, s) There's not an obvious way to get the object in a set that matches a value you have.
- masklinn 5y agoWhile this works for dedup'ing a single set of strings, it doesn't work for dedup'ing string identities across the application lifetime because you don't have a fast way of fetching the deduplicated version from the base.
- masklinn 5y agoSimpler scheme for deduplicating strings in Python: use `sys.intern`. It also specifically increases hash performances because interned strings can be compared by address after a hash collision (and furthermore hash caching means interned strings almost certainly already have their hash computed and stored) for s in strings: s = sys.intern(s) # Do something with s Depending on your application lifecycle you may want to shove the dedup'd strings in a set (or in a list, after checking if the identity of what you got from `sys.intern` is different than what you put in) in order to keep them alive: `sys.intern` does not do that.
- fulafel 5y agoInterestingly according to docs this doesn't prevent your stirngs from being GC'd either. "Interned strings are not immortal; you must keep a reference to the return value of intern() around to benefit from it"
- masklinn 5y ago> Interestingly according to docs Also the second paragraph of my comment.
- ciupicri 5y agoI use something like: @functools.cache def dedup(s): return sys.intern(s)
- twic 5y agoIsn't sys.intern a cache itself? What does adding another layer of cacheing on top add?
- ciupicri 5y agoYou're right, PyUnicode_InternInPlace (which is called by sys.intern) already caches results using a regular Python dictionary, so @functools.cache wouldn't add any value. It happened a long time ago, but I think I misread something about sys.intern not being able to handle lots and lots of strings, so I added an extra layer of caching, but it looks like it's not needed anymore. [1]: https://github.com/python/cpython/blob/3.10/Objects/unicodeobject.c#L15794 https://github.com/python/cpython/blob/3.10/Objects/unicodeo...