3 ms·
Simpler scheme for deduplicating strings in Python: use `sys.intern`. It also specifically increases hash performances because interned strings can be compared
by masklinn 5y ago
Simpler scheme for deduplicating strings in Python: use `sys.intern`. It also specifically increases hash performances because interned strings can be compared by address after a hash collision (and furthermore hash caching means interned strings almost certainly already have their hash computed and stored)
for s in strings:
s = sys.intern(s)
# Do something with s
Depending on your application lifecycle you may want to shove the dedup'd strings in a set (or in a list, after checking if the identity of what you got from `sys.intern` is different than what you put in) in order to keep them alive: `sys.intern` does not do that.
- fulafel 5y agoInterestingly according to docs this doesn't prevent your stirngs from being GC'd either. "Interned strings are not immortal; you must keep a reference to the return value of intern() around to benefit from it"
- masklinn 5y ago> Interestingly according to docs Also the second paragraph of my comment.