5 ms·
Reducing Python String Memory Use in Apache Arrow 0.12 (2019)
- taeric 5y agoRelatedly, perhaps the most impactful performance optimization I let some of the science side know about on my team is to declare columns as categorical in this same basic instance. The memory and speed gain is impressive.
- wumpus 5y agoIt would be fun to compare the two. Interesting that the blog author is the creator of Pandas and didn't include a comparison.
- vletal 5y agoThere should be "string deduplication" in the original title. That would be informative and targeted more to Arrow users.
- polskibus 5y ago2019
- rav 5y agoSimple scheme for deduplicating strings in Python: dedup = {} for s in strings: s = dedup.setdefault(s, s) # Do something with s It hashes each string "s" and looks in the dictionary "dedup" to see if there's an existing string object previously in the list "strings", in which case it uses it, and otherwise it stores the current string object in "dedup".
- bombela 5y agodedup = set(strings)
- quietbritishjim 5y agoThen what do you replace this line with? s = dedup.setdefault(s, s) There's not an obvious way to get the object in a set that matches a value you have.
- masklinn 5y agoWhile this works for dedup'ing a single set of strings, it doesn't work for dedup'ing string identities across the application lifetime because you don't have a fast way of fetching the deduplicated version from the base.
- masklinn 5y agoSimpler scheme for deduplicating strings in Python: use `sys.intern`. It also specifically increases hash performances because interned strings can be compared by address after a hash collision (and furthermore hash caching means interned strings almost certainly already have their hash computed and stored) for s in strings: s = sys.intern(s) # Do something with s Depending on your application lifecycle you may want to shove the dedup'd strings in a set (or in a list, after checking if the identity of what you got from `sys.intern` is different than what you put in) in order to keep them alive: `sys.intern` does not do that.
- fulafel 5y agoInterestingly according to docs this doesn't prevent your stirngs from being GC'd either. "Interned strings are not immortal; you must keep a reference to the return value of intern() around to benefit from it"
- masklinn 5y ago> Interestingly according to docs Also the second paragraph of my comment.
- ciupicri 5y agoI use something like: @functools.cache def dedup(s): return sys.intern(s)
- twic 5y agoIsn't sys.intern a cache itself? What does adding another layer of cacheing on top add?
- ciupicri 5y agoYou're right, PyUnicode_InternInPlace (which is called by sys.intern) already caches results using a regular Python dictionary, so @functools.cache wouldn't add any value. It happened a long time ago, but I think I misread something about sys.intern not being able to handle lots and lots of strings, so I added an extra layer of caching, but it looks like it's not needed anymore. [1]: https://github.com/python/cpython/blob/3.10/Objects/unicodeobject.c#L15794 https://github.com/python/cpython/blob/3.10/Objects/unicodeo...
- deleted 5y ago[deleted]