5 ms·
That implementation looks pretty inefficient, creating lots of strings that are thrown away immediately. Could it not be refactored into a generator combined wi
by orf 10y ago
That implementation looks pretty inefficient, creating lots of strings that are thrown away immediately. Could it not be refactored into a generator combined with a string.join?
- dom0 10y agoUsual use would be "translate pattern once, compile it and then match 1000 file names against it".
- orf 10y agoAhh yes, it's cached in fnmatchcase. It's pretty fast as well, <10µs for a basic match.
- bluetech 10y agoA cache without any eviction! Probably should not expose `fnmatch` to the network where the pattern varies.
- dom0 10y agoThe RE engine here is only used as a fast VM; so the pathological cases from the article should still apply. So apart from the memory use issues there's also that.
- orf 10y agoIt should be updated with a functools.lru_cache decorator I think.
- benhoyt 10y agoThat's an old version of Python source from a mirror. It uses functools.lru_cache with a maxsize of 256 now: https://github.com/python/cpython/blob/fcfe80ec2592fed8b3941c79056a8737abef7d3b/Lib/fnmatch.py#L38 https://github.com/python/cpython/blob/fcfe80ec2592fed8b3941...
- js2 10y agoGah, thanks for the proper link. I should've double checked (and realized it since no updates in 8 years is unlikely).
- munificent 10y agoI think Python's string type has an optimization that concatenation on a string where the LHS only has a single reference to it will reuse the existing string when possible. You still have to grow the buffer periodically, of course, but it's not creating as much garbage as it looks like.
- masklinn 10y ago> I think Python's string type has an optimization that concatenation on a string where the LHS only has a single reference to it will reuse the existing string when possible. It's specifically a CPython optimisation, it may (and usually will) not hold up on alternate implementations like pypy or jyton.
- munificent 10y agoSure, but the function in question lives in CPython's own implementation of its core libraries, so it seems reasonable for it to rely on that implementation. If PyPy or Jython want to reuse that core library code, it's on them to make whatever changes are needed to make it fast on those implementations.