2 ms·
import mmap, codecs from collections import Counter def word_count(filepath): freq = Counter() decode = codecs.getincrementaldec
by est 7mo ago
import mmap, codecs
from collections import Counter
def word_count(filepath):
freq = Counter()
decode = codecs.getincrementaldecoder('utf-8')().decode
with open(filepath, 'rb') as f, mmap.mmap(f.fileno(), 0, access=mmap.ACCESS_READ) as mm:
for chunk in iter(lambda: mm.read(65536), b''):
freq.update(decode(chunk).split())
freq.update(decode(b'', final=True).split())
return freq
- contravariant 6mo agoOh that's neat, though I might split this into two functions in most cases, no need to entangle opening the file and counting the words in a filelike object. That's two neat tricks that I'm definitely adding to my bag of python trickery.