2 ms·
A more obvious optimisation (to me) would be to leverage the native functions and avoid creating a list of ~80 million strings in memory. On my machine, the ba
by masklinn 4y ago
A more obvious optimisation (to me) would be to leverage the native functions and avoid creating a list of ~80 million strings in memory.
On my machine, the base script pretty reliably takes ~10s:
Reading : 0.1935129165649414
Processing: 9.955206871032715
Sorting : 0.0067043304443359375
Outputting: 0.01335597038269043
TOTAL : 10.168780088424683
Switching content to a no-op (`content = sys.stdin`) and feeding `Counter` from a native iterators pipeline:
counts = collections.Counter(chain.from_iterable(map(str.split, map(str.lower, content))))
is a pretty reliable 10% gain:
Reading : 1.1920928955078125e-06
Processing: 8.863707780838013
Sorting : 0.004117012023925781
Outputting: 0.012418985366821289
TOTAL : 8.880244970321655
As far as I can tell, the bottleneck is about half the preprocessing (lowercasing and splitting) and half filling the Counter.
You won't get a 10x gain out of that though.