I tried a perl script versus the shell pipeline, see my other comments in the thread. It's significantly faster, because using sort and uniq -c is a pretty high

I tried a perl script versus the shell pipeline, see my other comments in the thread. It's significantly faster, because using sort and uniq -c is a pretty high effort way to count words, especially for big lists. Your words seem to work fine with it, it's just splitting on whitespace.