3 ms·
I tried it on the text of the article itself. Here's my cmdline: # cat file | preserve only spaces and letters (remove punctuation) | convert to lowercase and
by gerbilly 8y ago
I tried it on the text of the article itself.
Here's my cmdline:
# cat file | preserve only spaces and letters (remove punctuation) | convert to lowercase and put each word on one line | use awk maps to keep a count for each word and print map in two columns, count followed by word | sort in reverse numerically by count column, then by word ascending | print top 10 words by frequency
cat input.txt | sed -e 's/[^a-zA-Z ]//g' | tr -s ' A-Z' '\na-z' | awk -F ' ' '{ if($1 in count) { count[$1]=count[$1]+1 } else { count [$1] = 1 } } END { for(k in count) { printf("%d\t%s\n", count[k],k) } }' | sort -k1,1nr -k2,2 | sed -n -e '1,10 p'
Output:
27 the
21 to
15 a
11 is
10 of
10 that
9 but
8 text
7 was
6 and
- tln 8y agoHere's my version using a different "command line": chrome devtools Object.entries(document.body.innerText.replace(/[^a-zA-Z\s]/g, '').toLowerCase().split(/\s+/).reduce((o, word) => {o[word] = (o[word] || 0) + 1; return o}, {})).sort((a,b) => b[1] - a[1]).slice(0, 10) Plain JS is missing some niceties, but being able to crunch some data from a website using the same iterative approach as a shell command line has its place!
- pcnix 8y agoThis is pretty cool, I haven't really used the console for something like this in a while, and it's a good reminder for its power.
- yesenadam 8y agoInstead of { if($1 in count) { count[$1]=count[$1]+1 } else { count [$1] = 1 } } , {count[$1]++} does and means exactly the same thing. Why bother 'changing' the input field separator? No need. By default it's already space. awk '{count[$1]++} END {for(k in count) print count[k],k}}'
- gerbilly 8y agoNeat. I wasn't sure what awk would do if there was no entry when it first encountered a new word. And yeah, I left the -F in there by mistake. But I don't get overly fussy with pipelines like this. I usually just hack them up as a one off then forget about them.
- majewsky 8y agoSome more simplification suggestions: first sed --> tr -dc 'a-zA-Z \n' second sed --> head