4 ms·

For a sys-admin job: tr 'a-z' 'A-Z' | sed 's/[^A-Z][^A-Z]*/\ /g' | grep -v '^$' | sort | uniq -c | sort -nrk1 | head -10 edit: I don't know enough about HN,th

by mzs 13y ago

stass 13y ago
This is too complicated... echo $sentence | rs -T | sort | uniq -c | sort -rn|head -10
- mzs 13y ago
  Ooh that's a clever use of reshape, but wrong sadly. Yet you're right I could make it simpler, it was just what came first to mind, I did not even check it. Imagine typical sentences: $ echo Foo foo foo. | rs -T Foo foo foo. $ echo Foo foo foo. | tr 'a-z' 'A-Z' | sed 's/[^A-Z]/\ /g' | grep -v '^$' FOO FOO FOO Also the link mentioned using wiki articles for testing, so they would have paragraphs and that's where reshape shines for things like emails, but fails here: $ cat foo Foo foo foo. $ <foo rs -T Foo foo foo. $ <foo tr 'a-z' 'A-Z' | sed 's/[^A-Z]/\ /g' | grep -v '^$' FOO FOO FOO That that's though makes me think, should I treat contractions special? What about plurals? It's starting to get silly now.
  - stass 13y ago
    I think it goes beyond the original question now. :) If one wants to do proper thing, then definitely contractions, word variations, etc should be considered. For this, I guess something like OpenNLP can be used. What's interesting though is how UNIX shell, being essentially a symbolic FP language, allows one to solve the problem in a clear and concise way. And if one desires, the program can be easily modified to read sentences, say, from network from ssh tunneled via HTTPS. That kind of flexibility can rarely be achieved in languages like C# with tight coupling between the units of abstraction.
kragen 13y ago
You can use tr instead of sed and grep, too. Also -k1 and -n 10 are default for sort and head: $ < bible-pg10.txt tr -cs a-zA-Z '\n' | sort | uniq -c | sort -nr | head 62265 the 38915 and 34588 of 13474 to 12846 And 12589 that 12387 in 9762 shall 9668 he 8942 unto I have a `bins` script that is just `sort | uniq -c | sort -nr`, so that reduces to `tr -cs a-zA-Z '\n' | bins | head`. Unix for Poets, man. It's the shit.
- mzs 13y ago
  I ALWAYS forget about squeeze, thanks!

by mzs 13y ago

stass 13y ago

mzs 13y ago
Ooh that's a clever use of reshape, but wrong sadly. Yet you're right I could make it simpler, it was just what came first to mind, I did not even check it. Imagine typical sentences: $ echo Foo foo foo. | rs -T Foo foo foo. $ echo Foo foo foo. | tr 'a-z' 'A-Z' | sed 's/[^A-Z]/\ /g' | grep -v '^$' FOO FOO FOO Also the link mentioned using wiki articles for testing, so they would have paragraphs and that's where reshape shines for things like emails, but fails here: $ cat foo Foo foo foo. $ <foo rs -T Foo foo foo. $ <foo tr 'a-z' 'A-Z' | sed 's/[^A-Z]/\ /g' | grep -v '^$' FOO FOO FOO That that's though makes me think, should I treat contractions special? What about plurals? It's starting to get silly now.
- stass 13y ago
  I think it goes beyond the original question now. :) If one wants to do proper thing, then definitely contractions, word variations, etc should be considered. For this, I guess something like OpenNLP can be used. What's interesting though is how UNIX shell, being essentially a symbolic FP language, allows one to solve the problem in a clear and concise way. And if one desires, the program can be easily modified to read sentences, say, from network from ssh tunneled via HTTPS. That kind of flexibility can rarely be achieved in languages like C# with tight coupling between the units of abstraction.

kragen 13y ago

You can use tr instead of sed and grep, too. Also -k1 and -n 10 are default for sort and head: $ < bible-pg10.txt tr -cs a-zA-Z '\n' | sort | uniq -c | sort -nr | head 62265 the 38915 and 34588 of 13474 to 12846 And 12589 that 12387 in 9762 shall 9668 he 8942 unto I have a `bins` script that is just `sort | uniq -c | sort -nr`, so that reduces to `tr -cs a-zA-Z '\n' | bins | head`. Unix for Poets, man. It's the shit.

mzs 13y ago
I ALWAYS forget about squeeze, thanks!