4 ms·

I actually wrote a sed utility to collapse arbitrary whitespace before piping into cut before I learned about Awk. I never knew about -w before, but looks like

I actually wrote a sed utility to collapse arbitrary whitespace before piping into cut before I learned about Awk. I never knew about -w before, but looks like neither of the systems I'm currently using for work support it.

vram22 8y ago
>I actually wrote a sed utility to collapse arbitrary whitespace Can be done with tr, without writing a utility (if by "collapse" you mean what I think you do): This file t: $ cat t the quick brown fox jumped over the lazy dog containing many combinations of spaces, tabs and newlines (whitespace) can be changed to this file t2: $ cat t2 the quick brown fox jumped over the lazy dog by this tr command: tr -cs "[a-zA-Z]" "\012" < t > t2 That also makes the output more amenable to further processing, including common tasks like finding the frequencies of the words in the input, as mentioned in the "More shell, less egg" post mentioned in this post: The Bentley-Knuth problem and solutions: https://jugad2.blogspot.com/2012/07/the-bentley-knuth-problem-and-solutions.html https://jugad2.blogspot.com/2012/07/the-bentley-knuth-proble...
- e12e 8y ago
  Does that work with utf8 files? I hardly ever work with us ascii files anymore.
  - vram22 8y ago
    Don't know. It was a while back, and in Python 2 (if you mean the Python version I wrote). I didn't take any special steps to support Unicode, so likely not. Same for the shell version I wrote.
    - vram22 8y ago
      Just realized you may have meant the tr command I used - whether it supports UTF-8. Don't know about that either.
    - e12e 8y ago
      I should've been more explicit, but was thinking about: tr -cs "[a-zA-Z]" "\012" < t > t2 (which I somehow managed to read as an awk invocation). Re: python - I believe if using things like \w, \d or \s you should be Unicode safe. Come to think of it, I seem to recall gnu tools should also have Unicode aware pattern/character classes, eg: https://www.gnu.org/software/gawk/manual/html_node/Bracket-Expressions.html https://www.gnu.org/software/gawk/manual/html_node/Bracket-E... > For example, before the POSIX standard, you had to write /[A-Za-z0-9]/ to match alphanumeric characters. If your character set had other alphabetic characters in it, this would not match them. With the POSIX character classes, you can write /[[:alnum:]]/ to match the alphabetic and numeric characters in your character set. https://docs.python.org/3/library/re.html#re-syntax https://docs.python.org/3/library/re.html#re-syntax [ed: Apparently gnu tr is still out in the cold re:unicode; https://www.gnu.org/software/coreutils/manual/html_node/tr-invocation.html#tr-invocation https://www.gnu.org/software/coreutils/manual/html_node/tr-i... > Currently tr fully supports only single-byte characters. Eventually it will support multibyte characters; when it does, the -C option will cause it to complement the set of characters, whereas -c will cause it to complement the set of values. This distinction will matter only when some values are not characters, and this is possible only in locales using multibyte encodings when the input contains encoding errors. ]
      - vram22 8y ago
        Got it. Good info, thanks.

vram22 8y ago

>I actually wrote a sed utility to collapse arbitrary whitespace Can be done with tr, without writing a utility (if by "collapse" you mean what I think you do): This file t: $ cat t the quick brown fox jumped over the lazy dog containing many combinations of spaces, tabs and newlines (whitespace) can be changed to this file t2: $ cat t2 the quick brown fox jumped over the lazy dog by this tr command: tr -cs "[a-zA-Z]" "\012" < t > t2 That also makes the output more amenable to further processing, including common tasks like finding the frequencies of the words in the input, as mentioned in the "More shell, less egg" post mentioned in this post: The Bentley-Knuth problem and solutions: https://jugad2.blogspot.com/2012/07/the-bentley-knuth-problem-and-solutions.html https://jugad2.blogspot.com/2012/07/the-bentley-knuth-proble...

e12e 8y ago
Does that work with utf8 files? I hardly ever work with us ascii files anymore.
- vram22 8y ago
  Don't know. It was a while back, and in Python 2 (if you mean the Python version I wrote). I didn't take any special steps to support Unicode, so likely not. Same for the shell version I wrote.
  - vram22 8y ago
    Just realized you may have meant the tr command I used - whether it supports UTF-8. Don't know about that either.
  - e12e 8y ago
    I should've been more explicit, but was thinking about: tr -cs "[a-zA-Z]" "\012" < t > t2 (which I somehow managed to read as an awk invocation). Re: python - I believe if using things like \w, \d or \s you should be Unicode safe. Come to think of it, I seem to recall gnu tools should also have Unicode aware pattern/character classes, eg: https://www.gnu.org/software/gawk/manual/html_node/Bracket-Expressions.html https://www.gnu.org/software/gawk/manual/html_node/Bracket-E... > For example, before the POSIX standard, you had to write /[A-Za-z0-9]/ to match alphanumeric characters. If your character set had other alphabetic characters in it, this would not match them. With the POSIX character classes, you can write /[[:alnum:]]/ to match the alphabetic and numeric characters in your character set. https://docs.python.org/3/library/re.html#re-syntax https://docs.python.org/3/library/re.html#re-syntax [ed: Apparently gnu tr is still out in the cold re:unicode; https://www.gnu.org/software/coreutils/manual/html_node/tr-invocation.html#tr-invocation https://www.gnu.org/software/coreutils/manual/html_node/tr-i... > Currently tr fully supports only single-byte characters. Eventually it will support multibyte characters; when it does, the -C option will cause it to complement the set of characters, whereas -c will cause it to complement the set of values. This distinction will matter only when some values are not characters, and this is possible only in locales using multibyte encodings when the input contains encoding errors. ]
    - vram22 8y ago
      Got it. Good info, thanks.