3 ms·

Don't know. It was a while back, and in Python 2 (if you mean the Python version I wrote). I didn't take any special steps to support Unicode, so likely not. Sa

by vram22 8y ago

Don't know. It was a while back, and in Python 2 (if you mean the Python version I wrote). I didn't take any special steps to support Unicode, so likely not. Same for the shell version I wrote.

vram22 8y ago
Just realized you may have meant the tr command I used - whether it supports UTF-8. Don't know about that either.
e12e 8y ago
I should've been more explicit, but was thinking about: tr -cs "[a-zA-Z]" "\012" < t > t2 (which I somehow managed to read as an awk invocation). Re: python - I believe if using things like \w, \d or \s you should be Unicode safe. Come to think of it, I seem to recall gnu tools should also have Unicode aware pattern/character classes, eg: https://www.gnu.org/software/gawk/manual/html_node/Bracket-Expressions.html https://www.gnu.org/software/gawk/manual/html_node/Bracket-E... > For example, before the POSIX standard, you had to write /[A-Za-z0-9]/ to match alphanumeric characters. If your character set had other alphabetic characters in it, this would not match them. With the POSIX character classes, you can write /[[:alnum:]]/ to match the alphabetic and numeric characters in your character set. https://docs.python.org/3/library/re.html#re-syntax https://docs.python.org/3/library/re.html#re-syntax [ed: Apparently gnu tr is still out in the cold re:unicode; https://www.gnu.org/software/coreutils/manual/html_node/tr-invocation.html#tr-invocation https://www.gnu.org/software/coreutils/manual/html_node/tr-i... > Currently tr fully supports only single-byte characters. Eventually it will support multibyte characters; when it does, the -C option will cause it to complement the set of characters, whereas -c will cause it to complement the set of values. This distinction will matter only when some values are not characters, and this is possible only in locales using multibyte encodings when the input contains encoding errors. ]
- vram22 8y ago
  Got it. Good info, thanks.

by vram22 8y ago

Don't know. It was a while back, and in Python 2 (if you mean the Python version I wrote). I didn't take any special steps to support Unicode, so likely not. Same for the shell version I wrote.

vram22 8y ago

Just realized you may have meant the tr command I used - whether it supports UTF-8. Don't know about that either.

e12e 8y ago

I should've been more explicit, but was thinking about: tr -cs "[a-zA-Z]" "\012" < t > t2 (which I somehow managed to read as an awk invocation). Re: python - I believe if using things like \w, \d or \s you should be Unicode safe. Come to think of it, I seem to recall gnu tools should also have Unicode aware pattern/character classes, eg: https://www.gnu.org/software/gawk/manual/html_node/Bracket-Expressions.html https://www.gnu.org/software/gawk/manual/html_node/Bracket-E... > For example, before the POSIX standard, you had to write /[A-Za-z0-9]/ to match alphanumeric characters. If your character set had other alphabetic characters in it, this would not match them. With the POSIX character classes, you can write /[[:alnum:]]/ to match the alphabetic and numeric characters in your character set. https://docs.python.org/3/library/re.html#re-syntax https://docs.python.org/3/library/re.html#re-syntax [ed: Apparently gnu tr is still out in the cold re:unicode; https://www.gnu.org/software/coreutils/manual/html_node/tr-invocation.html#tr-invocation https://www.gnu.org/software/coreutils/manual/html_node/tr-i... > Currently tr fully supports only single-byte characters. Eventually it will support multibyte characters; when it does, the -C option will cause it to complement the set of characters, whereas -c will cause it to complement the set of values. This distinction will matter only when some values are not characters, and this is possible only in locales using multibyte encodings when the input contains encoding errors. ]

vram22 8y ago
Got it. Good info, thanks.