3 ms·
This is cool for English text. But once you get Unicode with various ways to represent é, whew lad. This get shitty quickly in the shell.
by fnord123 7y ago
This is cool for English text. But once you get Unicode with various ways to represent é, whew lad. This get shitty quickly in the shell.
- zokier 7y agoI would generalize that further and say that shell processing works well when your data is well-behaved wrt the thing you are trying to do. But once you start venturing into untrusted data, escaping, and yes, Unicode, things can become hairy. So basically you should always be doing some sanity checks when working in shell
- bmn__ 7y agoEnglish is Unicode. Pretending otherwise would be quite naïve. https://www.azabani.com/pages/gbu/#slide4 https://www.azabani.com/pages/gbu/#slide4
- fnord123 7y agoYou claim is not clear. "Unicode with various ways to represent é" is a shit show to parse using shell tools. e.g. Try scraping Spanish language Twitter feeds. When I have done this kind of work, I made a tool to canonicalize glyphs and had to put it between every step of a pipeline.
- dublin 7y agoJust drop the eighth bit and use 7-bit ASCII like God intended and it's not a problem... :-)