5 ms·

The problem is not in the filesystem, it's in the shell. Stop writing shell scripts for tasks that need to be robust! None of these problems hit you when using

by catern 8y ago

The problem is not in the filesystem, it's in the shell. Stop writing shell scripts for tasks that need to be robust! None of these problems hit you when using a better programming language like Python. (Yes, Python 3 has a bit of an impedance mismatch between its native Unicode string type and the filesystem, but that just makes for slightly uglier code, not actual bugs)

blablabla123 8y ago
I wonder how much effort it would take, to make the shell robust when it comes to this. Obviously in Python 3 it's not a big deal to process UTF8 strings with control characters, store them in a database and still have decent XSS protection with no ("manual") input filter. But with Python you loose the flexibility of pipes, combining commands tersely on the fly.
- JdeBP 8y ago
  Of course, the problem here is in thinking that there is "the" shell, and that one needs to always use a POSIX compatible shell for interactive login. (The C shell users of the world should disabuse one of that particular notion, if nothing else.) * https://news.ycombinator.com/item?id=11672023 https://news.ycombinator.com/item?id=11672023 * https://news.ycombinator.com/item?id=17989710 https://news.ycombinator.com/item?id=17989710
benj111 8y ago
Surely the problem is the 'Everything is a stream of bytes' model? Eg: Ls | foo | bar Ls just prints nl separated strings, no metadata, nothing to tell you what it is. In current practical terms that means avoiding the shell, that isn't inherent though. I guess powershell avoids this issue.
- JdeBP 8y ago
  Hence Greg Wooledge's famous advice not to parse the output of ls. As catern points out, in a fully-fledged systems programming language, one can just directly call the library routines that read directories. And thereby obtain metadata without having to parse it from locale-dependent forms, too. * https://mywiki.wooledge.org/ParsingLs https://mywiki.wooledge.org/ParsingLs * https://unix.stackexchange.com/a/503711/5132 https://unix.stackexchange.com/a/503711/5132

by catern 8y ago

blablabla123 8y ago

I wonder how much effort it would take, to make the shell robust when it comes to this. Obviously in Python 3 it's not a big deal to process UTF8 strings with control characters, store them in a database and still have decent XSS protection with no ("manual") input filter. But with Python you loose the flexibility of pipes, combining commands tersely on the fly.

JdeBP 8y ago
Of course, the problem here is in thinking that there is "the" shell, and that one needs to always use a POSIX compatible shell for interactive login. (The C shell users of the world should disabuse one of that particular notion, if nothing else.) * https://news.ycombinator.com/item?id=11672023 https://news.ycombinator.com/item?id=11672023 * https://news.ycombinator.com/item?id=17989710 https://news.ycombinator.com/item?id=17989710

benj111 8y ago

Surely the problem is the 'Everything is a stream of bytes' model? Eg: Ls | foo | bar Ls just prints nl separated strings, no metadata, nothing to tell you what it is. In current practical terms that means avoiding the shell, that isn't inherent though. I guess powershell avoids this issue.

JdeBP 8y ago
Hence Greg Wooledge's famous advice not to parse the output of ls. As catern points out, in a fully-fledged systems programming language, one can just directly call the library routines that read directories. And thereby obtain metadata without having to parse it from locale-dependent forms, too. * https://mywiki.wooledge.org/ParsingLs https://mywiki.wooledge.org/ParsingLs * https://unix.stackexchange.com/a/503711/5132 https://unix.stackexchange.com/a/503711/5132