7 ms·

The problem is not in the filesystem, it's in the shell. Stop writing shell scripts for tasks that need to be robust! None of these problems hit you when using

by catern 8y ago

The problem is not in the filesystem, it's in the shell. Stop writing shell scripts for tasks that need to be robust! None of these problems hit you when using a better programming language like Python. (Yes, Python 3 has a bit of an impedance mismatch between its native Unicode string type and the filesystem, but that just makes for slightly uglier code, not actual bugs)

xc1111 8y ago
The proposal just wants to eliminate problematic names that no one uses anyway. Python 3 is full of problems with fsencoding, to the point that some people want to deprecate the plain C locale (!). This is hardly a solution.
- pbhjpbhj 8y ago
  There's a manpage that cautions how to handle this of you called your file "-" (without the quotes) IIRC ... some people just want to watch the World burn.
- dwheeler 8y ago
  That's right, no one really uses these kinds of filenames anyway unless they are an attacker. Supporting them simply makes it easy to end up with security vulnerabilities, and it also makes it harder to write code. Let's make things easy.
  - moby_click 8y ago
    On the other hand, people may try to create "Fixing Unix/Linux/Posix Filenames.html". I'd much rather have any bytestring as filename and explicit structure.
unilynx 8y ago
It wouldn't just be shell scripts that need replacement - the command line is generally also a shell with the same problems, so that needs replacement too The chances of getting the kernel to outright ban some of the more dangerous filenames seem a lot higher than simply eliminating sh/bash...
- dwheeler 8y ago
  I think the best would be to create an LSM module. I created one a while back but have not had time to complete it. I called it safename: https://github.com/david-a-wheeler/linux/blob/squelch-1/security/safename/safename_lsm.c https://github.com/david-a-wheeler/linux/blob/squelch-1/secu...
  - geofft 8y ago
    Unfortunately this wouldn't catch names from network filesystems, removable media, etc., which comes up fairly frequently. (Or, more generally, from existing disks.) Is there a place in the Linux kernel you can intervene on existing filesystems to escape bad names from existing filesystems and refuse the creation of new, unescaped bad names? Could this be something unionfs-ish?
dwheeler 8y ago
False. Many of these problems affect all languages, including Python3. There are still many cases where you need to execute external programs. It's true that some problems are worse in shell, but even then, most languages (including Python) have constructs that call shell. Since the problems hit all languages, we should work on trying to make things better. Where possible, the simple and obvious thing should be the safe thing.
- kazinator 8y ago
  I completely agree with your idea; I've long believed that the OS kernel should reject dubious filenames.
  - kazinator 8y ago
    However, from past thoughts and discussions, there are issues. (1) We have to be able to mount filesystems which have bad filenames and allow users to work with them. In that situation, that means the kernel cannot reject all requests for bad file names, only creation requests. And perhaps not even that. It can be a mount option with three values: off; reject all dubious names in all filesystem syscalls; or just in creating syscalls: open with O_CREAT, mknod, link, ... It could also be a capability. Superuser could work with the dubious filenames and set up capabilities for non-privileged child processes to do same, so people needing to work with a filesystem full of these names can get into an environment where they can do that. (2) Enforcement at the kernel level is too late in some situations. It doesn't entirely compensate for broken programs. Here is a trivial example of what this means. Suppose that some program receives an argument "foo bar" and decides to split that into "foo" and "bar", treating this as two file names to create or process. Those filenames look fine to the kernel, so it is allowed. In shell scripts, not all name-related issues arise from files that are already in the system. In fact, shell scripts are often robust in the face of existing files with poor names. For instance names generated with globbing expansion are safe, even if they contain spaces and shell meta-characters. The one danger is that the prefix of the expanded list looks like command line options to the program. The issues that arise in shell programming from failing to quote variable expansions cannot be fixed outside of the shell. The OS cannot fix the lack of quoting in command $VAR; it can't prevent a shell internal variable VAR from containing spaces.
ufo 8y ago
As the article already points out, not using the shell (or fixing the shell) would fix many of the problems with spaces and newlines in filenames but there are still others that are more fundamental. For example: * Filenames starting with hyphens can get parsed as command line options * Filenames containing control sequences cause the terminal to misbehave if they are printed to the screen * Filenames containing newlines can't be easily included in all sorts of line-by-line file formats. Finally, as the article also points out, the shell is not going away anytime soon. Getting rid of it is not the easy way out.
nerdponx 8y ago
Why should we stop using a popular and effective tool in order to accommodate an unpopular and archaic set of non-features?
hedora 8y ago
I’ve had to deal with maintenance of multiple python programs that could have been done in shell instead. They were written in python 2 and 3 by different teams of strong developers at different organizations at different times. They were universally unmaintainable, unreliable and undebuggable. Some narrowly avoided installing pip-distributed malware on internal networks. There is a reason shell is still widely used. I have never worked with anyone that actually knows one of the shell languages that prefers python for shell scripting tasks.
- alerighi 8y ago
  You have dealt with bad written python programs, sure, but if you have to deal with a bad written shell script, it's much worse! The shell is full of strange and dangerous behaviors, that make even experienced programmers write buggy scripts. The main reason is that the shell wants to be a programming language but works differently from every other programming language, and has a special and complicated syntax. Also shell scripts are really not that portable at all, if you write a script on a GNU system it's nearly guaranteed that it will not work on a non-GNU system, and vice versa. Not only the shell but more importantly the commands are different, GNU awk is different than standard UNIX awk, GNU sed is different that UNIX sed, grep is different, and so on. Maybe minor difference that produces bug difficult to find and fix. For me if a shell script becomes more long than 50/100 lines of code you should consider rewriting it in python, shell scripts are only a valid solutions to rapidly write programs that invokes some commands to automate some things, not a valid solution for doing complex tasks.
pmoriarty 8y ago
In addition to all of the other problems with Python mentioned in this thread, Python can also be incredibly slow to start compared to sh. Another problem is that doing many things that would be simple in the shell require a lot of boilerplate in Python. That kind of boilerplate might be worth putting up with for large programs, but for something that could be easily handled with a short shell script it's really annoying.

by catern 8y ago

xc1111 8y ago

The proposal just wants to eliminate problematic names that no one uses anyway. Python 3 is full of problems with fsencoding, to the point that some people want to deprecate the plain C locale (!). This is hardly a solution.

pbhjpbhj 8y ago
There's a manpage that cautions how to handle this of you called your file "-" (without the quotes) IIRC ... some people just want to watch the World burn.
dwheeler 8y ago
That's right, no one really uses these kinds of filenames anyway unless they are an attacker. Supporting them simply makes it easy to end up with security vulnerabilities, and it also makes it harder to write code. Let's make things easy.
- moby_click 8y ago
  On the other hand, people may try to create "Fixing Unix/Linux/Posix Filenames.html". I'd much rather have any bytestring as filename and explicit structure.

unilynx 8y ago

It wouldn't just be shell scripts that need replacement - the command line is generally also a shell with the same problems, so that needs replacement too The chances of getting the kernel to outright ban some of the more dangerous filenames seem a lot higher than simply eliminating sh/bash...

dwheeler 8y ago
I think the best would be to create an LSM module. I created one a while back but have not had time to complete it. I called it safename: https://github.com/david-a-wheeler/linux/blob/squelch-1/security/safename/safename_lsm.c https://github.com/david-a-wheeler/linux/blob/squelch-1/secu...
- geofft 8y ago
  Unfortunately this wouldn't catch names from network filesystems, removable media, etc., which comes up fairly frequently. (Or, more generally, from existing disks.) Is there a place in the Linux kernel you can intervene on existing filesystems to escape bad names from existing filesystems and refuse the creation of new, unescaped bad names? Could this be something unionfs-ish?

dwheeler 8y ago

False. Many of these problems affect all languages, including Python3. There are still many cases where you need to execute external programs. It's true that some problems are worse in shell, but even then, most languages (including Python) have constructs that call shell. Since the problems hit all languages, we should work on trying to make things better. Where possible, the simple and obvious thing should be the safe thing.

kazinator 8y ago
I completely agree with your idea; I've long believed that the OS kernel should reject dubious filenames.
- kazinator 8y ago
  However, from past thoughts and discussions, there are issues. (1) We have to be able to mount filesystems which have bad filenames and allow users to work with them. In that situation, that means the kernel cannot reject all requests for bad file names, only creation requests. And perhaps not even that. It can be a mount option with three values: off; reject all dubious names in all filesystem syscalls; or just in creating syscalls: open with O_CREAT, mknod, link, ... It could also be a capability. Superuser could work with the dubious filenames and set up capabilities for non-privileged child processes to do same, so people needing to work with a filesystem full of these names can get into an environment where they can do that. (2) Enforcement at the kernel level is too late in some situations. It doesn't entirely compensate for broken programs. Here is a trivial example of what this means. Suppose that some program receives an argument "foo bar" and decides to split that into "foo" and "bar", treating this as two file names to create or process. Those filenames look fine to the kernel, so it is allowed. In shell scripts, not all name-related issues arise from files that are already in the system. In fact, shell scripts are often robust in the face of existing files with poor names. For instance names generated with globbing expansion are safe, even if they contain spaces and shell meta-characters. The one danger is that the prefix of the expanded list looks like command line options to the program. The issues that arise in shell programming from failing to quote variable expansions cannot be fixed outside of the shell. The OS cannot fix the lack of quoting in command $VAR; it can't prevent a shell internal variable VAR from containing spaces.

ufo 8y ago

As the article already points out, not using the shell (or fixing the shell) would fix many of the problems with spaces and newlines in filenames but there are still others that are more fundamental. For example: * Filenames starting with hyphens can get parsed as command line options * Filenames containing control sequences cause the terminal to misbehave if they are printed to the screen * Filenames containing newlines can't be easily included in all sorts of line-by-line file formats. Finally, as the article also points out, the shell is not going away anytime soon. Getting rid of it is not the easy way out.

nerdponx 8y ago

Why should we stop using a popular and effective tool in order to accommodate an unpopular and archaic set of non-features?

hedora 8y ago

I’ve had to deal with maintenance of multiple python programs that could have been done in shell instead. They were written in python 2 and 3 by different teams of strong developers at different organizations at different times. They were universally unmaintainable, unreliable and undebuggable. Some narrowly avoided installing pip-distributed malware on internal networks. There is a reason shell is still widely used. I have never worked with anyone that actually knows one of the shell languages that prefers python for shell scripting tasks.

alerighi 8y ago
You have dealt with bad written python programs, sure, but if you have to deal with a bad written shell script, it's much worse! The shell is full of strange and dangerous behaviors, that make even experienced programmers write buggy scripts. The main reason is that the shell wants to be a programming language but works differently from every other programming language, and has a special and complicated syntax. Also shell scripts are really not that portable at all, if you write a script on a GNU system it's nearly guaranteed that it will not work on a non-GNU system, and vice versa. Not only the shell but more importantly the commands are different, GNU awk is different than standard UNIX awk, GNU sed is different that UNIX sed, grep is different, and so on. Maybe minor difference that produces bug difficult to find and fix. For me if a shell script becomes more long than 50/100 lines of code you should consider rewriting it in python, shell scripts are only a valid solutions to rapidly write programs that invokes some commands to automate some things, not a valid solution for doing complex tasks.

pmoriarty 8y ago

In addition to all of the other problems with Python mentioned in this thread, Python can also be incredibly slow to start compared to sh. Another problem is that doing many things that would be simple in the shell require a lot of boilerplate in Python. That kind of boilerplate might be worth putting up with for large programs, but for something that could be easily handled with a short shell script it's really annoying.