3 ms·

That's kind of cure worse than disease. Just ditch escaping completely.

by rini17 2y ago

yjftsjthsd-h 2y ago
If you completely ditch escaping, how do you handle filenames that contain special characters (in this context, mostly ? and *, but ()[] are also perennial favorites)? And to preempt the most obvious answer: No, you can't just ban them, because existing OSs and filesystems allow them and you need interoperability.
- rini17 2y ago
  There are ways, no idea why doing anything here is so reviled. Find and xargs can delimite filenames by NUL, which is not allowed in filenames. Best practice in SQL was to abandon parameters escaping completely and pass them out of band. For internal representation, use array datastructures with length information. Actually, would it be that bad, to ban * and ? in filenames? If you accept them in the name of interop, something inevitably breaks later. Better to fail upfront. Many applications do sanitize filenames already and when they need to use binary data as file name, convert it to hex instead. It's a hassle otherwise.
  - eru 2y ago
    > Actually, would it be that bad, to ban * and ? in filenames? That's possible, if you design your filesystem from scratch. But if you take your filesystem as given for now (with its ability to represent all kinds of interesting characters), and just want to design globbing you have to solve this problem. Otherwise you have a tool that can only handle some files. That's what Gnu Make does, btw. Try handling any file or output with whitespace in the name in Make, if you want some frustration. Yes, null-termination works for the specific problem of termination. Though if you just use program-to-program communication, you can also prefix your strings with their length. > If you accept them in the name of interop, something inevitably breaks later. Why? That's only the case when you have legacy software written by less than careful people. There's no reason to expect breakage when you are designing new software, just like the people in the article where doing. (Of course, back then they didn't know what they were doing, so we have a lot of breakage historically.) But for the very specific purpose of the shell talking to a helper program for globbing, you can control exactly what's happening, including all the encoding and decoding (or escaping and unescaping). So there's no unexpected breakage. And btw, you also need to give the human a way to specify a literal * in a filename, too. Not just for communication between programs. > Best practice in SQL was to abandon parameters escaping completely and pass them out of band. Yes, that's partially because SQL is such a complicated language, and because you are talking about program-to-program communication anyway, so you don't need to be human-friendly there. So communicating them on a separate is the simplest thing that covers all cases.
eru 2y ago
Why? This is just for communication between the shell and its helper programs, the user wouldn't even see. What do you not like about escaping? Of course, for program-to-program communication you can also use different techniques, instead of escaping. Escaping is just the most human-readable and human-producible. (As a simple example, to be able to represent all characters in a string, you can either escape quotes like \" or you can prefix the string with its length. Computers can work with either convention, but humans will hate you if they have to prefix every string literal with its length and keep that length in sync with the string.)
- rini17 2y ago
  Are you aware that the main issue here is not with string literals, but with glob expansions? Literals are quite easy to check statically as mistakes usually cause havoc with surrounding code syntax. Even so, I avoid nontrivial use of them. But expansions and substitutions with escaping are the can of worms.

That's kind of cure worse than disease. Just ditch escaping completely.

by rini17 2y ago

That's kind of cure worse than disease. Just ditch escaping completely.

If you completely ditch escaping, how do you handle filenames that contain special characters (in this context, mostly ? and *, but ()[] are also perennial favorites)? And to preempt the most obvious answer: No, you can't just ban them, because existing OSs and filesystems allow them and you need interoperability.

rini17 2y ago
There are ways, no idea why doing anything here is so reviled. Find and xargs can delimite filenames by NUL, which is not allowed in filenames. Best practice in SQL was to abandon parameters escaping completely and pass them out of band. For internal representation, use array datastructures with length information. Actually, would it be that bad, to ban * and ? in filenames? If you accept them in the name of interop, something inevitably breaks later. Better to fail upfront. Many applications do sanitize filenames already and when they need to use binary data as file name, convert it to hex instead. It's a hassle otherwise.
- eru 2y ago
  > Actually, would it be that bad, to ban * and ? in filenames? That's possible, if you design your filesystem from scratch. But if you take your filesystem as given for now (with its ability to represent all kinds of interesting characters), and just want to design globbing you have to solve this problem. Otherwise you have a tool that can only handle some files. That's what Gnu Make does, btw. Try handling any file or output with whitespace in the name in Make, if you want some frustration. Yes, null-termination works for the specific problem of termination. Though if you just use program-to-program communication, you can also prefix your strings with their length. > If you accept them in the name of interop, something inevitably breaks later. Why? That's only the case when you have legacy software written by less than careful people. There's no reason to expect breakage when you are designing new software, just like the people in the article where doing. (Of course, back then they didn't know what they were doing, so we have a lot of breakage historically.) But for the very specific purpose of the shell talking to a helper program for globbing, you can control exactly what's happening, including all the encoding and decoding (or escaping and unescaping). So there's no unexpected breakage. And btw, you also need to give the human a way to specify a literal * in a filename, too. Not just for communication between programs. > Best practice in SQL was to abandon parameters escaping completely and pass them out of band. Yes, that's partially because SQL is such a complicated language, and because you are talking about program-to-program communication anyway, so you don't need to be human-friendly there. So communicating them on a separate is the simplest thing that covers all cases.

eru 2y ago

Why? This is just for communication between the shell and its helper programs, the user wouldn't even see. What do you not like about escaping? Of course, for program-to-program communication you can also use different techniques, instead of escaping. Escaping is just the most human-readable and human-producible. (As a simple example, to be able to represent all characters in a string, you can either escape quotes like \" or you can prefix the string with its length. Computers can work with either convention, but humans will hate you if they have to prefix every string literal with its length and keep that length in sync with the string.)

rini17 2y ago
Are you aware that the main issue here is not with string literals, but with glob expansions? Literals are quite easy to check statically as mistakes usually cause havoc with surrounding code syntax. Even so, I avoid nontrivial use of them. But expansions and substitutions with escaping are the can of worms.