3 ms·

> PS: I don't know why expanding shell wildcards used a separate program in V6 and earlier, but part of it may have been to keep the shell smaller and more mini

> PS: I don't know why expanding shell wildcards used a separate program in V6 and earlier, but part of it may have been to keep the shell smaller and more minimal so that it required less memory. See, I thought it was a nice separation of concerns and wondered why we lost such a nice approach, until I read: > How escaping wildcards works in the V5 and V6 shell is that all characters in commands and arguments are restricted to being seven-bit ASCII. The shell and /etc/glob both use the 8th bit to mark quoted characters, which means that such quoted characters don't match their unquoted versions and won't be seen as wildcards by either the shell at which point I suddenly became a fan of ditching it. I do wonder if there's not some better way to factor that functionality out...

deleted 2y ago
[deleted]
hnlmorg 2y ago
The way Murex works is each parameter is first compiled into an AST, and then globing only works against the unquoted tokens. Globbing is also a separate built in, which allows for other types of wildcard matches like regex too. Eg https://murex.rocks/tour.html#filesystem-wildcards-globbing https://murex.rocks/tour.html#filesystem-wildcards-globbing So you have have the best of both worlds: inline globbing for convenience and also wildcard matching as a function too.
eru 2y ago
> at which point I suddenly became a fan of ditching it. I do wonder if there's not some better way to factor that functionality out... Just use backslash escaping like we do practically everywhere else in the Unix world?
- rini17 2y ago
  That's kind of cure worse than disease. Just ditch escaping completely.
  - yjftsjthsd-h 2y ago
    If you completely ditch escaping, how do you handle filenames that contain special characters (in this context, mostly ? and *, but ()[] are also perennial favorites)? And to preempt the most obvious answer: No, you can't just ban them, because existing OSs and filesystems allow them and you need interoperability.
    - rini17 2y ago
      There are ways, no idea why doing anything here is so reviled. Find and xargs can delimite filenames by NUL, which is not allowed in filenames. Best practice in SQL was to abandon parameters escaping completely and pass them out of band. For internal representation, use array datastructures with length information. Actually, would it be that bad, to ban * and ? in filenames? If you accept them in the name of interop, something inevitably breaks later. Better to fail upfront. Many applications do sanitize filenames already and when they need to use binary data as file name, convert it to hex instead. It's a hassle otherwise.
      - eru 2y ago
        > Actually, would it be that bad, to ban * and ? in filenames? That's possible, if you design your filesystem from scratch. But if you take your filesystem as given for now (with its ability to represent all kinds of interesting characters), and just want to design globbing you have to solve this problem. Otherwise you have a tool that can only handle some files. That's what Gnu Make does, btw. Try handling any file or output with whitespace in the name in Make, if you want some frustration. Yes, null-termination works for the specific problem of termination. Though if you just use program-to-program communication, you can also prefix your strings with their length. > If you accept them in the name of interop, something inevitably breaks later. Why? That's only the case when you have legacy software written by less than careful people. There's no reason to expect breakage when you are designing new software, just like the people in the article where doing. (Of course, back then they didn't know what they were doing, so we have a lot of breakage historically.) But for the very specific purpose of the shell talking to a helper program for globbing, you can control exactly what's happening, including all the encoding and decoding (or escaping and unescaping). So there's no unexpected breakage. And btw, you also need to give the human a way to specify a literal * in a filename, too. Not just for communication between programs. > Best practice in SQL was to abandon parameters escaping completely and pass them out of band. Yes, that's partially because SQL is such a complicated language, and because you are talking about program-to-program communication anyway, so you don't need to be human-friendly there. So communicating them on a separate is the simplest thing that covers all cases.
Joker_vD 2y ago
Why would I want to factor out some syntactic functionality of one specific (and not very well thought out) shell to reuse, again? But if you really insist, you can write your own glob(1) that would invoke glob(3) for you, sure. There is also wordexp(3) although I believe its implementations had security problems for quite some time?
p_l 2y ago
Important thing to remember is that even after the move to PDP-11, early Unix systems had to deal with 32kB as entire space available to userland program, both code and data (including stack)
- kjs3 2y ago
  You mean 32k words, not 32k bytes, right[1]? And AFAIK by V5 or V6, Unix could use split instruction and data if the MMU supported it giving a bit more headroom. But, yeah, memory was very tight, and a lot of very clever tricks were used to get around it. [1] Even worse, the top 4kW/8kB was reserved for I/O.
  - p_l 2y ago
    I meant 32k bytes - PDP-11 was byte-addressed, not word-addressed. The 64kB address space was split in half between kernel and userland in so called "high and low moby" scheme (as it required minimal logic latching onto single address line). And for I/D split you needed appropriate CPU model. the top 8kB "I/O page" is reserved as part of the kernel space, not userspace, so it does not impact as much the userspace part.
    - kjs3 2y ago
      Ah, I misunderstood your point. And while the PDP-11 was byte addressable, the doco often talked about memory size in words. Carry on.
BoingBoomTschak 2y ago
There's a sane language that never went that route: https://www.tcl.tk/man/tcl9.0/TclCmd/glob.html https://www.tcl.tk/man/tcl9.0/TclCmd/glob.html It also ditched another special case recently: the leading ~.

deleted 2y ago

[deleted]

hnlmorg 2y ago

The way Murex works is each parameter is first compiled into an AST, and then globing only works against the unquoted tokens. Globbing is also a separate built in, which allows for other types of wildcard matches like regex too. Eg https://murex.rocks/tour.html#filesystem-wildcards-globbing https://murex.rocks/tour.html#filesystem-wildcards-globbing So you have have the best of both worlds: inline globbing for convenience and also wildcard matching as a function too.

eru 2y ago

> at which point I suddenly became a fan of ditching it. I do wonder if there's not some better way to factor that functionality out... Just use backslash escaping like we do practically everywhere else in the Unix world?

rini17 2y ago
That's kind of cure worse than disease. Just ditch escaping completely.
- yjftsjthsd-h 2y ago
  If you completely ditch escaping, how do you handle filenames that contain special characters (in this context, mostly ? and *, but ()[] are also perennial favorites)? And to preempt the most obvious answer: No, you can't just ban them, because existing OSs and filesystems allow them and you need interoperability.
  - rini17 2y ago
    There are ways, no idea why doing anything here is so reviled. Find and xargs can delimite filenames by NUL, which is not allowed in filenames. Best practice in SQL was to abandon parameters escaping completely and pass them out of band. For internal representation, use array datastructures with length information. Actually, would it be that bad, to ban * and ? in filenames? If you accept them in the name of interop, something inevitably breaks later. Better to fail upfront. Many applications do sanitize filenames already and when they need to use binary data as file name, convert it to hex instead. It's a hassle otherwise.
    - eru 2y ago
      > Actually, would it be that bad, to ban * and ? in filenames? That's possible, if you design your filesystem from scratch. But if you take your filesystem as given for now (with its ability to represent all kinds of interesting characters), and just want to design globbing you have to solve this problem. Otherwise you have a tool that can only handle some files. That's what Gnu Make does, btw. Try handling any file or output with whitespace in the name in Make, if you want some frustration. Yes, null-termination works for the specific problem of termination. Though if you just use program-to-program communication, you can also prefix your strings with their length. > If you accept them in the name of interop, something inevitably breaks later. Why? That's only the case when you have legacy software written by less than careful people. There's no reason to expect breakage when you are designing new software, just like the people in the article where doing. (Of course, back then they didn't know what they were doing, so we have a lot of breakage historically.) But for the very specific purpose of the shell talking to a helper program for globbing, you can control exactly what's happening, including all the encoding and decoding (or escaping and unescaping). So there's no unexpected breakage. And btw, you also need to give the human a way to specify a literal * in a filename, too. Not just for communication between programs. > Best practice in SQL was to abandon parameters escaping completely and pass them out of band. Yes, that's partially because SQL is such a complicated language, and because you are talking about program-to-program communication anyway, so you don't need to be human-friendly there. So communicating them on a separate is the simplest thing that covers all cases.

Joker_vD 2y ago

Why would I want to factor out some syntactic functionality of one specific (and not very well thought out) shell to reuse, again? But if you really insist, you can write your own glob(1) that would invoke glob(3) for you, sure. There is also wordexp(3) although I believe its implementations had security problems for quite some time?

p_l 2y ago

Important thing to remember is that even after the move to PDP-11, early Unix systems had to deal with 32kB as entire space available to userland program, both code and data (including stack)

kjs3 2y ago
You mean 32k words, not 32k bytes, right[1]? And AFAIK by V5 or V6, Unix could use split instruction and data if the MMU supported it giving a bit more headroom. But, yeah, memory was very tight, and a lot of very clever tricks were used to get around it. [1] Even worse, the top 4kW/8kB was reserved for I/O.
- p_l 2y ago
  I meant 32k bytes - PDP-11 was byte-addressed, not word-addressed. The 64kB address space was split in half between kernel and userland in so called "high and low moby" scheme (as it required minimal logic latching onto single address line). And for I/D split you needed appropriate CPU model. the top 8kB "I/O page" is reserved as part of the kernel space, not userspace, so it does not impact as much the userspace part.
  - kjs3 2y ago
    Ah, I misunderstood your point. And while the PDP-11 was byte addressable, the doco often talked about memory size in words. Carry on.

BoingBoomTschak 2y ago

There's a sane language that never went that route: https://www.tcl.tk/man/tcl9.0/TclCmd/glob.html https://www.tcl.tk/man/tcl9.0/TclCmd/glob.html It also ditched another special case recently: the leading ~.