5 ms·

If you are that concerned about this specific problem, shell allows you to "set noclobber" (or "set -C") to not overwrite existing files.

by zvr 5y ago

chubot 5y ago
Yup, I only learned about that while writing https://www.oilshell.org https://www.oilshell.org ! I'm sure that has been around for decades; shell also has discovery and documentation problems :)
- wahern 5y ago
  > shell also has discovery and documentation problems Only because people refuse to read the specification. The POSIX Shell & Utilities volume is exceedingly clear and straight-forward. (I vehemently disagree w/ its characterization in that paper.) It helps if you use the frames-based HTML version to make it easier to navigate and explore the various utilities, and especially to see how the syntax specification is broken down--while that section is short, it's difficult to appreciate without explicitly seeing how the subsections are related. I only code in POSIX shell because I really can't be bothered to remember all the minutiae of other shells, almost all of which provide strong POSIX compatibility, Bash especially. I've never lamented the lack of arrays; but I regularly wish that POSIX would adopt ksh-style coprocesses so dash would have them. The shell is really all about stitching together processes (including subshell contexts), and coprocesses fill a conspicuous gap in functionality.
  - chubot 5y ago
    I've never used ksh-style coprocesses so I'd be interested to hear about some use cases for them, and what you like about them. Examples would be even better -- I've never seen them, and I've looked at a lot of shell. Bash has a different implementation of coprocesses that doesn't seem to be used much, and is very limited. I believe there can only be one coprocess active at a time, which is very un-shell-like! I got an e-mail once from someone who was using them, and I felt it could be achieved by other means. I have a good idea coprocesses in Oil which basically is a startup time optimization. The coprocess will basically take argv over a unix domain socket, and have other commands to change the process state. It solves "the problem of VMs and containers that start slowly". https://www.oilshell.org/blog/2021/04/release-0.8.9.html#whats-next https://www.oilshell.org/blog/2021/04/release-0.8.9.html#wha... Unix domain sockets are already used for the headless shell in Oil (more coming on the blog about that) https://www.oilshell.org/release/latest/doc/headless.html#implement-the-fanos-protocol https://www.oilshell.org/release/latest/doc/headless.html#im...
    - wahern 5y ago
      The simplest description I've found is at https://www.dartmouth.edu/~rc/classes/ksh/coprocesses.html https://www.dartmouth.edu/~rc/classes/ksh/coprocesses.html, which also provides a couple of examples. Basically, you start a coprocess using the "|&" operator: somecommand |& This starts somecommand asynchronously, with stdin and stdout attached to anonymous pipes with descriptor numbers that can be referenced by a literal descriptor name, "p". So running cat <&p will shuttle the output of the somecommand coprocess to stdout. "p" always references the most recently created coprocess, so to juggle multiple coprocesses you have to dup "p" to explicit descriptors: exec 4>&p somecommand2 5<&p |& That's not as great as having named references, but it's rare to need to independently juggle more than 1 or 2 coprocesses from the same context. You're almost always plugging them together in series or in a complex graph using subshells. In either case you're only needing to manage the same few descriptor constants. It's sort of like writing in a stack-based language, where you really only use explicit references to the top of the stack. coprocesses provide a simple construct for cases where you would otherwise have to manually create a named pipe using mkfifo. The need for this pattern comes up surprisingly often, IME, yet because it's so messy--now you need to worry about /tmp races, cleaning things up, etc--people rarely bother, myself included. From the perspective of a shell it's a relatively simple construct to implement. The semantic benefit/cost ratio is exceptional. This basic functionality is complemented with extension options to the read and print (non-standard) builtins: -p reads from or writes to the latest coprocess, and -u the specified descriptor number. I haven't used coprocesses much because I can't rely on them, but I don't think these are strictly needed. Maybe they were required for proper buffering in early implementations (or even current ones), where redirection to and from builtins didn't utilize the same buffered I/O context? That seems more like an implementation detail. ksh93, bash, and some other shells provide the process substitution construct, "<()", which can provide similar capabilities to coprocesses in many cases. But I'm pretty sure anything you could do with process substitution you could do with coprocesses, making the latter the more powerful construct. coprocesses seems like the critical missing primitive from the traditional Unix shell model, and would round it out. (Of course, the sky is the limit for evolving the Unix shell model in new directions, as Oil has shown.) ksh88 had coprocesses so I'm curious why it wasn't included in POSIX Maybe because bash lacked an implementation? Or incompatible quirks between ksh88 and ksh93 related to stdout inheritance across coprocesses?

If you are that concerned about this specific problem, shell allows you to "set noclobber" (or "set -C") to not overwrite existing files.

by zvr 5y ago

If you are that concerned about this specific problem, shell allows you to "set noclobber" (or "set -C") to not overwrite existing files.

chubot 5y ago

Yup, I only learned about that while writing https://www.oilshell.org https://www.oilshell.org ! I'm sure that has been around for decades; shell also has discovery and documentation problems :)

wahern 5y ago
> shell also has discovery and documentation problems Only because people refuse to read the specification. The POSIX Shell & Utilities volume is exceedingly clear and straight-forward. (I vehemently disagree w/ its characterization in that paper.) It helps if you use the frames-based HTML version to make it easier to navigate and explore the various utilities, and especially to see how the syntax specification is broken down--while that section is short, it's difficult to appreciate without explicitly seeing how the subsections are related. I only code in POSIX shell because I really can't be bothered to remember all the minutiae of other shells, almost all of which provide strong POSIX compatibility, Bash especially. I've never lamented the lack of arrays; but I regularly wish that POSIX would adopt ksh-style coprocesses so dash would have them. The shell is really all about stitching together processes (including subshell contexts), and coprocesses fill a conspicuous gap in functionality.
- chubot 5y ago
  I've never used ksh-style coprocesses so I'd be interested to hear about some use cases for them, and what you like about them. Examples would be even better -- I've never seen them, and I've looked at a lot of shell. Bash has a different implementation of coprocesses that doesn't seem to be used much, and is very limited. I believe there can only be one coprocess active at a time, which is very un-shell-like! I got an e-mail once from someone who was using them, and I felt it could be achieved by other means. I have a good idea coprocesses in Oil which basically is a startup time optimization. The coprocess will basically take argv over a unix domain socket, and have other commands to change the process state. It solves "the problem of VMs and containers that start slowly". https://www.oilshell.org/blog/2021/04/release-0.8.9.html#whats-next https://www.oilshell.org/blog/2021/04/release-0.8.9.html#wha... Unix domain sockets are already used for the headless shell in Oil (more coming on the blog about that) https://www.oilshell.org/release/latest/doc/headless.html#implement-the-fanos-protocol https://www.oilshell.org/release/latest/doc/headless.html#im...
  - wahern 5y ago
    The simplest description I've found is at https://www.dartmouth.edu/~rc/classes/ksh/coprocesses.html https://www.dartmouth.edu/~rc/classes/ksh/coprocesses.html, which also provides a couple of examples. Basically, you start a coprocess using the "|&" operator: somecommand |& This starts somecommand asynchronously, with stdin and stdout attached to anonymous pipes with descriptor numbers that can be referenced by a literal descriptor name, "p". So running cat <&p will shuttle the output of the somecommand coprocess to stdout. "p" always references the most recently created coprocess, so to juggle multiple coprocesses you have to dup "p" to explicit descriptors: exec 4>&p somecommand2 5<&p |& That's not as great as having named references, but it's rare to need to independently juggle more than 1 or 2 coprocesses from the same context. You're almost always plugging them together in series or in a complex graph using subshells. In either case you're only needing to manage the same few descriptor constants. It's sort of like writing in a stack-based language, where you really only use explicit references to the top of the stack. coprocesses provide a simple construct for cases where you would otherwise have to manually create a named pipe using mkfifo. The need for this pattern comes up surprisingly often, IME, yet because it's so messy--now you need to worry about /tmp races, cleaning things up, etc--people rarely bother, myself included. From the perspective of a shell it's a relatively simple construct to implement. The semantic benefit/cost ratio is exceptional. This basic functionality is complemented with extension options to the read and print (non-standard) builtins: -p reads from or writes to the latest coprocess, and -u the specified descriptor number. I haven't used coprocesses much because I can't rely on them, but I don't think these are strictly needed. Maybe they were required for proper buffering in early implementations (or even current ones), where redirection to and from builtins didn't utilize the same buffered I/O context? That seems more like an implementation detail. ksh93, bash, and some other shells provide the process substitution construct, "<()", which can provide similar capabilities to coprocesses in many cases. But I'm pretty sure anything you could do with process substitution you could do with coprocesses, making the latter the more powerful construct. coprocesses seems like the critical missing primitive from the traditional Unix shell model, and would round it out. (Of course, the sky is the limit for evolving the Unix shell model in new directions, as Oil has shown.) ksh88 had coprocesses so I'm curious why it wasn't included in POSIX Maybe because bash lacked an implementation? Or incompatible quirks between ksh88 and ksh93 related to stdout inheritance across coprocesses?