12 ms·

"Rules" that terminal programs follow

298 points by charlieok 2y ago

matheusmoreira 2y ago
> programs should print “regular” output to stdout and errors to stderr This is really important. I'd like to expand on this. Standard output is for the data the program was asked to produce, no more and no less. If user asked for some JSON data, standard output should contain that exact JSON object and absolutely nothing else. Standard "error" is actually a misnomer. It should have been called the standard user stream. Anything meant for the user to read on the terminal is supposed to go there. Error messages are of course included in that set but so are status messages and verbose output. This ensures the output of programs can be piped into other programs seamlessly. Non-output data still gets sent to the terminal or redirected somewhere else. Would have been great if programs were able to easily create new terminal-connected file descriptors for specific purposes. They could document those numbers in their manuals just like they document exit codes. Then users would get "ports" for every output. Could cut down on parsing significantly. For compatibility, they could all redirect to either standard output or standard error by default... I think I'm gonna experiment with this a bit.
- packetlost 2y ago
  I'm not sure I like the "standard user stream" name, but I otherewise agree with everything here.
  - matheusmoreira 2y ago
    Yeah, naming is hard. If there is a better word for "non-output", I don't know it.
    - packetlost 2y ago
      I would maybe call it stdlog or stdinfo vs stderr, but yeah naming things is hard.
      - matheusmoreira 2y ago
        I think stdinfo would exclude error messages from the definition, just like stderr excludes info messages. I really like stdlog. Standard log stream is a pretty awesome name. Short and terse, the word "log" doesn't even need abbreviation and it's correct since it's a superset of error and info streams and also generic enough to cover other unforeseen categories. I'll use it from now on!
        packetlost 2y ago
        The idea for info is that it's informational, which is always the case regardless of whether it's printing error messages due to a failure or informational messages because the user specified a `-v` flag or whatever. Whether the command failed or not is communicated via the exit code. That being said, I do prefer stdlog for the reasons: it communicates the intent and usage as unambiguously and tersely as possible.
        matheusmoreira 2y ago
        > because the user specified a `-v` flag or whatever The way I see it is the version information and help text belong on standard output if the user passes --version and --help since that's what the user explicitly asked for. When the command is invoked incorrectly, the help text should go to standard error while standard output should be empty. I agree about the exit code. People like to parse error messages and that's always wrong. Everything should be done via exit codes instead.
        packetlost 2y ago
        The `-v` was in reference to the common convention of one or more `v` flags increasing the verbosity of logging.
      - ryandrake 2y ago
        C++ has std::clog[1], but it's basically cerr, and the OS still only knows about stdin, stdout, and stderr. 1: https://cplusplus.com/reference/iostream/clog/ https://cplusplus.com/reference/iostream/clog/
  - paulddraper 2y ago
    I'd name it "standard log." That's what it is....a log. Of errors, warnings, informational messages, whatever to complement the primary output. BONUS: stdlog fits nicely :)
  - saghm 2y ago
    What about "debug"? I think the important part about stderr is that it's "meta" info about how the program is running (or ran) rather than what the program is intended to produce as output. This seems pretty similar to what often is referred to as the "debug" level of logging (which includes lower levels of logging like warnings and errors).
- lionkor 2y ago
  Unless your app doesn't generate output,then grepping the output will not work if it's stderr and it will be confusing
  - bombcar 2y ago
    You can learn the 2>&1 thing, but it would be nice if there was a feature of one of the greps to "slurp up" the error output. Of course, the real problem is there's no standard, the standards that do exist are ignored, and each new command-line tool generates a new standard.
    - sudobash1 2y ago
      It generally isn't possible for grep to "slurp up" the error output because stderr does not get passed through pipes (by default). This is shell behavior and grep cannot do anything about it. As a side note, some shells have started implementing a shorthand for `foo 2>&1 | grep` which is `foo |& grep`.
      - ElevenLathe 2y ago
        Probably you could do a hack like finding the program at the other end of the pipe and looking at its open file descriptors, but lots of corner cases to consider.
      - kelnos 2y ago
        Right, the problem is that apparently shells haven't seen user need for selectively redirecting one or the other. Like I might want to see foo's stdout on the terminal, but pipe stderr (and only stderr) to grep, perhaps so I only see some specific error messages that I care about. `2>&1` will send stderr to stdout, and then the pipe will send everything to grep (and I believe `|&` does the same). (To be fair, that does seem like a pretty rare use case.)
        sudobash1 2y ago
        Actually, you can. If you run: foo 2>&1 >&- | grep ... then you will grep only the stderr. The stdout will be discarded. Alternatively, if you only want to grep to see the stderr, but you still want to see stdout, you can swap stderr and stdout like this: foo 3>&2 2>&1 1>&3 | grep ... And in many shells, you can easily split the stdout and stderr into two separate pipelines like this: foo > >(grep stdout) 2> >(grep stderr)
- AndrewDucker 2y ago
  This is definitely one of the things that PowerShell got right. 6 different streams, each of which can be intercepted separately and configured differently. https://learn.microsoft.com/en-us/powershell/module/microsoft.powershell.core/about/about_output_streams?view=powershell-7.4 https://learn.microsoft.com/en-us/powershell/module/microsof...
  - matheusmoreira 2y ago
    There's a lot of good ideas in power shell. I like how they broke up the log stream into separate streams by severity. It still seems arbitrary and insufficiently general though. Why severity and not some other criteria? What if programs could define any number of output streams? By default they could all coalesce into the terminal but other programs could connect to each one separately if they needed. Like an audio mixer of sorts: by default you get the mixed audio but there are ways to access each individual voice if needed.
  - immibis 2y ago
    I don't think that 6 different streams that will always go to the same place but some of them are usually turned off are very useful. Standard-error/user is useful because it goes to a different place. Levels would be better suited to a simple global variable, or a level associated with each printed line. The chance that someone redirects errors to error.log is reasonable, but the chance that someone redirects errors to error.log and warnings to warning.log and debug to debug.log and verbose to verbose.log is pretty low.
    - AndrewDucker 2y ago
      I turn on Verbose logging sometimes, to get more info. Error logging can be set to either spit out the error and continue, or to stop processing, which is very useful. Warning and error messages come out in their own colours automatically, which is nice. And sure, you can just have them all come out to the same place. But having the options to configure them is very handy.
      - 2y ago
bobbiechen 2y ago
Nice writeup. Since she mentioned how hard it is to learn these conventions, I'll plug my preferred reference when thinking about CLIs specifically (rather than TUIs and REPLs) - the Command Line Interface Guidelines https://clig.dev https://clig.dev It does include the blog post's rules on exiting on Ctrl-C, accepting `-` for stdin, disabling color in pipes, and much more.
- deleted 2y ago
  [deleted]
- thiisguy 2y ago
  This is referenced at the very beginning, just after the table of contents.
  - bobbiechen 2y ago
    Oops, I missed it when I was reading, and assumed it wasn't referenced. Thanks!
- mcint 2y ago
  I would also recommend, call to attention, the Further Reading from CLIG, https://clig.dev/#further-reading https://clig.dev/#further-reading. POSIX, GNU, Unix resources, Heroku CLI guide, and 12-factor CLI app guide.
- snthpy 2y ago
  Thanks. Didn't know about that one. There is also: https://usage.jdx.dev/ https://usage.jdx.dev/
felixhummel 2y ago
I'd add "long-running processes should reload their configuration on SIGHUP". :)
- rocqua 2y ago
  This isn't a list of shoulds. It's a list of, generally dones. Big difference. This one is still nowhere near universal enough to count in the original lisr sadly.
- throwaway127482 2y ago
  This behavior seems difficult to actually implement in practice, no? Your application has to be careful not to accidentally cache any configuration state, as well as any state derived from the configuration, otherwise it will go out of sync. Maybe easy for trivial apps but most apps requiring configuration files are non trivial.
- abbeyj 2y ago
  This is probably technically out of scope for this article. Long-running processes that respond to SIGHUP will usually be running detached (with no controlling TTY). Thus it is a bit of stretch to call them "terminal programs".
anthk 2y ago
Don't hardcode readline keybindings.
Sesse__ 2y ago
“rule 5.1: Ctrl-W should delete the last word […] I can’t think of any exceptions to this other than text editors but if there are I’d love to hear about them!” mysql(1) only links to editline instead of readline, where Ctrl-W by default deletes everything to the start of the line, not just the last word. It drove me mad in the period where I had to use it; you just see your entire nice query disappear. :-)
- saghm 2y ago
  It could be worse! Back in college, I had the misfortune to decide to try to use sqlplus, Oracle's CLI to try to connect to their database. Not only did terminal shortcuts like Ctrl-W not work at all, but you couldn't even move the cursor earlier in the line, and there was no history navigation to get to the previous command, so any typo forced you to retype the entire thing from scratch. Entering a single forward slash would allow you to run the last command verbatim, but that didn't help with typos. Ctrl-L also didn't work for clearing the screen, but you could manually run `clear scr` to do it instead, because I guess having the command just be `clear` wouldn't be obvious enough.
  - cafard 2y ago
    SQL*Plus is pretty basic, but L will show you the previous command, a number will give you that line of the previous command, to edit, etc.
  - Sesse__ 2y ago
    If you ever need to do so again, try gqlplus, which wraps sqlplus to add such quality-of-life features :-)
    - saghm 2y ago
      I hope my streak of almost a decade of not needing to use Oracle will continue, but if not, I definitely will be looking for alternatives to sqlplus...
      - 2y ago
CorrectHorseBat 2y ago
[flagged]
- blueflow 2y ago
  Do an "stty -a" in the terminal - you will see these two keys are handled by the kernel itself and will also work with programs reading dumbly from stdin, like "cat".
- kps 2y ago
  Ctrl-W originates (on Un*x) from the ‘new’ BSD tty driver in the late '70s. Ctrl-U vs Ctrl-X was one of those BSD-vs-AT&T things (where V7 defaulted to ‘@’ 'cause your terminal might not have those fancy control characters). `vi` respected the current tty settings.
- d3VwsX 2y ago
  Ctrl-w and ctrl-u in a default emacs do not even do the same things as in readline. C-w is (usually) kill-region (similar to what most editors call "cut"). C-u is universal-argument, that is used as a prefix for other commands to add arguments.
sysread 2y ago
Non-interactive programs that emit informational output should only do so to stderr or a log file so that they may be used in a pipe line.
- deleted 2y ago
  [deleted]
model-15-DAV 2y ago
As an addendum to Rule 7, `cd -` takes you to the last opened directory. Or is `cd` considered part of the terminal emulator's job, as a built-in?
- saghm 2y ago
  I don't think it's part of the terminal emulator (e.g. xterm, gnome-terminal) but the shell (bash, zsh, etc.). You're correct that it's not something that's implemented as an external program though; the "current directory" is state for a currently running terminal session, so changing that state is done via the shell interface (either directly by built-in commands like cd or indirectly via external commands that use those transitively).
  - model-15-DAV 2y ago
    True, I meant the shell, as `cd` is a shell builtin. However, `git-checkout` does implement this behavior. `git checkout -` will checkout the previously checked out branch.
  - machinestops 2y ago
    Note: Child processes can't change the working directory of the parent. An external command (that is, not a shell builtin, shell function, or externally loaded module) cannot change the working directory, because they're launched as child processes.
    - saghm 2y ago
      Good point. I'm not sure why I was thinking that it was possible to do via some other command invoking `cd` or something, but you're right that the only examples I can think of are all using other builtins (e.g. invoking `source` to have a script change the current state).
      - d3VwsX 2y ago
        There is a workaround I saw used by wcd, a tool for changing directories. To install it you have to add a wrapper function in your shell that executes the actual wcd binary, and after possibly interacting with the user to figure out what directory to change to the executable will print out the destination, and then the wrapper function will make the call to cd, affecting the shell it runs in. https://wcd.sourceforge.io/ https://wcd.sourceforge.io/
lyxell 2y ago
I’d like to add: Programs should not add files to your home directory and should respect XDG_CONFIG_HOME and friends.
- kelnos 2y ago
  I agree, but this article is -- as the author tries to make clear -- descriptive, not prescriptive. She's listing out the things she's seen commonly in applications, not trying to convince applications that they should behave in certain ways.
chriswarbo 2y ago
Ctrl-D for REPLs always bites me with GHCi. My usual approach to quitting GHCi is: - Press Ctrl-D, like normal - Get confused when nothing happens - Remember that it doesn't work in GHCi, so run `:q` instead - Get an error message about "lexical error at character '\EOT'", due to Ctrl-D inserting an invisible char at the start of the input - Try `:q` again, without any invisible prefix - GHCi successfully quits
- jeffrallen 2y ago
  You think that's hard, try getting out of vi when you're on a keyboard from an unfamiliar country.
  - lilyball 2y ago
    Do keyboards in unfamiliar countries not have the colon key?
    - WolfeReader 2y ago
      That or the Q key
    - Philpax 2y ago
      Both the colon and Q keys can be in different places, which can really mess with your muscle memory.
  - from-nibly 2y ago
    Or if you are like me and have caps mapped as escape. Vim works really weird when you start typing random capital letters after thinking you were moving somewhere.
  - saint_yossarian 2y ago
    There's also the `ZZ` / `ZQ` alternatives, so you just have to find those letters :)
  - linhns 2y ago
jeffrallen 2y ago
If you are a young sysadmin, take the time to.learn Emacs. Not because Emacs is good (but it is) but because deadline keys are Emacs keys, so once you know Emacs you know shell, MySQL, etc. Play your terminal like a piano . Your livelihood depends on it.
neilv 2y ago
Additional suggestions: * Respect the user's default foreground and background color. Don't change them without good reason. * If you use colors, make them legible regardless of what the default background and foreground colors are, and regardless of the terminal's color map. * Don't use color as the only indication of something. The user's terminal might not display it, and it probably won't be preserved in copy&paste into notes. * Use emoji only judiciously, if at all. Similar with gratuitous non-ASCII characters. It doesn't display everywhere, it doesn't paste well everywhere, and emoji can be a bit much when copy&pasted into some notes. * In a scrolling (non-full-screen) stdout-ish output, don't delete important information that you showed temporarily. For example, hiding warnings or filenames compiled, to display a green checkmark for done. For another example, clearing the screen of Web app build information (including package security warnings!), to display a message that it's running in dev mode, is also not wanted. People might want to see that information, or copy&paste it into notes. * If you went full angry fruit salad with your command line program, because it's your baby, and you're having fun hamming it up, that's fine, but please provide an easy preference setting for people to opt out of that. Your program is probably only one of many things on user's workstation display, where other programs might be using color and visuals more meaningfully, so animated throbbing red explosions for the code reformatter is a bit much.
- deleted 2y ago
  [deleted]
- hatthew 2y ago
  As the article explicitly stated, these are descriptive not prescriptive rules. They're things that you can generally assume all terminal programs already follow.
  - neilv 2y ago
    The ones I listed were mostly descriptive, until recent years.
- teddyh 2y ago
  > please provide an easy preference setting for people to opt out of that. There is some effort to standardize this: <https://bixense.com/clicolors/ https://bixense.com/clicolors/>
ucarion 2y ago
What does "cooked" mode mean in the context of this article?
- kelnos 2y ago
  It means that the keystrokes you type aren't just immediately handled to the application in raw form; something higher up the chain (kernel, terminal emulator, shell) is pre-processing ("cooking") them in some way, possibly taking actions before (or in place of) passing the keystrokes to the application. For example, if an application you're using has left things in cooked mode, and you press ctrl+c, the application will never "see" that keystroke; something higher up in the input chain (I believe Linux's TTY driver) will see it, swallow it, and send SIGINT to the application. Applications can also put the tty into "raw mode", where this won't happen; in that case the app is responsible for implementing those "expected" keystroke behaviors, if it makes sense for the application to do so.
  - lilyball 2y ago
    cooked mode also generally gives input to the program a line at a time and handles things like backspace to delete characters, and it handles program output too, doing things like replacing \n with \r\n.
    - teddyh 2y ago
      No, line-by-line vs. character-by-character is a separate mode from “cooked” mode. See “stty extproc”.
      - lilyball 2y ago
        I read up on this and "cooked but character-by-character" is apparently something called "cbreak mode" (or "rare mode").
        teddyh 2y ago
        According to stty(1), ”cbreak” is a negative alias of “icanon”, which simply enables the special functions of the ‘erase’ (backspace), ‘kill’ (Ctrl-U), ‘werase’ (Ctrl-W), and ‘rprnt’ (Ctrl-R) keybindings. “cooked” mode includes “icanon”. So no, I do not think that what you wrote is correct. From what I can tell, the line-by-line mode is “extproc”.
mattofak 2y ago
In the same category of command line program guidelines: https://clig.dev/ https://clig.dev/
lieks 2y ago
The main reason I enjoy CLIs so much more than GUIs (or eves TUIs sometimes) is that it feels so consistent. There are conventions, but following all the conventions in a CLI is a lot easier than designing a good GUI. So they tend to be higher quality as a result. I spend a lot of time thinking how to bring this property to GUIs, but my best answers are still "lots of effort" or "lower your expectations".
- tpmoney 2y ago
  A large part of this I think is that other than MacOS, GUI conventions aren’t heavily implemented by the OS and easy to opt into. And because no other OS landed on “meta key for GUI shortcuts” so there are a lot more conflicts (e.g. ctrl-c). The Application Framework defaults which underly most native macOS application build a lot of common keyboard controls in. Use the default menu bar classes with the default basic commands, and Command O, N, Q, X, C,V and probably others come for free, you just implement the code that you need to for those functions. Use a standard text field and you automatically get Command/Option/Fn Left, Right, Up, Down for navigation. It’s more notabke when a macOS application doesn’t follow convention (e.g. InteliJ uses Shift-Opt up/down to move lines rather than expand the selection by paragraph) than when one does. Windows does decently well on this front, but ctrl as a default modifier can hurt terminal based app usage and there are a number of UI frameworks even within the OS that appear to get different defaults. And in the Linux world, I think the only way you could do this would be for someone to design (and a distro to standardize on and port apps to) a full on application framework. The window managers don’t want to be in the business of dictating the behavior of stuff in windows. The GUI toolkits don’t want to be in the business of defining os wide defaults and the DEs and distros don’t want to be in the business of if porting or dictating UI frameworks. And realistically there’s no one “on high” that could make the sort of dictation that for example “hence forth copy and paste will be Meta-C and Meta-V”
ruricolist 2y ago
For forcing colorized output: I personally prefer pipetty (from the colorized-logs package on Debian) to unbuffer, since unbuffer merges stdout and stdin.
cbhl 2y ago
> rule 3: REPLs should quit when you press Ctrl-D on an empty line If memory serves, this behavior depends on the OS. On Windows I believe the norm there is to type "<Ctrl-Z><Enter>"
- hatthew 2y ago
  I think it's assumed this article is about *nix, not windows
cpif 2y ago
I can use Ctrl-A, Ctrl-E, and Ctrl-U in text fields in the lynx browser, but not Ctrl-W. I just checked to see if Ctrl-F and Ctrl-B work, and found that the former kills one word forward and the latter acts like Ctrl-W ought to?
jrockway 2y ago
One that's missing is treating ~ as the home directory. This appears to be a shell thing and not a POSIX API thing. For example, this doesn't work: func main() { if _, err := os.ReadFile("~/.bashrc"); err != nil { log.Fatal(err) } fmt.Println("ok") } Meanwhile over in shell land: $ echo ~/~/~ /home/jrockway/~/~ The behavior is actually kind of amazing. I mention it because while "yourprogram ~/path/to/file" always works, having a repl that asks for a filename might not work. I've seen a lot of software where this DOES work, so I think it counts as a "most TUI programs do this" thing.
- wahern 2y ago
  Tilde expansion is part of shell word expansion: https://pubs.opengroup.org/onlinepubs/9799919799/utilities/V3_chap02.html#tag_19_06_01 https://pubs.opengroup.org/onlinepubs/9799919799/utilities/V... Tilde expansion is the first operation in word expansion: > The expansions that are performed for a given word shall be performed in the following order: 1) Tilde expansion, parameter expansion, command substitution, and arithmetic expansion shall be performed, beginning to end. 2) Field splitting shall be performed on the portions of the fields generated by step 1. 3) Pathname expansion shall be performed, unless set -f is in effect. 4) Quote removal, if performed, shall always be performed last. See https://pubs.opengroup.org/onlinepubs/9799919799/utilities/V3_chap02.html#tag_19_06 https://pubs.opengroup.org/onlinepubs/9799919799/utilities/V... To see the whole shell specification, goto the main page at https://pubs.opengroup.org/onlinepubs/9799919799/ https://pubs.opengroup.org/onlinepubs/9799919799/. Select "Shell & Utilities" from the top-left frame, then "Shell Command Language" from the bottom-left frame.
- Sharlin 2y ago
  That's a good point, particularly given that .. is handled for you at the OS API level so you might be inclined to expect ~ is too on POSIX.
  - jasomill 2y ago
    This would massively complicate kernel path resolution on many systems. Taking Linux as an example, even in the simplest cases, ~user expansion as typically implemented — via the getpwnam(3) libc function, which in itself implies a rather unfortunate dependency — requires reading and parsing /etc/passwd, and may also require access to network directory services like LDAP.
jez 2y ago
Some more notes: - If this is your first time hearing about the readline/emacs keybindings like Ctrl-E and Ctrl-W, you'll be pleased to know that most macOS input sources use these keybindings. If you're on macOS, feel free to try Ctrl-E, Ctrl-W, or Ctrl-U in your browser's address bar right now - If you're using a command line program that doesn't support _any_ line editing (e.g. no readline keybindings, and no other keybindings), you can install the `rlwrap` program and launch the REPL under rlwrap. For example Standard ML of New Jersey has a REPL but no line editing functionality, but you can recover that via `rlwrap smlnj` - "don’t use more than 16 colours" — I would go so far as to say "don't use more than 8 colors, or at least make your colors configurable." Many popular color schemes, including Solaraized and the default Base 16 color scheme, use the "bright" colors to hold various shades of gray. What you think is "bright green" might actually be the same shade of gray that normal text is colored.
- Ghoelian 2y ago
  > If you're on macOS, feel free to try Ctrl-E, Ctrl-W, or Ctrl-U in your browser's address bar right now Most browsers I've used close the current tab when you press Ctrl-W. Actually, the terminal emulator I use, Alacritty, also does this, and most file explorers that have tabs also do. Iirc even windows explorer does this now, but it's been a while since I've actually used windows.
  - Ghoelian 2y ago
    Actually no I'm wrong, I forgot that command isn't control on a mac. Zen browser just does nothing when I press ctrl-W in the address bar.
  - poincaredisk 2y ago
    >Actually, the terminal emulator I use, Alacritty, also does this This can't be right. I use ctrl+w all the time, and occasionally use Alacritty. I'd notice if this shortcut closed my terminal window (it's extremely annoying when I use a web-based ssh, because I have this shortcut deep in my muscle memory).
shmerl 2y ago
> don’t use more than 16 colours We aren't in the '80s. Use true color if you want to, modern terminals should support it (built in Linux tty is a weird outlier that should have supported true color years ago). But that also depends on the context. For example if something implements its own TUI with a lot of elements - it makes more sense to use more colors than the barebones set. Most programs that do care about colors, check what terminal capabilities are before using them.
- zokier 2y ago
  As Evans explains, its not about technical capability, but about respecting users choice of colors/theme.
adiabatty 2y ago
One thing that I’ve noticed: On UNIX, expanding globs (*.txt) is the shell’s job. On Windows, expanding globs is the program’s job. I used to have a bunch of four-line Python programs to, essentially, run `flac --best --replay-gain *.wav`.
- zokier 2y ago
  On Unix programs get arguments as array of strings. On Windows programs get command line as single string. Strictly speaking that does not explain where globbing happens, but it does help understanding where they come from.
teddyh 2y ago
See also The Art of Unix Programming: <http://www.catb.org/~esr/writings/taoup/html/ http://www.catb.org/~esr/writings/taoup/html/>
xuhu 2y ago
How did copy-pasting text between programs fit in the initial shell design ? You can't paste from a remote tmux to a local one, you can't mouse-copy multiple lines from a vim vertical split, etc. Where's the terminal clipboard that works across programs ?
- _kst_ 2y ago
  You can (usually) run your remote tmux in a window in your local tmux session; then you can use the local tmux session's copy/paste features. I commonly run "ssh remote-system" in a tmux window, then attach to a tmux session on the remote system. If you nest tmux sessions like this, you have to type the prefix character (Ctrl-B by default; I use Ctrl-Space) twice for the nested session to see it.
eviks 2y ago
Some of the rules codify many bad practices, from poor color support to unergonomic keybindings. Given that the support isn't universal and breaks in parts anyway, it's better to break it competely and use something more ergonomic
- poincaredisk 2y ago
  If you write a command line program, please don't introduce random keybindings for simple actions, or unnecessarily fancy colors when standard 8 would suffice. These rules (or rather conventions) exist for a reason.
  - eviks 2y ago
    Sure there is a reason, just not a good one. Bad defaults are rather sticky. And the keybinding would be ergonomic, not random
Sophira 2y ago
One thing I'm wondering about is what text encoding the program should use to output. I tend to write scripts that output exclusively in UTF-8, but I realise this might not be a given. (And, of course, the user's terminal expected encoding setting doesn't necessarily mean that files should be written in that encoding.) Presumably, you would ideally output text in whatever encoding is specified by the LANG environment variable, but this seems like something that only comes with full i18n/l10n support, since it also specifies the actual language to use. Are there any actual established guidelines on this?
- Sharlin 2y ago
  I think these days on POSIX you can just assume UTF-8. On Windows the situation is... not great and entirely depends on what terminal/shell combination you're using.
yencabulator 2y ago
> then the operating system will return an EOF when you press Ctrl-D on an empty line. This is akshully not correct. Control-D makes the read(2) return with the data currently in the input buffer. If there's no data in the buffer, that results in a 0-length read, which is how EOF is signaled. Try this: run cat, type foo, press control-D. "foo" will be echoed, without any newline.
- wizzwizz4 2y ago
  It is correct, because when you press Ctrl+D on an empty line, the OS will return an EOF. It's not a complete answer, but neither's yours, because you didn't consider what happens when the OOM killer reaps cat. However, it's not intended to be a complete description, because it's describing a UI convention.
eschaton 2y ago
Other rules for command line tools: 1. Don’t assume a terminal type. Look at `TERM` and use termcap/terminfo or a library built atop them for anything beyond line-oriented plain text output, or least assume a plain teletype unless you specifically recognize the user’s terminal type. 2. Don’t assume the presence of a terminal at all. Check `isatty()` before doing anything fancy, and be sure to work without a terminal so your tool can be used in pipelines and called by other programs via `exec()`. 3. Follow the common conventions in your arguments and output structure. For example, if your tool takes an open-ended set of arguments, support specifying a response file with `@path/to/file` to avoid argument limits. If your tool supports record-oriented output, support a `-0` argument to `NUL`-separate the output for use with `xargs` in pipelines. 4. Use the standard `<sysexits.h>` exit codes. They exist for a reason and they make use of your tool within pipelines and programs more straightforward because they make it easier to trace why a failure occurred. 5. Include both in-binary `--help`/usage information and a man page. Often a user will just need a quick refresher on argument syntax, which is what the built-in help text is for; the man page should be a comprehensive reference with examples. It should *never* defer to a web page or GNU `info`—it’s fine if those exist too and are pointed out, but they should not be the primary user reference. Lots of Linux-oriented tools have one or more of these failure modes, and behave poorly on real terminals that aren’t VT100 derivatives or are awkward to use in anything but an interactive setting.
- _kst_ 2y ago
  > 1. Don’t assume a terminal type. Look at `TERM` and use termcap/terminfo or a library built atop them for anything beyond line-oriented plain text output, or least assume a plain teletype unless you specifically recognize the user’s terminal type. I agree, but these days I think you can mostly get away with assuming VT100-compatible behavior. > 4. Use the standard `<sysexits.h>` exit codes. They exist for a reason and they make use of your tool within pipelines and programs more straightforward because they make it easier to trace why a failure occurred. I just took a look at this header file. (It defines exit codes starting at 64.) I'm not sure I've ever seen a program that uses these codes. Many programs for UNIX-like systems just use exit(1) for generic errors, or maybe something to distinguish between data errors and usage errors such as unrecognized command-line options. I mostly use Linux; maybe it's more common on BSD-based systems (it appeared "somewhere after 4.3BSD"). For example curl defines nearly 100 distinct error codes; none of them are based on <sysexit.h>. > 5. Include both in-binary `--help`/usage information and a man page. Often a user will just need a quick refresher on argument syntax, which is what the built-in help text is for; the man page should be a comprehensive reference with examples. It should never defer to a web page or GNU `info`—it’s fine if those exist too and are pointed out, but they should not be the primary user reference. In practice, GNU `info` tends to be the primary reference for GNU programs. The man page is often missing a lot of information.
EasyMark 2y ago
Another one that comes in really handy for ssh connections: hit -> [return] ~ . to break a jammed up ssh session. A lot of people don't seem to know that one.
xg15 2y ago
It's also important to know which rules are "hard", i.e. implemented or enforced by the first three (or implemented by the program, but would be a complete nonstarter to break) and which rules are just "conventions", like the "short options/long options" rules in POSIX. In particular, those can change between OSes: On all linuxes, programs already get a pre-parsed array of command line args on start, so parsing and quoting behavior depend on the shell, not individual programs. On Windows, programs get passed the entire command line invocation as a string and have to do the parsing themselves. While all "well-behaved" programs let the libc do this, it's perfectly possible for a program to break the rules here.
noisycarlos 2y ago
I can't be the only one that accidentally closes tabs in my browser when I want to delete a word with Ctrl+W

298 points by charlieok 2y ago

> programs should print “regular” output to stdout and errors to stderr This is really important. I'd like to expand on this. Standard output is for the data the program was asked to produce, no more and no less. If user asked for some JSON data, standard output should contain that exact JSON object and absolutely nothing else. Standard "error" is actually a misnomer. It should have been called the standard user stream. Anything meant for the user to read on the terminal is supposed to go there. Error messages are of course included in that set but so are status messages and verbose output. This ensures the output of programs can be piped into other programs seamlessly. Non-output data still gets sent to the terminal or redirected somewhere else. Would have been great if programs were able to easily create new terminal-connected file descriptors for specific purposes. They could document those numbers in their manuals just like they document exit codes. Then users would get "ports" for every output. Could cut down on parsing significantly. For compatibility, they could all redirect to either standard output or standard error by default... I think I'm gonna experiment with this a bit.

packetlost 2y ago
I'm not sure I like the "standard user stream" name, but I otherewise agree with everything here.
- matheusmoreira 2y ago
  Yeah, naming is hard. If there is a better word for "non-output", I don't know it.
  - packetlost 2y ago
    I would maybe call it stdlog or stdinfo vs stderr, but yeah naming things is hard.
    - matheusmoreira 2y ago
      I think stdinfo would exclude error messages from the definition, just like stderr excludes info messages. I really like stdlog. Standard log stream is a pretty awesome name. Short and terse, the word "log" doesn't even need abbreviation and it's correct since it's a superset of error and info streams and also generic enough to cover other unforeseen categories. I'll use it from now on!
      - packetlost 2y ago
        The idea for info is that it's informational, which is always the case regardless of whether it's printing error messages due to a failure or informational messages because the user specified a `-v` flag or whatever. Whether the command failed or not is communicated via the exit code. That being said, I do prefer stdlog for the reasons: it communicates the intent and usage as unambiguously and tersely as possible.
        matheusmoreira 2y ago
        > because the user specified a `-v` flag or whatever The way I see it is the version information and help text belong on standard output if the user passes --version and --help since that's what the user explicitly asked for. When the command is invoked incorrectly, the help text should go to standard error while standard output should be empty. I agree about the exit code. People like to parse error messages and that's always wrong. Everything should be done via exit codes instead.
        packetlost 2y ago
        The `-v` was in reference to the common convention of one or more `v` flags increasing the verbosity of logging.
    - ryandrake 2y ago
      C++ has std::clog[1], but it's basically cerr, and the OS still only knows about stdin, stdout, and stderr. 1: https://cplusplus.com/reference/iostream/clog/ https://cplusplus.com/reference/iostream/clog/
- paulddraper 2y ago
  I'd name it "standard log." That's what it is....a log. Of errors, warnings, informational messages, whatever to complement the primary output. BONUS: stdlog fits nicely :)
- saghm 2y ago
  What about "debug"? I think the important part about stderr is that it's "meta" info about how the program is running (or ran) rather than what the program is intended to produce as output. This seems pretty similar to what often is referred to as the "debug" level of logging (which includes lower levels of logging like warnings and errors).
lionkor 2y ago
Unless your app doesn't generate output,then grepping the output will not work if it's stderr and it will be confusing
- bombcar 2y ago
  You can learn the 2>&1 thing, but it would be nice if there was a feature of one of the greps to "slurp up" the error output. Of course, the real problem is there's no standard, the standards that do exist are ignored, and each new command-line tool generates a new standard.
  - sudobash1 2y ago
    It generally isn't possible for grep to "slurp up" the error output because stderr does not get passed through pipes (by default). This is shell behavior and grep cannot do anything about it. As a side note, some shells have started implementing a shorthand for `foo 2>&1 | grep` which is `foo |& grep`.
    - ElevenLathe 2y ago
      Probably you could do a hack like finding the program at the other end of the pipe and looking at its open file descriptors, but lots of corner cases to consider.
    - kelnos 2y ago
      Right, the problem is that apparently shells haven't seen user need for selectively redirecting one or the other. Like I might want to see foo's stdout on the terminal, but pipe stderr (and only stderr) to grep, perhaps so I only see some specific error messages that I care about. `2>&1` will send stderr to stdout, and then the pipe will send everything to grep (and I believe `|&` does the same). (To be fair, that does seem like a pretty rare use case.)
      - sudobash1 2y ago
        Actually, you can. If you run: foo 2>&1 >&- | grep ... then you will grep only the stderr. The stdout will be discarded. Alternatively, if you only want to grep to see the stderr, but you still want to see stdout, you can swap stderr and stdout like this: foo 3>&2 2>&1 1>&3 | grep ... And in many shells, you can easily split the stdout and stderr into two separate pipelines like this: foo > >(grep stdout) 2> >(grep stderr)
AndrewDucker 2y ago
This is definitely one of the things that PowerShell got right. 6 different streams, each of which can be intercepted separately and configured differently. https://learn.microsoft.com/en-us/powershell/module/microsoft.powershell.core/about/about_output_streams?view=powershell-7.4 https://learn.microsoft.com/en-us/powershell/module/microsof...
- matheusmoreira 2y ago
  There's a lot of good ideas in power shell. I like how they broke up the log stream into separate streams by severity. It still seems arbitrary and insufficiently general though. Why severity and not some other criteria? What if programs could define any number of output streams? By default they could all coalesce into the terminal but other programs could connect to each one separately if they needed. Like an audio mixer of sorts: by default you get the mixed audio but there are ways to access each individual voice if needed.
- immibis 2y ago
  I don't think that 6 different streams that will always go to the same place but some of them are usually turned off are very useful. Standard-error/user is useful because it goes to a different place. Levels would be better suited to a simple global variable, or a level associated with each printed line. The chance that someone redirects errors to error.log is reasonable, but the chance that someone redirects errors to error.log and warnings to warning.log and debug to debug.log and verbose to verbose.log is pretty low.
  - AndrewDucker 2y ago
    I turn on Verbose logging sometimes, to get more info. Error logging can be set to either spit out the error and continue, or to stop processing, which is very useful. Warning and error messages come out in their own colours automatically, which is nice. And sure, you can just have them all come out to the same place. But having the options to configure them is very handy.
    - 2y ago

bobbiechen 2y ago

Nice writeup. Since she mentioned how hard it is to learn these conventions, I'll plug my preferred reference when thinking about CLIs specifically (rather than TUIs and REPLs) - the Command Line Interface Guidelines https://clig.dev https://clig.dev It does include the blog post's rules on exiting on Ctrl-C, accepting `-` for stdin, disabling color in pipes, and much more.

deleted 2y ago
[deleted]
thiisguy 2y ago
This is referenced at the very beginning, just after the table of contents.
- bobbiechen 2y ago
  Oops, I missed it when I was reading, and assumed it wasn't referenced. Thanks!
mcint 2y ago
I would also recommend, call to attention, the Further Reading from CLIG, https://clig.dev/#further-reading https://clig.dev/#further-reading. POSIX, GNU, Unix resources, Heroku CLI guide, and 12-factor CLI app guide.
snthpy 2y ago
Thanks. Didn't know about that one. There is also: https://usage.jdx.dev/ https://usage.jdx.dev/

felixhummel 2y ago

I'd add "long-running processes should reload their configuration on SIGHUP". :)

rocqua 2y ago
This isn't a list of shoulds. It's a list of, generally dones. Big difference. This one is still nowhere near universal enough to count in the original lisr sadly.
throwaway127482 2y ago
This behavior seems difficult to actually implement in practice, no? Your application has to be careful not to accidentally cache any configuration state, as well as any state derived from the configuration, otherwise it will go out of sync. Maybe easy for trivial apps but most apps requiring configuration files are non trivial.
abbeyj 2y ago
This is probably technically out of scope for this article. Long-running processes that respond to SIGHUP will usually be running detached (with no controlling TTY). Thus it is a bit of stretch to call them "terminal programs".

anthk 2y ago

Don't hardcode readline keybindings.

Sesse__ 2y ago

“rule 5.1: Ctrl-W should delete the last word […] I can’t think of any exceptions to this other than text editors but if there are I’d love to hear about them!” mysql(1) only links to editline instead of readline, where Ctrl-W by default deletes everything to the start of the line, not just the last word. It drove me mad in the period where I had to use it; you just see your entire nice query disappear. :-)

saghm 2y ago
It could be worse! Back in college, I had the misfortune to decide to try to use sqlplus, Oracle's CLI to try to connect to their database. Not only did terminal shortcuts like Ctrl-W not work at all, but you couldn't even move the cursor earlier in the line, and there was no history navigation to get to the previous command, so any typo forced you to retype the entire thing from scratch. Entering a single forward slash would allow you to run the last command verbatim, but that didn't help with typos. Ctrl-L also didn't work for clearing the screen, but you could manually run `clear scr` to do it instead, because I guess having the command just be `clear` wouldn't be obvious enough.
- cafard 2y ago
  SQL*Plus is pretty basic, but L will show you the previous command, a number will give you that line of the previous command, to edit, etc.
- Sesse__ 2y ago
  If you ever need to do so again, try gqlplus, which wraps sqlplus to add such quality-of-life features :-)
  - saghm 2y ago
    I hope my streak of almost a decade of not needing to use Oracle will continue, but if not, I definitely will be looking for alternatives to sqlplus...
    - 2y ago

CorrectHorseBat 2y ago

[flagged]

blueflow 2y ago
Do an "stty -a" in the terminal - you will see these two keys are handled by the kernel itself and will also work with programs reading dumbly from stdin, like "cat".
kps 2y ago
Ctrl-W originates (on Un*x) from the ‘new’ BSD tty driver in the late '70s. Ctrl-U vs Ctrl-X was one of those BSD-vs-AT&T things (where V7 defaulted to ‘@’ 'cause your terminal might not have those fancy control characters). `vi` respected the current tty settings.
d3VwsX 2y ago
Ctrl-w and ctrl-u in a default emacs do not even do the same things as in readline. C-w is (usually) kill-region (similar to what most editors call "cut"). C-u is universal-argument, that is used as a prefix for other commands to add arguments.

sysread 2y ago

Non-interactive programs that emit informational output should only do so to stderr or a log file so that they may be used in a pipe line.

deleted 2y ago
[deleted]

model-15-DAV 2y ago

As an addendum to Rule 7, `cd -` takes you to the last opened directory. Or is `cd` considered part of the terminal emulator's job, as a built-in?

saghm 2y ago
I don't think it's part of the terminal emulator (e.g. xterm, gnome-terminal) but the shell (bash, zsh, etc.). You're correct that it's not something that's implemented as an external program though; the "current directory" is state for a currently running terminal session, so changing that state is done via the shell interface (either directly by built-in commands like cd or indirectly via external commands that use those transitively).
- model-15-DAV 2y ago
  True, I meant the shell, as `cd` is a shell builtin. However, `git-checkout` does implement this behavior. `git checkout -` will checkout the previously checked out branch.
- machinestops 2y ago
  Note: Child processes can't change the working directory of the parent. An external command (that is, not a shell builtin, shell function, or externally loaded module) cannot change the working directory, because they're launched as child processes.
  - saghm 2y ago
    Good point. I'm not sure why I was thinking that it was possible to do via some other command invoking `cd` or something, but you're right that the only examples I can think of are all using other builtins (e.g. invoking `source` to have a script change the current state).
    - d3VwsX 2y ago
      There is a workaround I saw used by wcd, a tool for changing directories. To install it you have to add a wrapper function in your shell that executes the actual wcd binary, and after possibly interacting with the user to figure out what directory to change to the executable will print out the destination, and then the wrapper function will make the call to cd, affecting the shell it runs in. https://wcd.sourceforge.io/ https://wcd.sourceforge.io/

lyxell 2y ago

I’d like to add: Programs should not add files to your home directory and should respect XDG_CONFIG_HOME and friends.

kelnos 2y ago
I agree, but this article is -- as the author tries to make clear -- descriptive, not prescriptive. She's listing out the things she's seen commonly in applications, not trying to convince applications that they should behave in certain ways.

chriswarbo 2y ago

Ctrl-D for REPLs always bites me with GHCi. My usual approach to quitting GHCi is: - Press Ctrl-D, like normal - Get confused when nothing happens - Remember that it doesn't work in GHCi, so run `:q` instead - Get an error message about "lexical error at character '\EOT'", due to Ctrl-D inserting an invisible char at the start of the input - Try `:q` again, without any invisible prefix - GHCi successfully quits

jeffrallen 2y ago
You think that's hard, try getting out of vi when you're on a keyboard from an unfamiliar country.
- lilyball 2y ago
  Do keyboards in unfamiliar countries not have the colon key?
  - WolfeReader 2y ago
    That or the Q key
  - Philpax 2y ago
    Both the colon and Q keys can be in different places, which can really mess with your muscle memory.
- from-nibly 2y ago
  Or if you are like me and have caps mapped as escape. Vim works really weird when you start typing random capital letters after thinking you were moving somewhere.
- saint_yossarian 2y ago
  There's also the `ZZ` / `ZQ` alternatives, so you just have to find those letters :)
- linhns 2y ago

jeffrallen 2y ago

If you are a young sysadmin, take the time to.learn Emacs. Not because Emacs is good (but it is) but because deadline keys are Emacs keys, so once you know Emacs you know shell, MySQL, etc. Play your terminal like a piano . Your livelihood depends on it.

neilv 2y ago

Additional suggestions: * Respect the user's default foreground and background color. Don't change them without good reason. * If you use colors, make them legible regardless of what the default background and foreground colors are, and regardless of the terminal's color map. * Don't use color as the only indication of something. The user's terminal might not display it, and it probably won't be preserved in copy&paste into notes. * Use emoji only judiciously, if at all. Similar with gratuitous non-ASCII characters. It doesn't display everywhere, it doesn't paste well everywhere, and emoji can be a bit much when copy&pasted into some notes. * In a scrolling (non-full-screen) stdout-ish output, don't delete important information that you showed temporarily. For example, hiding warnings or filenames compiled, to display a green checkmark for done. For another example, clearing the screen of Web app build information (including package security warnings!), to display a message that it's running in dev mode, is also not wanted. People might want to see that information, or copy&paste it into notes. * If you went full angry fruit salad with your command line program, because it's your baby, and you're having fun hamming it up, that's fine, but please provide an easy preference setting for people to opt out of that. Your program is probably only one of many things on user's workstation display, where other programs might be using color and visuals more meaningfully, so animated throbbing red explosions for the code reformatter is a bit much.

deleted 2y ago
[deleted]
hatthew 2y ago
As the article explicitly stated, these are descriptive not prescriptive rules. They're things that you can generally assume all terminal programs already follow.
- neilv 2y ago
  The ones I listed were mostly descriptive, until recent years.
teddyh 2y ago
> please provide an easy preference setting for people to opt out of that. There is some effort to standardize this: <https://bixense.com/clicolors/ https://bixense.com/clicolors/>

ucarion 2y ago

What does "cooked" mode mean in the context of this article?

kelnos 2y ago
It means that the keystrokes you type aren't just immediately handled to the application in raw form; something higher up the chain (kernel, terminal emulator, shell) is pre-processing ("cooking") them in some way, possibly taking actions before (or in place of) passing the keystrokes to the application. For example, if an application you're using has left things in cooked mode, and you press ctrl+c, the application will never "see" that keystroke; something higher up in the input chain (I believe Linux's TTY driver) will see it, swallow it, and send SIGINT to the application. Applications can also put the tty into "raw mode", where this won't happen; in that case the app is responsible for implementing those "expected" keystroke behaviors, if it makes sense for the application to do so.
- lilyball 2y ago
  cooked mode also generally gives input to the program a line at a time and handles things like backspace to delete characters, and it handles program output too, doing things like replacing \n with \r\n.
  - teddyh 2y ago
    No, line-by-line vs. character-by-character is a separate mode from “cooked” mode. See “stty extproc”.
    - lilyball 2y ago
      I read up on this and "cooked but character-by-character" is apparently something called "cbreak mode" (or "rare mode").
      - teddyh 2y ago
        According to stty(1), ”cbreak” is a negative alias of “icanon”, which simply enables the special functions of the ‘erase’ (backspace), ‘kill’ (Ctrl-U), ‘werase’ (Ctrl-W), and ‘rprnt’ (Ctrl-R) keybindings. “cooked” mode includes “icanon”. So no, I do not think that what you wrote is correct. From what I can tell, the line-by-line mode is “extproc”.

mattofak 2y ago

In the same category of command line program guidelines: https://clig.dev/ https://clig.dev/

lieks 2y ago

The main reason I enjoy CLIs so much more than GUIs (or eves TUIs sometimes) is that it feels so consistent. There are conventions, but following all the conventions in a CLI is a lot easier than designing a good GUI. So they tend to be higher quality as a result. I spend a lot of time thinking how to bring this property to GUIs, but my best answers are still "lots of effort" or "lower your expectations".

tpmoney 2y ago
A large part of this I think is that other than MacOS, GUI conventions aren’t heavily implemented by the OS and easy to opt into. And because no other OS landed on “meta key for GUI shortcuts” so there are a lot more conflicts (e.g. ctrl-c). The Application Framework defaults which underly most native macOS application build a lot of common keyboard controls in. Use the default menu bar classes with the default basic commands, and Command O, N, Q, X, C,V and probably others come for free, you just implement the code that you need to for those functions. Use a standard text field and you automatically get Command/Option/Fn Left, Right, Up, Down for navigation. It’s more notabke when a macOS application doesn’t follow convention (e.g. InteliJ uses Shift-Opt up/down to move lines rather than expand the selection by paragraph) than when one does. Windows does decently well on this front, but ctrl as a default modifier can hurt terminal based app usage and there are a number of UI frameworks even within the OS that appear to get different defaults. And in the Linux world, I think the only way you could do this would be for someone to design (and a distro to standardize on and port apps to) a full on application framework. The window managers don’t want to be in the business of dictating the behavior of stuff in windows. The GUI toolkits don’t want to be in the business of defining os wide defaults and the DEs and distros don’t want to be in the business of if porting or dictating UI frameworks. And realistically there’s no one “on high” that could make the sort of dictation that for example “hence forth copy and paste will be Meta-C and Meta-V”

ruricolist 2y ago

For forcing colorized output: I personally prefer pipetty (from the colorized-logs package on Debian) to unbuffer, since unbuffer merges stdout and stdin.

cbhl 2y ago

> rule 3: REPLs should quit when you press Ctrl-D on an empty line If memory serves, this behavior depends on the OS. On Windows I believe the norm there is to type "<Ctrl-Z><Enter>"

hatthew 2y ago
I think it's assumed this article is about *nix, not windows

cpif 2y ago

I can use Ctrl-A, Ctrl-E, and Ctrl-U in text fields in the lynx browser, but not Ctrl-W. I just checked to see if Ctrl-F and Ctrl-B work, and found that the former kills one word forward and the latter acts like Ctrl-W ought to?

jrockway 2y ago

One that's missing is treating ~ as the home directory. This appears to be a shell thing and not a POSIX API thing. For example, this doesn't work: func main() { if _, err := os.ReadFile("~/.bashrc"); err != nil { log.Fatal(err) } fmt.Println("ok") } Meanwhile over in shell land: $ echo ~/~/~ /home/jrockway/~/~ The behavior is actually kind of amazing. I mention it because while "yourprogram ~/path/to/file" always works, having a repl that asks for a filename might not work. I've seen a lot of software where this DOES work, so I think it counts as a "most TUI programs do this" thing.

wahern 2y ago
Tilde expansion is part of shell word expansion: https://pubs.opengroup.org/onlinepubs/9799919799/utilities/V3_chap02.html#tag_19_06_01 https://pubs.opengroup.org/onlinepubs/9799919799/utilities/V... Tilde expansion is the first operation in word expansion: > The expansions that are performed for a given word shall be performed in the following order: 1) Tilde expansion, parameter expansion, command substitution, and arithmetic expansion shall be performed, beginning to end. 2) Field splitting shall be performed on the portions of the fields generated by step 1. 3) Pathname expansion shall be performed, unless set -f is in effect. 4) Quote removal, if performed, shall always be performed last. See https://pubs.opengroup.org/onlinepubs/9799919799/utilities/V3_chap02.html#tag_19_06 https://pubs.opengroup.org/onlinepubs/9799919799/utilities/V... To see the whole shell specification, goto the main page at https://pubs.opengroup.org/onlinepubs/9799919799/ https://pubs.opengroup.org/onlinepubs/9799919799/. Select "Shell & Utilities" from the top-left frame, then "Shell Command Language" from the bottom-left frame.
Sharlin 2y ago
That's a good point, particularly given that .. is handled for you at the OS API level so you might be inclined to expect ~ is too on POSIX.
- jasomill 2y ago
  This would massively complicate kernel path resolution on many systems. Taking Linux as an example, even in the simplest cases, ~user expansion as typically implemented — via the getpwnam(3) libc function, which in itself implies a rather unfortunate dependency — requires reading and parsing /etc/passwd, and may also require access to network directory services like LDAP.

jez 2y ago

Some more notes: - If this is your first time hearing about the readline/emacs keybindings like Ctrl-E and Ctrl-W, you'll be pleased to know that most macOS input sources use these keybindings. If you're on macOS, feel free to try Ctrl-E, Ctrl-W, or Ctrl-U in your browser's address bar right now - If you're using a command line program that doesn't support _any_ line editing (e.g. no readline keybindings, and no other keybindings), you can install the `rlwrap` program and launch the REPL under rlwrap. For example Standard ML of New Jersey has a REPL but no line editing functionality, but you can recover that via `rlwrap smlnj` - "don’t use more than 16 colours" — I would go so far as to say "don't use more than 8 colors, or at least make your colors configurable." Many popular color schemes, including Solaraized and the default Base 16 color scheme, use the "bright" colors to hold various shades of gray. What you think is "bright green" might actually be the same shade of gray that normal text is colored.

Ghoelian 2y ago
> If you're on macOS, feel free to try Ctrl-E, Ctrl-W, or Ctrl-U in your browser's address bar right now Most browsers I've used close the current tab when you press Ctrl-W. Actually, the terminal emulator I use, Alacritty, also does this, and most file explorers that have tabs also do. Iirc even windows explorer does this now, but it's been a while since I've actually used windows.
- Ghoelian 2y ago
  Actually no I'm wrong, I forgot that command isn't control on a mac. Zen browser just does nothing when I press ctrl-W in the address bar.
- poincaredisk 2y ago
  >Actually, the terminal emulator I use, Alacritty, also does this This can't be right. I use ctrl+w all the time, and occasionally use Alacritty. I'd notice if this shortcut closed my terminal window (it's extremely annoying when I use a web-based ssh, because I have this shortcut deep in my muscle memory).

shmerl 2y ago

> don’t use more than 16 colours We aren't in the '80s. Use true color if you want to, modern terminals should support it (built in Linux tty is a weird outlier that should have supported true color years ago). But that also depends on the context. For example if something implements its own TUI with a lot of elements - it makes more sense to use more colors than the barebones set. Most programs that do care about colors, check what terminal capabilities are before using them.

zokier 2y ago
As Evans explains, its not about technical capability, but about respecting users choice of colors/theme.

adiabatty 2y ago

One thing that I’ve noticed: On UNIX, expanding globs (*.txt) is the shell’s job. On Windows, expanding globs is the program’s job. I used to have a bunch of four-line Python programs to, essentially, run `flac --best --replay-gain *.wav`.

zokier 2y ago
On Unix programs get arguments as array of strings. On Windows programs get command line as single string. Strictly speaking that does not explain where globbing happens, but it does help understanding where they come from.

teddyh 2y ago

See also The Art of Unix Programming: <http://www.catb.org/~esr/writings/taoup/html/ http://www.catb.org/~esr/writings/taoup/html/>

xuhu 2y ago

How did copy-pasting text between programs fit in the initial shell design ? You can't paste from a remote tmux to a local one, you can't mouse-copy multiple lines from a vim vertical split, etc. Where's the terminal clipboard that works across programs ?

_kst_ 2y ago
You can (usually) run your remote tmux in a window in your local tmux session; then you can use the local tmux session's copy/paste features. I commonly run "ssh remote-system" in a tmux window, then attach to a tmux session on the remote system. If you nest tmux sessions like this, you have to type the prefix character (Ctrl-B by default; I use Ctrl-Space) twice for the nested session to see it.

eviks 2y ago

Some of the rules codify many bad practices, from poor color support to unergonomic keybindings. Given that the support isn't universal and breaks in parts anyway, it's better to break it competely and use something more ergonomic

poincaredisk 2y ago
If you write a command line program, please don't introduce random keybindings for simple actions, or unnecessarily fancy colors when standard 8 would suffice. These rules (or rather conventions) exist for a reason.
- eviks 2y ago
  Sure there is a reason, just not a good one. Bad defaults are rather sticky. And the keybinding would be ergonomic, not random

Sophira 2y ago

One thing I'm wondering about is what text encoding the program should use to output. I tend to write scripts that output exclusively in UTF-8, but I realise this might not be a given. (And, of course, the user's terminal expected encoding setting doesn't necessarily mean that files should be written in that encoding.) Presumably, you would ideally output text in whatever encoding is specified by the LANG environment variable, but this seems like something that only comes with full i18n/l10n support, since it also specifies the actual language to use. Are there any actual established guidelines on this?

Sharlin 2y ago
I think these days on POSIX you can just assume UTF-8. On Windows the situation is... not great and entirely depends on what terminal/shell combination you're using.

yencabulator 2y ago

> then the operating system will return an EOF when you press Ctrl-D on an empty line. This is akshully not correct. Control-D makes the read(2) return with the data currently in the input buffer. If there's no data in the buffer, that results in a 0-length read, which is how EOF is signaled. Try this: run cat, type foo, press control-D. "foo" will be echoed, without any newline.

wizzwizz4 2y ago
It is correct, because when you press Ctrl+D on an empty line, the OS will return an EOF. It's not a complete answer, but neither's yours, because you didn't consider what happens when the OOM killer reaps cat. However, it's not intended to be a complete description, because it's describing a UI convention.

eschaton 2y ago

Other rules for command line tools: 1. Don’t assume a terminal type. Look at `TERM` and use termcap/terminfo or a library built atop them for anything beyond line-oriented plain text output, or least assume a plain teletype unless you specifically recognize the user’s terminal type. 2. Don’t assume the presence of a terminal at all. Check `isatty()` before doing anything fancy, and be sure to work without a terminal so your tool can be used in pipelines and called by other programs via `exec()`. 3. Follow the common conventions in your arguments and output structure. For example, if your tool takes an open-ended set of arguments, support specifying a response file with `@path/to/file` to avoid argument limits. If your tool supports record-oriented output, support a `-0` argument to `NUL`-separate the output for use with `xargs` in pipelines. 4. Use the standard `<sysexits.h>` exit codes. They exist for a reason and they make use of your tool within pipelines and programs more straightforward because they make it easier to trace why a failure occurred. 5. Include both in-binary `--help`/usage information and a man page. Often a user will just need a quick refresher on argument syntax, which is what the built-in help text is for; the man page should be a comprehensive reference with examples. It should *never* defer to a web page or GNU `info`—it’s fine if those exist too and are pointed out, but they should not be the primary user reference. Lots of Linux-oriented tools have one or more of these failure modes, and behave poorly on real terminals that aren’t VT100 derivatives or are awkward to use in anything but an interactive setting.

_kst_ 2y ago
> 1. Don’t assume a terminal type. Look at `TERM` and use termcap/terminfo or a library built atop them for anything beyond line-oriented plain text output, or least assume a plain teletype unless you specifically recognize the user’s terminal type. I agree, but these days I think you can mostly get away with assuming VT100-compatible behavior. > 4. Use the standard `<sysexits.h>` exit codes. They exist for a reason and they make use of your tool within pipelines and programs more straightforward because they make it easier to trace why a failure occurred. I just took a look at this header file. (It defines exit codes starting at 64.) I'm not sure I've ever seen a program that uses these codes. Many programs for UNIX-like systems just use exit(1) for generic errors, or maybe something to distinguish between data errors and usage errors such as unrecognized command-line options. I mostly use Linux; maybe it's more common on BSD-based systems (it appeared "somewhere after 4.3BSD"). For example curl defines nearly 100 distinct error codes; none of them are based on <sysexit.h>. > 5. Include both in-binary `--help`/usage information and a man page. Often a user will just need a quick refresher on argument syntax, which is what the built-in help text is for; the man page should be a comprehensive reference with examples. It should never defer to a web page or GNU `info`—it’s fine if those exist too and are pointed out, but they should not be the primary user reference. In practice, GNU `info` tends to be the primary reference for GNU programs. The man page is often missing a lot of information.

EasyMark 2y ago

Another one that comes in really handy for ssh connections: hit -> [return] ~ . to break a jammed up ssh session. A lot of people don't seem to know that one.

xg15 2y ago

It's also important to know which rules are "hard", i.e. implemented or enforced by the first three (or implemented by the program, but would be a complete nonstarter to break) and which rules are just "conventions", like the "short options/long options" rules in POSIX. In particular, those can change between OSes: On all linuxes, programs already get a pre-parsed array of command line args on start, so parsing and quoting behavior depend on the shell, not individual programs. On Windows, programs get passed the entire command line invocation as a string and have to do the parsing themselves. While all "well-behaved" programs let the libc do this, it's perfectly possible for a program to break the rules here.

noisycarlos 2y ago

I can't be the only one that accidentally closes tabs in my browser when I want to delete a word with Ctrl+W