13 ms·
Zsh and Fish’s simple but clever trick for highlighting missing linefeeds
- enriquto 6y agoI don't know if I'm disgusted or delighted by this solution, but it doesn't leave me indifferent!
- messe 6y agoAs I said in my other comment: Both. It's vomit-inducingly beautiful.
- Smaug123 6y agoThat is distressingly clever, and quite beautiful. It's one of those solutions that makes me wonder whether I could ever have come up with it.
- messe 6y agoIt's the kind of solution that were I to have come up with it, I would be both disgusted and impressed with myself.
- takeda 6y agoYou probably would feel the same if you would implement your own pty including re-implementing entire terminal emulation. That is until someone would show this solution.
- messe 6y agoCan't disagree with that, as that may be my hobby project in the next few weeks. I'm currently evaluating options for implementing a decent terminal emulator for jailbroken iPadOS. 95% chance I'm just gonna go with a webview and hterm just for simplicity on the development end, but there's a sick and twisted masochistic part of me that wants to write a full terminal emulator myself. You'd think working tech support would've beaten the masochism out of me.
- aloknnikhil 6y agoAnd also one of those things you stumble upon rather than think of on your own, I guess.
- GuB-42 6y agoIt is almost like a magic trick. You probably know some of them along the lines of : think of a number, then do a series of operations, and then I can guess the result. It can be done with cards too. In reality no matter what your initial choice is, the end result is always the same. The trick is to combine relatively complex functions (ex: "take the sum all digits of the number", "add 1 if odd") in a way that produce something simple (ex: "the result is 5"). Here the trick is to express the function "pos(x,y)=(0, y+1)" in term of the functions "pos(x,y)=(0,y)" and "pos(x,y)=if x<$column then (x+1,y) else (0,y+1). It is also the basis of the "abstraction inversion" anti-pattern.
- messe 6y agoThis is almost certainly going to be somebody's horrific programming interview question at some point in the next six months.
- jchook 6y agoNecessity is the mother of invention.
- psychoslave 6y agoWell, sure, but it doesn't necessarily incite to the most elegant solution. I mean, the article gives a sample of other possible approaches that one might fall into which are all far most eager it seems. Even taking into account ComputerGuru insights on all the bumps of the actual implementation, it's still seem far more cleaver and elegant, at least to my mind.
- ComputerGuru 6y agoOne of the fish devs here. I definitely didn’t come up with this hack but I’ve worked on improving some edge cases in the past. Our implementation differs from zsh some although I think we’d originally adopted their approach. This feature is still surprisingly complicated with many caveats. For example, some terminals will wrap to the next line if you are at $COLUMNS, some at $COLUMNS+1, and some not at all. Some advertise they do one thing and do another. If you miscalculate, you’ll end up skipping an extra line or none at all, maybe overwriting existing output or emitting the missing line break glyph when it should be hidden. We’ve ironed out most of the cases but every once in a while someone will be running an esoteric Pseudo tty (usually not under X) and run into issues. There are also issues with selection of content. Some terminal emulators will bug out when you try to copy-and-paste content because they get lost with the output past the current location in cases where a line break was already emitted. It’s always the terminal emulator that’s broken but you’d be surprised at how broken some of the most popular emulators are. The entire codebase is littered with workarounds for tty emulators; if someone comes to you and says “fish doesn’t work in $terminal but bash does” (either because it’s minimalistic and is missing a feature that would cause the breakage or because it has a workaround), we almost always end up treating it as a fish bug unless we’re able to get it fixed in the upstream terminal package/OS/software/whatever, whom we file reports with all the time. It’s insane the configurations and platforms people use shells under and I’m personally deeply appreciative of everyone that takes the time to file these bugs - in addition to making the software better it really does shed light on some extremely archaic bits of posix standards and Unix behavior and I’ve learned so much from the many deep-dives that start off with “this is kind of quirky behavior observed under certain circumstances” and ends up requiring architectural changes deep within because some underlying assumption doesn’t hold universally true as exemplified by a *nix that hasn’t seen a new user in fifteen years or something. It’s why I love hacking on fish; I must have one of the craziest collection of virtual machines to test edge cases, as do many of the other devs (especially @zanchey and @faho, although no one does a deep dive like @ridiculousfish).
- teddyh 6y agoWhy output spaces when “tput el” works?
- twic 6y agoWhat happens if you resize the window after this trick has been used?
- ComputerGuru 6y agoIt depends if you make it bigger or smaller and if the line break was missing or not. Under some combinations of those two parameters, previous lines overflow with white space.
- scarlac 6y agoI just tested in Terminal.app (macOS) with zsh and several things happen: If you do nothing but run `echo -n 'hello world'` and then resize the window, the next line prompt will be jittery (it'll jump left and right as you resize but ultimately return to the correct position). My prompt is "14:41:44 ~ ️ <cursor>" and if I resize fast enough, the previous line (the one with the output `hello world%` starts displaying characters on the right side. Characters from the next line. If I do it fast, it becomes obvious what it's doing: Fragments of the next line is copied to the previous line, so the right part of 'hello world%' is filled with: `111141414:314:3914:3914:39:` after a few resizes. If you press return again (executing no command) then the buggy output is now fixed and no magical behavior occurs. It may be easier to just see, so here's an example: https://www.loom.com/share/5eab7cfb4590475a8881fe3e91d92738 https://www.loom.com/share/5eab7cfb4590475a8881fe3e91d92738
- kccqzy 6y agoMy understanding is that zsh registers a signal handler for SIGWINCH and that handler fixes things up. --- SIGWINCH {si_signo=SIGWINCH, si_code=SI_KERNEL} --- rt_sigprocmask(SIG_BLOCK, ~[RTMIN RT_1], [WINCH], 8) = 0 rt_sigprocmask(SIG_SETMASK, [WINCH], ~[KILL STOP RTMIN RT_1], 8) = 0 ioctl(10, TIOCGWINSZ, {ws_row=45, ws_col=108, ws_xpixel=756, ws_ypixel=585}) = 0 getuid() = 1000 geteuid() = 1000 capget({version=_LINUX_CAPABILITY_VERSION_3, pid=0}, NULL) = 0 capget({version=_LINUX_CAPABILITY_VERSION_3, pid=0}, {effective=0, permitted=0, inheritable=0}) = 0 ioctl(10, TIOCSPGRP, [20139]) = 0 ioctl(10, TCGETS, {B38400 opost isig -icanon -echo ...}) = 0 ioctl(10, SNDCTL_TMR_STOP or TCSETSW, {B38400 opost isig -icanon -echo ...}) = 0 ioctl(10, TCGETS, {B38400 opost isig -icanon -echo ...}) = 0 write(10, "\r\r\33[0m\33[27m\33[24m\33[J\33[44mubuntu@i"..., 90) = 90 rt_sigreturn({mask=[]}) = -1 EINTR (Interrupted system call) It essentially queries the new window size, and then writes a new prompt beginning with two carriage returns.
- nneonneo 6y agoA slight improvement: adding "\\033[K" to the end, i.e.: PROMPT_COMMAND='printf "⏎%$((COLUMNS-1))s\\r\\033[K"' kills the extra spaces at the end, which can help alleviate some of the weirder line wrapping that occurs when resizing a terminal window. (I believe zsh uses something like \033[J${PS1}\033[K for similar effect).
- JetSpiegel 6y agoThis is a great addition to the post, thanks for this.
- simias 6y agoI love this article, as a long time ZSH user it's one of these things I've always wondered how they were implemented but never cared to investigate myself. It's really a clever hack too, if I had been asked to implement this feature my default approach would definitely have been to find a way to intercept the command's output somehow.
- ridiculous_fish 6y agowaves Great article about shell hackery. It's good to clear up the misconception that shells know what's on the screen. They don't even know where the cursor is! One important reason for this hack is right prompts. If rprompt has width 7, the shell moves right by $COLUMNS-7, outputs the rprompt, and then moves left to return the cursor. What happens if during the move-right phase, the cursor wraps to the next line? move-left doesn't "wrap back", it just pins against the left side! So your prompts get split across lines, your right prompt floats somewhere in space, and your input may even overlap it. It's quite confusing to the user. So if you have a right prompt, the shell has to be very sure it's on a new line when it starts!
- JdeBP 6y agoCUB does not wrap at left margin, but BS sometimes does. * https://unix.stackexchange.com/a/198445/5132 https://unix.stackexchange.com/a/198445/5132
- zokier 6y agoThis made me think how trailing newline is actually bit weird. Wouldn't it make more sense to just have shell prompts always start with newline and commands not outputting trailing newlines as gratuitously as they do now?
- patrec 6y agoIn unix every text file is basically a (possibly degenerate, Nx1) matrix and every row is terminated (and demarcated) by a newline. This makes it much easier to filter, concatenate and rearrange rows. Try rewriting this: cat <(cmd1...) <(cmd2...) >> some_output for a world without final terminating newlines.
- 1996 6y agoAs the path can be long, I do this: a first line for the things the shell want to displays, a second line for the things I type # (user@server):/where/I/am # This gives something like # (user@server):/where/I/am # cd now # (user@server):/where/I/am/now # The # is used because I sometimes copy-paste, and I want to avoid problems
- Ericson2314 6y agoI want to swap the layering shell and the terminal emulator, so each shell command (that needs it) gets it's own pty. I think this will enable a much nicer UI and also simplify things.
- opk 6y agoThis is what tmux is for.
- Ericson2314 6y agoNo this is not what tmux is fore. IMO tmux and screens are giant hacks: - multiplexing: better to use ssh multiplexing, which is just nice in general - persistence: Yes, you want to open a pty on a the host, but you should actually emulate the terminal on the guest: one should just forward all the pty messages between the guest and host (graphical emlator). In other words, a lot more like regular ssh - sharing: graphical emulators should just understand that multiple can be hooked up, have some support for this, we can relay the input from one emulator to the others as needed. For my terminal-inside-shell, I would use a customer server + protocol for managing all the ptys (remember because backgrounded commands there can be multiple).
- gnachman 6y agoI just discovered this yesterday when a user filed a bug against iTerm2. It caused a problem for reflow because it looks like one long line. Took a minute to makes sense of what was going on, and then I thought it was a neat trick but it wouldn’t have occurred to me to post on hn about it. Glad others appreciate it!
- macintux 6y agoSince it can’t be uttered often enough: thank you for iTerm2. I’m not a power user, but it and Emacs are generally my first two software installs on any new Mac.
- spurgu 6y ago+1 I've thought about switching back to Linux but I wouldn't want to live without iTerm2. Edit: This prompted me to make a modest donation.
- chungy 6y agoThis is such a wonderful little hack, I've placed it into my .bashrc. Thank you! I did actually modify it a little bit, got rid of the two beginning %s in favor of ⏎. It's not a character I expect to really happen to end legitimate output of a program, so it seems more than good enough to keep it from being accidentally intended.
- opk 6y agoApply some attributes to it if you want it to be clearer that it isn't part of the legitimate output of a program. This is why the zsh default is a % in standout. And for zsh user's if you have a setup where unicode fonts can be replied upon, `PROMPT_EOL_MARK` can be set to change the mark. It allows all the usual zsh prompt expansions for stuff like coloring.
- amelius 6y ago> Contrary to popular belief, the shell does not sit between programs and the terminal. Perhaps it should?
- soraminazuki 6y agoFor better or for worse, unix simply wasn't designed that way. Changing this now would mean breaking so many assumptions hard-coded into a vast majority of unix applications. Considering how complex unix TTYs are, there is practically zero chance this can be done transparently to existing programs. It's going to need a redesign of the I/O system, at which point, we might as well call this new system Plan10. I also think there are problems with the approach itself. Making the shell sit between programs would complicate the interaction between various programs and introduce new bottlenecks in the shell. Additionally, it likely won't be an improvement over the current situation. Rather, it would just introduce another layer of incompatibility in the shell that programs would have to deal with.
- hnlmorg 6y agoI did this originally with my shell and it's actually not as painful as you'd think. Most CLI tools don't care where fd 0, 1 and 2 are coming from / going to (after all, if you're piping those programs then they're no longer writing to the TTY anyway). The only programs that cause an issue are tools that send ANSI escape sequences (eg for ncurses) and check if STDOUT is a TTY before sending them. In those cases you'll just get a message written saying something along the lines of "STDOUT is not a TTY". In theory you could write another workaround to fix that workaround but it was a horrible kludge that I didn't like in the first place so ended up finding another solution (which isn't as elegant as the zsh / fish fix and I'll soon be adapting that into my own shell too).
- soraminazuki 6y ago> The only programs that cause an issue are tools that send ANSI escape sequences (eg for ncurses) and check if STDOUT is a TTY before sending them. Sure, but aren't those programs the whole point of having sophisticated terminal emulators? If plain streams of text was all we cared about for CLI applications, it would indeed make things a lot easier. So much easier, in fact, that we can dump our current terminal emulators in favor of a more simple alternative without ever having to complicate the shell. But since people do care about terminal UIs, this isn't a realistic solution.
- ChristianBundy 6y agoI learned a new thing today, thanks for posting this!
- gumby 6y agoThis is indeed a clever approach -- and I've been using printing terminals for over 40 years.
- beaker52 6y agoThis explains why I've seen % when the prompt has been missing for whatever reason.
- sime2009 6y agoOMG, as a developer of a terminal application I've encountered this little symbol but just couldn't figure out where it was coming from. I assumed it was something I was doing wrong on the emulation side.
- enriquto 6y agoI do not like this feature. How can you distinguish the output of a program that outputs a newline from one that doesn't? This is intentional obfuscation. If you are bothered from where your prompt starts, add a newline at the beginning of your prompt.
- Rogach 6y agoThere is a "missing linefeed indicator" symbol (usually %) that is output if there is no trailing newline. So the output of the program that outputs a newline looks like: $ program output $ And output of the program without trailing newline: $ program output% $ There is an issue if the program is expected to output "%" at the end, of course.
- maddyboo 6y agoTo make it a bit easier to distinguish, zsh seems to display the indicator with inverted fg/bg colors.
- qwerty456127 6y ago> The shell could use pipes to intercept all output, and relay it onto the terminal. While it works in trivial cases like whoami, some programs check whether stdout is a terminal and change their behavior, others go over your head and talk to the TTY directly (e.g. ssh‘s password prompt), and some use TTY specific ioctls that fail if the output is not a TTY, such as querying window size or disabling local echo for password input. Where do I read more about this? I'd like to understand terminals, pipes and the differences. And how does the mouse (in case of terminal apps which support mouse, e.g. Midnight Commander) fit in this model.
- macintux 6y agoMy goto book for these concepts has always been APUE, although the 2nd edition at least doesn’t cover mice that I can see. http://www.apuebook.com/ http://www.apuebook.com/
- Dylan16807 6y agoA pipe is like a special file that takes text from one program and puts it into another. A terminal is either a physical object with a screen and a keyboard, or software that pretends to be such an object. The terminal can display text, but also certain sequences of text do special things. For example, if the terminal receives the character for the escape key followed by [?1000h then it doesn't display it. Instead it turns on mouse click support. So a program connected to your terminal that wants mouse support can send that sequence. Then if you click, your terminal types an escape key, followed by [M, followed by three characters to say what buttons you're holding and where the mouse is located. Now the program will know you clicked.
- qwerty456127 6y agoThank you very much, now I have an idea of how does mouse support work in a Linux terminal. However, I still don't understand how does a program know it's connected to a terminal rather than some pipes.
- surajrmal 6y ago
- stared 6y agoI am waiting for GPT-2-based models for ZSH. Something in the line of TabNine, but for the terminal.
- kazinator 6y agoInterrogating the terminal emulator to get the current column is in fact a way smarter, more robust solution that will work regardless of the terminal's behavior when printing at the rightmost column. Also, fewer characters are exchanged with the TTY in the happy case. Proof of concept, using Bash on Ubuntu 18.04: Define this function: getcol() { local savetty=$(stty -g < /dev/tty) local ttyesp stty raw min 16 time 100 < /dev/tty printf '\e[6n' > /dev/tty read -s -r -n 16 -d R ttyresp < /dev/tty stty $savetty < /dev/tty printf "%s\n" ${ttyresp#*;} } Then this PS1 for testing: PS1='$(if [ $(getcol) != 1 ]; then echo '[noeol]' > /dev/tty; fi)\$ ' Test: $ echo good output good output $ echo -n bad output bad output[noeol] $ echo -n # no output case $ WFM
- JdeBP 6y agoUntil you can name at least one common current terminal emulator that does not respond to DSR 6, you have not tested this enough. (-:
- kazinator 6y agoWhy would I search for a situation that's going to be a definite WONTFIX? I made up my mind some fifteen years ago that if a terminal isn't ANSI conforming, it can be casually disregarded as something unsupported. It's like testing that web pages work with Mosaic from 1993.
- JdeBP 6y agoBecause you are claiming to be providing something that is "smarter" and "more robust". Not actually testing it enough to find the common case where it doesn't work gives the lie to that claim. It indicates if anything a lack of smartness, and is not robust. (Even Thomas Dickey's Stack Exchange answer on the subject, which is better than what you've given here, isn't robust, as M. Dickey didn't describe how to robustly parse a CPR sequence.) When people talk about "ANSI sequences" and "ANSI conformant", they also indicate that likely their knowledge comes from old wives' tales and samizdat. The actual existing standards are ECMA-35 and ECMA-48, and the "E" in "ECMA" does not stand for "ANSI". The world of terminals is alas full of people who've worked off informal "escape code lists", or the Appendix for ANSI.SYS in the back of the Microsoft manual for MS-DOS, and don't actually understand in the first place what it is to be standards conformant. For starters, it is ECMA-48 in both directions, and your code has to be standards conformant, which that absurdly fragile string matching is not. Would you have your own "it's not conformant, it can be disregarded" applied to you? Because it very much does. In truth, (partly as a result of people working from samizdat and old wives' tales) most terminals and terminal emulators are like your code, not conformant, and by your metric everything should be "casually disregarded", leaving nothing at all; which is hardly a sensible position.