6 ms·

The TTY demystified (2008)

243 points by Ivoah 10y ago

CaliforniaKarl 10y ago
One of the most important commands I learned was "reset", especially when I was working with anything that would want to do something special. From the man page: "When invoked as reset, tset sets cooked and echo modes, turns off and raw modes, turns on newline translation and resets any unset characters to their default values before doing the terminal described above. This is useful after a program dies a terminal in an abnormal state." Except, it is also useful when you exit something like a Serial-Over-LAN session, after watching the BIOS do inappropriate things to your terminal window.
- jclulow 10y ago
  If the SOL client is leaving the terminal in that state, it's almost certainly broken. The only time it's acceptable not to clean up after yourself is if you literally can't, because you've been whacked with a signal you can't handle (SIGKILL); or because you've _tried_ to clean up and for some reason the correct sequence of restoration calls has failed for some unknown operational reason.
  - MaulingMonkey 10y ago
    Some edge cases: - cat /dev/urandom (cat can't reasonably tell if you've piped to another command that expects raw binary data or a terminal that expects to be left in a sane state) - ssh remotehost cat /dev/urandom (ditto) - Emergency exit to avoid data loss when detecting e.g. memory corruption (lesser evil) - Hardware bugs/failures erroneously emitting escape codes (memory bit flips, loose RS-232 connections) You're of course correct that the vastly preferred thing to do when practical is to clean up when reasonable to do so - and it's probably reasonable to do so.
    - sdkmvx 10y ago
      cat could tell but it probably wouldn't be a good design. Calling isatty is easy, and I think GNU cat already does so and buffers differently. But how can it tell what raw binary data did to your terminal? First you'd need to program all the different terminal types into cat (probably by using termcap) which isn't great simple "Unix" design to begin with. Then you can't just reset the terminal after running cat. You need to see if the binary data screwed up the terminal. That isn't in termcap and is probably intractable in the general case. While catting actual random data is contrived, using ssh is not. What if the connection is broken while you're in vi. Vi switches to the alternate screen and then scrolling stops working. If you don't know what happened it's not clear why. How can ssh fix this other than acting like tmux and parsing every terminal command sequence? I'm not convinced there is a simple fix. This doesn't even get into encoding issues. Some UTF-8 sequences are terminal commands. The UTF-8 encoding for U+00DF (the German eszett) contains a sequence which will cause some terminals to wait for a termination byte you wont ever send. Plan 9 had the right idea: do away with terminal command sequences altogether. But that only works there because you can take your "terminal" and turn it into a GUI window.
  - CaliforniaKarl 10y ago
    One reason might be because of an escape character conflict For ipmitool, one sequence for exiting a SOL session is <return><tilde><period>. Turns out, SSH uses the same sequence, so if you're using ipmitool over SSH (such as via a jump host), you'll disconnect SSH instead of ending the SOL session. (BTW, you can change the "EscapeChar" ssh config item to something else, or you can send <Return><tilde><n><o><n><e> to disable it for a session.)
    - asdfaoeu 10y ago
      You can also send \n~~. and SSH will send a single ~. Also useful for quoting nested SSH sessions.
- vram22 10y ago
  >One of the most important commands I learned was "reset" Used to use 'stty sane' to fix such issues, sometimes. Remember 'reset' too. Also set many of the settings manually by doing this (if lucky enough to get the chance in advance, not always the case at customer sites): When the terminal was in an okay state, do: stty -a where -a means show All settings. Then read up on them (in man pages and manuals), and then when the terminal got borked (by a crashed curses or other program, or manual human error), manually set some of those settings back to normal values again, often all in one stty command, like: stty ocrnl inlcr echo erase ^h intr ^c ... (writing from memory, some may be wrong). AFAIK many of the settings were poorly or not documented well - at least in the man pages. Solving many such issues for clients as a system engineer could be frustrating as well as exhilarating ...
tejasmanohar 10y ago
I stumbled across this a few months ago while implementing basic TTY resizing and ended up learning way more than I needed to. Super impressive, thorough article. :-)
ZenoArrow 10y ago
This one is a bit of a classic, for the last 8 years (apart from in 2015) it has been on HN at least once a year: https://hn.algolia.com/?query=TTY%20demystified&sort=byDate&dateRange=all&type=all&storyText=false&prefix=false&page=0 https://hn.algolia.com/?query=TTY%20demystified&sort=byDate&...
- mkesper 10y ago
  Maybe worth another api entry (maybe 'frequent' or 'classic')?
laumars 10y ago
Thanks for the submission, I've missed this previous times it has been posted. I'm currently in the process of writing my own shell so if anyone has any other good resources you'd recommend then I'd be hugely grateful :) (the simpler the better too hehehe)
- lutusp 10y ago
  When writing shells it's possible to invoke standard libraries on both ends of the circuit to handle these communications/protocol issues without having to rewrite them.
  - laumars 10y ago
    I've got file streams working, that's the easy part. However I'm having issues with console applications that detect and require a PTY. Also handling ncurses is causing me a bit of bother.
  - deleted 10y ago
    [deleted]
- vram22 10y ago
  Not sure from your question, whether you are only looking for resources on TTYs, or general resources on writing a Unix shell. If the latter, this book is quite good: Search for "advanced unix programming marc rochkind" and also see this specific page (it's one of the hits of that search): http://basepath.com/aup/ http://basepath.com/aup/ I had read it some years ago. (It used to come bundled with HP-UX servers and workstations, and I was working in an HP joint venture, so got to read it and try out some of the stuff.) Thought it was really good. Gave a lot of insights into appropriate use of Unix system calls. An example used in the book is creating a simple shell (covers right ways to fork, exec, dup, set up pipes with the pipe system call, etc.). (IIRC, there's even another one about creating a simple client-server DBMS!) And IIRC, he showed a good bit about writing code portable between different versions of Unix, in a sane way.
  - laumars 10y ago
skissane 10y ago
> Beware, though: What you are about to see is not particularly elegant. In fact, the TTY subsystem — while quite functional from a user's point of view — is a twisty little mess of special cases. I wonder if it is about time someone tried to build something more elegant. If someone was inventing something like the TTY subsystem from scratch today, with no requirements for backward compatibility, what would they do differently?
- rogual 10y ago
  I'd want to investigate not relying on in-band signalling for terminal control. So, perhaps have two input streams: stdin and stdctrl? Also, it'd be tempting to try and add more structure, but I honestly think bare streams of bytes are the right abstraction at this level.
  - skissane 10y ago
    > Also, it'd be tempting to try and add more structure, but I honestly think bare streams of bytes are the right abstraction at this level. Most contemporary terminal streams use escape sequences which are based on ECMA-48 with a bunch of de facto standard extensions, so that isn't really a bare stream of bytes. It is an overly complex encoding and ideally would be replaced with something more elegant. (Of course, strictly speaking kernel space largely doesn't deal with ECMA-48 and that is left to user-space libraries such as terminfo/readline/ncurses/etc, but it is part of the overall architecture.) As well as escape sequences, most contemporary terminal streams use UTF-8 as well, so that's another way in which they are no longer bare byte streams. Do we need line disciplines in the kernel? I think the answer is that historically line disciplines were implemented in Unix before shared libraries were. If shared libraries had been implemented earlier, line disciplines could have been implemented in user space as shared libraries instead of in the kernel.
  - pjc50 10y ago
    Careful: when you have OOB signalling, how do you keep the control codes in sync with the data, bearing in mind that both streams may be buffered differently?
ape4 10y ago
Should add "(2008)" to this.
- sonthonax 10y ago
  Why? The only reason to put a date on a title is when the content itself is of historical interest, or may no longer be accurate.
CJefferson 10y ago
This isn't entirely relevant to the article, but I hope some shell/TTY expert might read this, and know the answer :) Hypothetically, could a TTY, in combination with the help of a shell, be able to separate the stdin, stdout and stderr of every running program? By default the shell could do what we normally see, but then easily (with some kind of GUI control) let me pull a particular program's stdout/stderr out to a seperate window, including (if I stored it) all it's previous output, and redraw the current terminal removing that program's output. I've always been tempted to do something like this, and I'm curious if there is any massive reasons it's simply not possible.
- EdiX 10y ago
  Are you trying to do mathematica notebooks for the shell? There's no easy way to do this, you have to run each command into a separate pseudo-tty. You can only do this with a combination of a special terminal emulator and a special shell that only runs inside the special terminal emulator. You can't do this and then just use bash.
  - CJefferson 10y ago
    Oh, I know it would be a serious undertaking (I'd probably try hacking it into fish, because I've looked into it's code-base). I was just wondering if it's hypothetically possible, and why no-one has done it :)
    - EdiX 10y ago
      I don't know. The reason I haven't done it is because it's too much work, there is no pleasant way to do GUI programming and I don't have a clear design for how the UI.
  - bitwize 10y ago
    You can do it with bash and tmux.
- dsr_ 10y ago
  Yes. http://tldp.org/LDP/abs/html/io-redirection.html http://tldp.org/LDP/abs/html/io-redirection.html shows how to do it in bash for any given process.
birchb 10y ago
ASCII was invented in 1963, so the author's statement that Telex was ASCII-based is probably wrong. Is the rest of the article also as inaccurate?
- kpil 10y ago
  According to Wikipedia, it seems that ASCII was first used by AT&T's TWX teleprinter network, that previously used another encoding, ITA2. The keyword is "evolved into" as a long chain of changes, from Morse code, Murray code, a plethora of other variants of encodings and transmission methods, eventually standardised into ASCII on top of a modem in 1963.
deleted 10y ago
[deleted]
snhorne 10y ago
If you're interested in the history surrounding the TTY, I just published a newsletter about it on Friday which you can read (without signing up for anything) here: http://www.rubyletter.com/newsletter/2017/02/03/terminal.html http://www.rubyletter.com/newsletter/2017/02/03/terminal.htm...

243 points by Ivoah 10y ago

One of the most important commands I learned was "reset", especially when I was working with anything that would want to do something special. From the man page: "When invoked as reset, tset sets cooked and echo modes, turns off and raw modes, turns on newline translation and resets any unset characters to their default values before doing the terminal described above. This is useful after a program dies a terminal in an abnormal state." Except, it is also useful when you exit something like a Serial-Over-LAN session, after watching the BIOS do inappropriate things to your terminal window.

jclulow 10y ago
If the SOL client is leaving the terminal in that state, it's almost certainly broken. The only time it's acceptable not to clean up after yourself is if you literally can't, because you've been whacked with a signal you can't handle (SIGKILL); or because you've _tried_ to clean up and for some reason the correct sequence of restoration calls has failed for some unknown operational reason.
- MaulingMonkey 10y ago
  Some edge cases: - cat /dev/urandom (cat can't reasonably tell if you've piped to another command that expects raw binary data or a terminal that expects to be left in a sane state) - ssh remotehost cat /dev/urandom (ditto) - Emergency exit to avoid data loss when detecting e.g. memory corruption (lesser evil) - Hardware bugs/failures erroneously emitting escape codes (memory bit flips, loose RS-232 connections) You're of course correct that the vastly preferred thing to do when practical is to clean up when reasonable to do so - and it's probably reasonable to do so.
  - sdkmvx 10y ago
    cat could tell but it probably wouldn't be a good design. Calling isatty is easy, and I think GNU cat already does so and buffers differently. But how can it tell what raw binary data did to your terminal? First you'd need to program all the different terminal types into cat (probably by using termcap) which isn't great simple "Unix" design to begin with. Then you can't just reset the terminal after running cat. You need to see if the binary data screwed up the terminal. That isn't in termcap and is probably intractable in the general case. While catting actual random data is contrived, using ssh is not. What if the connection is broken while you're in vi. Vi switches to the alternate screen and then scrolling stops working. If you don't know what happened it's not clear why. How can ssh fix this other than acting like tmux and parsing every terminal command sequence? I'm not convinced there is a simple fix. This doesn't even get into encoding issues. Some UTF-8 sequences are terminal commands. The UTF-8 encoding for U+00DF (the German eszett) contains a sequence which will cause some terminals to wait for a termination byte you wont ever send. Plan 9 had the right idea: do away with terminal command sequences altogether. But that only works there because you can take your "terminal" and turn it into a GUI window.
- CaliforniaKarl 10y ago
  One reason might be because of an escape character conflict For ipmitool, one sequence for exiting a SOL session is <return><tilde><period>. Turns out, SSH uses the same sequence, so if you're using ipmitool over SSH (such as via a jump host), you'll disconnect SSH instead of ending the SOL session. (BTW, you can change the "EscapeChar" ssh config item to something else, or you can send <Return><tilde><n><o><n><e> to disable it for a session.)
  - asdfaoeu 10y ago
    You can also send \n~~. and SSH will send a single ~. Also useful for quoting nested SSH sessions.
vram22 10y ago
>One of the most important commands I learned was "reset" Used to use 'stty sane' to fix such issues, sometimes. Remember 'reset' too. Also set many of the settings manually by doing this (if lucky enough to get the chance in advance, not always the case at customer sites): When the terminal was in an okay state, do: stty -a where -a means show All settings. Then read up on them (in man pages and manuals), and then when the terminal got borked (by a crashed curses or other program, or manual human error), manually set some of those settings back to normal values again, often all in one stty command, like: stty ocrnl inlcr echo erase ^h intr ^c ... (writing from memory, some may be wrong). AFAIK many of the settings were poorly or not documented well - at least in the man pages. Solving many such issues for clients as a system engineer could be frustrating as well as exhilarating ...

tejasmanohar 10y ago

I stumbled across this a few months ago while implementing basic TTY resizing and ended up learning way more than I needed to. Super impressive, thorough article. :-)

ZenoArrow 10y ago

This one is a bit of a classic, for the last 8 years (apart from in 2015) it has been on HN at least once a year: https://hn.algolia.com/?query=TTY%20demystified&sort=byDate&dateRange=all&type=all&storyText=false&prefix=false&page=0 https://hn.algolia.com/?query=TTY%20demystified&sort=byDate&...

mkesper 10y ago
Maybe worth another api entry (maybe 'frequent' or 'classic')?

laumars 10y ago

Thanks for the submission, I've missed this previous times it has been posted. I'm currently in the process of writing my own shell so if anyone has any other good resources you'd recommend then I'd be hugely grateful :) (the simpler the better too hehehe)

lutusp 10y ago
When writing shells it's possible to invoke standard libraries on both ends of the circuit to handle these communications/protocol issues without having to rewrite them.
- laumars 10y ago
  I've got file streams working, that's the easy part. However I'm having issues with console applications that detect and require a PTY. Also handling ncurses is causing me a bit of bother.
- deleted 10y ago
  [deleted]
vram22 10y ago
Not sure from your question, whether you are only looking for resources on TTYs, or general resources on writing a Unix shell. If the latter, this book is quite good: Search for "advanced unix programming marc rochkind" and also see this specific page (it's one of the hits of that search): http://basepath.com/aup/ http://basepath.com/aup/ I had read it some years ago. (It used to come bundled with HP-UX servers and workstations, and I was working in an HP joint venture, so got to read it and try out some of the stuff.) Thought it was really good. Gave a lot of insights into appropriate use of Unix system calls. An example used in the book is creating a simple shell (covers right ways to fork, exec, dup, set up pipes with the pipe system call, etc.). (IIRC, there's even another one about creating a simple client-server DBMS!) And IIRC, he showed a good bit about writing code portable between different versions of Unix, in a sane way.
- laumars 10y ago

skissane 10y ago

> Beware, though: What you are about to see is not particularly elegant. In fact, the TTY subsystem — while quite functional from a user's point of view — is a twisty little mess of special cases. I wonder if it is about time someone tried to build something more elegant. If someone was inventing something like the TTY subsystem from scratch today, with no requirements for backward compatibility, what would they do differently?

rogual 10y ago
I'd want to investigate not relying on in-band signalling for terminal control. So, perhaps have two input streams: stdin and stdctrl? Also, it'd be tempting to try and add more structure, but I honestly think bare streams of bytes are the right abstraction at this level.
- skissane 10y ago
  > Also, it'd be tempting to try and add more structure, but I honestly think bare streams of bytes are the right abstraction at this level. Most contemporary terminal streams use escape sequences which are based on ECMA-48 with a bunch of de facto standard extensions, so that isn't really a bare stream of bytes. It is an overly complex encoding and ideally would be replaced with something more elegant. (Of course, strictly speaking kernel space largely doesn't deal with ECMA-48 and that is left to user-space libraries such as terminfo/readline/ncurses/etc, but it is part of the overall architecture.) As well as escape sequences, most contemporary terminal streams use UTF-8 as well, so that's another way in which they are no longer bare byte streams. Do we need line disciplines in the kernel? I think the answer is that historically line disciplines were implemented in Unix before shared libraries were. If shared libraries had been implemented earlier, line disciplines could have been implemented in user space as shared libraries instead of in the kernel.
- pjc50 10y ago
  Careful: when you have OOB signalling, how do you keep the control codes in sync with the data, bearing in mind that both streams may be buffered differently?

ape4 10y ago

Should add "(2008)" to this.

sonthonax 10y ago
Why? The only reason to put a date on a title is when the content itself is of historical interest, or may no longer be accurate.

CJefferson 10y ago

This isn't entirely relevant to the article, but I hope some shell/TTY expert might read this, and know the answer :) Hypothetically, could a TTY, in combination with the help of a shell, be able to separate the stdin, stdout and stderr of every running program? By default the shell could do what we normally see, but then easily (with some kind of GUI control) let me pull a particular program's stdout/stderr out to a seperate window, including (if I stored it) all it's previous output, and redraw the current terminal removing that program's output. I've always been tempted to do something like this, and I'm curious if there is any massive reasons it's simply not possible.

EdiX 10y ago
Are you trying to do mathematica notebooks for the shell? There's no easy way to do this, you have to run each command into a separate pseudo-tty. You can only do this with a combination of a special terminal emulator and a special shell that only runs inside the special terminal emulator. You can't do this and then just use bash.
- CJefferson 10y ago
  Oh, I know it would be a serious undertaking (I'd probably try hacking it into fish, because I've looked into it's code-base). I was just wondering if it's hypothetically possible, and why no-one has done it :)
  - EdiX 10y ago
    I don't know. The reason I haven't done it is because it's too much work, there is no pleasant way to do GUI programming and I don't have a clear design for how the UI.
- bitwize 10y ago
  You can do it with bash and tmux.
dsr_ 10y ago
Yes. http://tldp.org/LDP/abs/html/io-redirection.html http://tldp.org/LDP/abs/html/io-redirection.html shows how to do it in bash for any given process.

birchb 10y ago

ASCII was invented in 1963, so the author's statement that Telex was ASCII-based is probably wrong. Is the rest of the article also as inaccurate?

kpil 10y ago
According to Wikipedia, it seems that ASCII was first used by AT&T's TWX teleprinter network, that previously used another encoding, ITA2. The keyword is "evolved into" as a long chain of changes, from Morse code, Murray code, a plethora of other variants of encodings and transmission methods, eventually standardised into ASCII on top of a modem in 1963.

deleted 10y ago

[deleted]

snhorne 10y ago

If you're interested in the history surrounding the TTY, I just published a newsletter about it on Friday which you can read (without signing up for anything) here: http://www.rubyletter.com/newsletter/2017/02/03/terminal.html http://www.rubyletter.com/newsletter/2017/02/03/terminal.htm...