12 ms·

The TTY Demystified (2008)

187 points by csomar 6y ago

mesofile 6y ago
A veritable Internet classic, well worth a read by anyone who uses a text terminal -- real or emulated. Previous discussions: https://hn.algolia.com/?q=TTY+Demystified https://hn.algolia.com/?q=TTY+Demystified
- dang 6y ago
  The TTY demystified - https://news.ycombinator.com/item?id=10631513 https://news.ycombinator.com/item?id=10631513 - Nov 2015 (40 comments) The TTY demystified - https://news.ycombinator.com/item?id=8094186 https://news.ycombinator.com/item?id=8094186 - July 2014 (21 comments) The TTY demystified - https://news.ycombinator.com/item?id=4062981 https://news.ycombinator.com/item?id=4062981 - June 2012 (17 comments) The TTY demystified - https://news.ycombinator.com/item?id=658155 https://news.ycombinator.com/item?id=658155 - June 2009 (13 comments)
galacticaactual 6y ago
I really, really enjoy coming to HN for articles like this.
AceJohnny2 6y ago
What would a clean-slate implementation of a serial terminal look like today? I work in embedded, and a serial terminal is practically a must in any system, if only for debug. A way to provide input and receive output. You'll definitely want a FIFOs for RX/TX. You also likely want control chars and a way to shortcut the FIFO, if only to interrupt output. The FIFO means you can abstract away the exact physical communication medium. TUI interfaces mean you also want a way to tell the program what the dimensions of your screen are, through the control chars. What else?
- R0b0t1 6y ago
  I've given this some thought. The way to improve the TTY is to give it channels, like SSH or RDP. This way you can transit unrelated data over the same connection. And, if you think about it, most of the complexity of a serial interface is handling mode settings and state; streams abstract that away for you. Streams get rid of the need for control characters and in band signalling. Streams could be given priority and interrupt levels.
  - ksherlock 6y ago
    If you add a header with the channel number and data length and a CRC, it starts looking like UDP.
  - tyingq 6y ago
    RS-485 is essentially RS-232 with channels. Never used it, but Linux supports it at some level.
    - nrclark 6y ago
      This might not be accurate. RS-485 is an electrical signaling standard for carrying arbitrary binary signals. It's usually used for carrying UART data, as a longer-range / higher-speed replacement for RS-232. It has no opinion about what is transmitted, and has no concept of channels or even any particular encoding format.
      - tyingq 6y ago
        It could be the basis for a tty with multiple channels, fairly easily.
        catblast01 6y ago
        How does that make it an RS-232 with channels? You’re confused. RS-485 just like RS-232 is an electrical specification for serial signaling (with multiple access), nothing more nor less. There are USB to 485 converters that basically operate exactly the same as their serial port counterparts and will show up as a tty device in Linux/Unix flavor or a COM port in Windows.
- kragen 6y ago
  Are you talking about a physical machine or a protocol design? I don't think you can abstract away the exact physical communication medium from a physical machine (or your cable won't fit in the hole, or it'll burn out your circuitry), and the protocol design probably doesn't need a FIFO. Protocol design: maybe typewriter text and shitty VT100 escape sequences were a bad idea? Let's use HTML-style tags, but in an endless tag soup similar to VT100 semantics, not a tree.
  - AceJohnny2 6y ago
    I was thinking mostly protocol design, I suppose, with the implicit requirement that it'd have to work over various physical lines.
    - kragen 6y ago
      I have lots of ideas. But we have 55 years of experience with the advantages and disadvantages of the ADM3A, VT100, and similar terminals, and we've gradually improved it over that time. Now it's ten thousand times faster than a physical VT100, supports 24-bit color, auto-highlights URLs, has enormous scrollback, supports copy and paste, allows applications to handle mouse events, can be recorded with script or ttyrec or asciinema, can be detached and reattached later with screen or tmux or mosh, can get local echo with mosh, can share a screen and keyboard between people in different locations with screen or tmux, and has hundreds of thousands of programs already written for it — aside from backward-compatibility with teletype software like ex, ls, sh, grep, ps, ping, vmstat, and so on, you have user interfaces designed for character-cell terminals like Emacs, Vim, links, mtr, dstat, top, htop, dialog/whiptail, less, mutt, irssi, ircII, weechat, nethack, sl, robotfindskitten, command-line editing in things like tcsh and bash, progress bars in things like apt, nano, and of course the aa demo bb (if run with env -u DISPLAY). There are a lot of things that can't really be done without breaking compatibility with that ecosystem. Proportional fonts of continuously varying sizes, or padding? Now screen positions are no longer integer multiples of a character size, and you can't get things to vertically align by inserting space characters. (This is already a problem with Unicode, where different monospace fonts are monospace over different amounts of the character set; see http://canonical.org/~kragen/sw/dev3/randomlayout.py http://canonical.org/~kragen/sw/dev3/randomlayout.py for some attempts to test the limits.) Pixel graphics? Are those going to pass through tmux unharmed? (Some recent attempts to jimmy pixel graphics into terminals won't even work over ssh.) How will copy and paste work with them? Will they send another megabyte over my cellphone connection every time I press ^L to redraw the screen? We can definitely do better. But doing better is going to require a lot of experimentation to be a Pareto improvement over 55 years of tweaks to the VT100. Most of the ideas we come up with are going to turn out not to work as well in practice as we hoped they would in theory. But if you're interested in reading them anyway, some of my notes are in https://dercuano.github.io/topics/terminals.html https://dercuano.github.io/topics/terminals.html.
- zamadatix 6y ago
  Clean slate definitely object/packet based - just because we need to find the dimensions of the screen doesn't mean it needs to be done via control characters.
- Koshkin 6y ago
  And not just the terminal. Why on Earth people thought that the regular PC keyboard and the mouse required anything more complex than the good old RS-232.
- wanderingjew 6y ago
  Fairly easy, a few thousand lines of C. Bit of work, though. Parser is the hardest, a lot of details to get right. You don't really need a buffer for the output, because you can't type that fast. Of course, driving the display is a bit harder. I'm using a parallel tft, wrote the driver, too. Keyboard code is kludgy because I'm not using diodes for cost reduction on the bom. You can see it here: violence.works yes that's a url.
  - girvo 6y ago
    I'm both amazed at what you've accomplished and a little miffed that someone beat me to it: I've been working on-and-off on a similar project for a couple of years now haha. Awesome work What display did you end up settling on, where'd you source it, and how many columns do you end up fitting? That was the biggest sticking point for me.
tbrock 6y ago
This gets posted here every so often but to be honest it still isn’t demystified for me even on the third read.
sn41 6y ago
Oh yeah, I remember that this is where I really understood line discipline. For example, why backspace/Ctrl-H worked even if you are typing in things directly to a terminal (e.g. cat >> output file) and it did not need an editor to "interpret" the key, whereas things like Ctrl+A would need an editor. I know it's pretty basic, but this is the article that really helped me understand it.
timonoko 6y ago
Sorry, I am still mystified. Especially with software flow control. Sometimes using emacs "search-forward" everything just stops and it takes time to figure out what just happened. But this phenomenon has becomen increasingly rare in recent years. It is truly mystifying.
u678u 6y ago
Its amazing to me that despite our enormously powerful PCs and wonderful modern software tools we still rely on foundations built in a different time tty, awk, even bash scripts. Half the time its great to work with such finely tuned Darwinian survivors, the other half I want to start again. Doing helm k8s yaml templates makes me wonder if its possible for us to write something simple and clean that could replace those old building blocks.
- gpanders 6y ago
  I won't defend TTY or bash, but I will say that awk has stood the test of time because it's an amazingly potent tool for doing ad-hoc processing of textual information, which is still a frequent task even in the era of Kubernetes and yaml templates. Why does it need replacing? When I learned awk I felt like I unlocked a superpower.
  - saurik 6y ago
    Yeah: it's like, the problem we are trying to solve largely hasn't changed, so the solutions are maybe a bit lacking in elegance sometimes, but they are generally fine... and the concern would be that someone trying to do it today would be like "the correct way to do this is to have some ridiculously complex stack built on top of 70 pseudo-standards from Google with a single bespoke implementation in JavaScript that no one could ever reimplement".
- anthk 6y ago
  Windows 10 is still using the CP/M unit labeling scheme.
- kjeetgill 6y ago
  I definitely found jq to be my modern addition to the pantheon alongside awk, sed, and grep.
gumby 6y ago
This is a pretty good article. A few comments: > Teletypes were once connected across the world in a large network, called Telex, which was used for transferring commercial telegrams, but the teletypes weren't connected to any computers yet. Telex was a private system disjoint from the telegram system which was, in most countries, run by the PTT (government agency called Post, Telegraph and Telephone). In the US and some other countries it was private, of course, where Western Union was the dominant player. I remember staying in a hotel in India the early 80s that had four different systems (with its own model of printing terminals, each one a clunky electromechanical beast). One was telex and one some iron curtain alternative, I remember, but I don’t remember the other two. > ... a backspace key is often useful. This could of course be implemented by the applications themselves, but in accordance with the UNIX design philosophy, applications should be kept as simple as possible. So as a convenience, the operating system provides an editing buffer and some rudimentary editing commands This approach far predates Unix, and Unix got it from Multics. Mainframes implemented it in their channel controllers to save time on the CPU (it was central, after all), while minicomputers typically did it the way Unix did. Non-“glass tty” video terminals didn’t have frame buffers in the sense today, but just character storage. Characters were rendered by looking up pixel data in a ROM (before that, hardwared!) which might only have been a rectangle of 32 pixels or less. > In a similar way, it is possible to configure the TTY to react on a stop flow byte and a start flow byte. These are typically ^S (ASCII code 19) and ^Q (ASCII code 17) respectively. Old hardware terminals transmit these bytes automatically, and expect the operating system to regulate its flow of data accordingly. This is called flow control, and it's the reason why your xterm sometimes appears to lock up when you accidentally press ^S. I don’t remember any terminals that recognised soft (inline) flow control though they may have existed. Instead, they signaled it through the transmission cable itself. If you only used three wires in your serial cable this was impossible, but check out the full pin out of a DB-25 or DE-9 (sometimes ignorantly referred to as “DB9”) connector.
- fuckf4ce 6y ago
  > I don’t remember any terminals that recognised soft (inline) flow control XON/XOFF. The VT100, as well as many other contemporary terminals of the time did. Not uncommon at all. http://bitsavers.trailing-edge.com/pdf/dec/terminal/vt100/EK-VT100-TM-003_VT100_Technical_Manual_Jul82.pdf http://bitsavers.trailing-edge.com/pdf/dec/terminal/vt100/EK...

187 points by csomar 6y ago

A veritable Internet classic, well worth a read by anyone who uses a text terminal -- real or emulated. Previous discussions: https://hn.algolia.com/?q=TTY+Demystified https://hn.algolia.com/?q=TTY+Demystified

dang 6y ago
The TTY demystified - https://news.ycombinator.com/item?id=10631513 https://news.ycombinator.com/item?id=10631513 - Nov 2015 (40 comments) The TTY demystified - https://news.ycombinator.com/item?id=8094186 https://news.ycombinator.com/item?id=8094186 - July 2014 (21 comments) The TTY demystified - https://news.ycombinator.com/item?id=4062981 https://news.ycombinator.com/item?id=4062981 - June 2012 (17 comments) The TTY demystified - https://news.ycombinator.com/item?id=658155 https://news.ycombinator.com/item?id=658155 - June 2009 (13 comments)

galacticaactual 6y ago

I really, really enjoy coming to HN for articles like this.

AceJohnny2 6y ago

What would a clean-slate implementation of a serial terminal look like today? I work in embedded, and a serial terminal is practically a must in any system, if only for debug. A way to provide input and receive output. You'll definitely want a FIFOs for RX/TX. You also likely want control chars and a way to shortcut the FIFO, if only to interrupt output. The FIFO means you can abstract away the exact physical communication medium. TUI interfaces mean you also want a way to tell the program what the dimensions of your screen are, through the control chars. What else?

R0b0t1 6y ago
I've given this some thought. The way to improve the TTY is to give it channels, like SSH or RDP. This way you can transit unrelated data over the same connection. And, if you think about it, most of the complexity of a serial interface is handling mode settings and state; streams abstract that away for you. Streams get rid of the need for control characters and in band signalling. Streams could be given priority and interrupt levels.
- ksherlock 6y ago
  If you add a header with the channel number and data length and a CRC, it starts looking like UDP.
- tyingq 6y ago
  RS-485 is essentially RS-232 with channels. Never used it, but Linux supports it at some level.
  - nrclark 6y ago
    This might not be accurate. RS-485 is an electrical signaling standard for carrying arbitrary binary signals. It's usually used for carrying UART data, as a longer-range / higher-speed replacement for RS-232. It has no opinion about what is transmitted, and has no concept of channels or even any particular encoding format.
    - tyingq 6y ago
      It could be the basis for a tty with multiple channels, fairly easily.
      - catblast01 6y ago
        How does that make it an RS-232 with channels? You’re confused. RS-485 just like RS-232 is an electrical specification for serial signaling (with multiple access), nothing more nor less. There are USB to 485 converters that basically operate exactly the same as their serial port counterparts and will show up as a tty device in Linux/Unix flavor or a COM port in Windows.
kragen 6y ago
Are you talking about a physical machine or a protocol design? I don't think you can abstract away the exact physical communication medium from a physical machine (or your cable won't fit in the hole, or it'll burn out your circuitry), and the protocol design probably doesn't need a FIFO. Protocol design: maybe typewriter text and shitty VT100 escape sequences were a bad idea? Let's use HTML-style tags, but in an endless tag soup similar to VT100 semantics, not a tree.
- AceJohnny2 6y ago
  I was thinking mostly protocol design, I suppose, with the implicit requirement that it'd have to work over various physical lines.
  - kragen 6y ago
    I have lots of ideas. But we have 55 years of experience with the advantages and disadvantages of the ADM3A, VT100, and similar terminals, and we've gradually improved it over that time. Now it's ten thousand times faster than a physical VT100, supports 24-bit color, auto-highlights URLs, has enormous scrollback, supports copy and paste, allows applications to handle mouse events, can be recorded with script or ttyrec or asciinema, can be detached and reattached later with screen or tmux or mosh, can get local echo with mosh, can share a screen and keyboard between people in different locations with screen or tmux, and has hundreds of thousands of programs already written for it — aside from backward-compatibility with teletype software like ex, ls, sh, grep, ps, ping, vmstat, and so on, you have user interfaces designed for character-cell terminals like Emacs, Vim, links, mtr, dstat, top, htop, dialog/whiptail, less, mutt, irssi, ircII, weechat, nethack, sl, robotfindskitten, command-line editing in things like tcsh and bash, progress bars in things like apt, nano, and of course the aa demo bb (if run with env -u DISPLAY). There are a lot of things that can't really be done without breaking compatibility with that ecosystem. Proportional fonts of continuously varying sizes, or padding? Now screen positions are no longer integer multiples of a character size, and you can't get things to vertically align by inserting space characters. (This is already a problem with Unicode, where different monospace fonts are monospace over different amounts of the character set; see http://canonical.org/~kragen/sw/dev3/randomlayout.py http://canonical.org/~kragen/sw/dev3/randomlayout.py for some attempts to test the limits.) Pixel graphics? Are those going to pass through tmux unharmed? (Some recent attempts to jimmy pixel graphics into terminals won't even work over ssh.) How will copy and paste work with them? Will they send another megabyte over my cellphone connection every time I press ^L to redraw the screen? We can definitely do better. But doing better is going to require a lot of experimentation to be a Pareto improvement over 55 years of tweaks to the VT100. Most of the ideas we come up with are going to turn out not to work as well in practice as we hoped they would in theory. But if you're interested in reading them anyway, some of my notes are in https://dercuano.github.io/topics/terminals.html https://dercuano.github.io/topics/terminals.html.
zamadatix 6y ago
Clean slate definitely object/packet based - just because we need to find the dimensions of the screen doesn't mean it needs to be done via control characters.
Koshkin 6y ago
And not just the terminal. Why on Earth people thought that the regular PC keyboard and the mouse required anything more complex than the good old RS-232.
wanderingjew 6y ago
Fairly easy, a few thousand lines of C. Bit of work, though. Parser is the hardest, a lot of details to get right. You don't really need a buffer for the output, because you can't type that fast. Of course, driving the display is a bit harder. I'm using a parallel tft, wrote the driver, too. Keyboard code is kludgy because I'm not using diodes for cost reduction on the bom. You can see it here: violence.works yes that's a url.
- girvo 6y ago
  I'm both amazed at what you've accomplished and a little miffed that someone beat me to it: I've been working on-and-off on a similar project for a couple of years now haha. Awesome work What display did you end up settling on, where'd you source it, and how many columns do you end up fitting? That was the biggest sticking point for me.

tbrock 6y ago

This gets posted here every so often but to be honest it still isn’t demystified for me even on the third read.

sn41 6y ago

Oh yeah, I remember that this is where I really understood line discipline. For example, why backspace/Ctrl-H worked even if you are typing in things directly to a terminal (e.g. cat >> output file) and it did not need an editor to "interpret" the key, whereas things like Ctrl+A would need an editor. I know it's pretty basic, but this is the article that really helped me understand it.

timonoko 6y ago

Sorry, I am still mystified. Especially with software flow control. Sometimes using emacs "search-forward" everything just stops and it takes time to figure out what just happened. But this phenomenon has becomen increasingly rare in recent years. It is truly mystifying.

u678u 6y ago

Its amazing to me that despite our enormously powerful PCs and wonderful modern software tools we still rely on foundations built in a different time tty, awk, even bash scripts. Half the time its great to work with such finely tuned Darwinian survivors, the other half I want to start again. Doing helm k8s yaml templates makes me wonder if its possible for us to write something simple and clean that could replace those old building blocks.

gpanders 6y ago
I won't defend TTY or bash, but I will say that awk has stood the test of time because it's an amazingly potent tool for doing ad-hoc processing of textual information, which is still a frequent task even in the era of Kubernetes and yaml templates. Why does it need replacing? When I learned awk I felt like I unlocked a superpower.
- saurik 6y ago
  Yeah: it's like, the problem we are trying to solve largely hasn't changed, so the solutions are maybe a bit lacking in elegance sometimes, but they are generally fine... and the concern would be that someone trying to do it today would be like "the correct way to do this is to have some ridiculously complex stack built on top of 70 pseudo-standards from Google with a single bespoke implementation in JavaScript that no one could ever reimplement".
anthk 6y ago
Windows 10 is still using the CP/M unit labeling scheme.
kjeetgill 6y ago
I definitely found jq to be my modern addition to the pantheon alongside awk, sed, and grep.

gumby 6y ago

This is a pretty good article. A few comments: > Teletypes were once connected across the world in a large network, called Telex, which was used for transferring commercial telegrams, but the teletypes weren't connected to any computers yet. Telex was a private system disjoint from the telegram system which was, in most countries, run by the PTT (government agency called Post, Telegraph and Telephone). In the US and some other countries it was private, of course, where Western Union was the dominant player. I remember staying in a hotel in India the early 80s that had four different systems (with its own model of printing terminals, each one a clunky electromechanical beast). One was telex and one some iron curtain alternative, I remember, but I don’t remember the other two. > ... a backspace key is often useful. This could of course be implemented by the applications themselves, but in accordance with the UNIX design philosophy, applications should be kept as simple as possible. So as a convenience, the operating system provides an editing buffer and some rudimentary editing commands This approach far predates Unix, and Unix got it from Multics. Mainframes implemented it in their channel controllers to save time on the CPU (it was central, after all), while minicomputers typically did it the way Unix did. Non-“glass tty” video terminals didn’t have frame buffers in the sense today, but just character storage. Characters were rendered by looking up pixel data in a ROM (before that, hardwared!) which might only have been a rectangle of 32 pixels or less. > In a similar way, it is possible to configure the TTY to react on a stop flow byte and a start flow byte. These are typically ^S (ASCII code 19) and ^Q (ASCII code 17) respectively. Old hardware terminals transmit these bytes automatically, and expect the operating system to regulate its flow of data accordingly. This is called flow control, and it's the reason why your xterm sometimes appears to lock up when you accidentally press ^S. I don’t remember any terminals that recognised soft (inline) flow control though they may have existed. Instead, they signaled it through the transmission cable itself. If you only used three wires in your serial cable this was impossible, but check out the full pin out of a DB-25 or DE-9 (sometimes ignorantly referred to as “DB9”) connector.

fuckf4ce 6y ago
> I don’t remember any terminals that recognised soft (inline) flow control XON/XOFF. The VT100, as well as many other contemporary terminals of the time did. Not uncommon at all. http://bitsavers.trailing-edge.com/pdf/dec/terminal/vt100/EK-VT100-TM-003_VT100_Technical_Manual_Jul82.pdf http://bitsavers.trailing-edge.com/pdf/dec/terminal/vt100/EK...