6 ms·

Character by character TTY input in Unix, then and now

72 points by gbrown_ 8y ago

willvk 8y ago
Correct me if I'm wrong, but I don't think that protected mode existed in old Unix systems and so was not delineated between user space and kernel space as described. Protected mode and the separation of kernel and user space was only available since 80286 processors starting in 1982. Refs: https://en.wikipedia.org/wiki/User_space https://en.wikipedia.org/wiki/User_space https://en.wikipedia.org/wiki/Protected_mode https://en.wikipedia.org/wiki/Protected_mode
- bitwize 8y ago
  Virtual memory systems and protected memory certainly did exist before the 286, and Unix supported them well before 1982. Hell, the most popular CPU architecture for production Unix systems -- VAX -- was 32-bit and dates from the mid-1970s.
- rjsw 8y ago
  Sorry, but you are wrong. Have a look at the architectures of the PDP-11, VAX, 68000 and many other CPUs.
  - tralarpa 8y ago
    By the way, an interesting story: When Sun (?) wanted to implement their Unix with paging on the 68000 they had a problem: If you access a memory page that has been swapped out to disk, the OS has to (1) load the page from disk to RAM and (2) let the CPU repeat the instruction that caused the memory access. But the last step could not be properly done on the 68000 because it did not store its internal state completely when an (address) error happened (this was fixed in the 68010). Their solution: Run two 68000 in parallel on the same code, one of them delayed by one instruction. When the first CPU triggers the page fault, the system can stop the second CPU before it reaches the instruction that caused the fault.
    - jandrese 8y ago
      That's astounding. 68ks were not cheap. I assume the system then stalls for a couple of instructions to switch back to the first chip after the page fault completes? I have to assume they reached that solution after exhausting every other option. On the other hand, this could also be used as a form of error checking. If CPU2 ever returns something different you know there was an error somewhere in the system and you can hard stop to prevent further data corruption.
      - tralarpa 8y ago
        > I have to assume they reached that solution after exhausting every other option I assume that the 68000 was so powerful for its price (probably much cheaper than the existing mainframe CPUs) that it made this a viable solution. Or maybe the company had promised their customers a 68k-based Unix system (with the 68451 MMU) and the 68010 was delayed or too expensive and they had to find a quick solution? (I have no idea) Btw, I was probably wrong about Sun being the company behind this. Apollo and MassComp have been mentioned in mailing lists.
        jandrese 8y ago
        IIRC they cost around $350 each back in the 80s. That's roughly $1000 in today's dollars. A lot of companies wanted to use them instead of the Mickey Mouse Intel chips of the day, but the price point was too severe.
      - anyfoo 8y ago
        Speaking of which: http://www.os2museum.com/wp/the-nearly-ultimate-386-board/ http://www.os2museum.com/wp/the-nearly-ultimate-386-board/ The article describes a motherboard with both a soldered on 386 CPU, and a separate socket for another one. This was not a multiprocessor board, to make use of the socket you were supposed to disable the onboard CPU using a jumper. But if you plugged in a similar enough CPU without setting the jumper, it appears that both CPUs ran at the same time. And since they had the same clock and they should have no difference in behavior, the system, at least as tested, ran fine.
    - 8y ago
- icedchai 8y ago
  There is a world outside of Intel, especially back in the 80's and 90's...
  - mwfunk 8y ago
    And especially back in the '60s and '70s, when Unix and its predecessors were originally developed.
- cat199 8y ago
  wrong chip. https://en.wikipedia.org/wiki/VAX#Virtual_memory_map https://en.wikipedia.org/wiki/VAX#Virtual_memory_map (although paging was in pdp-11 unix too, and neither was the 1st to have memory levels) see also: https://svnweb.freebsd.org/base/head/share/misc/bsd-family-tree?revision=339484&view=markup https://svnweb.freebsd.org/base/head/share/misc/bsd-family-t... https://www.levenez.com/unix/ https://www.levenez.com/unix/ https://www.tuhs.org/ https://www.tuhs.org/
- willvk 8y ago
  Happy to be wrong on this and good to know it did exist before x86. Thanks for the replies.
  - apostacy 8y ago
    This is a very PC-centric way of looking at things. The 68K processors did not have a concept of protected mode. It just had user mode and supervisor mode[1] [1]: https://retrocomputing.stackexchange.com/questions/2784/how-did-the-original-apple-macintosh-and-atari-st-use-protected-mode/2786 https://retrocomputing.stackexchange.com/questions/2784/how-...
bluetomcat 8y ago
Would you agree that the whole TTY subsystem makes very little sense in this day and age? Why not let programs operate in raw mode by default and provide user-level library calls for input line buffering?
- jstimpfle 8y ago
  It's important for job control. Like Ctrl+c or Ctrl+z.
  - JoshTriplett 8y ago
    Arguably the shell or terminal could handle those.
    - jstimpfle 8y ago
      I don't know. It is one of the recurring questions in software development (or politics...) what should be centralised and what should be distributed. Job control might be one thing that we need a good default for anyway, so why not implement it in a central place?
  - jml7c5 8y ago
    Why not do that screen/tmux style, where shell takes input and passes it to foreground program, unless it's one of those special key sequences?
    - jstimpfle 8y ago
      That's not what happens with screen or tmux (or xterm, or sshd on the remote side of an ssh connection...). These allocate a pseudo terminal (PTY; which is a structure in the kernel) and connect to the master end. Using that they emulate a terminal, which has a virtual keyboard and a virtual screen. A shell (or another program) is started, with its input and output(s) connected to the slave end. The virtual keyboard might be X11 key events in the case of xterm, or a network stream in the case of sshd, or simply the stdin downstream from another terminal in the case of tmux. That keyboard character stream is fed into the write fd of the master end of the PTY. The virtual display is implemented by processing data read from the read fd of the master end of the PTY. That might involve drawing to an X11 window in the case of xterm, or sending back data over the network in the case of sshd, or writing to stdout connected to another terminal in the case of tmux. The important part is this: The job control is still done in the kernel as part of the PTY implementation. That implementation is where for example a Ctrl+c (0x03 byte) is converted to a SIGINT which is then sent to the slave program.
rubatuga 8y ago
This really helps me understand how read() works as I am currently learning C programming. Didn't know that ctrl+D still sent everything before it in the buffer to the read call, and that a read of length zero and EOF was the same thing.
- majewsky 8y ago
  Ctrl-D is maybe the hardest part of TTYs to emulate if you want to bypass the accumulated legacy of the kernel-level TTY device API. It's very easy to make read() return 0 forever, but it's next-to-impossible to make read() return 0 exactly once, except for TTY slaves.
3pt14159 8y ago
This is incredibly timely. I've been trying to figure out if it is possible to create a universal shim that would always give me readline-like behaviour while supplying input to an ongoing program. Something like this: ./shim.sh | ./some_horrible_cli.rb That way I could always rely on my linux shortcuts. Ie, no "^A^K" when I try to kill a line; it would just kill the line. Is this possible?
- dTal 8y ago
  Good news! This already exists. It's called 'rlwrap' and it's awesome.
  - 3pt14159 8y ago
    Thank you! :)
gumby 8y ago
Th discussion of ssh is interesting and calls to mind the SUPDUP protocol, which moves all the character echoing, cooked mode, and even some local editing — including screen movement for Emacs!) to the local (user end) of the network connection. Might be worth reviving.
caf 8y ago
The fact that the kernel TTY driver is responsible for handling erase in cooked mode leads to a wrinkle: in order to do that correctly it needs to know if you're using a multibyte character set, and if so, which one (because a single erase must erase an entire multibyte codepoint). Linux uses the (non-POSIX) IUTF8 termios(2) flag to indicate a terminal is using UTF8. If this is set on a terminal which isn't UTF-8, you'll see some odd results - for example, in a terminal using the single byte character set ISO-8859-1 but with IUTF8 set and in cooked mode, if you type: Ã¡Ã¡ and then backspace once and hit enter, the process reading cooked mode will recieve "Ã¡\n" - the erase has eaten two ISO-8859-1 codepoints because it thought that they were a single UTF-8 codepoint. I've never used a non-UTF-8 multibyte terminal (like BIG5 or SHIFT-JIS) but I don't think erase would work correctly in cooked mode on such terminals.

72 points by gbrown_ 8y ago

Correct me if I'm wrong, but I don't think that protected mode existed in old Unix systems and so was not delineated between user space and kernel space as described. Protected mode and the separation of kernel and user space was only available since 80286 processors starting in 1982. Refs: https://en.wikipedia.org/wiki/User_space https://en.wikipedia.org/wiki/User_space https://en.wikipedia.org/wiki/Protected_mode https://en.wikipedia.org/wiki/Protected_mode

bitwize 8y ago
Virtual memory systems and protected memory certainly did exist before the 286, and Unix supported them well before 1982. Hell, the most popular CPU architecture for production Unix systems -- VAX -- was 32-bit and dates from the mid-1970s.
rjsw 8y ago
Sorry, but you are wrong. Have a look at the architectures of the PDP-11, VAX, 68000 and many other CPUs.
- tralarpa 8y ago
  By the way, an interesting story: When Sun (?) wanted to implement their Unix with paging on the 68000 they had a problem: If you access a memory page that has been swapped out to disk, the OS has to (1) load the page from disk to RAM and (2) let the CPU repeat the instruction that caused the memory access. But the last step could not be properly done on the 68000 because it did not store its internal state completely when an (address) error happened (this was fixed in the 68010). Their solution: Run two 68000 in parallel on the same code, one of them delayed by one instruction. When the first CPU triggers the page fault, the system can stop the second CPU before it reaches the instruction that caused the fault.
  - jandrese 8y ago
    That's astounding. 68ks were not cheap. I assume the system then stalls for a couple of instructions to switch back to the first chip after the page fault completes? I have to assume they reached that solution after exhausting every other option. On the other hand, this could also be used as a form of error checking. If CPU2 ever returns something different you know there was an error somewhere in the system and you can hard stop to prevent further data corruption.
    - tralarpa 8y ago
      > I have to assume they reached that solution after exhausting every other option I assume that the 68000 was so powerful for its price (probably much cheaper than the existing mainframe CPUs) that it made this a viable solution. Or maybe the company had promised their customers a 68k-based Unix system (with the 68451 MMU) and the 68010 was delayed or too expensive and they had to find a quick solution? (I have no idea) Btw, I was probably wrong about Sun being the company behind this. Apollo and MassComp have been mentioned in mailing lists.
      - jandrese 8y ago
        IIRC they cost around $350 each back in the 80s. That's roughly $1000 in today's dollars. A lot of companies wanted to use them instead of the Mickey Mouse Intel chips of the day, but the price point was too severe.
    - anyfoo 8y ago
      Speaking of which: http://www.os2museum.com/wp/the-nearly-ultimate-386-board/ http://www.os2museum.com/wp/the-nearly-ultimate-386-board/ The article describes a motherboard with both a soldered on 386 CPU, and a separate socket for another one. This was not a multiprocessor board, to make use of the socket you were supposed to disable the onboard CPU using a jumper. But if you plugged in a similar enough CPU without setting the jumper, it appears that both CPUs ran at the same time. And since they had the same clock and they should have no difference in behavior, the system, at least as tested, ran fine.
  - 8y ago
icedchai 8y ago
There is a world outside of Intel, especially back in the 80's and 90's...
- mwfunk 8y ago
  And especially back in the '60s and '70s, when Unix and its predecessors were originally developed.
cat199 8y ago
wrong chip. https://en.wikipedia.org/wiki/VAX#Virtual_memory_map https://en.wikipedia.org/wiki/VAX#Virtual_memory_map (although paging was in pdp-11 unix too, and neither was the 1st to have memory levels) see also: https://svnweb.freebsd.org/base/head/share/misc/bsd-family-tree?revision=339484&view=markup https://svnweb.freebsd.org/base/head/share/misc/bsd-family-t... https://www.levenez.com/unix/ https://www.levenez.com/unix/ https://www.tuhs.org/ https://www.tuhs.org/
willvk 8y ago
Happy to be wrong on this and good to know it did exist before x86. Thanks for the replies.
- apostacy 8y ago
  This is a very PC-centric way of looking at things. The 68K processors did not have a concept of protected mode. It just had user mode and supervisor mode[1] [1]: https://retrocomputing.stackexchange.com/questions/2784/how-did-the-original-apple-macintosh-and-atari-st-use-protected-mode/2786 https://retrocomputing.stackexchange.com/questions/2784/how-...

bluetomcat 8y ago

Would you agree that the whole TTY subsystem makes very little sense in this day and age? Why not let programs operate in raw mode by default and provide user-level library calls for input line buffering?

jstimpfle 8y ago
It's important for job control. Like Ctrl+c or Ctrl+z.
- JoshTriplett 8y ago
  Arguably the shell or terminal could handle those.
  - jstimpfle 8y ago
    I don't know. It is one of the recurring questions in software development (or politics...) what should be centralised and what should be distributed. Job control might be one thing that we need a good default for anyway, so why not implement it in a central place?
- jml7c5 8y ago
  Why not do that screen/tmux style, where shell takes input and passes it to foreground program, unless it's one of those special key sequences?
  - jstimpfle 8y ago
    That's not what happens with screen or tmux (or xterm, or sshd on the remote side of an ssh connection...). These allocate a pseudo terminal (PTY; which is a structure in the kernel) and connect to the master end. Using that they emulate a terminal, which has a virtual keyboard and a virtual screen. A shell (or another program) is started, with its input and output(s) connected to the slave end. The virtual keyboard might be X11 key events in the case of xterm, or a network stream in the case of sshd, or simply the stdin downstream from another terminal in the case of tmux. That keyboard character stream is fed into the write fd of the master end of the PTY. The virtual display is implemented by processing data read from the read fd of the master end of the PTY. That might involve drawing to an X11 window in the case of xterm, or sending back data over the network in the case of sshd, or writing to stdout connected to another terminal in the case of tmux. The important part is this: The job control is still done in the kernel as part of the PTY implementation. That implementation is where for example a Ctrl+c (0x03 byte) is converted to a SIGINT which is then sent to the slave program.

rubatuga 8y ago

This really helps me understand how read() works as I am currently learning C programming. Didn't know that ctrl+D still sent everything before it in the buffer to the read call, and that a read of length zero and EOF was the same thing.

majewsky 8y ago
Ctrl-D is maybe the hardest part of TTYs to emulate if you want to bypass the accumulated legacy of the kernel-level TTY device API. It's very easy to make read() return 0 forever, but it's next-to-impossible to make read() return 0 exactly once, except for TTY slaves.

3pt14159 8y ago

This is incredibly timely. I've been trying to figure out if it is possible to create a universal shim that would always give me readline-like behaviour while supplying input to an ongoing program. Something like this: ./shim.sh | ./some_horrible_cli.rb That way I could always rely on my linux shortcuts. Ie, no "^A^K" when I try to kill a line; it would just kill the line. Is this possible?

dTal 8y ago
Good news! This already exists. It's called 'rlwrap' and it's awesome.
- 3pt14159 8y ago
  Thank you! :)

gumby 8y ago

Th discussion of ssh is interesting and calls to mind the SUPDUP protocol, which moves all the character echoing, cooked mode, and even some local editing — including screen movement for Emacs!) to the local (user end) of the network connection. Might be worth reviving.

caf 8y ago

The fact that the kernel TTY driver is responsible for handling erase in cooked mode leads to a wrinkle: in order to do that correctly it needs to know if you're using a multibyte character set, and if so, which one (because a single erase must erase an entire multibyte codepoint). Linux uses the (non-POSIX) IUTF8 termios(2) flag to indicate a terminal is using UTF8. If this is set on a terminal which isn't UTF-8, you'll see some odd results - for example, in a terminal using the single byte character set ISO-8859-1 but with IUTF8 set and in cooked mode, if you type: Ã¡Ã¡ and then backspace once and hit enter, the process reading cooked mode will recieve "Ã¡\n" - the erase has eaten two ISO-8859-1 codepoints because it thought that they were a single UTF-8 codepoint. I've never used a non-UTF-8 multibyte terminal (like BIG5 or SHIFT-JIS) but I don't think erase would work correctly in cooked mode on such terminals.