5 ms·

I wanted to write a terminal emulator, the biggest hurdle is understanding the escape sequences. all documents seem to be unreadable, including those mentioned

I wanted to write a terminal emulator, the biggest hurdle is understanding the escape sequences. all documents seem to be unreadable, including those mentioned in the post, and those often referenced in projects, like https://vt100.net/emu/dec_ansi_parser https://vt100.net/emu/dec_ansi_parser the second difficulty is handling reflow. a real terminal can't resize its screen, but an emulator can. how to implement that correctly with cursor movements? the third difficulty is handling font fallback and rendering emojis and other combinatory glyphs correctly.

dilap 3y ago
haven't tried, but i'd guess chatgpt (especially 4) could help a ton w/ this.
SoftTalker 3y ago
Some real terminals had a few different row/column modes, e.g. 80x24 or 132x40 (IIRC). I don't recall if text was reflowed when switching.
- jws 3y ago
  The early DEC terminals did not reflow on switching.
- wyan 3y ago
  Usually the screen was cleared when switching modes, so no text reflow.
jws 3y ago
I've written ANSI terminal handling before. The original VT100 paper manual that came with the terminal is a nice start, because the complexity hadn't really happened yet and people used to write useful documentation. With that as a starting point for understanding it isn't hard to extend into handling the full spec. That diagram you linked is actually quite nice, but visually intimidating. If you think of it as a handful of regular expressions exploded into a state diagram that helps. For instance, the entire left half of the diagram is just the CSI code acceptor, see "CSI (Control Sequence Introducer) sequences" on the "ANSI escape code" wikipedia page. You can write a regular expression to match a CSI and carry on. This is 2023 and you aren't using an 8080 with 3k of RAM. (probably) The only tiny trick is that you have to handle the "incomplete trailing regex" and wait for more data to arrive and try again. As for handling reflow. I wouldn't call that an "implementation" problem. I'd call it a "specification" problem. I'd approach it by seeing what Apple's Terminal program does, write that down, and call that my specification.
alpaca128 3y ago
I always used Wikipedia's article on ANSI escape sequences. A few details could be explained a bit better but overall I found it useful. The diagram you linked is probably a more complete and compact overview of all possible combinations, but I don't find it very intuitive either.
- sureglymop 3y ago
  I think it lacks some important escape sequences. E.g. how do programs like vim and tmux switch to another buffer and then restore the buffer? I vaguely know about it but never saw an actually complete documentation.
  - vidarh 3y ago
    The diagram is complete. It shows the collection of the raw sequences, which includes a bunch of parameters that you then need to process separately to determine what to actually do. To switch to/from the alternate screen mode is \e[?47h and \e[?47l. "\e[?" is DEC private mode which are DEC private mode codes. The number specifies a range of settings to switch on or off. The "h" and "l" determines if you're setting or clearing the setting respectively. The parsing of those are handled by the escape, csi entry, and csi param boxes in the diagram.
  - deleted 3y ago
    [deleted]
  - ksherlock 3y ago
    They need to keep the screen and scrollback history in memory, then redraw it as needed. As mentioned next door, xterm has support for alternate screen mode which can save and restore the current contents. But tmux works fine with more than 2 buffers and tmux works fine if the terminal underneath it doesn't support alternate screen mode (like a physical vt100)
vidarh 3y ago
Ignore all of this, and start simple. You can get something going with just the most rudimentary escape handling just by spitting what programs write to your terminal to debug output in the terminal you run your new terminal from, and add a proper parser a bit later. You can totally ignore reflow. It'll look ugly. It doesn't matter. When running full screen applications you need to handle width/height reporting, that's all. Font fallback and nice font handling is a detail to worry about well down the line. There are libraries that can do a lot of the lifting for you depending on language/platform. Just pick a font with reasonable coverage and worry about the rest later.
blueflow 3y ago
Ignore all that ANSI stuff, implement escape sequences like you think its easy to implement and then write a terminfo file for that so applications know how to use it.
- yencabulator 3y ago
  And hope your users never SSH to other machines, because shuffling terminfo files around is simply painful.
  - blueflow 3y ago
    What else are you gonna do? Impersonate some other terminal type and then only support half of its features? Looking at the many xterm "clones".
duskwuff 3y ago
> the second difficulty is handling reflow. a real terminal can't resize its screen, but an emulator can. how to implement that correctly with cursor movements? The answer to this one, incidentally, is ¯\_(ツ)_/¯. Every terminal emulator engine I've examined handles reflow a little differently. There's no official spec, and everyone has their own idea about how to make it work "best". That being said, I've wrestled with these questions myself, and here's what I've come up with: * Each line should have a flag on it which tracks whether it should be reflowed with the next line. By default, this flag should not be set. * The reflow flag should only be set when the cursor leaves that line because a character was printed with the cursor in the last column ("wrapnext" state), and no special modes are set which change the wrapping columns. * Performing any other action which changes the contents of a line should clear the reflow flag on that line. The same logic can also be used to join wrapped lines for copy/paste. It won't work 100% of the time, but it will work for most situations.
wkat4242 3y ago
> the second difficulty is handling reflow. a real terminal can't resize its screen, but an emulator can. how to implement that correctly with cursor movements? That's not entirely true. Most serial terminals did have multiple text size modes eg 80x25 but also 128x50 or something. I still have a real VT520 here that can do that. Not a Dec labeled one sadly but one of the later boundless models.
deleted 3y ago
[deleted]

dilap 3y ago

haven't tried, but i'd guess chatgpt (especially 4) could help a ton w/ this.

SoftTalker 3y ago

Some real terminals had a few different row/column modes, e.g. 80x24 or 132x40 (IIRC). I don't recall if text was reflowed when switching.

jws 3y ago
The early DEC terminals did not reflow on switching.
wyan 3y ago
Usually the screen was cleared when switching modes, so no text reflow.

jws 3y ago

I've written ANSI terminal handling before. The original VT100 paper manual that came with the terminal is a nice start, because the complexity hadn't really happened yet and people used to write useful documentation. With that as a starting point for understanding it isn't hard to extend into handling the full spec. That diagram you linked is actually quite nice, but visually intimidating. If you think of it as a handful of regular expressions exploded into a state diagram that helps. For instance, the entire left half of the diagram is just the CSI code acceptor, see "CSI (Control Sequence Introducer) sequences" on the "ANSI escape code" wikipedia page. You can write a regular expression to match a CSI and carry on. This is 2023 and you aren't using an 8080 with 3k of RAM. (probably) The only tiny trick is that you have to handle the "incomplete trailing regex" and wait for more data to arrive and try again. As for handling reflow. I wouldn't call that an "implementation" problem. I'd call it a "specification" problem. I'd approach it by seeing what Apple's Terminal program does, write that down, and call that my specification.

alpaca128 3y ago

I always used Wikipedia's article on ANSI escape sequences. A few details could be explained a bit better but overall I found it useful. The diagram you linked is probably a more complete and compact overview of all possible combinations, but I don't find it very intuitive either.

sureglymop 3y ago
I think it lacks some important escape sequences. E.g. how do programs like vim and tmux switch to another buffer and then restore the buffer? I vaguely know about it but never saw an actually complete documentation.
- vidarh 3y ago
  The diagram is complete. It shows the collection of the raw sequences, which includes a bunch of parameters that you then need to process separately to determine what to actually do. To switch to/from the alternate screen mode is \e[?47h and \e[?47l. "\e[?" is DEC private mode which are DEC private mode codes. The number specifies a range of settings to switch on or off. The "h" and "l" determines if you're setting or clearing the setting respectively. The parsing of those are handled by the escape, csi entry, and csi param boxes in the diagram.
- deleted 3y ago
  [deleted]
- ksherlock 3y ago
  They need to keep the screen and scrollback history in memory, then redraw it as needed. As mentioned next door, xterm has support for alternate screen mode which can save and restore the current contents. But tmux works fine with more than 2 buffers and tmux works fine if the terminal underneath it doesn't support alternate screen mode (like a physical vt100)

vidarh 3y ago

Ignore all of this, and start simple. You can get something going with just the most rudimentary escape handling just by spitting what programs write to your terminal to debug output in the terminal you run your new terminal from, and add a proper parser a bit later. You can totally ignore reflow. It'll look ugly. It doesn't matter. When running full screen applications you need to handle width/height reporting, that's all. Font fallback and nice font handling is a detail to worry about well down the line. There are libraries that can do a lot of the lifting for you depending on language/platform. Just pick a font with reasonable coverage and worry about the rest later.

blueflow 3y ago

Ignore all that ANSI stuff, implement escape sequences like you think its easy to implement and then write a terminfo file for that so applications know how to use it.

yencabulator 3y ago
And hope your users never SSH to other machines, because shuffling terminfo files around is simply painful.
- blueflow 3y ago
  What else are you gonna do? Impersonate some other terminal type and then only support half of its features? Looking at the many xterm "clones".

duskwuff 3y ago

> the second difficulty is handling reflow. a real terminal can't resize its screen, but an emulator can. how to implement that correctly with cursor movements? The answer to this one, incidentally, is ¯\_(ツ)_/¯. Every terminal emulator engine I've examined handles reflow a little differently. There's no official spec, and everyone has their own idea about how to make it work "best". That being said, I've wrestled with these questions myself, and here's what I've come up with: * Each line should have a flag on it which tracks whether it should be reflowed with the next line. By default, this flag should not be set. * The reflow flag should only be set when the cursor leaves that line because a character was printed with the cursor in the last column ("wrapnext" state), and no special modes are set which change the wrapping columns. * Performing any other action which changes the contents of a line should clear the reflow flag on that line. The same logic can also be used to join wrapped lines for copy/paste. It won't work 100% of the time, but it will work for most situations.

wkat4242 3y ago

> the second difficulty is handling reflow. a real terminal can't resize its screen, but an emulator can. how to implement that correctly with cursor movements? That's not entirely true. Most serial terminals did have multiple text size modes eg 80x25 but also 128x50 or something. I still have a real VT520 here that can do that. Not a Dec labeled one sadly but one of the later boundless models.

deleted 3y ago

[deleted]