13 ms·

There seems to be desperate demand for graphics output in terminals. Sixels are a rather crude method dating from a time where terminals were optimized for low

by sprash 8y ago

There seems to be desperate demand for graphics output in terminals. Sixels are a rather crude method dating from a time where terminals were optimized for low bandwidth serial connections and very constrained resources in the device that actually renders the content. We have 3D graphic cards now! This is why right now I'm working on a terminal emulator that implements escape codes that allows a subset of "byte compiled" OpenGL commands to run directly in the terminal window. You now can render a complex 3D scene just by cat'ing a file without any boiler plate code. I think there are many potential use cases for this. It allows for example remote graphics accelerated applications running on another machine via ssh. Once all the textures are uploaded to the GPU RAM the bandwidth required for the display commands is rather low even for interactive games.

fxfan 8y ago
Can you, github.com/jwilm, github.com/kovidgoyal et al get together and instead give us a better standard that all 3 of you implement? I am sorry but I won't use your terminal just for one feature, I can use a dedicated image viewer.
- sprash 8y ago
  So far my version is just a crudely hacked together proof of concept on top of st I did as hobby. It just renders textured polygons without any shaders or more advanced stuff. Since this is my first OpenGL application ever it probably won't ever be released to the public. Somebody else with more experience should rather do it. A More advanced protocol like Valves Fossilize (https://github.com/ValveSoftware/Fossilize https://github.com/ValveSoftware/Fossilize) should be used for that which means I can throw away my code anyways at this point. However serialized graphics does not seem a project goal of either kitty or alacritty.
  - ubercow13 8y ago
    kitty has image support
    - sprash 8y ago
      It implements its own protocol (https://sw.kovidgoyal.net/kitty/graphics-protocol.html https://sw.kovidgoyal.net/kitty/graphics-protocol.html) to display images. This is something different from serialized OpenGL.
  - watersb 8y ago
    Nice.
csdvrx 8y ago
You should not dismiss "crude" methods so easily: the same output can be rendered inside GNU screen or tmux, which would be complicated otherwise. And low bandwidth is useful when doing SSH: I posted this quick example of what I use everyday as a reply "ROSshow: ASCII art visualizations for robot sensor data" (https://news.ycombinator.com/item?id=19519165 https://news.ycombinator.com/item?id=19519165) by ssh'ing to my remote host (which does not have a GPU) from my laptop (which doesn't have a GPU either). I wanted to share something that you can use right now, and that works reliably even in non optimal conditions. Notice how my loadavg is above 10, and how I'm using a mlterm binary downloaded straight from mlterm.sf.net/bin.html The same will work with a vanilla msys2 installation, or with the in-kernel console of "old powerless machines" as noted on https://saitoha.github.io/libsixel/#sayaka-chan https://saitoha.github.io/libsixel/#sayaka-chan
- sprash 8y ago
  With a serialized GPU commands the remote host does not need a GPU since everything is rendered locally. Only the machine running the terminal emulator needs one. And this would also work within screen or tmux if they don't meddle with unknown escape codes.
  - csdvrx 8y ago
    With the history of ^O bracketed paste handling exploits and the more recent spectre/meltdown exploits, I'm not sure serialized GPU rendering is such a bright idea. If the concern is the speed of the scrollback on device rendering the content, wouldn't adding GPU support to say mlterm be enough? Anyway, if someone has a desperate need of graphical output inside a terminal, this is an actual solution working with little risks and default tools: I only recompiled gnuplot with the proper compile-time flag, exported the GNUTERM variable, and voila!!
  - chubot 8y ago
    Even if they don't mess with unknown escape codes, wouldn't they have to know how wide what is being drawn is? I ran into this issue while writing a shell. All shells use wcwidth() to determine the width of a unicode character. I think all terminals need that information too, and they would also need the analogous information for unknown escapes.
jhallenworld 8y ago
Please do not. The best thing you can do it get sixel working on more terminal emulators, particularly Windows ones like PuTTY and TeraTerm. Do any free native windows terminal emulators support Sixel? I'm working on embedded software for some industrial equipment the runs over a serial port, and I'm running into all kinds of annoying problems. I use Linux, customer uses Windows. In Linux (Ubuntu 16), I found that large paste to Xterm does not work (I know for sure that this used to work in the past). It seems to discard data when Xon/Xoff flow control kicks in, or when a buffer is too large. I've tried minicom, PL2303, FTDI, picocom, my own serial program, XTerm and Gnome terminal. I'm pretty sure the problem is X or Unity (a C program that transfers data out the serial port works just fine). On the other hand, Xterm supports Sixel, so the embedded software can display graphs. [Followup: I just traced this problem to "picocom", but it has been fixed in a newer version]. The idea is that I should be able to select calibration data from a spreadsheet and paste it into the terminal emulator connected to my device. In Windows, embarrassingly, large paste works just fine, for example from TeraTerm. But Putty and TeraTerm do not support Sixel. I suppose XTerm in Cygwin might work, but it would better to have a native Windows program. I've not tried Mac, but nobody in the industrial world uses them.
- csdvrx 8y ago
  100% right. Adding sixel support to more terminals would do more to broaden graphical support. On linux, just adding sixel to gnome-terminal and libvte would go a long way. FYI, I picked up sixel because it's the closest thing to a standard, and because technically I saw it as good enough for most cases. > I use Linux I use Windows :-) > Do any free native windows terminal emulators support Sixel? mintty on msys2 and mlterm do. I prefer mintty, it feels faster. I'm eagerly waiting for conpty and hopefully even better consoles in Windows!
  - jhallenworld 8y ago
    Ah, on Windows, I would prefer a terminal emulator that can connect directly to a serial port. I don't think mintty and mlterm can do this.
- Eduard 8y ago
swiley 8y ago
ugh. Most people are using tty apps because they don't want more crap that breaks when opengl (read, drivers) misbehaves.

by sprash 8y ago

fxfan 8y ago

Can you, github.com/jwilm, github.com/kovidgoyal et al get together and instead give us a better standard that all 3 of you implement? I am sorry but I won't use your terminal just for one feature, I can use a dedicated image viewer.

sprash 8y ago
So far my version is just a crudely hacked together proof of concept on top of st I did as hobby. It just renders textured polygons without any shaders or more advanced stuff. Since this is my first OpenGL application ever it probably won't ever be released to the public. Somebody else with more experience should rather do it. A More advanced protocol like Valves Fossilize (https://github.com/ValveSoftware/Fossilize https://github.com/ValveSoftware/Fossilize) should be used for that which means I can throw away my code anyways at this point. However serialized graphics does not seem a project goal of either kitty or alacritty.
- ubercow13 8y ago
  kitty has image support
  - sprash 8y ago
    It implements its own protocol (https://sw.kovidgoyal.net/kitty/graphics-protocol.html https://sw.kovidgoyal.net/kitty/graphics-protocol.html) to display images. This is something different from serialized OpenGL.
- watersb 8y ago
  Nice.

csdvrx 8y ago

You should not dismiss "crude" methods so easily: the same output can be rendered inside GNU screen or tmux, which would be complicated otherwise. And low bandwidth is useful when doing SSH: I posted this quick example of what I use everyday as a reply "ROSshow: ASCII art visualizations for robot sensor data" (https://news.ycombinator.com/item?id=19519165 https://news.ycombinator.com/item?id=19519165) by ssh'ing to my remote host (which does not have a GPU) from my laptop (which doesn't have a GPU either). I wanted to share something that you can use right now, and that works reliably even in non optimal conditions. Notice how my loadavg is above 10, and how I'm using a mlterm binary downloaded straight from mlterm.sf.net/bin.html The same will work with a vanilla msys2 installation, or with the in-kernel console of "old powerless machines" as noted on https://saitoha.github.io/libsixel/#sayaka-chan https://saitoha.github.io/libsixel/#sayaka-chan

sprash 8y ago
With a serialized GPU commands the remote host does not need a GPU since everything is rendered locally. Only the machine running the terminal emulator needs one. And this would also work within screen or tmux if they don't meddle with unknown escape codes.
- csdvrx 8y ago
  With the history of ^O bracketed paste handling exploits and the more recent spectre/meltdown exploits, I'm not sure serialized GPU rendering is such a bright idea. If the concern is the speed of the scrollback on device rendering the content, wouldn't adding GPU support to say mlterm be enough? Anyway, if someone has a desperate need of graphical output inside a terminal, this is an actual solution working with little risks and default tools: I only recompiled gnuplot with the proper compile-time flag, exported the GNUTERM variable, and voila!!
- chubot 8y ago
  Even if they don't mess with unknown escape codes, wouldn't they have to know how wide what is being drawn is? I ran into this issue while writing a shell. All shells use wcwidth() to determine the width of a unicode character. I think all terminals need that information too, and they would also need the analogous information for unknown escapes.

jhallenworld 8y ago

Please do not. The best thing you can do it get sixel working on more terminal emulators, particularly Windows ones like PuTTY and TeraTerm. Do any free native windows terminal emulators support Sixel? I'm working on embedded software for some industrial equipment the runs over a serial port, and I'm running into all kinds of annoying problems. I use Linux, customer uses Windows. In Linux (Ubuntu 16), I found that large paste to Xterm does not work (I know for sure that this used to work in the past). It seems to discard data when Xon/Xoff flow control kicks in, or when a buffer is too large. I've tried minicom, PL2303, FTDI, picocom, my own serial program, XTerm and Gnome terminal. I'm pretty sure the problem is X or Unity (a C program that transfers data out the serial port works just fine). On the other hand, Xterm supports Sixel, so the embedded software can display graphs. [Followup: I just traced this problem to "picocom", but it has been fixed in a newer version]. The idea is that I should be able to select calibration data from a spreadsheet and paste it into the terminal emulator connected to my device. In Windows, embarrassingly, large paste works just fine, for example from TeraTerm. But Putty and TeraTerm do not support Sixel. I suppose XTerm in Cygwin might work, but it would better to have a native Windows program. I've not tried Mac, but nobody in the industrial world uses them.

csdvrx 8y ago
100% right. Adding sixel support to more terminals would do more to broaden graphical support. On linux, just adding sixel to gnome-terminal and libvte would go a long way. FYI, I picked up sixel because it's the closest thing to a standard, and because technically I saw it as good enough for most cases. > I use Linux I use Windows :-) > Do any free native windows terminal emulators support Sixel? mintty on msys2 and mlterm do. I prefer mintty, it feels faster. I'm eagerly waiting for conpty and hopefully even better consoles in Windows!
- jhallenworld 8y ago
  Ah, on Windows, I would prefer a terminal emulator that can connect directly to a serial port. I don't think mintty and mlterm can do this.
Eduard 8y ago

swiley 8y ago

ugh. Most people are using tty apps because they don't want more crap that breaks when opengl (read, drivers) misbehaves.