3 ms·

This isn't meant to detract from the broader point about the limitations of terminals, but a simple array of pixels is among the least efficient ways to interac

This isn't meant to detract from the broader point about the limitations of terminals, but a simple array of pixels is among the least efficient ways to interact with modern GPUs, especially if it doesn't support rectangular copy operations. The best way to interact with a GPU today and for the foreseeable future is through command buffers, not direct pixel access per se.

akkartik 1y ago
The point is well taken. I don't know much about interacting with GPUs. I don't particularly care so far about getting more performance, given the wildly fast computers I have and my use cases (I don't make or play games). I _do_ care about power efficiency; do GPUs help there? Modern GPU-based terminal implementations aren't particularly power efficient in my experience..
- kbolino 1y ago
  There's so many factors affecting power efficiency that it's hard to give a categorical answer. A lot of it is dependent on factors that vary widely, from the hardware in use, to the display setup (resolution and refresh rate), to the quality of the drivers, to the window system (composited or not), to the size (cols x rows) of the terminal window, to the feature set involved, etc. The problem with a lot of GPU accelerated terminals, if I had to wager a guess, is that they draw as fast as possible. Turning off GPU acceleration likely forces things to happen much slower thanks to various bottlenecks like memory bandwidth and sharing CPU time with other processes. GPU acceleration of most GUI apps puts them in a similar position as video games. It doesn't have to happen as fast as possible, and can be throttled through e.g. v-sync or lower-than-max FPS targets or turning on and off specific display features that might tax the GPU more (e.g. if shaders get involved, alpha blending is used, etc.). The sibling comment makes a good point about compatibility and simplicity, though those don't always translate into lower power usage.
  - vidarh 1y ago
    > It doesn't have to happen as fast as possible, and can be throttled through e.g. v-sync or lower-than-max FPS targets or turning on and off specific display features that might tax the GPU more (e.g. if shaders get involved, alpha blending is used, etc.). Exactly. E.g if you want to render as fast as possible, the logical way of doing it is to keep track of how many lines have been output (the easiest, but not necessarily most efficient, way is to render to a scrollback buffer) and then separately, synced to v-sync if you prefer, start rendering from what is at the top of the virtual text-version of the screen when the rendering starts a new frame. Do this in two threads, and you can then render to the bitmap at whatever FPS you can handle, while you can let the app running in the terminal output text as fast as it can produce it: If the text-output thread manages to add more than one line to the end of the buffer per frame rendered to the bitmap, your output will just scroll more than one line per frame. You've then decoupled the decision of the FPS necessary from how fast the app you're running can output text, and frankly, your FPS needs to dip fairly low before that looks janky.
kragen 1y ago
There are multiple axes of "best". The simplest, most portable, and most reproducible way to interact with a GPU is direct pixel access. Sometimes that's not fast enough, of course, but that's mainly when you're suffering from uncontrollable fits of interactivity. Most of the time, the best solution to that problem is to redesign your user interface to require less interaction: https://worrydream.com/MagicInk/ https://worrydream.com/MagicInk/ > The ubiquity of frustrating, unhelpful software interfaces has motivated decades of research into “Human-Computer Interaction.” In this paper, I suggest that the long-standing focus on “interaction” may be misguided. For a majority subset of software, called “information software,” I argue that interactivity is actually a curse for users and a crutch for designers, and users’ goals can be better satisfied through other means. But yeah if you're playing an FPS you probably want to talk to your GPU through command buffers rather than pixel buffers.
- kbolino 1y ago
  There's going to be a compatibility-performance tradeoff here, to be sure, though the compatibility issue is going to be more with "very old platforms" and the performance issue is going to be more with "very high resolutions on very high refresh rates". So it's a question of whether you want to produce something that works well on current and past hardware vs. works well on current and future hardware, with some allowance for "can't please everybody". I don't consider scrolling a large page to be an "uncontrollable fit of interactivity" but it's going to struggle to stay smooth using a single, simple linear array of pixels that's manipulated solely by the CPU. If you can at least work with multiple pixel buffers and operate on them at least somewhat abstractly so that even basic operations can be pushed down to the GPU, even if you don't work directly with command buffers, that will go a long way to bridging the gap between past and future, at least for 2D interfaces.
  - kragen 1y ago
    The compatibility issue is mostly going to be with future platforms that subtly change the semantics of the interfaces you're using or whose drivers have different bugs than the drivers you tested on. To take a trivial example, most GPUs don't bother to implement IEEE 754 gradual underflow. I think you're wrong about struggling to stay smooth scrolling a large page. Maybe it was true on the original iPhone in 02007? Or it's true of complex multilayered translucent vector art with a fixed background? But it's not true of things like text with inline images. Let's suppose that scrolling a large page involves filling a 4K pixel buffer, 3840×2160, with 32-bit color. If you have an in-memory image of the page, this is just 2160 memcpys of the appropriate 15360-byte pixel line; you're going to be memcpy-bandwidth-limited, because figuring out where to copy the pixels from is a relatively trivial calculation by comparison. On the laptop I'm typing this on (which incidentally doesn't have a 4K screen) memcpy bandwidth to main memory (not cache) is 10.8 gigabytes per second, according to http://canonical.org/~kragen/sw/dev3/memcpycost.c http://canonical.org/~kragen/sw/dev3/memcpycost.c. The whole pixel buffer you're filling is only 33.2 megabytes, so this takes 3.1 milliseconds. (Of one CPU core.) Even at 120fps this is less than half the time required. (For a large page you might want to not keep all your JPEGs decompressed in RAM, re-decoding them as required, but this is basically never done on the GPU.) But what if the page is full of text and you have to rerender the visible part from a font atlas every frame? That's not quite as fast on the CPU, but it's still not slow enough to be a problem. If you have a tree of glyph-index strings with page positions in memory already, finding the glyph strings that are on the screen is computationally trivial; perhaps in an 16-pixel-tall font, 2160 scan lines is 135 lines of text, each of which might contain five or six strings, and so you just have to find the 600 strings in the tree that overlap your viewport. Maybe each line has 400 glyphs in it, though 60 would be more typical, for a total of 55000 glyphs to draw. We're going to want to render one texel per pixel to avoid fuzzing out the letters, and by the same token we can, I think, presuppose that the text is not rotated. So again in our inner loop we're memcpying, but this time from the font atlas into the pixel buffer. Maybe we're only memcpying a few pixels at a time, like an average of 8, so we end up calling memcpy 55000×16 ≈ 900k times per frame, requiring on the order of 10 million instructions, which is on the order of an extra millisecond. So maybe instead of 3 milliseconds your frame time is 4 milliseconds. (It might actually be faster instead of slower, because the relevant parts of the font atlas are probably going to have a high data cache hit rate, so memcpy can go faster than 10 gigs a second.) I did test something similar to this in http://canonical.org/~kragen/sw/dev3/propfont.c http://canonical.org/~kragen/sw/dev3/propfont.c, which runs on one core of this laptop at 84 million glyphs per second (thus about 0.7ms for our hypothetical 55000-glyph screenful) but it's doing a somewhat harder job because it's word-wrapping the text as it goes. (It's using a small font, so it takes less memcpy time per glyph.) So maybe scrolling a 4K page might take 4 milliseconds per screen update on the CPU. If you only use one core. I would say it was "struggling to stay smooth" if the frame rate fell below 30fps, which is 33 milliseconds per frame. So you have almost an order of magnitude of performance headroom. If your window is only 1920×1080, you have 1½ orders of magnitude of headroom, 2 orders of magnitude if you're willing to use four cores.

akkartik 1y ago

The point is well taken. I don't know much about interacting with GPUs. I don't particularly care so far about getting more performance, given the wildly fast computers I have and my use cases (I don't make or play games). I _do_ care about power efficiency; do GPUs help there? Modern GPU-based terminal implementations aren't particularly power efficient in my experience..

kbolino 1y ago
There's so many factors affecting power efficiency that it's hard to give a categorical answer. A lot of it is dependent on factors that vary widely, from the hardware in use, to the display setup (resolution and refresh rate), to the quality of the drivers, to the window system (composited or not), to the size (cols x rows) of the terminal window, to the feature set involved, etc. The problem with a lot of GPU accelerated terminals, if I had to wager a guess, is that they draw as fast as possible. Turning off GPU acceleration likely forces things to happen much slower thanks to various bottlenecks like memory bandwidth and sharing CPU time with other processes. GPU acceleration of most GUI apps puts them in a similar position as video games. It doesn't have to happen as fast as possible, and can be throttled through e.g. v-sync or lower-than-max FPS targets or turning on and off specific display features that might tax the GPU more (e.g. if shaders get involved, alpha blending is used, etc.). The sibling comment makes a good point about compatibility and simplicity, though those don't always translate into lower power usage.
- vidarh 1y ago
  > It doesn't have to happen as fast as possible, and can be throttled through e.g. v-sync or lower-than-max FPS targets or turning on and off specific display features that might tax the GPU more (e.g. if shaders get involved, alpha blending is used, etc.). Exactly. E.g if you want to render as fast as possible, the logical way of doing it is to keep track of how many lines have been output (the easiest, but not necessarily most efficient, way is to render to a scrollback buffer) and then separately, synced to v-sync if you prefer, start rendering from what is at the top of the virtual text-version of the screen when the rendering starts a new frame. Do this in two threads, and you can then render to the bitmap at whatever FPS you can handle, while you can let the app running in the terminal output text as fast as it can produce it: If the text-output thread manages to add more than one line to the end of the buffer per frame rendered to the bitmap, your output will just scroll more than one line per frame. You've then decoupled the decision of the FPS necessary from how fast the app you're running can output text, and frankly, your FPS needs to dip fairly low before that looks janky.

kragen 1y ago

There are multiple axes of "best". The simplest, most portable, and most reproducible way to interact with a GPU is direct pixel access. Sometimes that's not fast enough, of course, but that's mainly when you're suffering from uncontrollable fits of interactivity. Most of the time, the best solution to that problem is to redesign your user interface to require less interaction: https://worrydream.com/MagicInk/ https://worrydream.com/MagicInk/ > The ubiquity of frustrating, unhelpful software interfaces has motivated decades of research into “Human-Computer Interaction.” In this paper, I suggest that the long-standing focus on “interaction” may be misguided. For a majority subset of software, called “information software,” I argue that interactivity is actually a curse for users and a crutch for designers, and users’ goals can be better satisfied through other means. But yeah if you're playing an FPS you probably want to talk to your GPU through command buffers rather than pixel buffers.

kbolino 1y ago
There's going to be a compatibility-performance tradeoff here, to be sure, though the compatibility issue is going to be more with "very old platforms" and the performance issue is going to be more with "very high resolutions on very high refresh rates". So it's a question of whether you want to produce something that works well on current and past hardware vs. works well on current and future hardware, with some allowance for "can't please everybody". I don't consider scrolling a large page to be an "uncontrollable fit of interactivity" but it's going to struggle to stay smooth using a single, simple linear array of pixels that's manipulated solely by the CPU. If you can at least work with multiple pixel buffers and operate on them at least somewhat abstractly so that even basic operations can be pushed down to the GPU, even if you don't work directly with command buffers, that will go a long way to bridging the gap between past and future, at least for 2D interfaces.
- kragen 1y ago
  The compatibility issue is mostly going to be with future platforms that subtly change the semantics of the interfaces you're using or whose drivers have different bugs than the drivers you tested on. To take a trivial example, most GPUs don't bother to implement IEEE 754 gradual underflow. I think you're wrong about struggling to stay smooth scrolling a large page. Maybe it was true on the original iPhone in 02007? Or it's true of complex multilayered translucent vector art with a fixed background? But it's not true of things like text with inline images. Let's suppose that scrolling a large page involves filling a 4K pixel buffer, 3840×2160, with 32-bit color. If you have an in-memory image of the page, this is just 2160 memcpys of the appropriate 15360-byte pixel line; you're going to be memcpy-bandwidth-limited, because figuring out where to copy the pixels from is a relatively trivial calculation by comparison. On the laptop I'm typing this on (which incidentally doesn't have a 4K screen) memcpy bandwidth to main memory (not cache) is 10.8 gigabytes per second, according to http://canonical.org/~kragen/sw/dev3/memcpycost.c http://canonical.org/~kragen/sw/dev3/memcpycost.c. The whole pixel buffer you're filling is only 33.2 megabytes, so this takes 3.1 milliseconds. (Of one CPU core.) Even at 120fps this is less than half the time required. (For a large page you might want to not keep all your JPEGs decompressed in RAM, re-decoding them as required, but this is basically never done on the GPU.) But what if the page is full of text and you have to rerender the visible part from a font atlas every frame? That's not quite as fast on the CPU, but it's still not slow enough to be a problem. If you have a tree of glyph-index strings with page positions in memory already, finding the glyph strings that are on the screen is computationally trivial; perhaps in an 16-pixel-tall font, 2160 scan lines is 135 lines of text, each of which might contain five or six strings, and so you just have to find the 600 strings in the tree that overlap your viewport. Maybe each line has 400 glyphs in it, though 60 would be more typical, for a total of 55000 glyphs to draw. We're going to want to render one texel per pixel to avoid fuzzing out the letters, and by the same token we can, I think, presuppose that the text is not rotated. So again in our inner loop we're memcpying, but this time from the font atlas into the pixel buffer. Maybe we're only memcpying a few pixels at a time, like an average of 8, so we end up calling memcpy 55000×16 ≈ 900k times per frame, requiring on the order of 10 million instructions, which is on the order of an extra millisecond. So maybe instead of 3 milliseconds your frame time is 4 milliseconds. (It might actually be faster instead of slower, because the relevant parts of the font atlas are probably going to have a high data cache hit rate, so memcpy can go faster than 10 gigs a second.) I did test something similar to this in http://canonical.org/~kragen/sw/dev3/propfont.c http://canonical.org/~kragen/sw/dev3/propfont.c, which runs on one core of this laptop at 84 million glyphs per second (thus about 0.7ms for our hypothetical 55000-glyph screenful) but it's doing a somewhat harder job because it's word-wrapping the text as it goes. (It's using a small font, so it takes less memcpy time per glyph.) So maybe scrolling a 4K page might take 4 milliseconds per screen update on the CPU. If you only use one core. I would say it was "struggling to stay smooth" if the frame rate fell below 30fps, which is 33 milliseconds per frame. So you have almost an order of magnitude of performance headroom. If your window is only 1920×1080, you have 1½ orders of magnitude of headroom, 2 orders of magnitude if you're willing to use four cores.