14 ms·

GPU-Backed User Interfaces

146 points by paulgb 4y ago

Kukumber 4y ago
That's weird that the author completely ignores popular projects like ImGui [1] or lvgl, maybe the author wanted to talk about W.I.P. GPU based UIs written in Rust only? In that case, there doesn't seems to be a lot of choice [1] - https://github.com/ocornut/imgui/issues/5886 https://github.com/ocornut/imgui/issues/5886
- paulgb 4y ago
  I had some notes on Dear ImGui (and the prevalence of immediate-mode across other frameworks that it inspired), but ultimately cut it for length because I wanted to focus on applications rather than frameworks/GUI toolkits.
  - Kukumber 4y ago
    https://github.com/ocornut/imgui/issues/3075#issuecomment-604711590 https://github.com/ocornut/imgui/issues/3075#issuecomment-60...
- lionkor 4y ago
  ImGui, by default, is run at like 60+fps. No GUI app that doesnt display real time changes needs to render at 60+ fps
  - seanw444 4y ago
    Maybe not a constant 60+ FPS, but if I scroll, and I can tell it's rendering at 20-30 FPS max on my 144 Hz monitor, I'm going to be annoyed. I want smoothness.
  - jakelazaroff 4y ago
    Dear ImGui doesn't actually include a draw loop, does it? AFAIK you're in control of how often that happens.
    - lionkor 4y ago
      Yes, and the example and sample code does it with max framerate.
      - jakelazaroff 4y ago
        Okay, but that’s not the same thing as “Dear ImGui runs at 60fps by default”.
danielvaughn 4y ago
Considering how GPU-backed UI is viewed as a performance win, I wonder why Flutter web performs so poorly.
- dahart 4y ago
  Using GPUs only enables high performance. It definitely doesn’t guarantee it. ;)
- mhoad 4y ago
  Flutter web is in the process of moving to a new rendering pipeline that is C++ compiled to WASM and backed by WebGPU. I don’t think they are going to have troubles performance wise for long. This is actually an interesting read I came across by someone from their team recently which poses some hypothetical directions they could take things in the future depending on how the web platform evolves over time. https://docs.google.com/document/d/1peUSMsvFGvqD5yKh3GprskLC3KVdAlLGOsK6gFoEOD0/edit?resourcekey=0-bPajpoo9IBZpG__-uCBE6w https://docs.google.com/document/d/1peUSMsvFGvqD5yKh3GprskLC...
AshleysBrain 4y ago
AFAIK modern browsers all use GPU accelerated rendering, so using HTML/CSS does give you a GPU-backed user interface. Also desktop PWAs are IMO underrated and a really good way to distribute a desktop app without having to bundle a whole browser engine with your app. We do this for Construct (www.construct.net) - a complete game development IDE - and I think it's worked out great.
- pjmlp 4y ago
  Not really it is a bit more subtle than that, unless you do some magic incantations with CSS, like z-ordering or transforms, it may bet that software rendering is used instead. There are a couple of talks regarding CSS tricks to force GPU redrawing.
  - AshleysBrain 4y ago
    I think that was the case several years ago but I thought these days much more of the browser rendering pipeline was automatically GPU accelerated. It's hard to find up-to-date references (would appreciate any links) but for example going to chrome://gpu shows it using GPU acceleration for both compositing and rasterization (including "multiple raster threads"), as well as the usual suspects like video decoding, canvas and WebGL.
    - pjmlp 4y ago
      Don't be fooled by canvas/WebGL support. I remember seeing this kind of content on Google IO and WebDev talks. Would need to also go treasure hunting to find them.
      - AshleysBrain 4y ago
        Treasure hunting to find an API widely supported on 95%+ of devices?
        pjmlp 4y ago
matlin 4y ago
This is pretty much what Flutter is. Except Flutter has system-specific UI's created by Google so each app can have that native look depending on where it's running.
- ArtWomb 4y ago
  Impeller. New shader compilation backend. Heroic coding. Metal/Vulkan/WebGPU translators. The bar is pixel perfect across gpus which makes it so challanging. My current favorite open source project: Android & Win ARM ports ;) https://github.com/flutter/flutter/wiki/Impeller https://github.com/flutter/flutter/wiki/Impeller
- actionfromafar 4y ago
  Almost that native look. Diluting what a native app feels like, but that price is something Google is willing for you to pay.
c-smile 4y ago
> Unlike the UIs provided by the operating system, where you can say "draw a button with the label 'delete'", you have to tell OpenGL how to make a button out of triangles. Just use Sciter, by default for rendering it uses Direct2d/DirectX (+ Vulkan||OpenGL) on Windows, Metal||OpenGL on Mac and Vulkan||OpenGL on Linux. With support of a11y by the way.
- fassssst 4y ago
  But the UIs provided by the operating system are making DirectX/Metal calls for you... they are GPU accelerated. And they have been for like 16 years now.
  - c-smile 4y ago
    > UIs provided by the operating system ... GPU accelerated That depends on what you mean by "system UI". Most system utilities on Windows are still using GDI. GDI is not GPU accelerated (other than BitBlt). Same apply to MacOS - CoreGraphics there is pure CPU rasterization library. And the same situation on Linux/GTK, system uses Cairo - pure CPU rasterization library. Usually, on modern OSes, window/desktop manager is the only GPU accelerated thing - it composes window bitmaps on screen.
    - fassssst 4y ago
      WPF and WinUI on Windows are both GPU accelerated. They render with Direct3D calls under the hood.
      - c-smile 4y ago
        Sciter greatly overlaps with those. It is just that instead of XAML it uses HTML/CSS, JS/C/C++ instead of C# and works on all desktop platforms. For the note: historically Sciter appeared before WPF and WinUI/UWP : https://sciter.com/sciter/sciter-vs-wpf/ https://sciter.com/sciter/sciter-vs-wpf/ By any means WPF and WinUI are not UIs provided by the OS. These are userland UI libraries. Same thing as Sciter.
xlazom00 4y ago
On windows 8 and newer, microsoft use new framework to create UI for apps. They are all GPU accelerated. Calculator, Settings,.. And it is up to the framework if it support all feature like accesiblity or system spellcheck or whatever. And on Mac OS Cocoa use GPU acceleration as well. As if you want to show something nice and fency you will need GPU acceleration. But still you can fall back to CPU rendering. There is Open GL on CPU in MESA. same for DirectX. It is shower for some fency stuff but good enought for basic UI
ClumsyPilot 4y ago
No mention of Unity or games in General? Thats what every game does - draws a UI directly on the GPU. Game engines like Unity have in-built frameworks for creating UI and then can package the same application for Windows, Linux, playstation, etc. Many people have (ab)used it to create cross-platform software. It's very high-performance, you just get large applications and load times.
- emaro 4y ago
  > It takes a lot of work to make a usable UI out of triangles, so most applications did not go this path. One big exception was games, which were already heavy GPU users, and usually had minimal UIs. The 3D renderer Blender is another example. Only briefly, but seems covered to me. Edit: now I get it — they mention games, but not game engines as a means of creating other applications.
- paulgb 4y ago
  > Many people have (ab)used it to create cross-platform software. It's very high-performance, you just get large applications and load times. (author here) Part of the appeal of posting here is that I learn stuff like this :) Are there some good examples of these that come to mind?
  - jb1991 4y ago
    A lot of cross-platform desktop applications have also been written in Godot.
    - arminiusreturns 4y ago
      This is one more reason I am loving godot (nightly compile from main). I see this as a real opportunity to learn an eco system I can do more than just make games with.
alokedesai 4y ago
Disclaimer: I work at Warp (warp.dev) and wrote the initial How Warp Works blog post (https://www.warp.dev/blog/how-warp-works https://www.warp.dev/blog/how-warp-works) that discusses why we chose to write our own UI framework. This is a great article! Writing a custom UI framework to render on the GPU is a big cost--but it has been worth it for us. It has let us build a UI-heavy terminal while continuing to match the performance of some of the fastest terminals on the market (such as Alacritty). There is certainly an initial velocity hit for new engineers as they have to learn the semantics of our framework, but overall I wouldn't say it has a big detriment on our velocity as a consumer of the framework. The callout to a11y is a good one--we have basic a11y support but it is by no means robust. There are additional downsides to building your own GPU-rendered UI framework other than a11y that I want to call out: you have to build most of the interactions with the platform yourself instead of getting it for free when rendering using system APIs. One example of this is properly supporting i18n. A robust framework should support RTL text, positioning IME / the emoji picker, so forth. While implementing these yourself is doable (we have decent support for IME at Warp, for example) it requires an engineer or two to spend a few weeks to implement. Text rendering is also a beast, see https://www.warp.dev/blog/adventures-text-rendering-kerning-glyph-atlases https://www.warp.dev/blog/adventures-text-rendering-kerning-.... For Warp, this has definitely been worth it, but it hints that the community needs better cross-platform GPU-rendered UI frameworks so people don't need to consistently build them themselves.
- aconbere 4y ago
  I was shocked to find that there wasn’t a fully fleshed out text rendering engine for WebGL. On the one hand I think you could argue that if you have a lot of text maybe html would suit your needs better. On the other hand a lot of UIs just need some text and it’s so east for it to look like junk.
  - flohofwoe 4y ago
    There's an entire layer of mid-level web APIs missing to better connect WebGL and WebGPU to all the other browser systems (like text and DOM rendering). Ideally canvas with WebGL and WebGPU would sit at the bottom of the browser API layer cake, the DOM at the top and squeezed inbetween public APIs for text, shape, CSS+HTML rendering. But instead we got this weird canvas thing dangling off as another high-level DOM element.
aconbere 4y ago
I’ve spent the last week playing around with running WASM apps executing WebGL. I’m exhausted by the modern web. There are so many layers and layers and layers, and none of it works very well! But WebGL rendering on the web offers a unique and compelling alternative. I can write my frontend and backend in Rust. I can use gui libraries like egui directly if I want or write my own UIs if I prefer. It’s almost like going back to the bad flash days, but with open web standards. There are a bunch of downsides (some the article mentions. Accessibility isn’t there yet, the platform is immature so a lot of basic stuff you have to write yourself (form handling, etc), and a big one for me is: text rendering! There appears to be few if any mature text rendering libraries that you can drop into a webgl project and get good crisp text, shaping, and kerning.
- martin_drapeau 4y ago
  15 years ago I was at the same place you are. Believe it or not, I built an entire web-like UI using Tcl/Tk! But then I hit limits and started re-implementing the wheel. So I bit the bullet and learned all those layers of web development. Never looked back. When you get to that point, may I suggest you master HTML and submitting forms. You can do a lot without Javascript. Look at Laravel for an easy to comprehend framework (PHP). It is just perfect.
  - aconbere 4y ago
    I think you might be confused about where I'm coming from and starting at. I've been writing rich web applications professionally since 2004.
  - jimmySixDOF 4y ago
    Have you looked at TrokiaJS ?
- traverseda 4y ago
  My ideal stack is plain html, enhanced with HTMX for things like form validation. Then for more complicated stuff I use self contained web components. That web component might be a terminal, or it might be a text editor, or it might be some kind of video calling widget. One example, you don't need javascript for a chat interface (aside from htmx). You can use htmx to subscribe to a server-sent-event stream for receiving messages, and for posting them you just use a normal form element. It lets you write a lot more app-like functionality as standard html, and for when you can't do that a custom element is often the answer. Don't know why web components haven't caught on more.
PaulHoule 4y ago
Isn’t Windows fundamentally GPU based since WDDM came with Windows Vista?
- incrudible 4y ago
  Fundamentally? No. It's really just compositing whatever buffers represent the content of the windows, but those may well be wholly or partially rendered on the CPU. Almost nobody does text rendering on the GPU, vector graphics are often CPU-based as well.
jillesvangurp 4y ago
As others are pointing out, most UIs are hardware accelerated directly or indirectly. Most "native" apps, are native in the sense that they use some some UI toolkit that renders using the platform specific and typically hardware accelerated ui infrastructure. With webgl and wasm, those same toolkits can now work in a browser. As for accessibility, that is indeed a challenge. You would need to use some ui toolkit that supports accessibility features. Those of course do exist, browsers don't have a monopoly on accessibility. A related challenge is the integrating that with the browser properly so that the native accessibility facilities in your OS can do whatever it needs to do (screen reading, enhancing contrast, etc.). This is not a solved problem. But it sounds like it could be a solvable problem that would require some changes to wasm and webgl.
- actionfromafar 4y ago
  When you have made all these changes to the browser, is it even a browser then? More like a Javascript shim on top of native controls.
  - necessary 4y ago
    Maybe not compared to what browsers were a decade or two ago, but I think the definition has changed. These days browsers are just sandboxes with convenient tooling for finding, downloading, and running arbitrary code. And I don't think that's a bad thing: that's what allows Photopea and Figma to exist in the browser at a moment's notice. Once you generalize the definition of browser, both ends of the spectrum coexist: from basic HTML pages to fully WebGL rendered WebAssembly apps.
    - actionfromafar 4y ago
      It's already that, but what I got from the OP was to add native controls, like what is necessary to (re-)implement accessability. (That builtin accessability mechanism of the browser itself won't do for various reasons.)
  - jillesvangurp 4y ago
naillo 4y ago
Regarding the last point, does this mean developing UIs like this is illegal in some parts of the world? I have no idea asking if someone is more knowledgable than me (e.g. europe).
- CGamesPlay 4y ago
  I don’t think developing a UI that isn’t accessible is ever illegal? I believe the illegal thing is providing a service that doesn’t have “reasonable affordances”. For example, providing a “m.coolco.io” where that subdomain has fundamentally the same service in an accessible fashion is allowed.
  - alex_suzuki 4y ago
    For government services, it can be a legal requirement for them to be accessible.
th0ma5 4y ago
What is the accessibility story for GPU backed interface software? On the web?
- alex_suzuki 4y ago
  As mentioned in TFA, it’s not there yet.
HellDunkel 4y ago
Independence of screen resolution and vector graphics do not align very well with rendering via OpenGL. And what if your app makes heavy use of rendering 3D on the GPU? You would better stay away from creating too much workload for the UI.
ognarb 4y ago
Another example of a GPU-Backed user interface is QtQuick.
FpUser 4y ago
I just briefly looked at makepad and the underlying framework. Seems like very nice project.
mastax 4y ago
> It takes a lot of work to make a usable UI out of triangles, so most applications did not go this path. One big exception was games, which were already heavy GPU users In the past, Scaleform was close to ubiquitous, at least for large games. Scaleform is Flash-based, so it was a vector graphics engine with a JavaScript-like language. It was presumably faster and less bloated than a browser engine would be, as a game middleware from the Xbox 360 era. But from a developer experience point of view, it closer to a browser engine than to "[making] a usable UI out of triangles." Scaleform is still pretty popular, but it has become popular to create game UIs with web technologies. You can find libCEF.dll in pretty much every game nowadays. Sometimes it's just used to render the update news or in-game store, but often the entire UI is HTML.
wolframhempel 4y ago
Having built a few GPU/HTML hybrids in the past, (e.g. https://arcentry.com/ https://arcentry.com/) I found that it teaches you a lot about the things you take for granted when developing for browsers and that you now have to manage yourself. This includes for example: - figuring out when to redraw the screen or sections of it - organizing your UI as a hierarchical tree that can be traversed - handling events in general. what did the user click on? What Z order are things in? Can we find out using 2D boxes or do we need to do 3D ray intersection? - combining UI hierarchies and events into event propagation - rendering text and images crisply is surprisingly tricky - blending layers over each other requires care. - layout flows (margins, paddings, things pushing against other things) need managing. ...and a host of other things. For anyone interested, I wrote a summary in my early days of development here: https://medium.com/@WolframHempel_82303/seven-things-ive-learned-building-a-browser-app-in-webgl-89c7c13c12e8 https://medium.com/@WolframHempel_82303/seven-things-ive-lea... - though this misses a lot of what I figured out later.
- dekhn 4y ago
  With some exceptions, the Qt Graphics View Framework satisfies all these goals with very little additional code from the user. It won't automatically render things as a tree (but the widget system does that for you) or doing layout flow (it's not a layout manager). But it does all the other things so well it's amazing it hasn't been cloned into every other graphics framework. https://doc.qt.io/qt-6/graphicsview.html https://doc.qt.io/qt-6/graphicsview.html
- outworlder 4y ago
  > figuring out when to redraw (...) sections of it What happens if you just brute-force and re-render everything when there's a change? I'm all for optimizing but is this really required these days? Even considering laptops and mobile power requirements, rendering off-screen and flipping is lightning fast. Layout flows are a different matter and can be expensive to compute.
  - kaba0 4y ago
    > Even considering laptops and mobile power requirements, rendering off-screen and flipping is lightning fast I believe it is not speed, but battery life. Most of the time nothing happens on the screen besides some small widget changing like once per second (e.g. clock), doing 59 full rerenders every second for that (or even more with high refresh rate screens) is significantly higher drain. But I am absolutely not an expert on this.
audidude 4y ago
GTK 4 renders with OpenGL, has native a11y built-in, styled with a comprehensive CSS engine, runs on Linux/bsd/Mac/windows, has bindings for almost every noteworthy language…
- aninteger 4y ago
  I've heard that native a11y support only works on GNOME, although I don't know for sure since Orca is just another GTK application. I believe a11y support was available starting back in GTK 2.x but I don't know ATK's full history.
  - traverseda 4y ago
    QT also works alright with screen readers for what it's worth.
- c-smile 4y ago
  > GTK 4 renders with OpenGL It may do that but generally it still uses Cairo. BTW, I would not rely long run on OpenGL existence on Windows and Mac (especially). In any case GTK working on Windows and Mac is an artefact rather than something really usable. I suspect that efforts to support such multiplatform feature makes Linux native GUI development less developed. Generally Linux cannot be considered as a desktop OS as it has no stable and uniform system UI (window manager, graphics, etc.)
  - audidude 4y ago
    > It may do that but generally it still uses Cairo. Uh, no it doesn't. I literally wrote the GL renderer, I know what I'm talking about. > In any case GTK working on Windows and Mac is an artefact rather than something really usable. Maybe pre-4, which is why it got redesigned. > I suspect that efforts to support such multiplatform feature makes Linux native GUI development less developed. What? > Generally Linux cannot be considered as a desktop OS as it has no stable and uniform system UI (window manager, graphics, etc.) Huh? You're out of your mind.
deleted 4y ago
[deleted]
noobermin 4y ago
This article doesn't answer why. I can come up with some uses of a GPU backed interface, but you know people will start using this to render their six page web app in due time. Why not just use the already existing GUI frameworks???
_jezell_ 4y ago
Flutter is quickly shaping up to be the best way to accomplish this. The new renderers landing on all platforms including web are really nice. I've ignored Flutter until now, but it's getting too good to ignore for much longer.
- nzoschke 4y ago
  Came here to say this. Flutter draws to a canvas at 60 FPS and uses the GPU. https://buildflutter.com/how-flutter-works/ https://buildflutter.com/how-flutter-works/ Anecdotally, I see it getting faster and faster on new engine versions without app changes.
  - von_lohengramm 4y ago
    I absolutely hate how 60 FPS is the goal for these canvas-based solutions. I can render DOM at 240 FPS (and much higher if my monitor had a better refresh rate) no problem, but things like Flutter barely ever even make it to the triple digits. It's an embarrassing case of contentment. 60 FPS should not be considered ideal. It should be considered the absolute bare minimum and really only applicable for resource-constrained systems. If you're not doing absurd amounts of number crunching like raytracing, then you should aiming for 1000 FPS.
thewebcount 4y ago
Aren't most native UI toolboxes also GPU-backed? For example Quartz on macOS is. I assume that Windows has something similar, too. Someone else mentioned GTK4 is on Linux. If you want a fast, accessible, user-friendly GPU-back UI, just use the native one. I keep hearing that learning 2 or 3 native UI frameworks is too much work and too costly, and the answer keeps being to learn yet another new cross-platform UI framework that creates apps that are some combination of slow, lack useful features, and don't look right to the users. At this point, wouldn't it be less costly to just learn the native UI frameworks?
- audidude 4y ago
  Does Cocoa draw controls like checkboxes and what not with the GPU? In GTK 4's case, drawing the widgets, CSS transitions, shadows, etc is all on the GPU. Video from GStreamer arrives as GL textures, same for WebKit integration and therefore can be used with all of the other high-level render-tree snapshotting features.
  - kaba0 4y ago
    I’m way out if my depth here, but isn’t certain things faster to render on the CPU still? I believe it is mostly vector graphics - I have seen a few papers showing that it can be done fast(er?) on the GPU but that may not be that widespread yet.
    - vbezhenar 4y ago
      Windows 95 rendered its UI on CPU and it was fast. You need GPU for big bitmaps and useless effects. For square with border and text CPU is more than enough. Then you send this small square bitmap to GPU to add shadows and whatnot to it.
      - Strom 4y ago
        Windows 95 doesn't even support Unicode. The level of UI you can draw there is a very small subset of what you can do in say Chrome. Indeed correct unicode text rendering is one of the difficult problems. It's the same thing that is missing from basically all games. So when people say stuff like games can run at 120fps, how hard can UI be? it's basically the same Windows 95 argument. The UI work being done is a small subset.
iTokio 4y ago
a11y There are efforts to support a cross platform accessibility library: https://github.com/AccessKit/accesskit https://github.com/AccessKit/accesskit
graiz 4y ago
The thing that holds many web UI's back is the web/html DOM and the respective CSS recalculations. You get get close to 1000x performance on web apps if you make sure the DOM will never cause a UI block or re-computation on the layout. The web is great for simple apps but if you're building anything complex you have to play a bunch of tricks to get web UI to work. Going full GPU and building your own controls is certainly possible but I'm hoping there's a middle ground.
- The5thElephant 4y ago
  This would be way easier if everyone didn't default to using React or even worse React-Native-Web. Not only do you have to deal with handling CSS recalcs (not so bad on its own), you also have to deal with React rerendering the entire DOM when you don't want it to and then the layers of div-soup and absurd style abstractions React-Native-Web uses. Tauri + Vue/Svelte/Solid/Etc seems like a great way to build a decent desktop and web app, but you just don't have teams picking anything but React React React React.
- pjc50 4y ago
  Do we have "transactional" DOM yet? The ability to batch up a bunch of stuff and then hit the UI update.
- bartq 4y ago
  We could have amazing HTML/CSS performance if we had programmable "HTML to bitmap" caching. But: a) it's not secure if we can render HTML to bitmap and send over network, b) cache invalidation problem - we'd have to invent mechanisms to lock HTML against further changes or track changes and rerun "HTML to bitmap" procedure. All this is not suitable for web, but for thing like Elector - hell yeah. HTML to bitmap extension would be really welcome. Then you could render UI in WebGL or WebGPU. Adobe Flash had something similar if I recall correctly.
teucris 4y ago
DIY accessibility is harder than it looks, even if you plan to hook into each platform’s native stack for it. For all its flaws, the web stack standardizes the structure of UI, greatly reducing the dev cost of making accessible apps.
madeofpalk 4y ago
I find it strange how, in an industry where people will eke out minor percentage improvements on conversation rates, "we" are so happy to just intentionally slice off a portion of the potential user base to never be able to use the software.
- password11 4y ago
  2.3% of the population have a visual impairment (https://ux.stackexchange.com/questions/57340/percentage-of-screen-readers-users-in-usa https://ux.stackexchange.com/questions/57340/percentage-of-s...). But because conversion rates are typically low and because of how the math works out for a funnel product, there's a larger business impact depending on which 2.3% is cut out. (Ignoring the negative business impact of being non inclusive/a11y) 12% vs. 10% is a 20% decrease in performance. 100% vs. 98% is a 2% decrease in performance. I'm not arguing against a11y. I'm just saying, that's why ADA exists. It's not in the business interest without the law. Also it's just good ethics to be accessible.
  - madeofpalk 4y ago
    > 2.3% of the population have a visual impairment I'm not sure exactly what this 2.3% is (for "blindness"?), but significantly more than 2.3% of the population wear perscription glasses. Those people have visual impairment and use assistive technologies Screen readers are not the only assistive technology. Especially on phones, many people use assistive tech to increase font weight or size. Ignoring these options will remove customers.
outworlder 4y ago
That's something that I have always wondered. And, having built a few of these myself on my free time for personal projects (starting with a university homework), I have my $.02 It is really difficult to do this correctly. The simplest systems to build with are the ones where you have a large visualization window and you only need some knobs to control it. The things that you think are going to be tricky (say, skeuomorphic knobs) are actually almost trivial to implement, even if compared to standard GUI toolkits(but not browsers). Things that you have taken for granted (textboxes) require months to implement correctly. And you will still find corner cases, be it keystrokes(even without taking different operating systems into account). It is easy to make it look 'flashy'. It is surprisingly difficult to make it look 'good'. We see some beautiful UIs in games and we think our applications can look the same. They really can't, not without great graphic designers. And no matter how great it looks, it will look 'alien' compared to other applications in the system. If you need any kind of dynamic layout (which you normally do to handle resizing), you are now looking into an entire system and months of development. Native OS controls usually have ways to hook up to them from external apps. Not so when you are drawing yourself. So that breaks accessibility but also any chance of automation. All those drawbacks (and more I haven't touched) aside, there are benefits. They can be incredibly fast, faster than native OS controls. Rendering a few quads is no problem for any GPU. You can have incredible interactivity(this needs to be explored more by non-Games), since rendering is so fast, faster than the older days of blitting stuff in framebuffers(and a lot of it can be handled by the GPU). You can also implement transitions that are impossible to do in anything else (maybe except for web browsers). You can transition between 2D and 3D seamlessly. For color blindness you can just apply a very simple shader over the whole thing. I think there is a lot of potential hidden by this approach. Think of some the of GUIs imagined by works of science fiction. The main issue is that there's a lot of work to be done before you have anything functional and if you use something out of the box, it will be probably difficult to extend. That said, I've considered taking this idea to an absurd level and write, say, a text editor on something like Unreal Engine and see how far into absurdity we can extend this idea.
snickmy 4y ago
I feel there is something wrong with this approach but I struggle in putting my finger on it. Something about using a level of abstraction at the wrong layer of the stack. I would expect GPU rendering to be used way deeper in the stack (OS level).
spankalee 4y ago
The browser is a GPU-backed renderer. At some point with your own renderer you have to implement a large part of the DOM, styling and rendering layers. And why should a text editor team have to do that? I think a lot of teams would be better off by learning how to make faster web apps, where there's a subset of HTML/CSS that approaches a tree of GPU textures with layout and style on top. Do things like: use fewer elements, very simple CSS selectors, `contain: content`, `contain: strict`, or `content-visibility: auto` everywhere possible, use shadow roots, use a fast vdom-less template system, and a framework where each component is its own render root, use virtualization for large lists... If you do that you can also ship a fast web app. And as VS Code shows by popping up in more and more sites, being an actual web app has some very nice distribution and reuse advantages.
- aconbere 4y ago
  Being a webapp and adding GPU rendering are not orthogonal. Figma for instance famously renders it's UI in WebGL. While I mostly agree that creators of today's web forget that there are a number of simple techniques to render fast pages, there are also a bunch of places where the HTML+CSS+JS web technologies serve us poorly. * Content Editable is a terrible editing canvas * Dom and CSS are pretty terrible core animation primitives * DOM and CSS make for pretty poor interactive graphics capabilities And in these cases WebGL offers some interesting alternatives to explore.
  - spankalee 4y ago
    Figma renders its documents with WASM/WebGL, but the UI of the app is just HTML. A fully GPU rendered web app is almost the worst of both worlds. You'll generally have massive code bundles and terrible a11y.
    - DaiPlusPlus 4y ago
      I remember when I first tried Flutter+Dart and was really put-off by how it rendered to <canvas>, then later versions did use the DOM but looked like the worst case of divitis I’ve ever seen, it even made React.js’s output look handwritten. But why do people do this? I note that a big problem with the web, as-is today, is how anemic and inflexible HTML’s standard <input> elements are, which forces devs to build their own widgets (often poorly), maybe use a framework like Angular, and then eventually go all-in on web abstractions like Flutter/Blazor/etc. If HTML were better then many (but far from all) of these cases could be avoided - for instance, I don’t understand why HTML still doesn’t have absolutely basic widgets like a combobox, a single-line textbox that wraps, a date-range input, hierarchical drop-downs (optgroup is single-level), defined styling and non-text content of <select>, and so on… (though we did recently get color and range/sliders - though there’s still cross-browser quirks which hamper adoption). I hated the days of ActiveX like everyone, but when you’re building an internal-only (I.e. IE-only) webpage it was neat that we could pull-in any of the standard Windows controls - and plenty others - into a page and it would just-work. And you got accessibility for free too, which is more than what you get with <canvas>-based frameworks still - or a multi-megabyte JS/WASM bundle download either (or an npm-based build system that breaks every time you update your dependencies…) ]. Okay yes I’m jaded.
cal85 4y ago
Off topic, but I wish more tech articles were written like this. The writing style is really easy to follow, succinct, plain-spoken, but with enough technical detail. Difficult balance to get right.
neilv 4y ago
One of the things I liked about Blender UI implementation, eons ago, is that I could fire it up on a very modest old Linux laptop, and it started up in an instant. What I'd think was the most demanding GUI application on my laptop also seemed the most responsive.
- aequitas 4y ago
  Also their save file format was in a way that allowed super fast saves of really complex projects.
zokier 4y ago
> Traditionally, if you wanted to write desktop software, you would use the operating system’s APIs to create a user interface. That’s why Windows apps looked like Windows apps, and Mac apps looked like Mac apps That is oft repeated but having lived through those times that seems just untrue, at least on Windows which was far more dominant than it is today. So many applications, big and small, were using completely custom UIs instead of native things. From WinAmp to Microsoft Office to all of Adobe/Macromedia suites to all cross-platform apps using GTK/Qt (or god forbid, Java). One particular example is the pre-electron Spotify app, which was wonderful small single-exe thing.
SpartanJ 4y ago
This is an ongoing trend that will continue until some of the new GUI-backed UIs gain some popularity. What we need right now is a real alternative to Electron based applications, and probably carrying new technologies. I know that we already have alternatives, like Qt, which is an amazing framework, but it doesn't fit all developer's needs and sometimes looks also "too big". I think that many devs are saturated of the HTML+CSS+JS ecosystem and want something new, maybe something less bloated in the sense of the tech specs. I celebrate the numerous projects that are appearing these days, to name a few: egui, iced, imgui, GuiLite, RmlUI, elements. They need more exposure, more collaborators, and more development, they need to grow. Having said that, I'm working on an alternative GPU accelerated and retained-mode GUI called eepp (https://github.com/SpartanJ/eepp/ https://github.com/SpartanJ/eepp/) while I work on a new code editor similar to Zed called ecode (https://github.com/SpartanJ/ecode https://github.com/SpartanJ/ecode). The framework still needs a lot of work but presents some ideas that might be interesting to some. Retained-mode GUIs are a complex beast and it's hard to achieve simplicity (I think immediate-mode GUIs can't be beaten in that regard, but they have their own downsides). So I invite anyone interested in GPU accelerated GUIs to take a look at my projects (online demos are available), I'm looking for collaborators! (sadly C++ isn't very appealing these days, but my code-base is huge and I can't write everything from scratch in a new fancy and cool language).

146 points by paulgb 4y ago

That's weird that the author completely ignores popular projects like ImGui [1] or lvgl, maybe the author wanted to talk about W.I.P. GPU based UIs written in Rust only? In that case, there doesn't seems to be a lot of choice [1] - https://github.com/ocornut/imgui/issues/5886 https://github.com/ocornut/imgui/issues/5886

paulgb 4y ago
I had some notes on Dear ImGui (and the prevalence of immediate-mode across other frameworks that it inspired), but ultimately cut it for length because I wanted to focus on applications rather than frameworks/GUI toolkits.
- Kukumber 4y ago
  https://github.com/ocornut/imgui/issues/3075#issuecomment-604711590 https://github.com/ocornut/imgui/issues/3075#issuecomment-60...
lionkor 4y ago
ImGui, by default, is run at like 60+fps. No GUI app that doesnt display real time changes needs to render at 60+ fps
- seanw444 4y ago
  Maybe not a constant 60+ FPS, but if I scroll, and I can tell it's rendering at 20-30 FPS max on my 144 Hz monitor, I'm going to be annoyed. I want smoothness.
- jakelazaroff 4y ago
  Dear ImGui doesn't actually include a draw loop, does it? AFAIK you're in control of how often that happens.
  - lionkor 4y ago
    Yes, and the example and sample code does it with max framerate.
    - jakelazaroff 4y ago
      Okay, but that’s not the same thing as “Dear ImGui runs at 60fps by default”.

danielvaughn 4y ago

Considering how GPU-backed UI is viewed as a performance win, I wonder why Flutter web performs so poorly.

dahart 4y ago
Using GPUs only enables high performance. It definitely doesn’t guarantee it. ;)
mhoad 4y ago
Flutter web is in the process of moving to a new rendering pipeline that is C++ compiled to WASM and backed by WebGPU. I don’t think they are going to have troubles performance wise for long. This is actually an interesting read I came across by someone from their team recently which poses some hypothetical directions they could take things in the future depending on how the web platform evolves over time. https://docs.google.com/document/d/1peUSMsvFGvqD5yKh3GprskLC3KVdAlLGOsK6gFoEOD0/edit?resourcekey=0-bPajpoo9IBZpG__-uCBE6w https://docs.google.com/document/d/1peUSMsvFGvqD5yKh3GprskLC...

AshleysBrain 4y ago

AFAIK modern browsers all use GPU accelerated rendering, so using HTML/CSS does give you a GPU-backed user interface. Also desktop PWAs are IMO underrated and a really good way to distribute a desktop app without having to bundle a whole browser engine with your app. We do this for Construct (www.construct.net) - a complete game development IDE - and I think it's worked out great.

pjmlp 4y ago
Not really it is a bit more subtle than that, unless you do some magic incantations with CSS, like z-ordering or transforms, it may bet that software rendering is used instead. There are a couple of talks regarding CSS tricks to force GPU redrawing.
- AshleysBrain 4y ago
  I think that was the case several years ago but I thought these days much more of the browser rendering pipeline was automatically GPU accelerated. It's hard to find up-to-date references (would appreciate any links) but for example going to chrome://gpu shows it using GPU acceleration for both compositing and rasterization (including "multiple raster threads"), as well as the usual suspects like video decoding, canvas and WebGL.
  - pjmlp 4y ago
    Don't be fooled by canvas/WebGL support. I remember seeing this kind of content on Google IO and WebDev talks. Would need to also go treasure hunting to find them.
    - AshleysBrain 4y ago
      Treasure hunting to find an API widely supported on 95%+ of devices?
      - pjmlp 4y ago

matlin 4y ago

This is pretty much what Flutter is. Except Flutter has system-specific UI's created by Google so each app can have that native look depending on where it's running.

ArtWomb 4y ago
Impeller. New shader compilation backend. Heroic coding. Metal/Vulkan/WebGPU translators. The bar is pixel perfect across gpus which makes it so challanging. My current favorite open source project: Android & Win ARM ports ;) https://github.com/flutter/flutter/wiki/Impeller https://github.com/flutter/flutter/wiki/Impeller
actionfromafar 4y ago
Almost that native look. Diluting what a native app feels like, but that price is something Google is willing for you to pay.

c-smile 4y ago

> Unlike the UIs provided by the operating system, where you can say "draw a button with the label 'delete'", you have to tell OpenGL how to make a button out of triangles. Just use Sciter, by default for rendering it uses Direct2d/DirectX (+ Vulkan||OpenGL) on Windows, Metal||OpenGL on Mac and Vulkan||OpenGL on Linux. With support of a11y by the way.

fassssst 4y ago
But the UIs provided by the operating system are making DirectX/Metal calls for you... they are GPU accelerated. And they have been for like 16 years now.
- c-smile 4y ago
  > UIs provided by the operating system ... GPU accelerated That depends on what you mean by "system UI". Most system utilities on Windows are still using GDI. GDI is not GPU accelerated (other than BitBlt). Same apply to MacOS - CoreGraphics there is pure CPU rasterization library. And the same situation on Linux/GTK, system uses Cairo - pure CPU rasterization library. Usually, on modern OSes, window/desktop manager is the only GPU accelerated thing - it composes window bitmaps on screen.
  - fassssst 4y ago
    WPF and WinUI on Windows are both GPU accelerated. They render with Direct3D calls under the hood.
    - c-smile 4y ago
      Sciter greatly overlaps with those. It is just that instead of XAML it uses HTML/CSS, JS/C/C++ instead of C# and works on all desktop platforms. For the note: historically Sciter appeared before WPF and WinUI/UWP : https://sciter.com/sciter/sciter-vs-wpf/ https://sciter.com/sciter/sciter-vs-wpf/ By any means WPF and WinUI are not UIs provided by the OS. These are userland UI libraries. Same thing as Sciter.

xlazom00 4y ago

On windows 8 and newer, microsoft use new framework to create UI for apps. They are all GPU accelerated. Calculator, Settings,.. And it is up to the framework if it support all feature like accesiblity or system spellcheck or whatever. And on Mac OS Cocoa use GPU acceleration as well. As if you want to show something nice and fency you will need GPU acceleration. But still you can fall back to CPU rendering. There is Open GL on CPU in MESA. same for DirectX. It is shower for some fency stuff but good enought for basic UI

ClumsyPilot 4y ago

No mention of Unity or games in General? Thats what every game does - draws a UI directly on the GPU. Game engines like Unity have in-built frameworks for creating UI and then can package the same application for Windows, Linux, playstation, etc. Many people have (ab)used it to create cross-platform software. It's very high-performance, you just get large applications and load times.

emaro 4y ago
> It takes a lot of work to make a usable UI out of triangles, so most applications did not go this path. One big exception was games, which were already heavy GPU users, and usually had minimal UIs. The 3D renderer Blender is another example. Only briefly, but seems covered to me. Edit: now I get it — they mention games, but not game engines as a means of creating other applications.
paulgb 4y ago
> Many people have (ab)used it to create cross-platform software. It's very high-performance, you just get large applications and load times. (author here) Part of the appeal of posting here is that I learn stuff like this :) Are there some good examples of these that come to mind?
- jb1991 4y ago
  A lot of cross-platform desktop applications have also been written in Godot.
  - arminiusreturns 4y ago
    This is one more reason I am loving godot (nightly compile from main). I see this as a real opportunity to learn an eco system I can do more than just make games with.

alokedesai 4y ago

Disclaimer: I work at Warp (warp.dev) and wrote the initial How Warp Works blog post (https://www.warp.dev/blog/how-warp-works https://www.warp.dev/blog/how-warp-works) that discusses why we chose to write our own UI framework. This is a great article! Writing a custom UI framework to render on the GPU is a big cost--but it has been worth it for us. It has let us build a UI-heavy terminal while continuing to match the performance of some of the fastest terminals on the market (such as Alacritty). There is certainly an initial velocity hit for new engineers as they have to learn the semantics of our framework, but overall I wouldn't say it has a big detriment on our velocity as a consumer of the framework. The callout to a11y is a good one--we have basic a11y support but it is by no means robust. There are additional downsides to building your own GPU-rendered UI framework other than a11y that I want to call out: you have to build most of the interactions with the platform yourself instead of getting it for free when rendering using system APIs. One example of this is properly supporting i18n. A robust framework should support RTL text, positioning IME / the emoji picker, so forth. While implementing these yourself is doable (we have decent support for IME at Warp, for example) it requires an engineer or two to spend a few weeks to implement. Text rendering is also a beast, see https://www.warp.dev/blog/adventures-text-rendering-kerning-glyph-atlases https://www.warp.dev/blog/adventures-text-rendering-kerning-.... For Warp, this has definitely been worth it, but it hints that the community needs better cross-platform GPU-rendered UI frameworks so people don't need to consistently build them themselves.

aconbere 4y ago
I was shocked to find that there wasn’t a fully fleshed out text rendering engine for WebGL. On the one hand I think you could argue that if you have a lot of text maybe html would suit your needs better. On the other hand a lot of UIs just need some text and it’s so east for it to look like junk.
- flohofwoe 4y ago
  There's an entire layer of mid-level web APIs missing to better connect WebGL and WebGPU to all the other browser systems (like text and DOM rendering). Ideally canvas with WebGL and WebGPU would sit at the bottom of the browser API layer cake, the DOM at the top and squeezed inbetween public APIs for text, shape, CSS+HTML rendering. But instead we got this weird canvas thing dangling off as another high-level DOM element.

aconbere 4y ago

I’ve spent the last week playing around with running WASM apps executing WebGL. I’m exhausted by the modern web. There are so many layers and layers and layers, and none of it works very well! But WebGL rendering on the web offers a unique and compelling alternative. I can write my frontend and backend in Rust. I can use gui libraries like egui directly if I want or write my own UIs if I prefer. It’s almost like going back to the bad flash days, but with open web standards. There are a bunch of downsides (some the article mentions. Accessibility isn’t there yet, the platform is immature so a lot of basic stuff you have to write yourself (form handling, etc), and a big one for me is: text rendering! There appears to be few if any mature text rendering libraries that you can drop into a webgl project and get good crisp text, shaping, and kerning.

martin_drapeau 4y ago
15 years ago I was at the same place you are. Believe it or not, I built an entire web-like UI using Tcl/Tk! But then I hit limits and started re-implementing the wheel. So I bit the bullet and learned all those layers of web development. Never looked back. When you get to that point, may I suggest you master HTML and submitting forms. You can do a lot without Javascript. Look at Laravel for an easy to comprehend framework (PHP). It is just perfect.
- aconbere 4y ago
  I think you might be confused about where I'm coming from and starting at. I've been writing rich web applications professionally since 2004.
- jimmySixDOF 4y ago
  Have you looked at TrokiaJS ?
traverseda 4y ago
My ideal stack is plain html, enhanced with HTMX for things like form validation. Then for more complicated stuff I use self contained web components. That web component might be a terminal, or it might be a text editor, or it might be some kind of video calling widget. One example, you don't need javascript for a chat interface (aside from htmx). You can use htmx to subscribe to a server-sent-event stream for receiving messages, and for posting them you just use a normal form element. It lets you write a lot more app-like functionality as standard html, and for when you can't do that a custom element is often the answer. Don't know why web components haven't caught on more.

PaulHoule 4y ago

Isn’t Windows fundamentally GPU based since WDDM came with Windows Vista?

incrudible 4y ago
Fundamentally? No. It's really just compositing whatever buffers represent the content of the windows, but those may well be wholly or partially rendered on the CPU. Almost nobody does text rendering on the GPU, vector graphics are often CPU-based as well.

jillesvangurp 4y ago

As others are pointing out, most UIs are hardware accelerated directly or indirectly. Most "native" apps, are native in the sense that they use some some UI toolkit that renders using the platform specific and typically hardware accelerated ui infrastructure. With webgl and wasm, those same toolkits can now work in a browser. As for accessibility, that is indeed a challenge. You would need to use some ui toolkit that supports accessibility features. Those of course do exist, browsers don't have a monopoly on accessibility. A related challenge is the integrating that with the browser properly so that the native accessibility facilities in your OS can do whatever it needs to do (screen reading, enhancing contrast, etc.). This is not a solved problem. But it sounds like it could be a solvable problem that would require some changes to wasm and webgl.

actionfromafar 4y ago
When you have made all these changes to the browser, is it even a browser then? More like a Javascript shim on top of native controls.
- necessary 4y ago
  Maybe not compared to what browsers were a decade or two ago, but I think the definition has changed. These days browsers are just sandboxes with convenient tooling for finding, downloading, and running arbitrary code. And I don't think that's a bad thing: that's what allows Photopea and Figma to exist in the browser at a moment's notice. Once you generalize the definition of browser, both ends of the spectrum coexist: from basic HTML pages to fully WebGL rendered WebAssembly apps.
  - actionfromafar 4y ago
    It's already that, but what I got from the OP was to add native controls, like what is necessary to (re-)implement accessability. (That builtin accessability mechanism of the browser itself won't do for various reasons.)
- jillesvangurp 4y ago

naillo 4y ago

Regarding the last point, does this mean developing UIs like this is illegal in some parts of the world? I have no idea asking if someone is more knowledgable than me (e.g. europe).

CGamesPlay 4y ago
I don’t think developing a UI that isn’t accessible is ever illegal? I believe the illegal thing is providing a service that doesn’t have “reasonable affordances”. For example, providing a “m.coolco.io” where that subdomain has fundamentally the same service in an accessible fashion is allowed.
- alex_suzuki 4y ago
  For government services, it can be a legal requirement for them to be accessible.

th0ma5 4y ago

What is the accessibility story for GPU backed interface software? On the web?

alex_suzuki 4y ago
As mentioned in TFA, it’s not there yet.

HellDunkel 4y ago

Independence of screen resolution and vector graphics do not align very well with rendering via OpenGL. And what if your app makes heavy use of rendering 3D on the GPU? You would better stay away from creating too much workload for the UI.

ognarb 4y ago

Another example of a GPU-Backed user interface is QtQuick.

FpUser 4y ago

I just briefly looked at makepad and the underlying framework. Seems like very nice project.

mastax 4y ago

> It takes a lot of work to make a usable UI out of triangles, so most applications did not go this path. One big exception was games, which were already heavy GPU users In the past, Scaleform was close to ubiquitous, at least for large games. Scaleform is Flash-based, so it was a vector graphics engine with a JavaScript-like language. It was presumably faster and less bloated than a browser engine would be, as a game middleware from the Xbox 360 era. But from a developer experience point of view, it closer to a browser engine than to "[making] a usable UI out of triangles." Scaleform is still pretty popular, but it has become popular to create game UIs with web technologies. You can find libCEF.dll in pretty much every game nowadays. Sometimes it's just used to render the update news or in-game store, but often the entire UI is HTML.

wolframhempel 4y ago

Having built a few GPU/HTML hybrids in the past, (e.g. https://arcentry.com/ https://arcentry.com/) I found that it teaches you a lot about the things you take for granted when developing for browsers and that you now have to manage yourself. This includes for example: - figuring out when to redraw the screen or sections of it - organizing your UI as a hierarchical tree that can be traversed - handling events in general. what did the user click on? What Z order are things in? Can we find out using 2D boxes or do we need to do 3D ray intersection? - combining UI hierarchies and events into event propagation - rendering text and images crisply is surprisingly tricky - blending layers over each other requires care. - layout flows (margins, paddings, things pushing against other things) need managing. ...and a host of other things. For anyone interested, I wrote a summary in my early days of development here: https://medium.com/@WolframHempel_82303/seven-things-ive-learned-building-a-browser-app-in-webgl-89c7c13c12e8 https://medium.com/@WolframHempel_82303/seven-things-ive-lea... - though this misses a lot of what I figured out later.

dekhn 4y ago
With some exceptions, the Qt Graphics View Framework satisfies all these goals with very little additional code from the user. It won't automatically render things as a tree (but the widget system does that for you) or doing layout flow (it's not a layout manager). But it does all the other things so well it's amazing it hasn't been cloned into every other graphics framework. https://doc.qt.io/qt-6/graphicsview.html https://doc.qt.io/qt-6/graphicsview.html
outworlder 4y ago
> figuring out when to redraw (...) sections of it What happens if you just brute-force and re-render everything when there's a change? I'm all for optimizing but is this really required these days? Even considering laptops and mobile power requirements, rendering off-screen and flipping is lightning fast. Layout flows are a different matter and can be expensive to compute.
- kaba0 4y ago
  > Even considering laptops and mobile power requirements, rendering off-screen and flipping is lightning fast I believe it is not speed, but battery life. Most of the time nothing happens on the screen besides some small widget changing like once per second (e.g. clock), doing 59 full rerenders every second for that (or even more with high refresh rate screens) is significantly higher drain. But I am absolutely not an expert on this.

audidude 4y ago

GTK 4 renders with OpenGL, has native a11y built-in, styled with a comprehensive CSS engine, runs on Linux/bsd/Mac/windows, has bindings for almost every noteworthy language…

aninteger 4y ago
I've heard that native a11y support only works on GNOME, although I don't know for sure since Orca is just another GTK application. I believe a11y support was available starting back in GTK 2.x but I don't know ATK's full history.
- traverseda 4y ago
  QT also works alright with screen readers for what it's worth.
c-smile 4y ago
> GTK 4 renders with OpenGL It may do that but generally it still uses Cairo. BTW, I would not rely long run on OpenGL existence on Windows and Mac (especially). In any case GTK working on Windows and Mac is an artefact rather than something really usable. I suspect that efforts to support such multiplatform feature makes Linux native GUI development less developed. Generally Linux cannot be considered as a desktop OS as it has no stable and uniform system UI (window manager, graphics, etc.)
- audidude 4y ago
  > It may do that but generally it still uses Cairo. Uh, no it doesn't. I literally wrote the GL renderer, I know what I'm talking about. > In any case GTK working on Windows and Mac is an artefact rather than something really usable. Maybe pre-4, which is why it got redesigned. > I suspect that efforts to support such multiplatform feature makes Linux native GUI development less developed. What? > Generally Linux cannot be considered as a desktop OS as it has no stable and uniform system UI (window manager, graphics, etc.) Huh? You're out of your mind.

deleted 4y ago

[deleted]

noobermin 4y ago

This article doesn't answer why. I can come up with some uses of a GPU backed interface, but you know people will start using this to render their six page web app in due time. Why not just use the already existing GUI frameworks???

_jezell_ 4y ago

Flutter is quickly shaping up to be the best way to accomplish this. The new renderers landing on all platforms including web are really nice. I've ignored Flutter until now, but it's getting too good to ignore for much longer.

nzoschke 4y ago
Came here to say this. Flutter draws to a canvas at 60 FPS and uses the GPU. https://buildflutter.com/how-flutter-works/ https://buildflutter.com/how-flutter-works/ Anecdotally, I see it getting faster and faster on new engine versions without app changes.
- von_lohengramm 4y ago
  I absolutely hate how 60 FPS is the goal for these canvas-based solutions. I can render DOM at 240 FPS (and much higher if my monitor had a better refresh rate) no problem, but things like Flutter barely ever even make it to the triple digits. It's an embarrassing case of contentment. 60 FPS should not be considered ideal. It should be considered the absolute bare minimum and really only applicable for resource-constrained systems. If you're not doing absurd amounts of number crunching like raytracing, then you should aiming for 1000 FPS.

thewebcount 4y ago

Aren't most native UI toolboxes also GPU-backed? For example Quartz on macOS is. I assume that Windows has something similar, too. Someone else mentioned GTK4 is on Linux. If you want a fast, accessible, user-friendly GPU-back UI, just use the native one. I keep hearing that learning 2 or 3 native UI frameworks is too much work and too costly, and the answer keeps being to learn yet another new cross-platform UI framework that creates apps that are some combination of slow, lack useful features, and don't look right to the users. At this point, wouldn't it be less costly to just learn the native UI frameworks?

audidude 4y ago
Does Cocoa draw controls like checkboxes and what not with the GPU? In GTK 4's case, drawing the widgets, CSS transitions, shadows, etc is all on the GPU. Video from GStreamer arrives as GL textures, same for WebKit integration and therefore can be used with all of the other high-level render-tree snapshotting features.
- kaba0 4y ago
  I’m way out if my depth here, but isn’t certain things faster to render on the CPU still? I believe it is mostly vector graphics - I have seen a few papers showing that it can be done fast(er?) on the GPU but that may not be that widespread yet.
  - vbezhenar 4y ago
    Windows 95 rendered its UI on CPU and it was fast. You need GPU for big bitmaps and useless effects. For square with border and text CPU is more than enough. Then you send this small square bitmap to GPU to add shadows and whatnot to it.
    - Strom 4y ago
      Windows 95 doesn't even support Unicode. The level of UI you can draw there is a very small subset of what you can do in say Chrome. Indeed correct unicode text rendering is one of the difficult problems. It's the same thing that is missing from basically all games. So when people say stuff like games can run at 120fps, how hard can UI be? it's basically the same Windows 95 argument. The UI work being done is a small subset.

iTokio 4y ago

a11y There are efforts to support a cross platform accessibility library: https://github.com/AccessKit/accesskit https://github.com/AccessKit/accesskit

graiz 4y ago

The thing that holds many web UI's back is the web/html DOM and the respective CSS recalculations. You get get close to 1000x performance on web apps if you make sure the DOM will never cause a UI block or re-computation on the layout. The web is great for simple apps but if you're building anything complex you have to play a bunch of tricks to get web UI to work. Going full GPU and building your own controls is certainly possible but I'm hoping there's a middle ground.

The5thElephant 4y ago
This would be way easier if everyone didn't default to using React or even worse React-Native-Web. Not only do you have to deal with handling CSS recalcs (not so bad on its own), you also have to deal with React rerendering the entire DOM when you don't want it to and then the layers of div-soup and absurd style abstractions React-Native-Web uses. Tauri + Vue/Svelte/Solid/Etc seems like a great way to build a decent desktop and web app, but you just don't have teams picking anything but React React React React.
pjc50 4y ago
Do we have "transactional" DOM yet? The ability to batch up a bunch of stuff and then hit the UI update.
bartq 4y ago
We could have amazing HTML/CSS performance if we had programmable "HTML to bitmap" caching. But: a) it's not secure if we can render HTML to bitmap and send over network, b) cache invalidation problem - we'd have to invent mechanisms to lock HTML against further changes or track changes and rerun "HTML to bitmap" procedure. All this is not suitable for web, but for thing like Elector - hell yeah. HTML to bitmap extension would be really welcome. Then you could render UI in WebGL or WebGPU. Adobe Flash had something similar if I recall correctly.

teucris 4y ago

DIY accessibility is harder than it looks, even if you plan to hook into each platform’s native stack for it. For all its flaws, the web stack standardizes the structure of UI, greatly reducing the dev cost of making accessible apps.

madeofpalk 4y ago

I find it strange how, in an industry where people will eke out minor percentage improvements on conversation rates, "we" are so happy to just intentionally slice off a portion of the potential user base to never be able to use the software.

password11 4y ago
2.3% of the population have a visual impairment (https://ux.stackexchange.com/questions/57340/percentage-of-screen-readers-users-in-usa https://ux.stackexchange.com/questions/57340/percentage-of-s...). But because conversion rates are typically low and because of how the math works out for a funnel product, there's a larger business impact depending on which 2.3% is cut out. (Ignoring the negative business impact of being non inclusive/a11y) 12% vs. 10% is a 20% decrease in performance. 100% vs. 98% is a 2% decrease in performance. I'm not arguing against a11y. I'm just saying, that's why ADA exists. It's not in the business interest without the law. Also it's just good ethics to be accessible.
- madeofpalk 4y ago
  > 2.3% of the population have a visual impairment I'm not sure exactly what this 2.3% is (for "blindness"?), but significantly more than 2.3% of the population wear perscription glasses. Those people have visual impairment and use assistive technologies Screen readers are not the only assistive technology. Especially on phones, many people use assistive tech to increase font weight or size. Ignoring these options will remove customers.

outworlder 4y ago

That's something that I have always wondered. And, having built a few of these myself on my free time for personal projects (starting with a university homework), I have my $.02 It is really difficult to do this correctly. The simplest systems to build with are the ones where you have a large visualization window and you only need some knobs to control it. The things that you think are going to be tricky (say, skeuomorphic knobs) are actually almost trivial to implement, even if compared to standard GUI toolkits(but not browsers). Things that you have taken for granted (textboxes) require months to implement correctly. And you will still find corner cases, be it keystrokes(even without taking different operating systems into account). It is easy to make it look 'flashy'. It is surprisingly difficult to make it look 'good'. We see some beautiful UIs in games and we think our applications can look the same. They really can't, not without great graphic designers. And no matter how great it looks, it will look 'alien' compared to other applications in the system. If you need any kind of dynamic layout (which you normally do to handle resizing), you are now looking into an entire system and months of development. Native OS controls usually have ways to hook up to them from external apps. Not so when you are drawing yourself. So that breaks accessibility but also any chance of automation. All those drawbacks (and more I haven't touched) aside, there are benefits. They can be incredibly fast, faster than native OS controls. Rendering a few quads is no problem for any GPU. You can have incredible interactivity(this needs to be explored more by non-Games), since rendering is so fast, faster than the older days of blitting stuff in framebuffers(and a lot of it can be handled by the GPU). You can also implement transitions that are impossible to do in anything else (maybe except for web browsers). You can transition between 2D and 3D seamlessly. For color blindness you can just apply a very simple shader over the whole thing. I think there is a lot of potential hidden by this approach. Think of some the of GUIs imagined by works of science fiction. The main issue is that there's a lot of work to be done before you have anything functional and if you use something out of the box, it will be probably difficult to extend. That said, I've considered taking this idea to an absurd level and write, say, a text editor on something like Unreal Engine and see how far into absurdity we can extend this idea.

snickmy 4y ago

I feel there is something wrong with this approach but I struggle in putting my finger on it. Something about using a level of abstraction at the wrong layer of the stack. I would expect GPU rendering to be used way deeper in the stack (OS level).

spankalee 4y ago

The browser is a GPU-backed renderer. At some point with your own renderer you have to implement a large part of the DOM, styling and rendering layers. And why should a text editor team have to do that? I think a lot of teams would be better off by learning how to make faster web apps, where there's a subset of HTML/CSS that approaches a tree of GPU textures with layout and style on top. Do things like: use fewer elements, very simple CSS selectors, `contain: content`, `contain: strict`, or `content-visibility: auto` everywhere possible, use shadow roots, use a fast vdom-less template system, and a framework where each component is its own render root, use virtualization for large lists... If you do that you can also ship a fast web app. And as VS Code shows by popping up in more and more sites, being an actual web app has some very nice distribution and reuse advantages.

aconbere 4y ago
Being a webapp and adding GPU rendering are not orthogonal. Figma for instance famously renders it's UI in WebGL. While I mostly agree that creators of today's web forget that there are a number of simple techniques to render fast pages, there are also a bunch of places where the HTML+CSS+JS web technologies serve us poorly. * Content Editable is a terrible editing canvas * Dom and CSS are pretty terrible core animation primitives * DOM and CSS make for pretty poor interactive graphics capabilities And in these cases WebGL offers some interesting alternatives to explore.
- spankalee 4y ago
  Figma renders its documents with WASM/WebGL, but the UI of the app is just HTML. A fully GPU rendered web app is almost the worst of both worlds. You'll generally have massive code bundles and terrible a11y.
  - DaiPlusPlus 4y ago
    I remember when I first tried Flutter+Dart and was really put-off by how it rendered to <canvas>, then later versions did use the DOM but looked like the worst case of divitis I’ve ever seen, it even made React.js’s output look handwritten. But why do people do this? I note that a big problem with the web, as-is today, is how anemic and inflexible HTML’s standard <input> elements are, which forces devs to build their own widgets (often poorly), maybe use a framework like Angular, and then eventually go all-in on web abstractions like Flutter/Blazor/etc. If HTML were better then many (but far from all) of these cases could be avoided - for instance, I don’t understand why HTML still doesn’t have absolutely basic widgets like a combobox, a single-line textbox that wraps, a date-range input, hierarchical drop-downs (optgroup is single-level), defined styling and non-text content of <select>, and so on… (though we did recently get color and range/sliders - though there’s still cross-browser quirks which hamper adoption). I hated the days of ActiveX like everyone, but when you’re building an internal-only (I.e. IE-only) webpage it was neat that we could pull-in any of the standard Windows controls - and plenty others - into a page and it would just-work. And you got accessibility for free too, which is more than what you get with <canvas>-based frameworks still - or a multi-megabyte JS/WASM bundle download either (or an npm-based build system that breaks every time you update your dependencies…) ]. Okay yes I’m jaded.

cal85 4y ago

Off topic, but I wish more tech articles were written like this. The writing style is really easy to follow, succinct, plain-spoken, but with enough technical detail. Difficult balance to get right.

neilv 4y ago

One of the things I liked about Blender UI implementation, eons ago, is that I could fire it up on a very modest old Linux laptop, and it started up in an instant. What I'd think was the most demanding GUI application on my laptop also seemed the most responsive.

aequitas 4y ago
Also their save file format was in a way that allowed super fast saves of really complex projects.

zokier 4y ago

> Traditionally, if you wanted to write desktop software, you would use the operating system’s APIs to create a user interface. That’s why Windows apps looked like Windows apps, and Mac apps looked like Mac apps That is oft repeated but having lived through those times that seems just untrue, at least on Windows which was far more dominant than it is today. So many applications, big and small, were using completely custom UIs instead of native things. From WinAmp to Microsoft Office to all of Adobe/Macromedia suites to all cross-platform apps using GTK/Qt (or god forbid, Java). One particular example is the pre-electron Spotify app, which was wonderful small single-exe thing.

SpartanJ 4y ago

This is an ongoing trend that will continue until some of the new GUI-backed UIs gain some popularity. What we need right now is a real alternative to Electron based applications, and probably carrying new technologies. I know that we already have alternatives, like Qt, which is an amazing framework, but it doesn't fit all developer's needs and sometimes looks also "too big". I think that many devs are saturated of the HTML+CSS+JS ecosystem and want something new, maybe something less bloated in the sense of the tech specs. I celebrate the numerous projects that are appearing these days, to name a few: egui, iced, imgui, GuiLite, RmlUI, elements. They need more exposure, more collaborators, and more development, they need to grow. Having said that, I'm working on an alternative GPU accelerated and retained-mode GUI called eepp (https://github.com/SpartanJ/eepp/ https://github.com/SpartanJ/eepp/) while I work on a new code editor similar to Zed called ecode (https://github.com/SpartanJ/ecode https://github.com/SpartanJ/ecode). The framework still needs a lot of work but presents some ideas that might be interesting to some. Retained-mode GUIs are a complex beast and it's hard to achieve simplicity (I think immediate-mode GUIs can't be beaten in that regard, but they have their own downsides). So I invite anyone interested in GPU accelerated GUIs to take a look at my projects (online demos are available), I'm looking for collaborators! (sadly C++ isn't very appealing these days, but my code-base is huge and I can't write everything from scratch in a new fancy and cool language).