17 ms·

Leveraging Rust and the GPU to render user interfaces at 120 FPS

259 points by rck 4y ago

fxtentacle 4y ago
"Inspired by the gaming world, we realized that the only way to achieve the performance we needed was to build our own UI framework" I'm surprised you did not look at "Dear ImGui", "Noesis", and "JUCE". All three of them are heavily used in gaming, are rather clean C++, use full GPU acceleration, and have visual editors available. Especially JUCE is used for A LOT of hard-realtime professional audio applications. "When we started building Zed, arbitrary 2D graphics rendering on the GPU was still very much a research project." What are you talking about? JUCE has had GPU-accelerated spline shapes and SVG animations since 2012? BTW, I like the explanations for how they use SDFs for rendering basic primitives. But that technique looks an awful lot like the 2018 GPU renderer from KiCad ;) And lastly, that glyph atlas for font rendering is only 1 channel? KiCad uses a technique using RGB for gradients so that the rendered glyphs can be anti-aliased without accidentally rounding sharp corners. Overall, this reads to me like they did not do much research before starting, which is totally OK, but then they shouldn't say stuff like "did not exist" or "was still a research project".
- deleted 4y ago
  [deleted]
- wnkrshm 4y ago
  It's not the first text editor either, Sublime has GPU-accelerated rendering.
  - fxtentacle 4y ago
    Also, looking at their own performance numbers, it appears that all of this only managed to reduce the character insertion latency by 21% when compared to CLion. That's still useful, but maybe not even noticeable to the user.
    - gizmo 4y ago
      That latency can’t be render latency, because it’s trivial. It has got to be related to unifying the text buffer with the syntax tree. Otherwise you’ll get wrong syntax highlighting for a couple of frames and that’s jarring.
- swatcoder 4y ago
  I agree with your final statement and the seeming lack of research, but are you sure your characterization of JUCE is accurate? You’d surprise a lot of people if you were right about it using GPU acceleration for its UI framework. It does have an OpenGL container that fits into its view object hierarchy if you want to write your own accelerated component, but the rest of the UI is pretty much standard event-loop -> platform-API dispatch. It’s accelerated where the underlying platform-API is accelerated but not in any kind of explicit or fine-tuned way. The focus has always been more on portability than performance, even on the audio side. (And of course the high-priority audio processing runs in an entirely segregated context from the UI, so the performance characteristics of the two pieces are decoupled anyway.)
  - coldtea 4y ago
    >I agree with your final statement and the seeming lack of research, but are you sure your characterization of JUCE is accurate? That JUCE is "heavily used in gaming" for starters is widly innacurate
    - fxtentacle 4y ago
      I personally know multiple people using it in their games and game-related tooling and Roli (the company selling JUCE) even has an official Unreal Engine 4 plugin. So to me, it appears to be widely used.
      - irh 4y ago
        > Roli (the company selling JUCE) I know it's tangential to the point that you're making, but JUCE was acquired by PACE in 2020.
      - swatcoder 4y ago
        JUCE is sold by Raw Material Software, which was purchased by PACE Anti-piracy, which is part of Avid. There may be some people you know using JUCE for games, but it’s an odd choice. Trying to figure out what you might mean, JUCE was involved with the BLOCKS SDK for Roli’s hardware peripherals, but again, that’s not really about gaming and definitely not about any familiar kind of gaming. That said, I’m sure you’re relating the best information you have and very likely just transposed some detail by accident and are now caught in a relentless, exhausting online nitpick. We’ve all been there. If you figure out what it was, I’d love to hear it!
- rob74 4y ago
  Maybe you need to add an "(in Rust)" to all these sentences? Sure there are C++ frameworks, but they probably wanted a pure Rust UI framework? My 2 cents: it's nice to have smooth rendering in your editor, but I'm currently mostly using a Java-based IDE (IDEA family), and it's responsive enough for my taste. If I were to use the current prototype of their editor, I'm afraid the usage of fixed-width fonts all over the place (which I assume can be fixed, but may also be due to constraints of the UI framework?) would probably bother me more than the 120 FPS UI would impress me (assuming that I had a monitor that was fast enough, which I don't). On the plus side, it sounds like they support sub-pixel rendering - if that's the case, kudos to them!
  - DeathArrow 4y ago
    Not true, even if you add Rust. egui was released earlier: https://github.com/emilk/egui https://github.com/emilk/egui And egui isn't tied to Apple proprietary frameworks.
    - littlestymaar 4y ago
      Afaik egui isn't doing any kind of the fancy GPU based 2D graphics this blog post is about though.
      - pama 4y ago
        It uses glow, GL on whatever, so OpenGL when it exists.
        littlestymaar 4y ago
        That doesn't mean it's doing any kind of GPU acceleration of 2D graphics. Given that they don't talk about it in the README, I'm pretty sure they don't.
        decremental 4y ago
        I use egui. The library itself is agnostic but relies on backend libraries for rendering, all of which (or at least the official ones) render on the GPU.
- littlestymaar 4y ago
  > What are you talking about? JUCE has had GPU-accelerated spline shapes and SVG animations since 2012? I'm not familiar at all with JUCE, but the state of the art in GPU-accelerated 2D graphics has dramatically improved over the past 10 years so I doubt what JUCE did in 2012 is really comparable.
  - fxtentacle 4y ago
    I shipped a JUCE-based app in 2012 and back then it was "good enough" to have GPU-accelerated rendering of SVG icons moving around in realtime with antialiasing. Since we used their OpenGL context, we could even render the GUI inside customers' games for debug visualization.
- dorkwood 4y ago
  > KiCad uses a technique using RGB for gradients so that the rendered glyphs can be anti-aliased without accidentally rounding sharp corners. This is known as a “multi-channel signed distance field”, or “msdf”. https://github.com/Chlumsky/msdfgen https://github.com/Chlumsky/msdfgen
  - scotty79 4y ago
    Do you know any text editor that uses it for font rendering?
    - fxtentacle 4y ago
      You would have to use that if you want smooth antialiased zooming in and out of text. BTW, the KiCad schematics are pretty text-heavy. You typically write down all the parameters and IDs of all the electrical components.
    - rikarendsmp 4y ago
      I used to use it, but its extremely font and glyph-sensitive wether or not it works. And if it doesn't work there is no easy fix (or ability to recognise it nonvisually)
- incrudible 4y ago
  > KiCad uses a technique using RGB for gradients so that the rendered glyphs can be anti-aliased without accidentally rounding sharp corners They are not using SDFs for the text, they render the glyphs at the specified font size directly.
- shultays 4y ago
  ImGui is at least only used for debug rendering, not something that makes it to end user. At least in the small subset of companies I worked at.
  - selfmodruntime 4y ago
    You are correct. I've also encountered it sometimes in internal business gui wrappers.
  - andersa 4y ago
    I wonder if this is because the default theme for it is somewhat ugly, and most developers aren't designers to make it look better. It is perfectly capable of rendering standalone applications, if you want it to...
  - marginalia_nu 4y ago
    ImGui is amazing to work with though. Like holy fuck is it pleasant compared to basically every other UI-development paradigm ever in the history of user interfaces.
    - jspdown 4y ago
      Yes but that's because it's an immediate mode renderer. It feels intuitive and that's why it's the de facto standard for building debug tooling. But this intuitiveness comes at a price! No matter how hard you try, this will always be slower and computationally heavy compared to retained mode. Great for some stuff, terrible choice for some other.
      - legosexmagic 4y ago
        Your mixing up terms. immediate mode rendering and immediate mode gui are unrelated. immediate mode gui libraries generally dont use immediate mode rendering because its realy slow. immediate mode gui tends to be faster in practice because you just naturally end up with less code. also its very easy to write a retained mode ui ontop of an immediate mode gui library.
- sieabahlpark 4y ago
  [dead]
- ohgodplsno 4y ago
  As for text rendering, Slug [0] has existed for much more than ten years, and is pretty much the gold standard for GPU text rendering. [0] https://sluglibrary.com/ https://sluglibrary.com/
  - bobajeff 4y ago
    Thanks, I'm reading the paper* linked to on that site explaining the technique used in the library. It's encouraging me more to pursue implementing a 2D render on the GPU. I'm also inspired by a recent talk about gkurve**. * https://jcgt.org/published/0006/02/02/ https://jcgt.org/published/0006/02/02/ ** https://m.youtube.com/watch?v=QTybQ-5MlrE https://m.youtube.com/watch?v=QTybQ-5MlrE
    - ohgodplsno 4y ago
      Do note that as far as I know, this technique is patent-encumbered, at least in the US.
      - bobajeff 4y ago
        Thanks. I wonder what the patents cover. With all the work going into 2D rendering on the GPU I'd imagine others discovering similar methods.
- gabereiser 4y ago
  The absolutism of some of their statements when we have 30 years of GPU research at our disposal is pretty eye-opening. I get that maybe this stuff didn't exist as a crate for rust but c'mon! Splines and shapes, glyph rendering, I wrote a game engine in C# back in 2007 that did all these things, and more. I like the explanation and the breakdown of SDFs for primitives but they are standing on the shoulders of giants and act like they're on an island.
- fallat 4y ago
  I'm not even sure it was wise to use SDFs to draw some shaded rounded rectangles.
  - kevingadd 4y ago
    I use sdfs for my unified rasterization in my UI library and they definitely are not the optimal method from a performance standpoint. I've had to do a lot of tricky optimizations to get acceptable performance on low spec hardware. I mostly stick with them because the flexibility and image quality feel worth it.
- Jasper_ 4y ago
  When we talk about 2D graphics as a research problem, we're talking about native rendering of splines and strokes. JUCE does not have GPU-accelerated splines, it flattens the path to lines and rasterizes the coverage area into a texture that then gets uploaded to the GPU: https://github.com/juce-framework/JUCE/blob/2b16c1b94c90d0db3072f6dc9da481a9484d0435/modules/juce_graphics/geometry/juce_PathIterator.cpp https://github.com/juce-framework/JUCE/blob/2b16c1b94c90d0db... https://github.com/juce-framework/JUCE/blob/2b16c1b94c90d0db3072f6dc9da481a9484d0435/modules/juce_graphics/geometry/juce_EdgeTable.cpp https://github.com/juce-framework/JUCE/blob/2b16c1b94c90d0db... It also does stroke handling on the CPU: https://github.com/juce-framework/JUCE/blob/2b16c1b94c90d0db3072f6dc9da481a9484d0435/modules/juce_graphics/geometry/juce_PathStrokeType.cpp https://github.com/juce-framework/JUCE/blob/2b16c1b94c90d0db... Basically, this isn't really "GPU accelerated splines". It's a CPU coverage rasterizer with composting handled by the GPU.
  - fxtentacle 4y ago
    You linked to the software fallback renderer which can be used for cross-platform compatibility. But JUCE also has platform-specific rendering modules. CoreGraphicsContext::createPath will convert the CPU spline segments to CG spline segments which are then rasterized by CoreGraphics using Metal on the GPU. https://github.com/juce-framework/JUCE/blob/2b16c1b94c90d0db3072f6dc9da481a9484d0435/modules/juce_graphics/native/juce_mac_CoreGraphicsContext.mm#L743 https://github.com/juce-framework/JUCE/blob/2b16c1b94c90d0db... And on Windows 7 and up it'll use the hardware-accelerated Direct2D APIs: https://github.com/juce-framework/JUCE/blob/2b16c1b94c90d0db3072f6dc9da481a9484d0435/modules/juce_graphics/native/juce_win32_Direct2DGraphicsContext.cpp#L40 https://github.com/juce-framework/JUCE/blob/2b16c1b94c90d0db...
    - Jasper_ 4y ago
      You mentioned using the OpenGL context, and this code is used by the OpenGL context. CoreGraphics does not use the GPU. Direct2D uses a approach which tesselates paths into triangles on the CPU. It is similar to the JUCE code in that the GPU is used for coverage, but it still does not natively render splines on the GPU.
- qbasic_forever 4y ago
  Google has done a ton of work on GPU accelerated vector graphics too for Android and other platforms. Their skia library is pretty nice: https://skia.org/ https://skia.org/
  - fxtentacle 4y ago
    Oops I forgot to mention that one. They also have a pretty interesting cross-platform GUI framework using it: https://flutter.dev/ https://flutter.dev/
    - EddieRingle 4y ago
      And what was learned from Flutter has produced something I consider even more interesting: Compose UI (https://developer.android.com/jetpack/compose https://developer.android.com/jetpack/compose) and JetBrains' Compose Multiplatform (https://www.jetbrains.com/lp/compose-mpp/ https://www.jetbrains.com/lp/compose-mpp/)
- EngManagerIsMe 4y ago
  Meanwhile, the gaming world is moving to HTML/CSS/JS for game UI in many cases.
  - TylerE 4y ago
    I know of at least one shippping commercial game - in 2023 - that renders the UI using Adobe Flash.
    - EngManagerIsMe 4y ago
      That's super common. It was more common historically, these days it's largely gone away. (Scaleform was the technology that used flash for UI dev)
- kevingadd 4y ago
  At least as far as dear imgui is concerned, it's an amazing library for dev tools but the accessibility story is dire and it has poor support for high quality typography, so it's a non starter if you're serious about good UI. This is not to diminish the quality work that went into the library, but I passed over it for those and other reasons. I used nuklear for a bit since its typography was extensible but in the end it was too hard to overcome its other issues, so I use a custom mixed immediate/retained library now with rich text and accessibility support.
  - mwcampbell 4y ago
    > a custom mixed immediate/retained library now with rich text and accessibility support. I'm always interested in learning about GUI toolkits that have accessibility support. What level of accessibility support does yours have? e.g. does it implement platform accessibility APIs, and if so, on which platforms? Is your library open source? If not, can I see it in action in any commercially available product? Thanks.
    - kevingadd 4y ago
      I haven't found good platform accessibility API bindings for C#, so right now it just has narration support, adjustable text size/contrast, and full keyboard navigation. I'm hoping that later on in my project's development process I'll have the budget to hire someone to try and fully integrate with the platform APIs - it's designed so it will be possible by pushing updates to the retained model. The way it works is that it has an immediate mode API (see https://github.com/sq/Libraries/blob/0ca01d949e3df5fabb1440de3dbc437ae6ccca9c/Squared/PRGUI.Demo/Game.cs#L1334 https://github.com/sq/Libraries/blob/0ca01d949e3df5fabb1440d... for a simple example of the API) and then under the hood it uses a lightweight retained-mode graph, which means that when it's more convenient you can just write more traditional classes and components. In practice most of the UI I've written for my main project is a mix of both models, like for example an editor popup window uses IMGUI-style API while the items in a virtual listbox are represented by a small custom component. It does have a full imgui-style layout engine under the hood instead of doing retained-mode layout, so it's able to fully re-generate layout from scratch every frame, and to minimize page tearing there is a system where components can request a second relayout pass (typically used for things like text autosize). Here's a more complex mixed-mode example from one of my development tools: https://gist.github.com/kg/6a6ba42d5019b546858a2b18751de019 https://gist.github.com/kg/6a6ba42d5019b546858a2b18751de019 Almost all of the on-screen elements in this footage are either immediate-mode or retained-mode UI using the library: https://www.youtube.com/watch?v=ey3FtFWxbhA https://www.youtube.com/watch?v=ey3FtFWxbhA
- bsder 4y ago
  > But that technique looks an awful lot like the 2018 GPU renderer from KiCad ;) And lastly, that glyph atlas for font rendering is only 1 channel? KiCad uses a technique using RGB for gradients so that the rendered glyphs can be anti-aliased without accidentally rounding sharp corners. Is this stuff written down somewhere other than the code? I use KiCad all the time and certainly never noticed this.
s9w 4y ago
[dead]
maeln 4y ago
While I do enjoy a nice and smooth gpu-accelerated ui, I never use a gpu-ui framework for my own project for one simple reason: Almost none of them properly support accessibility. Electron (and in general the web), despite its sluggishness has a very good support for accessibility. Most "traditional" native ui toolkit also do. That would be my advice to anyone making a gpu-accelerated ui library in 2023: Try to support accessibility, and even better: make it a first class citizen.
- bruce343434 4y ago
  If you want to do this, where can you start? What are some patterns for making code that's not too spaghetti when you have to handle tabbing, focus, layout, speech of element contents, the actual hierarchy of the elements etc? Are there standardized OS accessibility API hooks or something?
  - Gigachad 4y ago
    I’ve heard it’s hard to even work this out as all of the screen reader tools are expensive, proprietary, and there are no standards. The typical way is to just make your program, and if it gets popular, the screen reader companies will find a way to make their product work.
    - lukastyrychtr 4y ago
      That's partially true, but fortunately not completely. There are widely use open-source screen readers for Windows and, of course, there's no proprietary screen reader on Linux. And, definitely, there are standard APIs which are used to communicate the accessibility tree between an app and a screen reader. Yes, they are specific for each platform, and Windows has multiple of these, but they are standardized at least for each platform.
    - illiarian 4y ago
      > work this out as all of the screen reader tools are expensive, proprietary, and there are no standards. ARIA is a good start, and screen readers built into the OS are a good start. Moreover, major OSes have accessibility APIs that screen readers will use: - MacOS https://developer.apple.com/library/archive/documentation/Accessibility/Conceptual/AccessibilityMacOSX/index.html https://developer.apple.com/library/archive/documentation/Ac... - Windows: https://learn.microsoft.com/en-us/windows/apps/develop/accessibility https://learn.microsoft.com/en-us/windows/apps/develop/acces...
yason 4y ago
The bottleneck of UI is not the rendering. A measly 60 fps is plenty fast for UI that feels immediate. We had this in the 90's with software rendering, you don't need a GPU for that today. What causes user interfaces to hick up is that it's too easy to do stuff in the main UI thread. First it doesn't matter but stuff does accumulate, and eventually the UI begins to freeze briefly for example, after you press a button. The user interface gets intermingled with the program logic, and the execution of the program will visibly relay its operations to the user. It would be very much possible to keep the user interface running in a thread, dedicated to a single CPU on a priority task, updating at vsync rate as soon as there are dirty areas in the window, merely sending UI events to the processing thread, and doing absolutely nothing more. This is closer to how games work: the rendering thread does rendering and there are other, more slow-paced threads running physics, simulation, and game logic at a suitable pace. With games it's obvious because rendering is hard and it needs to be fast so anything else that might slow down rendering must be moved away but UIs shouldn't be any different. An instantly reacting UI feels natural to a human, one that takes its time to act will slow down the brain. But you don't need a GPU for that.
- DeathArrow 4y ago
  I think the bottleneck comes from updating each UI element instead updating them in batches and updating elements that don't need to be updated.
  - kaba0 4y ago
    That’s just retained mode GUIs calculating what got damaged and only updating those. That’s how most GUI frameworks work since many decades.
- eptcyka 4y ago
  It's significantly cheaper to render a fullscreen window on the GPU than it is on the CPU if you're running at 2560x1600 or 4K. To push that many pixels, you have to send significant amounts of data (framebuffers) at 60 FPS to the GPU anyway - which is no small feat and is bound to eat into the battery. It's just more efficient to run on the GPU.
deleted 4y ago
[deleted]
DeathArrow 4y ago
What's wrong with using platform APIs? I think that by 2023 most UI toolkits provided by the OS are hardware accelerated.
- nicoburns 4y ago
  It’s hard to make a cross-platform UI that way.
  - DeathArrow 4y ago
    But this UI is not cross platform either, as it is still using proprietary APIs.
  - pjmlp 4y ago
    That is what wrappers and platform plugins are for, no need to build a full blown API from scratch.
    - pornel 4y ago
      Such solutions are often in tension between using only the lowest common denominator, and the code having different implementation for each platform anyway. For example, their editor has tabs for editor buffers. Cocoa has a static tabbed widget, which has wrong look and odd UX for this. Cocoa also has a tabbed window type, which isn't a widget you can control. I imagine it'd be hard to abstract that away to work consistently with how Windows does tabbed views. I also haven't seen Windows' tabs being draggable, so that would probably need special DIY solution for Windows which Cocoa tabbed windows don't need. Anyway, I think native UI toolkits are dying. For most people the Web is their most familiar "toolkit" now, and native platforms instead of fighting that back with clear consistent design, went for flat design and multiple half-assed redesigns that messed up all remaining expectations of how "native" looks and feels.
      - pjmlp 4y ago
        There are no perfect solutions, but there is possible balance where OS features are abstracted into higher level concepts, not 1:1. For very contrived example, the cross platform settings panel widget exposes the business API required for settings in general, while the platform specific code takes care of using the host platform concepts to display and manage application specific settings. While it is more work than lowest common denominator approach, it is still much less than re-inventing the wheel. As proven by mobile OS platforms, there are still hope for native toolkits. Lets see how much long the Web will hold, now that the revenge of plugins is here thanks WASM.
DeathArrow 4y ago
According to Wiki, the technique was invented in 2005 by Casey Muratori: https://en.wikipedia.org/wiki/Immediate_mode_GUI https://en.wikipedia.org/wiki/Immediate_mode_GUI
- malf 4y ago
  That’s odd. I remember using it in the 90s. It’s the obvious way to do things if you don’t like small objects. (Also GPUI seems to be retained, so completely unrelated.)
- Hikikomori 4y ago
  Casey ranting about how slow visual studio is and compares it to RemedyBG that uses Dear Imgui (an implementation of Immediate mode GUI). https://www.youtube.com/watch?v=GC-0tCy4P1U https://www.youtube.com/watch?v=GC-0tCy4P1U
- KptMarchewa 4y ago
  Some very dedicated fan must have added this. I can see 2001 Java doc about some immediate mode classes: https://nick-lab.gs.washington.edu/java/jdk1.5b/guide/2d/spec/j2d-image.html https://nick-lab.gs.washington.edu/java/jdk1.5b/guide/2d/spe...
xlii 4y ago
That's exactly the rabbit hole I'm in. I love immediate feedback but getting it ranges from hard to neigh impossible. E.g. I have a complex Emacs setup for rendering Pikchr diagrams, but there are a lot of problems to solve from diagram conception to the end result, so I thought, hey, why not make my own cool RT editor - in Rust obviously. Unfortunately I learned that GUIs are though problem especially if idea is hobby-based so there's only one developer inside. Ultra responsive GUIs cool, I have a prototype in egui (not sure if that's as fast as Zed's premise but feels fast nonetheless) and yet it doesn't support multiple windows, which I wanted to have. 120 FPS with direct rendering sounds AWESOME just for sake of it, but I believe that for the end-user layout will be more important than refresh rate, and that's different beast to tame. Personally I "almost" settled for Dioxus (shameless plug: [1], there's link to YT video) and I'm quite happy with it. Having editor in WebView feels really quirky though (e.g. no textareas, I'm intercepting key events and rendering in div glyph-by-glyph directly). [1]: https://github.com/exlee/dioxus-editor https://github.com/exlee/dioxus-editor
- as-cii 4y ago
  Hey xlii! This is Antonio, author of the post. You're right that rendering is only part of the story. To stay within the ~8ms frame budget, however, every little bit counts. Maintaining application state, layout, painting, and finally pushing pixels to screen, all need to be as performant as they can be. For layout specifically we're using an approach inspired by Flutter, which lets us avoid complex algorithms but still have a lot of flexibility in the way elements can be positioned and produce a rich graphical experience. Thanks for reading and commenting!
  - xlii 4y ago
    I don't have experience with Flutter, but based on quick glance they're using widgeting and, what I found quite important - ability to develop GUI outside of the application. Something that I think libraries like egui are missing (and which is easily obtainable with Tauri/Dioxus). Rebuilding whole app to ensure that some box doesn't get cut off ruins development experience, especially for big apps. Kudos to you guys, I hope you'll make Zed extensible, so that instead of writing my own editor I can use yours ;-)
almostdigital 4y ago
Looking forward to trying this, VSCode is great but I really miss the performance of Sublime Text. I hope they get the plugin system right, killer feature would be if it could load VSCode plugins (incredibly hard to pull off, yes)
- as-cii 4y ago
  Thanks, almostdigital! After our past experience with Atom, getting the plugin system right is a top priority for the editor. The thought of cross compatibility with VSCode plugins definitely crossed our mind and it's not out of the question, although our current plan is to initially support plugins using WASM.
soulbadguy 4y ago
I am curious how this would compare to a ui written in flutter. It seems that fluter is also hardware accelerated and x-platform
- doodlesdev 4y ago
  In terms of performance it would possibly be even better in Flutter, BUT (and this is a big butt) text editing on the desktop in Flutter currently can only be described as broken. I last looked at this a few months ago and I think it's fixable, but as much as I like Flutter I don't think it's a good option for a text editor _just yet_.
RonnieOwnsLexus 4y ago
Why do we need a code editor with 120FPS support ?
- Strom 4y ago
  Because competing code editors support it and thus lacking this capability would make your code editor seem unusually incompetent.
  - RonnieOwnsLexus 4y ago
    I am sorry but i tried googling and binging which code editor supports 120fps. I also asked bing ai and it said it is not aware. Could you please point me here as to what I am missing ?
    - Strom 4y ago
      notepad.exe on Windows for example. If that's too lightweight, then VSCode is a more heavier example.
nottorp 4y ago
I don't understand. Why would you need to render a user interface constantly at 120 fps, instead of just updating it when something changes? Laptop batteries last too long these days? Electricity too cheap?
- sdflhasjd 4y ago
  "Because it looks good" is probably the most popular reason.
  - nottorp 4y ago
    But if nothing changes it looks as good at zero fps :) Edit: Yay, it's a text editor. What happened to only redrawing the line that's being edited and the status indicators?
- as-cii 4y ago
  Hey nottorp. Antonio here, author of the post. Zed and GPUI use energy very judiciously and only perform updates when needed. The idea is that we can render the whole application within ~8ms, and that shows everywhere: from typing a letter to scrolling up and down in a buffer. However, if the editor is sitting there idle, we won't waste precious CPU cycles. Thanks for the feedback!
  - nottorp 4y ago
    Yeah, might want to edit the title a bit. Or not, considering these concepts are getting lost. I mean, the win16 api from ages ago had support for invalidating specific screen regions etc. It probably got lost somewhere in the transition to javascript...
  - rubymamis 4y ago
    Will you allow developers access to your GUI framework? What about open-sourcing it?
- flohofwoe 4y ago
  It's not about rendering static screens at 120Hz, but rendering anything that's animated at a smooth 120Hz.
jug 4y ago
Wow, that’s some low level stuff. Most would just use an established UI framework because rendering performance is left to the window manager. I’m not sure I understand the need to go about it like this? Windows is not considered the epitome of performant interfaces but it has no trouble rendering UI’s at 120 fps. When people go and buy a 120 fps display, they are wowed by the smooth scrolling in a heavy application like Google Chrome. The window manager is already hardware accelerated (as for Windows since Vista) and the apps draw widgets on their surface.
- smoldesu 4y ago
  It's so weird. The devs of the Warp based terminal are doing something similar (Rust for single-platform low-level dev), and I'm also not sure what the point is. It feels like they're banking on Rust being a selling point, but forgot that the lang can drastically lower your iteration speed when it comes time to compete with other editors.
- ben-schaaf 4y ago
  > Windows is not considered the epitome of performant interfaces but it has no trouble rendering UI’s at 120 fps. When people go and buy a 120 fps display, they are wowed by the smooth scrolling in a heavy application like Google Chrome. Windows uses GPU-based rendering in most/all of their GUI frameworks. Chrome also makes heavy use of hardware acceleration. Rendering performance isn't left to the window manager. If you're making your own GUI this is exactly the kind of stuff you need to do to make it fast.
tgflynn 4y ago
What API are they using for interfacing with the GPU (ie. OpenGL, Vulkan, other ) ? I suspect a lot of time is likely to be spent on the CPU side updating vertex and other data and pushing it to the GPU so it would be useful to have some more detail on how they are handling that.
- tcldr 4y ago
  I saw a few Metal specific types in the source code contained within their demo video
avereveard 4y ago
I cannot stress how much I do not want my 300w gpu to be used to render text that change at most three times per second. And it's not just about electricity cost and heat stress, it will conflict with everything else that requires the gpu to do stuff, including watching 4k videos on the second monitor, which does have a legitimate case for requiring hardware acceleration since they move a lot of data 60 times per second, and your editor doesn't. And the limited resource is not the gpu itself but the nearby onboard memory is a scarce resource on its own. I'd be real mad at a software that prevents me to multitask.
- alexfromapex 4y ago
  This UI is so lightweight it seems like they should easily be able to toggle the GPU compute on or off
- ohgodplsno 4y ago
  I have horrible news to tell you about compositors: they already do use your GPU, even if you're watching a 4K video on the side. Even when requesting a full screen swapchain to avoid having to deal with the compositor, modern OSes will force you to go through that compositor and lie about it being fullscreen. Additionally, using your GPU doesn't mean locking your GPU on that task. Your 4K video most likely isn't taking up 100% of GPU time, nor 100% of the nearby onboard memory. Everyone already gets a share of that time.
  - avereveard 4y ago
    Compositor and ui framework that interact with them know that 120fps is not the target but that frame render time is, and know not to have active rendering contexts for static resources. Chasing 120fps rendering here is the problem.
  - rafram 4y ago
    > Even when requesting a full screen swapchain to avoid having to deal with the compositor, modern OSes will force you to go through that compositor and lie about it being fullscreen. I’m honestly thankful for this bit of lying. Windows 8 and below had horrible, horrible issues with fullscreen programs, especially games - they’d change your screen resolution and thus move and resize all your other windows, they’d freeze for ten seconds when you tried to alt-tab out, and they’d frequently just crash when you switched back to them. Those problems are essentially nonexistent nowadays, and it’s worth the small performance cost that comes from going through the compositor.
rafram 4y ago
[flagged]
- acdha 4y ago
  Why are you reacting so emotionally to something that far down the page? They’re talking about being aggressively multi-core and given their technical audience it hardly seems inappropriate to have that detail along with the other details like how they use GPU acceleration and the benefits of lower input latency.
  - rafram 4y ago
    I wouldn’t say I’m reacting emotionally, or that where it is on the page is relevant. It’s just a funny part of their marketing copy.
sebastianconcpt 4y ago
I'm intrigued. What's the applicability that you see for this?
fassssst 4y ago
Erm, native WinUI apps are GPU accelerated and render at vsync.
- doodlesdev 4y ago
  Also GTK4 is GPU accelerated whenever possible, with really well mantained Rust bindings for it I think the only thing missing would be macOS which I'm not sure what solutions are there. Another option in this front could be Flutter if write-once run-everywhere is a need for the project. Another advantage is that it's not only GPU accelerated but it's also retained mode.
- kridsdale1 4y ago
  So is everything written in Apple native UI frameworks since 2009.
  - kllrnohj 4y ago
    And Android and QT and GTK and Chrome and Firefox and etc... The article is talking about the same generic hybrid GPU-accelerated rendering architecture that everything uses. Seemingly the only "new" part is "in Rust!"
- pjc50 4y ago
  Nobody loves native WinUI, not even Microsoft. Which is kind of a shame. But it's the result of years of product management neglect as well as the pull away from desktop UIs to web UIs.
  - pjmlp 4y ago
    They only have themselves to blame, after the rewrites their forced their hardest advocates to go through, each one with worse tooling, dropping the UI designer, .NET Native and C++/CX along the way. Native AOT still can't compile WinUI, while C++/WinRT is like doing ATL in 2000, while bug issues grow exponentially. Only WinDev themselves, and WinUI MVPs, can still believe this is going somewhere. The rest of us have moved on.
scotty79 4y ago
I hoped they'll go for signed distance functions for glyph rendering as well. Rendering text with shaders is fascinating.
flohofwoe 4y ago
It's surprising that they jump immediately from the problem description into shader details. IME the main theme with achieving high performance not just in games, and not just in rendering, is to avoid frequent 'context switches' and instead queue/batch a lot of work (e.g. all rendering commands for one frame) and then submit all this work at once "to the other side" in a single call. This is not just how modern 3D APIs work, but is the same basic idea in 'miracle APIs' like io_uring This takes care of the 'throughput problem', but it can easily lead to a 'latency problem' (if the work to be done needs to travel through several such queues which are 'pipelined' together).
p0nce 4y ago
Fast software rasterizers are not slow at drawing text in the first place.
monkeydust 4y ago
What's the real world client experience of developing UIs to render @ 120FPS - is it like once you have tried it going back is really hard?
FpUser 4y ago
This guy Bero1985 wrote the 3D library / engine some years that has extensive 2D features including UI that uses SDF among the other things [0]. [0] - https://github.com/BeRo1985/pasvulkan https://github.com/BeRo1985/pasvulkan
29athrowaway 4y ago
You cannot see at 120 fps.
- Etherlord87 4y ago
  You're right, you can't see at 120 fps, because that's not how human vision works. You can't see at any fps. However the more fps there are, the sooner you get the information, and so regardless of the lag on your end (the biology behind the eyes, the brain, and the connection between them), you still add some delay coming from your computer.
- jamesgeck0 4y ago
  Check out https://www.testufo.com/ https://www.testufo.com/ on a 120Hz display; the higher FPS is absolutely perceptible.
- tayistay 4y ago
  If you're doing a drawing app, then 120fps helps you keep the stroke close to the stylus. Even then it may lag a bit because of a few frames of GPU latency and you have to do predictive stroke points and then adjust your stroke later. See https://developer.apple.com/documentation/uikit/touches_presses_and_gestures/handling_touches_in_your_view/minimizing_latency_with_predicted_touches https://developer.apple.com/documentation/uikit/touches_pres... Similarly, if you're dragging something around with your finger, 120fps keeps that thing closer to the finger. Just improves the UX.
computing 4y ago
First, this looks awesome. Can't wait to try zed. Second, forgive a naive question since I know nothing about graphics, but would the method described in the article perform better than Alacritty + Neovim?
btryder 4y ago
Lots of people nit-picking the 120 FPS but I think Zed looks super promising. The native support for collaborative editing looks fantastic, and I'm excited to try it out. Curious if you guys have thought about VR / AR possibilities with GPUI?
- exo-cortex 4y ago
  That would be interesting in VR to have syntax highlighting that moves variable names towards the viewer, or puts context menus with possible methods of some object above. Would probably just be stupid gimmick, but I want to see somebody do it :-D
  - rikarendsmp 4y ago
    I tried this, using https://makepad.dev https://makepad.dev our GPU accelerated UI and renderstack. And unfortunately it wasn't a great experience. Text popping forward for whatever reason is not really an improvement (i tried indent depth, syntax highlighting reasons, cursor behavior). Maybe 'veeeeery' subtly could do something, but otherwise you dont want it to break visual symmetry as we are used to
trollied 4y ago
Previous post: https://news.ycombinator.com/item?id=35057642 https://news.ycombinator.com/item?id=35057642
Animats 4y ago
This seems like the wrong portion of the problem on which to spend time. This is a text editor. Performance problems with text editors tend to involve long files and multiple tabs. Refresh speed isn't the problem, although keyboard response speed can be. I'd like to see "gedit", for Linux, fixed. It can stall on large files, and, in long edit sessions, will sometimes mess up the file name in the tab. Or "notepad++" for Linux.
msvan 4y ago
Lots of negativity in here. I for one am excited about the prospect of an editor that is as responsive as I remember Sublime being back in the day, with the feature set I've come to expect from VS Code. An editor like this simply does not exist today, and betting on the Rust ecosystem is entirely the right choice for building something like this in 2023.
- jakswa 4y ago
  Here here. I backed Onivim hoping it was going to shine a light in the darkness, and it seemed promising, but ultimately was abandoned? I think, unsure.
haberman 4y ago
This sounds really similar to the story we were hearing about Servo back in 2016: https://www.youtube.com/watch?v=erfnCaeLxSI https://www.youtube.com/watch?v=erfnCaeLxSI I was really excited when I saw that demo. Why didn't this turn into a final product that people could use?
- deleted 4y ago
  [deleted]
tiffanyh 4y ago
Nathan Sobo It should be noted that the main person behind Zed is Nathan Sobo, who created Atom while he was at Github, which is the basis of Visual Studio Code today. As such, I have high hopes Zed will be a much faster version of Visual Studio Code and am excited to see what him & his team make.
- 3836293648 4y ago
  Well, he created electron for Atom. Atom itself was never a part of vsc. VSC was targetting the browser for a while before electron came along
tayistay 4y ago
My rui library can render UIs at 120fps, uses similar SDF techniques (though uses a single shader for all rendering): https://github.com/audulus/rui https://github.com/audulus/rui Is their GPUI library open source?
- inamberclad 4y ago
  Sadly I didn't see any links.
bjconlan 4y ago
Beyond the rendering which as noted is nothing that hasn't been done before (in general) the inherent OT/multi user + tree sitter functionality is something that entices me. I'm surprised nobody pointed out lite/litexl here either it's rendering of ui is very similar (although fonts are via a texture; like a game would) and doesn't focus overly on the GPU but optimises those paths like games circa directx9/opengl 1.3 There are great details of the approach taken with lite at https://rxi.github.io https://rxi.github.io Lite-xl might have evolved the renderer but the code here is very consumable for me.

259 points by rck 4y ago

"Inspired by the gaming world, we realized that the only way to achieve the performance we needed was to build our own UI framework" I'm surprised you did not look at "Dear ImGui", "Noesis", and "JUCE". All three of them are heavily used in gaming, are rather clean C++, use full GPU acceleration, and have visual editors available. Especially JUCE is used for A LOT of hard-realtime professional audio applications. "When we started building Zed, arbitrary 2D graphics rendering on the GPU was still very much a research project." What are you talking about? JUCE has had GPU-accelerated spline shapes and SVG animations since 2012? BTW, I like the explanations for how they use SDFs for rendering basic primitives. But that technique looks an awful lot like the 2018 GPU renderer from KiCad ;) And lastly, that glyph atlas for font rendering is only 1 channel? KiCad uses a technique using RGB for gradients so that the rendered glyphs can be anti-aliased without accidentally rounding sharp corners. Overall, this reads to me like they did not do much research before starting, which is totally OK, but then they shouldn't say stuff like "did not exist" or "was still a research project".

deleted 4y ago
[deleted]
wnkrshm 4y ago
It's not the first text editor either, Sublime has GPU-accelerated rendering.
- fxtentacle 4y ago
  Also, looking at their own performance numbers, it appears that all of this only managed to reduce the character insertion latency by 21% when compared to CLion. That's still useful, but maybe not even noticeable to the user.
  - gizmo 4y ago
    That latency can’t be render latency, because it’s trivial. It has got to be related to unifying the text buffer with the syntax tree. Otherwise you’ll get wrong syntax highlighting for a couple of frames and that’s jarring.
swatcoder 4y ago
I agree with your final statement and the seeming lack of research, but are you sure your characterization of JUCE is accurate? You’d surprise a lot of people if you were right about it using GPU acceleration for its UI framework. It does have an OpenGL container that fits into its view object hierarchy if you want to write your own accelerated component, but the rest of the UI is pretty much standard event-loop -> platform-API dispatch. It’s accelerated where the underlying platform-API is accelerated but not in any kind of explicit or fine-tuned way. The focus has always been more on portability than performance, even on the audio side. (And of course the high-priority audio processing runs in an entirely segregated context from the UI, so the performance characteristics of the two pieces are decoupled anyway.)
- coldtea 4y ago
  >I agree with your final statement and the seeming lack of research, but are you sure your characterization of JUCE is accurate? That JUCE is "heavily used in gaming" for starters is widly innacurate
  - fxtentacle 4y ago
    I personally know multiple people using it in their games and game-related tooling and Roli (the company selling JUCE) even has an official Unreal Engine 4 plugin. So to me, it appears to be widely used.
    - irh 4y ago
      > Roli (the company selling JUCE) I know it's tangential to the point that you're making, but JUCE was acquired by PACE in 2020.
    - swatcoder 4y ago
      JUCE is sold by Raw Material Software, which was purchased by PACE Anti-piracy, which is part of Avid. There may be some people you know using JUCE for games, but it’s an odd choice. Trying to figure out what you might mean, JUCE was involved with the BLOCKS SDK for Roli’s hardware peripherals, but again, that’s not really about gaming and definitely not about any familiar kind of gaming. That said, I’m sure you’re relating the best information you have and very likely just transposed some detail by accident and are now caught in a relentless, exhausting online nitpick. We’ve all been there. If you figure out what it was, I’d love to hear it!
rob74 4y ago
Maybe you need to add an "(in Rust)" to all these sentences? Sure there are C++ frameworks, but they probably wanted a pure Rust UI framework? My 2 cents: it's nice to have smooth rendering in your editor, but I'm currently mostly using a Java-based IDE (IDEA family), and it's responsive enough for my taste. If I were to use the current prototype of their editor, I'm afraid the usage of fixed-width fonts all over the place (which I assume can be fixed, but may also be due to constraints of the UI framework?) would probably bother me more than the 120 FPS UI would impress me (assuming that I had a monitor that was fast enough, which I don't). On the plus side, it sounds like they support sub-pixel rendering - if that's the case, kudos to them!
- DeathArrow 4y ago
  Not true, even if you add Rust. egui was released earlier: https://github.com/emilk/egui https://github.com/emilk/egui And egui isn't tied to Apple proprietary frameworks.
  - littlestymaar 4y ago
    Afaik egui isn't doing any kind of the fancy GPU based 2D graphics this blog post is about though.
    - pama 4y ago
      It uses glow, GL on whatever, so OpenGL when it exists.
      - littlestymaar 4y ago
        That doesn't mean it's doing any kind of GPU acceleration of 2D graphics. Given that they don't talk about it in the README, I'm pretty sure they don't.
        decremental 4y ago
        I use egui. The library itself is agnostic but relies on backend libraries for rendering, all of which (or at least the official ones) render on the GPU.
littlestymaar 4y ago
> What are you talking about? JUCE has had GPU-accelerated spline shapes and SVG animations since 2012? I'm not familiar at all with JUCE, but the state of the art in GPU-accelerated 2D graphics has dramatically improved over the past 10 years so I doubt what JUCE did in 2012 is really comparable.
- fxtentacle 4y ago
  I shipped a JUCE-based app in 2012 and back then it was "good enough" to have GPU-accelerated rendering of SVG icons moving around in realtime with antialiasing. Since we used their OpenGL context, we could even render the GUI inside customers' games for debug visualization.
dorkwood 4y ago
> KiCad uses a technique using RGB for gradients so that the rendered glyphs can be anti-aliased without accidentally rounding sharp corners. This is known as a “multi-channel signed distance field”, or “msdf”. https://github.com/Chlumsky/msdfgen https://github.com/Chlumsky/msdfgen
- scotty79 4y ago
  Do you know any text editor that uses it for font rendering?
  - fxtentacle 4y ago
    You would have to use that if you want smooth antialiased zooming in and out of text. BTW, the KiCad schematics are pretty text-heavy. You typically write down all the parameters and IDs of all the electrical components.
  - rikarendsmp 4y ago
    I used to use it, but its extremely font and glyph-sensitive wether or not it works. And if it doesn't work there is no easy fix (or ability to recognise it nonvisually)
incrudible 4y ago
> KiCad uses a technique using RGB for gradients so that the rendered glyphs can be anti-aliased without accidentally rounding sharp corners They are not using SDFs for the text, they render the glyphs at the specified font size directly.
shultays 4y ago
ImGui is at least only used for debug rendering, not something that makes it to end user. At least in the small subset of companies I worked at.
- selfmodruntime 4y ago
  You are correct. I've also encountered it sometimes in internal business gui wrappers.
- andersa 4y ago
  I wonder if this is because the default theme for it is somewhat ugly, and most developers aren't designers to make it look better. It is perfectly capable of rendering standalone applications, if you want it to...
- marginalia_nu 4y ago
  ImGui is amazing to work with though. Like holy fuck is it pleasant compared to basically every other UI-development paradigm ever in the history of user interfaces.
  - jspdown 4y ago
    Yes but that's because it's an immediate mode renderer. It feels intuitive and that's why it's the de facto standard for building debug tooling. But this intuitiveness comes at a price! No matter how hard you try, this will always be slower and computationally heavy compared to retained mode. Great for some stuff, terrible choice for some other.
    - legosexmagic 4y ago
      Your mixing up terms. immediate mode rendering and immediate mode gui are unrelated. immediate mode gui libraries generally dont use immediate mode rendering because its realy slow. immediate mode gui tends to be faster in practice because you just naturally end up with less code. also its very easy to write a retained mode ui ontop of an immediate mode gui library.
sieabahlpark 4y ago
[dead]
ohgodplsno 4y ago
As for text rendering, Slug [0] has existed for much more than ten years, and is pretty much the gold standard for GPU text rendering. [0] https://sluglibrary.com/ https://sluglibrary.com/
- bobajeff 4y ago
  Thanks, I'm reading the paper* linked to on that site explaining the technique used in the library. It's encouraging me more to pursue implementing a 2D render on the GPU. I'm also inspired by a recent talk about gkurve**. * https://jcgt.org/published/0006/02/02/ https://jcgt.org/published/0006/02/02/ ** https://m.youtube.com/watch?v=QTybQ-5MlrE https://m.youtube.com/watch?v=QTybQ-5MlrE
  - ohgodplsno 4y ago
    Do note that as far as I know, this technique is patent-encumbered, at least in the US.
    - bobajeff 4y ago
      Thanks. I wonder what the patents cover. With all the work going into 2D rendering on the GPU I'd imagine others discovering similar methods.
gabereiser 4y ago
The absolutism of some of their statements when we have 30 years of GPU research at our disposal is pretty eye-opening. I get that maybe this stuff didn't exist as a crate for rust but c'mon! Splines and shapes, glyph rendering, I wrote a game engine in C# back in 2007 that did all these things, and more. I like the explanation and the breakdown of SDFs for primitives but they are standing on the shoulders of giants and act like they're on an island.
fallat 4y ago
I'm not even sure it was wise to use SDFs to draw some shaded rounded rectangles.
- kevingadd 4y ago
  I use sdfs for my unified rasterization in my UI library and they definitely are not the optimal method from a performance standpoint. I've had to do a lot of tricky optimizations to get acceptable performance on low spec hardware. I mostly stick with them because the flexibility and image quality feel worth it.
Jasper_ 4y ago
When we talk about 2D graphics as a research problem, we're talking about native rendering of splines and strokes. JUCE does not have GPU-accelerated splines, it flattens the path to lines and rasterizes the coverage area into a texture that then gets uploaded to the GPU: https://github.com/juce-framework/JUCE/blob/2b16c1b94c90d0db3072f6dc9da481a9484d0435/modules/juce_graphics/geometry/juce_PathIterator.cpp https://github.com/juce-framework/JUCE/blob/2b16c1b94c90d0db... https://github.com/juce-framework/JUCE/blob/2b16c1b94c90d0db3072f6dc9da481a9484d0435/modules/juce_graphics/geometry/juce_EdgeTable.cpp https://github.com/juce-framework/JUCE/blob/2b16c1b94c90d0db... It also does stroke handling on the CPU: https://github.com/juce-framework/JUCE/blob/2b16c1b94c90d0db3072f6dc9da481a9484d0435/modules/juce_graphics/geometry/juce_PathStrokeType.cpp https://github.com/juce-framework/JUCE/blob/2b16c1b94c90d0db... Basically, this isn't really "GPU accelerated splines". It's a CPU coverage rasterizer with composting handled by the GPU.
- fxtentacle 4y ago
  You linked to the software fallback renderer which can be used for cross-platform compatibility. But JUCE also has platform-specific rendering modules. CoreGraphicsContext::createPath will convert the CPU spline segments to CG spline segments which are then rasterized by CoreGraphics using Metal on the GPU. https://github.com/juce-framework/JUCE/blob/2b16c1b94c90d0db3072f6dc9da481a9484d0435/modules/juce_graphics/native/juce_mac_CoreGraphicsContext.mm#L743 https://github.com/juce-framework/JUCE/blob/2b16c1b94c90d0db... And on Windows 7 and up it'll use the hardware-accelerated Direct2D APIs: https://github.com/juce-framework/JUCE/blob/2b16c1b94c90d0db3072f6dc9da481a9484d0435/modules/juce_graphics/native/juce_win32_Direct2DGraphicsContext.cpp#L40 https://github.com/juce-framework/JUCE/blob/2b16c1b94c90d0db...
  - Jasper_ 4y ago
    You mentioned using the OpenGL context, and this code is used by the OpenGL context. CoreGraphics does not use the GPU. Direct2D uses a approach which tesselates paths into triangles on the CPU. It is similar to the JUCE code in that the GPU is used for coverage, but it still does not natively render splines on the GPU.
qbasic_forever 4y ago
Google has done a ton of work on GPU accelerated vector graphics too for Android and other platforms. Their skia library is pretty nice: https://skia.org/ https://skia.org/
- fxtentacle 4y ago
  Oops I forgot to mention that one. They also have a pretty interesting cross-platform GUI framework using it: https://flutter.dev/ https://flutter.dev/
  - EddieRingle 4y ago
    And what was learned from Flutter has produced something I consider even more interesting: Compose UI (https://developer.android.com/jetpack/compose https://developer.android.com/jetpack/compose) and JetBrains' Compose Multiplatform (https://www.jetbrains.com/lp/compose-mpp/ https://www.jetbrains.com/lp/compose-mpp/)
EngManagerIsMe 4y ago
Meanwhile, the gaming world is moving to HTML/CSS/JS for game UI in many cases.
- TylerE 4y ago
  I know of at least one shippping commercial game - in 2023 - that renders the UI using Adobe Flash.
  - EngManagerIsMe 4y ago
    That's super common. It was more common historically, these days it's largely gone away. (Scaleform was the technology that used flash for UI dev)
kevingadd 4y ago
At least as far as dear imgui is concerned, it's an amazing library for dev tools but the accessibility story is dire and it has poor support for high quality typography, so it's a non starter if you're serious about good UI. This is not to diminish the quality work that went into the library, but I passed over it for those and other reasons. I used nuklear for a bit since its typography was extensible but in the end it was too hard to overcome its other issues, so I use a custom mixed immediate/retained library now with rich text and accessibility support.
- mwcampbell 4y ago
  > a custom mixed immediate/retained library now with rich text and accessibility support. I'm always interested in learning about GUI toolkits that have accessibility support. What level of accessibility support does yours have? e.g. does it implement platform accessibility APIs, and if so, on which platforms? Is your library open source? If not, can I see it in action in any commercially available product? Thanks.
  - kevingadd 4y ago
    I haven't found good platform accessibility API bindings for C#, so right now it just has narration support, adjustable text size/contrast, and full keyboard navigation. I'm hoping that later on in my project's development process I'll have the budget to hire someone to try and fully integrate with the platform APIs - it's designed so it will be possible by pushing updates to the retained model. The way it works is that it has an immediate mode API (see https://github.com/sq/Libraries/blob/0ca01d949e3df5fabb1440de3dbc437ae6ccca9c/Squared/PRGUI.Demo/Game.cs#L1334 https://github.com/sq/Libraries/blob/0ca01d949e3df5fabb1440d... for a simple example of the API) and then under the hood it uses a lightweight retained-mode graph, which means that when it's more convenient you can just write more traditional classes and components. In practice most of the UI I've written for my main project is a mix of both models, like for example an editor popup window uses IMGUI-style API while the items in a virtual listbox are represented by a small custom component. It does have a full imgui-style layout engine under the hood instead of doing retained-mode layout, so it's able to fully re-generate layout from scratch every frame, and to minimize page tearing there is a system where components can request a second relayout pass (typically used for things like text autosize). Here's a more complex mixed-mode example from one of my development tools: https://gist.github.com/kg/6a6ba42d5019b546858a2b18751de019 https://gist.github.com/kg/6a6ba42d5019b546858a2b18751de019 Almost all of the on-screen elements in this footage are either immediate-mode or retained-mode UI using the library: https://www.youtube.com/watch?v=ey3FtFWxbhA https://www.youtube.com/watch?v=ey3FtFWxbhA
bsder 4y ago
> But that technique looks an awful lot like the 2018 GPU renderer from KiCad ;) And lastly, that glyph atlas for font rendering is only 1 channel? KiCad uses a technique using RGB for gradients so that the rendered glyphs can be anti-aliased without accidentally rounding sharp corners. Is this stuff written down somewhere other than the code? I use KiCad all the time and certainly never noticed this.

s9w 4y ago

[dead]

maeln 4y ago

While I do enjoy a nice and smooth gpu-accelerated ui, I never use a gpu-ui framework for my own project for one simple reason: Almost none of them properly support accessibility. Electron (and in general the web), despite its sluggishness has a very good support for accessibility. Most "traditional" native ui toolkit also do. That would be my advice to anyone making a gpu-accelerated ui library in 2023: Try to support accessibility, and even better: make it a first class citizen.

bruce343434 4y ago
If you want to do this, where can you start? What are some patterns for making code that's not too spaghetti when you have to handle tabbing, focus, layout, speech of element contents, the actual hierarchy of the elements etc? Are there standardized OS accessibility API hooks or something?
- Gigachad 4y ago
  I’ve heard it’s hard to even work this out as all of the screen reader tools are expensive, proprietary, and there are no standards. The typical way is to just make your program, and if it gets popular, the screen reader companies will find a way to make their product work.
  - lukastyrychtr 4y ago
    That's partially true, but fortunately not completely. There are widely use open-source screen readers for Windows and, of course, there's no proprietary screen reader on Linux. And, definitely, there are standard APIs which are used to communicate the accessibility tree between an app and a screen reader. Yes, they are specific for each platform, and Windows has multiple of these, but they are standardized at least for each platform.
  - illiarian 4y ago
    > work this out as all of the screen reader tools are expensive, proprietary, and there are no standards. ARIA is a good start, and screen readers built into the OS are a good start. Moreover, major OSes have accessibility APIs that screen readers will use: - MacOS https://developer.apple.com/library/archive/documentation/Accessibility/Conceptual/AccessibilityMacOSX/index.html https://developer.apple.com/library/archive/documentation/Ac... - Windows: https://learn.microsoft.com/en-us/windows/apps/develop/accessibility https://learn.microsoft.com/en-us/windows/apps/develop/acces...

yason 4y ago

The bottleneck of UI is not the rendering. A measly 60 fps is plenty fast for UI that feels immediate. We had this in the 90's with software rendering, you don't need a GPU for that today. What causes user interfaces to hick up is that it's too easy to do stuff in the main UI thread. First it doesn't matter but stuff does accumulate, and eventually the UI begins to freeze briefly for example, after you press a button. The user interface gets intermingled with the program logic, and the execution of the program will visibly relay its operations to the user. It would be very much possible to keep the user interface running in a thread, dedicated to a single CPU on a priority task, updating at vsync rate as soon as there are dirty areas in the window, merely sending UI events to the processing thread, and doing absolutely nothing more. This is closer to how games work: the rendering thread does rendering and there are other, more slow-paced threads running physics, simulation, and game logic at a suitable pace. With games it's obvious because rendering is hard and it needs to be fast so anything else that might slow down rendering must be moved away but UIs shouldn't be any different. An instantly reacting UI feels natural to a human, one that takes its time to act will slow down the brain. But you don't need a GPU for that.

DeathArrow 4y ago
I think the bottleneck comes from updating each UI element instead updating them in batches and updating elements that don't need to be updated.
- kaba0 4y ago
  That’s just retained mode GUIs calculating what got damaged and only updating those. That’s how most GUI frameworks work since many decades.
eptcyka 4y ago
It's significantly cheaper to render a fullscreen window on the GPU than it is on the CPU if you're running at 2560x1600 or 4K. To push that many pixels, you have to send significant amounts of data (framebuffers) at 60 FPS to the GPU anyway - which is no small feat and is bound to eat into the battery. It's just more efficient to run on the GPU.

deleted 4y ago

[deleted]

DeathArrow 4y ago

What's wrong with using platform APIs? I think that by 2023 most UI toolkits provided by the OS are hardware accelerated.

nicoburns 4y ago
It’s hard to make a cross-platform UI that way.
- DeathArrow 4y ago
  But this UI is not cross platform either, as it is still using proprietary APIs.
- pjmlp 4y ago
  That is what wrappers and platform plugins are for, no need to build a full blown API from scratch.
  - pornel 4y ago
    Such solutions are often in tension between using only the lowest common denominator, and the code having different implementation for each platform anyway. For example, their editor has tabs for editor buffers. Cocoa has a static tabbed widget, which has wrong look and odd UX for this. Cocoa also has a tabbed window type, which isn't a widget you can control. I imagine it'd be hard to abstract that away to work consistently with how Windows does tabbed views. I also haven't seen Windows' tabs being draggable, so that would probably need special DIY solution for Windows which Cocoa tabbed windows don't need. Anyway, I think native UI toolkits are dying. For most people the Web is their most familiar "toolkit" now, and native platforms instead of fighting that back with clear consistent design, went for flat design and multiple half-assed redesigns that messed up all remaining expectations of how "native" looks and feels.
    - pjmlp 4y ago
      There are no perfect solutions, but there is possible balance where OS features are abstracted into higher level concepts, not 1:1. For very contrived example, the cross platform settings panel widget exposes the business API required for settings in general, while the platform specific code takes care of using the host platform concepts to display and manage application specific settings. While it is more work than lowest common denominator approach, it is still much less than re-inventing the wheel. As proven by mobile OS platforms, there are still hope for native toolkits. Lets see how much long the Web will hold, now that the revenge of plugins is here thanks WASM.

DeathArrow 4y ago

According to Wiki, the technique was invented in 2005 by Casey Muratori: https://en.wikipedia.org/wiki/Immediate_mode_GUI https://en.wikipedia.org/wiki/Immediate_mode_GUI

malf 4y ago
That’s odd. I remember using it in the 90s. It’s the obvious way to do things if you don’t like small objects. (Also GPUI seems to be retained, so completely unrelated.)
Hikikomori 4y ago
Casey ranting about how slow visual studio is and compares it to RemedyBG that uses Dear Imgui (an implementation of Immediate mode GUI). https://www.youtube.com/watch?v=GC-0tCy4P1U https://www.youtube.com/watch?v=GC-0tCy4P1U
KptMarchewa 4y ago
Some very dedicated fan must have added this. I can see 2001 Java doc about some immediate mode classes: https://nick-lab.gs.washington.edu/java/jdk1.5b/guide/2d/spec/j2d-image.html https://nick-lab.gs.washington.edu/java/jdk1.5b/guide/2d/spe...

xlii 4y ago

That's exactly the rabbit hole I'm in. I love immediate feedback but getting it ranges from hard to neigh impossible. E.g. I have a complex Emacs setup for rendering Pikchr diagrams, but there are a lot of problems to solve from diagram conception to the end result, so I thought, hey, why not make my own cool RT editor - in Rust obviously. Unfortunately I learned that GUIs are though problem especially if idea is hobby-based so there's only one developer inside. Ultra responsive GUIs cool, I have a prototype in egui (not sure if that's as fast as Zed's premise but feels fast nonetheless) and yet it doesn't support multiple windows, which I wanted to have. 120 FPS with direct rendering sounds AWESOME just for sake of it, but I believe that for the end-user layout will be more important than refresh rate, and that's different beast to tame. Personally I "almost" settled for Dioxus (shameless plug: [1], there's link to YT video) and I'm quite happy with it. Having editor in WebView feels really quirky though (e.g. no textareas, I'm intercepting key events and rendering in div glyph-by-glyph directly). [1]: https://github.com/exlee/dioxus-editor https://github.com/exlee/dioxus-editor

as-cii 4y ago
Hey xlii! This is Antonio, author of the post. You're right that rendering is only part of the story. To stay within the ~8ms frame budget, however, every little bit counts. Maintaining application state, layout, painting, and finally pushing pixels to screen, all need to be as performant as they can be. For layout specifically we're using an approach inspired by Flutter, which lets us avoid complex algorithms but still have a lot of flexibility in the way elements can be positioned and produce a rich graphical experience. Thanks for reading and commenting!
- xlii 4y ago
  I don't have experience with Flutter, but based on quick glance they're using widgeting and, what I found quite important - ability to develop GUI outside of the application. Something that I think libraries like egui are missing (and which is easily obtainable with Tauri/Dioxus). Rebuilding whole app to ensure that some box doesn't get cut off ruins development experience, especially for big apps. Kudos to you guys, I hope you'll make Zed extensible, so that instead of writing my own editor I can use yours ;-)

almostdigital 4y ago

Looking forward to trying this, VSCode is great but I really miss the performance of Sublime Text. I hope they get the plugin system right, killer feature would be if it could load VSCode plugins (incredibly hard to pull off, yes)

as-cii 4y ago
Thanks, almostdigital! After our past experience with Atom, getting the plugin system right is a top priority for the editor. The thought of cross compatibility with VSCode plugins definitely crossed our mind and it's not out of the question, although our current plan is to initially support plugins using WASM.

soulbadguy 4y ago

I am curious how this would compare to a ui written in flutter. It seems that fluter is also hardware accelerated and x-platform

doodlesdev 4y ago
In terms of performance it would possibly be even better in Flutter, BUT (and this is a big butt) text editing on the desktop in Flutter currently can only be described as broken. I last looked at this a few months ago and I think it's fixable, but as much as I like Flutter I don't think it's a good option for a text editor _just yet_.

RonnieOwnsLexus 4y ago

Why do we need a code editor with 120FPS support ?

Strom 4y ago
Because competing code editors support it and thus lacking this capability would make your code editor seem unusually incompetent.
- RonnieOwnsLexus 4y ago
  I am sorry but i tried googling and binging which code editor supports 120fps. I also asked bing ai and it said it is not aware. Could you please point me here as to what I am missing ?
  - Strom 4y ago
    notepad.exe on Windows for example. If that's too lightweight, then VSCode is a more heavier example.

nottorp 4y ago

I don't understand. Why would you need to render a user interface constantly at 120 fps, instead of just updating it when something changes? Laptop batteries last too long these days? Electricity too cheap?

sdflhasjd 4y ago
"Because it looks good" is probably the most popular reason.
- nottorp 4y ago
  But if nothing changes it looks as good at zero fps :) Edit: Yay, it's a text editor. What happened to only redrawing the line that's being edited and the status indicators?
as-cii 4y ago
Hey nottorp. Antonio here, author of the post. Zed and GPUI use energy very judiciously and only perform updates when needed. The idea is that we can render the whole application within ~8ms, and that shows everywhere: from typing a letter to scrolling up and down in a buffer. However, if the editor is sitting there idle, we won't waste precious CPU cycles. Thanks for the feedback!
- nottorp 4y ago
  Yeah, might want to edit the title a bit. Or not, considering these concepts are getting lost. I mean, the win16 api from ages ago had support for invalidating specific screen regions etc. It probably got lost somewhere in the transition to javascript...
- rubymamis 4y ago
  Will you allow developers access to your GUI framework? What about open-sourcing it?
flohofwoe 4y ago
It's not about rendering static screens at 120Hz, but rendering anything that's animated at a smooth 120Hz.

jug 4y ago

Wow, that’s some low level stuff. Most would just use an established UI framework because rendering performance is left to the window manager. I’m not sure I understand the need to go about it like this? Windows is not considered the epitome of performant interfaces but it has no trouble rendering UI’s at 120 fps. When people go and buy a 120 fps display, they are wowed by the smooth scrolling in a heavy application like Google Chrome. The window manager is already hardware accelerated (as for Windows since Vista) and the apps draw widgets on their surface.

smoldesu 4y ago
It's so weird. The devs of the Warp based terminal are doing something similar (Rust for single-platform low-level dev), and I'm also not sure what the point is. It feels like they're banking on Rust being a selling point, but forgot that the lang can drastically lower your iteration speed when it comes time to compete with other editors.
ben-schaaf 4y ago
> Windows is not considered the epitome of performant interfaces but it has no trouble rendering UI’s at 120 fps. When people go and buy a 120 fps display, they are wowed by the smooth scrolling in a heavy application like Google Chrome. Windows uses GPU-based rendering in most/all of their GUI frameworks. Chrome also makes heavy use of hardware acceleration. Rendering performance isn't left to the window manager. If you're making your own GUI this is exactly the kind of stuff you need to do to make it fast.

tgflynn 4y ago

What API are they using for interfacing with the GPU (ie. OpenGL, Vulkan, other ) ? I suspect a lot of time is likely to be spent on the CPU side updating vertex and other data and pushing it to the GPU so it would be useful to have some more detail on how they are handling that.

tcldr 4y ago
I saw a few Metal specific types in the source code contained within their demo video

avereveard 4y ago

I cannot stress how much I do not want my 300w gpu to be used to render text that change at most three times per second. And it's not just about electricity cost and heat stress, it will conflict with everything else that requires the gpu to do stuff, including watching 4k videos on the second monitor, which does have a legitimate case for requiring hardware acceleration since they move a lot of data 60 times per second, and your editor doesn't. And the limited resource is not the gpu itself but the nearby onboard memory is a scarce resource on its own. I'd be real mad at a software that prevents me to multitask.

alexfromapex 4y ago
This UI is so lightweight it seems like they should easily be able to toggle the GPU compute on or off
ohgodplsno 4y ago
I have horrible news to tell you about compositors: they already do use your GPU, even if you're watching a 4K video on the side. Even when requesting a full screen swapchain to avoid having to deal with the compositor, modern OSes will force you to go through that compositor and lie about it being fullscreen. Additionally, using your GPU doesn't mean locking your GPU on that task. Your 4K video most likely isn't taking up 100% of GPU time, nor 100% of the nearby onboard memory. Everyone already gets a share of that time.
- avereveard 4y ago
  Compositor and ui framework that interact with them know that 120fps is not the target but that frame render time is, and know not to have active rendering contexts for static resources. Chasing 120fps rendering here is the problem.
- rafram 4y ago
  > Even when requesting a full screen swapchain to avoid having to deal with the compositor, modern OSes will force you to go through that compositor and lie about it being fullscreen. I’m honestly thankful for this bit of lying. Windows 8 and below had horrible, horrible issues with fullscreen programs, especially games - they’d change your screen resolution and thus move and resize all your other windows, they’d freeze for ten seconds when you tried to alt-tab out, and they’d frequently just crash when you switched back to them. Those problems are essentially nonexistent nowadays, and it’s worth the small performance cost that comes from going through the compositor.

rafram 4y ago

[flagged]

acdha 4y ago
Why are you reacting so emotionally to something that far down the page? They’re talking about being aggressively multi-core and given their technical audience it hardly seems inappropriate to have that detail along with the other details like how they use GPU acceleration and the benefits of lower input latency.
- rafram 4y ago
  I wouldn’t say I’m reacting emotionally, or that where it is on the page is relevant. It’s just a funny part of their marketing copy.

sebastianconcpt 4y ago

I'm intrigued. What's the applicability that you see for this?

fassssst 4y ago

Erm, native WinUI apps are GPU accelerated and render at vsync.

doodlesdev 4y ago
Also GTK4 is GPU accelerated whenever possible, with really well mantained Rust bindings for it I think the only thing missing would be macOS which I'm not sure what solutions are there. Another option in this front could be Flutter if write-once run-everywhere is a need for the project. Another advantage is that it's not only GPU accelerated but it's also retained mode.
kridsdale1 4y ago
So is everything written in Apple native UI frameworks since 2009.
- kllrnohj 4y ago
  And Android and QT and GTK and Chrome and Firefox and etc... The article is talking about the same generic hybrid GPU-accelerated rendering architecture that everything uses. Seemingly the only "new" part is "in Rust!"
pjc50 4y ago
Nobody loves native WinUI, not even Microsoft. Which is kind of a shame. But it's the result of years of product management neglect as well as the pull away from desktop UIs to web UIs.
- pjmlp 4y ago
  They only have themselves to blame, after the rewrites their forced their hardest advocates to go through, each one with worse tooling, dropping the UI designer, .NET Native and C++/CX along the way. Native AOT still can't compile WinUI, while C++/WinRT is like doing ATL in 2000, while bug issues grow exponentially. Only WinDev themselves, and WinUI MVPs, can still believe this is going somewhere. The rest of us have moved on.

scotty79 4y ago

I hoped they'll go for signed distance functions for glyph rendering as well. Rendering text with shaders is fascinating.

flohofwoe 4y ago

It's surprising that they jump immediately from the problem description into shader details. IME the main theme with achieving high performance not just in games, and not just in rendering, is to avoid frequent 'context switches' and instead queue/batch a lot of work (e.g. all rendering commands for one frame) and then submit all this work at once "to the other side" in a single call. This is not just how modern 3D APIs work, but is the same basic idea in 'miracle APIs' like io_uring This takes care of the 'throughput problem', but it can easily lead to a 'latency problem' (if the work to be done needs to travel through several such queues which are 'pipelined' together).

p0nce 4y ago

Fast software rasterizers are not slow at drawing text in the first place.

monkeydust 4y ago

What's the real world client experience of developing UIs to render @ 120FPS - is it like once you have tried it going back is really hard?

FpUser 4y ago

This guy Bero1985 wrote the 3D library / engine some years that has extensive 2D features including UI that uses SDF among the other things [0]. [0] - https://github.com/BeRo1985/pasvulkan https://github.com/BeRo1985/pasvulkan

29athrowaway 4y ago

You cannot see at 120 fps.

Etherlord87 4y ago
You're right, you can't see at 120 fps, because that's not how human vision works. You can't see at any fps. However the more fps there are, the sooner you get the information, and so regardless of the lag on your end (the biology behind the eyes, the brain, and the connection between them), you still add some delay coming from your computer.
jamesgeck0 4y ago
Check out https://www.testufo.com/ https://www.testufo.com/ on a 120Hz display; the higher FPS is absolutely perceptible.
tayistay 4y ago
If you're doing a drawing app, then 120fps helps you keep the stroke close to the stylus. Even then it may lag a bit because of a few frames of GPU latency and you have to do predictive stroke points and then adjust your stroke later. See https://developer.apple.com/documentation/uikit/touches_presses_and_gestures/handling_touches_in_your_view/minimizing_latency_with_predicted_touches https://developer.apple.com/documentation/uikit/touches_pres... Similarly, if you're dragging something around with your finger, 120fps keeps that thing closer to the finger. Just improves the UX.

computing 4y ago

First, this looks awesome. Can't wait to try zed. Second, forgive a naive question since I know nothing about graphics, but would the method described in the article perform better than Alacritty + Neovim?

btryder 4y ago

Lots of people nit-picking the 120 FPS but I think Zed looks super promising. The native support for collaborative editing looks fantastic, and I'm excited to try it out. Curious if you guys have thought about VR / AR possibilities with GPUI?

exo-cortex 4y ago
That would be interesting in VR to have syntax highlighting that moves variable names towards the viewer, or puts context menus with possible methods of some object above. Would probably just be stupid gimmick, but I want to see somebody do it :-D
- rikarendsmp 4y ago
  I tried this, using https://makepad.dev https://makepad.dev our GPU accelerated UI and renderstack. And unfortunately it wasn't a great experience. Text popping forward for whatever reason is not really an improvement (i tried indent depth, syntax highlighting reasons, cursor behavior). Maybe 'veeeeery' subtly could do something, but otherwise you dont want it to break visual symmetry as we are used to

trollied 4y ago

Previous post: https://news.ycombinator.com/item?id=35057642 https://news.ycombinator.com/item?id=35057642

Animats 4y ago

This seems like the wrong portion of the problem on which to spend time. This is a text editor. Performance problems with text editors tend to involve long files and multiple tabs. Refresh speed isn't the problem, although keyboard response speed can be. I'd like to see "gedit", for Linux, fixed. It can stall on large files, and, in long edit sessions, will sometimes mess up the file name in the tab. Or "notepad++" for Linux.

msvan 4y ago

Lots of negativity in here. I for one am excited about the prospect of an editor that is as responsive as I remember Sublime being back in the day, with the feature set I've come to expect from VS Code. An editor like this simply does not exist today, and betting on the Rust ecosystem is entirely the right choice for building something like this in 2023.

jakswa 4y ago
Here here. I backed Onivim hoping it was going to shine a light in the darkness, and it seemed promising, but ultimately was abandoned? I think, unsure.

haberman 4y ago

This sounds really similar to the story we were hearing about Servo back in 2016: https://www.youtube.com/watch?v=erfnCaeLxSI https://www.youtube.com/watch?v=erfnCaeLxSI I was really excited when I saw that demo. Why didn't this turn into a final product that people could use?

deleted 4y ago
[deleted]

tiffanyh 4y ago

Nathan Sobo It should be noted that the main person behind Zed is Nathan Sobo, who created Atom while he was at Github, which is the basis of Visual Studio Code today. As such, I have high hopes Zed will be a much faster version of Visual Studio Code and am excited to see what him & his team make.

3836293648 4y ago
Well, he created electron for Atom. Atom itself was never a part of vsc. VSC was targetting the browser for a while before electron came along

tayistay 4y ago

My rui library can render UIs at 120fps, uses similar SDF techniques (though uses a single shader for all rendering): https://github.com/audulus/rui https://github.com/audulus/rui Is their GPUI library open source?

inamberclad 4y ago
Sadly I didn't see any links.

bjconlan 4y ago

Beyond the rendering which as noted is nothing that hasn't been done before (in general) the inherent OT/multi user + tree sitter functionality is something that entices me. I'm surprised nobody pointed out lite/litexl here either it's rendering of ui is very similar (although fonts are via a texture; like a game would) and doesn't focus overly on the GPU but optimises those paths like games circa directx9/opengl 1.3 There are great details of the approach taken with lite at https://rxi.github.io https://rxi.github.io Lite-xl might have evolved the renderer but the code here is very consumable for me.