3 ms·

> If the web was being made form scratch today, it would probably start with a repurposed game engine. I still hold that reinventing the underlying operating s

> If the web was being made form scratch today, it would probably start with a repurposed game engine. I still hold that reinventing the underlying operating systems' text rendering and text input is almost always a bad idea. Some things this "game engine" would need to work with the underlying operating system to solve: - Accessibility (screen readers, etc) - OS-specific text kerning behaviour - Ligatures (including for Arabic and Korean text) - IME for Korean / Chinese / Japanese / etc. - Right-to-left and left-to-right language support. - Input events on every platform. There's about 20 keyboard shortcuts to interact with text on windows, macos and linux. They're different on every platform. iOS / Android are different again. - Native scrolling There's probably more. Its an obscene amount of work to reimplement this stuff correctly on top of a game engine-like API like Canvas. The browser already does almost all of this work and despite that its still a ridiculous hassle to implement web-native rich text editor components. (Though we may disagree on why). As a developer, there are a lot of advantages to having your application behave in an identical way on every platform. But as a user, I don't actually want that. I want your application to fit in with the rest of my operating system. (Be it windows / macos / linux+GTK / iOS / Android). Mind you, Raph Levien is smart and as I understand it, he disagrees with me on this. Here's a video of him talking about it: https://www.youtube.com/watch?v=zVUTZlNCb8U https://www.youtube.com/watch?v=zVUTZlNCb8U

raphlinus 4y ago
These things are hard, and I don't want to discount any of it. But three observations. First, browser engines do reimplement most of this stuff, though that's been an evolution, it used to be they relied a lot more on platform text layout, for example. And for the stuff that's not completely reinvented (accessibility), they have a put in a lot of work to make cross-platform abstractions. In many cases (interfacing with the compositor comes to mind), browsers are the only viable open source code bases you can read. Second, to a large extent OS platforms are stagnating. UWP was going to be a big advance over the old WinAPI ways of doing things, but it ended up being a dud. SwiftUI has performance problems for a number of reasons, including the fact they're still using software rendering for a lot of stuff, and their compositor model requires allocating huge amounts of memory for intermediate textures for all their UI elements. Third, the business structure around platforms requires them to each do things differently, which creates gratuitous incompatibility. You really only need one high quality library to do, say, text layout, rather than having slightly different and slightly incompatible versions on each platform. (And very few places are gratuitous incompatibilities more frustrating than GPU infrastructure) So for these reasons, I do believe it's worth exploring a new cross-platform GUI toolkit. It is pretty speculative, though; there are lots of ways it could fail. Game engines are decent at a lot of this stuff, of which of course GPU infrastructure (and graphics rendering in general) is quite good. Most of them could use work in the text department though :)
- josephg 4y ago
  I think there’s tremendous benefit to what you’re proposing. The future I’m the most worried about is where every application ships its own bespoke, slightly wrong text layout and rendering engine. We see this already in video games and it’s a mess. (Which games allow you to copy+paste in and out of their chat system? It’s a crapshoot!) The amount of work to do this well is high. Probably (easily) measured in engineer-years. All to reimplement something that - janky as it may be - already exists in some form in web browsers. But as you say, worth it if we can pull it off. But it’s a common good. The hardest part in my mind would be figuring out how to fund a project like this. My ideal application platform isn’t a video game engine. It’s probably something like electron, but with most of the browser stuff stripped out. (Dump javascript, the game pad api, midi, usb, Bluetooth, tab isolation, and probably the DOM. And use Houdini or something instead of CSS). And then I’d happily swap the rendering for pietgpu or whatever will perform the best on modern platforms. Basic desirada: Small and light, fast, cross platform and platform-native look and feel everywhere. Accessibility, IME, localisation, rtl- and ltr- language support, and so on. Mobile is still a big question. But solvable if sufficient engineer time was poured into it. It’s just a big project.
- mike_hearn 4y ago
  How would a new cross-platform UI toolkit be meaningfully better than e.g. Jetpack Compose or JavaFX?
  - raphlinus 4y ago
    To me, the single biggest opportunity is performance. There's a lot that can be done if you optimize for that: plumb incremental computation through the pipeline from the app logic all the way to the GPU, run your logic in multiple threads, and just generally minimize the work done. I gave a talk[1] which goes into more detail. Now, I freely admit that many applications don't need that much performance. But I think having a toolkit that is optimized along those lines could be a good basis for doing work on the other things (developer experience) that make a toolkit great. [1]: https://www.youtube.com/watch?v=zVUTZlNCb8U https://www.youtube.com/watch?v=zVUTZlNCb8U
jcelerier 4y ago
> But as a user, I don't actually want that Depends on the users I guess. Mine are really adamant that they want the exact same app no matter the OS. The OS is just a tool to launch actual productivity apps.

raphlinus 4y ago

These things are hard, and I don't want to discount any of it. But three observations. First, browser engines do reimplement most of this stuff, though that's been an evolution, it used to be they relied a lot more on platform text layout, for example. And for the stuff that's not completely reinvented (accessibility), they have a put in a lot of work to make cross-platform abstractions. In many cases (interfacing with the compositor comes to mind), browsers are the only viable open source code bases you can read. Second, to a large extent OS platforms are stagnating. UWP was going to be a big advance over the old WinAPI ways of doing things, but it ended up being a dud. SwiftUI has performance problems for a number of reasons, including the fact they're still using software rendering for a lot of stuff, and their compositor model requires allocating huge amounts of memory for intermediate textures for all their UI elements. Third, the business structure around platforms requires them to each do things differently, which creates gratuitous incompatibility. You really only need one high quality library to do, say, text layout, rather than having slightly different and slightly incompatible versions on each platform. (And very few places are gratuitous incompatibilities more frustrating than GPU infrastructure) So for these reasons, I do believe it's worth exploring a new cross-platform GUI toolkit. It is pretty speculative, though; there are lots of ways it could fail. Game engines are decent at a lot of this stuff, of which of course GPU infrastructure (and graphics rendering in general) is quite good. Most of them could use work in the text department though :)

josephg 4y ago
I think there’s tremendous benefit to what you’re proposing. The future I’m the most worried about is where every application ships its own bespoke, slightly wrong text layout and rendering engine. We see this already in video games and it’s a mess. (Which games allow you to copy+paste in and out of their chat system? It’s a crapshoot!) The amount of work to do this well is high. Probably (easily) measured in engineer-years. All to reimplement something that - janky as it may be - already exists in some form in web browsers. But as you say, worth it if we can pull it off. But it’s a common good. The hardest part in my mind would be figuring out how to fund a project like this. My ideal application platform isn’t a video game engine. It’s probably something like electron, but with most of the browser stuff stripped out. (Dump javascript, the game pad api, midi, usb, Bluetooth, tab isolation, and probably the DOM. And use Houdini or something instead of CSS). And then I’d happily swap the rendering for pietgpu or whatever will perform the best on modern platforms. Basic desirada: Small and light, fast, cross platform and platform-native look and feel everywhere. Accessibility, IME, localisation, rtl- and ltr- language support, and so on. Mobile is still a big question. But solvable if sufficient engineer time was poured into it. It’s just a big project.
mike_hearn 4y ago
How would a new cross-platform UI toolkit be meaningfully better than e.g. Jetpack Compose or JavaFX?
- raphlinus 4y ago
  To me, the single biggest opportunity is performance. There's a lot that can be done if you optimize for that: plumb incremental computation through the pipeline from the app logic all the way to the GPU, run your logic in multiple threads, and just generally minimize the work done. I gave a talk[1] which goes into more detail. Now, I freely admit that many applications don't need that much performance. But I think having a toolkit that is optimized along those lines could be a good basis for doing work on the other things (developer experience) that make a toolkit great. [1]: https://www.youtube.com/watch?v=zVUTZlNCb8U https://www.youtube.com/watch?v=zVUTZlNCb8U

jcelerier 4y ago

> But as a user, I don't actually want that Depends on the users I guess. Mine are really adamant that they want the exact same app no matter the OS. The OS is just a tool to launch actual productivity apps.