6 ms·

Here's my pitch for a better ecosystem: we develop an intermediate standard for fundamental UI building blocks (divs/blocks/layers, images, video, etc.) + a lay

Here's my pitch for a better ecosystem: we develop an intermediate standard for fundamental UI building blocks (divs/blocks/layers, images, video, etc.) + a layout engine, sort of like the DOM. Shells (browsers, native platforms like React Native but without the React requirement, etc.) would give JS renderers access to the lowest level painting calls available (ie. "repaint this node") along with programmatic access to the new standard UI API. Renders could then build on top of that API, implement the draw call optimizations themselves based on whatever makes sense for the rendering engine (for instance, React Fiber would be issuing draw calls differently than Vue's rendering engine.) This would allow developers to get native performance out of a web-based rendering context. It would also be theoretically easy to build new renderers around the standard UI API. So you could swap out your backend between browser-native and a mobile/desktop library backed by OpenGL/WebGL, or another headless solution, etc. Importantly, I think you need to limit the scope of this new UI standard (perhaps just to basic shapes, like how SVG is similarly limited), that way you leave UI implementations like dropdown boxes to ecosystem programmers (so you'd have a dropdown box for each React, Vue, etc. -- this sounds bad but people do this already.) Limiting the API scope also means making it easier for new renderers to enter the scene. Compare such a world to the current hellscape that is building your own HTML renderer. The API is so insane it's really only possible for big companies to take it on (Google, Microsoft, Apple, Mozilla.) How this ties back into this post: you could create an _____ Native, which would really just implement everything you want from React Native, but without React. Then you can just plug Vue or React into the renderer and boom you're done. Edit: Observant people will notice my ideal system resembles X11. Indeed, I think X11 is an interesting proposition in "shell" (browser, native) contexts, as shell applications are inherently window-managed and sandboxed. This allows for applications to reinvent the wheel without disrupting the fundamental UX of window management on the system OS.

mark_edward 8y ago
Web Components?
- lwansbrough 8y ago
  No, because web components assume you're using the DOM. Which we definitely do not want. I want to go lower level. Give me a hashmap of nodes currently on screen. Let me design how the nodes are arranged in a render tree. Let me design how the nodes are redrawn. Let me design how the nodes are styled. I shouldn't have to compile my styles to CSS. Make the browser a rendering slave with a small but effective layout engine + UI API.
  - gmueckl 8y ago
    What about accessibility? Accessibility helpers would need semantic information about the UI to be of any use. And I doubt that many homegrown or halfbaked Ui libraries (which would certainly sprout like mushrooms on a platform like yours) would provide correct annotations through an accessibility API. Most developers wouldn't care.
    - lwansbrough 8y ago
      Accessibility is definitely something I have thought about, too. Including accessibility features in the API is vital I think. I'm not very familiar with accessibility features as they exist today, but I think it would be possible for longer term projects built on top of such a UI system to emerge which would provide accessibility information by default, and the rest would resemble what we have with HTML today.
  - southerndrift 8y ago
    Can't you do this by yourself by implementing a thin layer on top of the canvas element? What do you need more than a z-map of all of your nodes to keep track of which one is at the top and should receive mouse and touch events?
  - jeremiep 8y ago
    A render list would be more efficient than a tree, simply because you're not jumping left and right in RAM with cache misses on every jump. A hashmap of nodes on screen is still very high level; you're basically replacing the DOM with data, yet keeping the exact same structure. But now you lose static types and everything is slower. Redrawing doesn't happen at the node level either; that would be terribly inefficient. Things are instead batched together. You don't have to compile your styles to CSS currently; just write them as JS objects and call the CSS constructors yourself from code. You'll quickly find its not productive for most styling. I simply don't believe the vast majority of developers to be able to properly handle rendering at that level. No offence but all your points about low-level rendering is not how it actually works. Its incredibly easy to fool yourself into thinking you wrote a well-designed, efficient piece of code and learn years later every single piece of it was far from optimal. Thats why we have layers of abstractions.
sanketsahu 8y ago
Flutter?
- lwansbrough 8y ago
  Flutter is closer, but it is too focused on being a framework. I'm thinking of a platform agnostic protocol with implementations in whatever language you like. Flutter is too many of things at once. If they basically ripped out each native platform's implementation and made it target the protocol I've described, then made their OOP framework target that protocol, then had the two interface using the protocol, that's what I'd like to see. That way you could write React Native on top of that new "Flutter Native Core".
mda 8y ago
Some of your points remind me of Flutter.
danpalmer 8y ago
While I admire your hopefulness that this could solve the problems, I don’t think it will for two reasons. 1. This is what the DOM is, to some extent. All those components on the page are “native” components, we have declarative markup for them (HTML) and we can script them. They are “rendered” by native code in browsers which are some of the most highly optimised codebases around. 2. The API design for JS and for higher performance systems is quite different. Take UITableViewDataSource for example, it does view prefetching, caching, object reuse, etc. As of iOS 12, this also feeds into the CPU throttling control to ensure enough compute is available on a cell fetch that it will meet the 8-16ms deadline so that no frames are dropped. This is just a totally different way of thinking about APIs and performance that the JS ecosystem is not designed to work with. It’s possible that a total rewrite of that stack could provide significant benefits, but I think that would be more from removing legacy cruft in the DOM, than from it being a paradigm shift.
- lwansbrough 8y ago
  Your UITableViewDataSource is a common problem with big JS applications, and has been solved to some extent with the same optimizations you've mentioned. However because JS and HTML is what it is, it has to be solved on a per-library basis (React Virtualized List, Vue Virtualized List, etc.) The problem remains that the JS renderer does not have access to those low level draw calls to optimize performance further. As for the DOM, again I think that while it's true that native HTML elements are highly optimized, many applications have much larger performance goals in mind and HTML/DOM has proven to be an ineffective system when trying to do very specific performance optimizations. Check out flipboard.com on your phone for an example of how much smoother a web app can feel when you bypass the DOM and let React take over the draw calls.
  - stefanfisk 8y ago
    AFAIK all virtual lists break native searching in the browser, so sadly the problem is far from solved
    - evv 8y ago
giancarlostoro 8y ago
Wonder where Cordova falls under in this regard? https://cordova.apache.org/ https://cordova.apache.org/ I think Cordova could be leveraged at the very least.
Aissen 8y ago
It's funny because Netflix's Gibbon and YouTube's Cobalt already use some of those ideas. At least the latter is open source. Flutter is built with the same thinking. There will be some kind of convergence in the coming years.
- hexmiles 8y ago
  What are Gibbon and Cobalt? I tried google but with no success. Do you have any link?
  - Aissen 8y ago
    Gibbon: - https://medium.com/netflix-techblog/crafting-a-high-performance-tv-user-interface-using-react-3350e5a6ad3b https://medium.com/netflix-techblog/crafting-a-high-performa... - https://medium.com/netflix-techblog/building-the-new-netflix-experience-for-tv-920d71d875de https://medium.com/netflix-techblog/building-the-new-netflix... - https://www.youtube.com/watch?v=eNC0mRYGWgc https://www.youtube.com/watch?v=eNC0mRYGWgc Cobalt: https://www.cobalt.dev/ https://www.cobalt.dev/
    - iaml 8y ago
      I find it funny that youtube made a cool new tech specifically to render UI on tv's and yet it works like shit on any tv I encountered.
      - Aissen 8y ago
        Most probably you're not using a cobalt-based YouTube app yet.
        iaml 8y ago
        Is there a list of platforms that use it?
        Aissen 8y ago
jcelerier 8y ago
> we develop an intermediate standard for fundamental UI building blocks (divs/blocks/layers, images, video, etc.) + a layout engine, sort of like the DOM http://qmlbook.github.io/en/ch04/index.html http://qmlbook.github.io/en/ch04/index.html
amelius 8y ago
The problem is: the platform owners (Apple and Google) have absolutely no incentive to make this see the day of light. They make money selling apps, and want to bind developers to their platform.
- lwansbrough 8y ago
  In some respects I think they've already paved the way. An adapter could be created to target canvas or WebGL. As for APIs like file system access, I don't think there's anything wrong with the JS web APIs as they exist today. I'd love to see Apple actually push ahead with the implementation of more of these APIs, as well as PWAs, etc. but I know it's not at all in their interest.
jeremiep 8y ago
This assumes the web rendering context is the proper abstraction to build application UIs. Its not. Its too complex and object-oriented to be efficient. It was intended for documents and is a general pain to work with for applications. For one, there's a lot to be learned from immediate-mode UIs (I'm thinking Dear IMGUI, not Unity3D's GUI); ie state inside components generates complexity, which is a bad thing. Your views should be nothing more than a pure FSM transforming app state to UI directives. IMGUIs have a ton of tricks to simplify UI development. With these in mind, FP becomes a much more interesting paradigm than OOP for this. The same logic applies whether you render the entire UI 60 times a second or you cache the previous render to only do the minimum amount of updates. A good abstraction is one where features can emerge from its design. For example you rarely have to test your views when they're just a simple pure transform. You don't need an entire framework and command-patterns to implement undo when your app state is immutable and modelled as a sequence of actions. With each emergent feature you remove the need for tons of code and tooling; that yields agility. I still believe most web application are much more complex than they should be, especially on the UI level, regardless of the framework used. And until we solve that problem, I don't think the web stack is the way to go for native. We're just replacing one set of problems for another one. I strongly believe we shouldn't make things easier, but simpler.
- lwansbrough 8y ago
  I'm trying really hard to understand your argument. So you believe such a UI API should be immutable (whatever that means in this context), and modelled as a sequence of actions? I think that's fine, but probably not practical in reality. I don't see the point of having a debate about the merits of functional programming vs. OOP in the middle of your post. People are going to build tools around the API that will abstract away whatever semantics you feel are best in order to fit their own needs. So the goal is to build an API that's simple at it's core to implement and use. Inevitably someone actually has to go and code the adapter layer between the protocol receiver and the actual low level rendering layer (be it OpenGL, UIKit, etc.) and unless you build a system that's easy to understand on that end as well, you're gonna end up with an API no one actually likes or wants to use. I don't think it's unreasonable to expect that the low-level rendering engine (ie. an implementation in UIKit) maintains some level of UI state. That is - after all - where most of the optimizations will come from. And that's how UIKit, OpenGL and most other rendering systems work. Things generally aren't immutable because it's a.) not efficient, and b.) a nightmare to work with at the lowest levels. It seems to me the appropriate abstraction is build the protocol around a UI state tree, because that's what everyone else is already working with. This is incredibly easy to maintain on either end, and higher level libraries can easily remove the pain of UI state management (a la React, Vue, etc.)
basch 8y ago
have you heard of React Canvas? https://engineering.flipboard.com/2015/02/mobile-web https://engineering.flipboard.com/2015/02/mobile-web

mark_edward 8y ago

Web Components?

lwansbrough 8y ago
No, because web components assume you're using the DOM. Which we definitely do not want. I want to go lower level. Give me a hashmap of nodes currently on screen. Let me design how the nodes are arranged in a render tree. Let me design how the nodes are redrawn. Let me design how the nodes are styled. I shouldn't have to compile my styles to CSS. Make the browser a rendering slave with a small but effective layout engine + UI API.
- gmueckl 8y ago
  What about accessibility? Accessibility helpers would need semantic information about the UI to be of any use. And I doubt that many homegrown or halfbaked Ui libraries (which would certainly sprout like mushrooms on a platform like yours) would provide correct annotations through an accessibility API. Most developers wouldn't care.
  - lwansbrough 8y ago
    Accessibility is definitely something I have thought about, too. Including accessibility features in the API is vital I think. I'm not very familiar with accessibility features as they exist today, but I think it would be possible for longer term projects built on top of such a UI system to emerge which would provide accessibility information by default, and the rest would resemble what we have with HTML today.
- southerndrift 8y ago
  Can't you do this by yourself by implementing a thin layer on top of the canvas element? What do you need more than a z-map of all of your nodes to keep track of which one is at the top and should receive mouse and touch events?
- jeremiep 8y ago
  A render list would be more efficient than a tree, simply because you're not jumping left and right in RAM with cache misses on every jump. A hashmap of nodes on screen is still very high level; you're basically replacing the DOM with data, yet keeping the exact same structure. But now you lose static types and everything is slower. Redrawing doesn't happen at the node level either; that would be terribly inefficient. Things are instead batched together. You don't have to compile your styles to CSS currently; just write them as JS objects and call the CSS constructors yourself from code. You'll quickly find its not productive for most styling. I simply don't believe the vast majority of developers to be able to properly handle rendering at that level. No offence but all your points about low-level rendering is not how it actually works. Its incredibly easy to fool yourself into thinking you wrote a well-designed, efficient piece of code and learn years later every single piece of it was far from optimal. Thats why we have layers of abstractions.

sanketsahu 8y ago

Flutter?

lwansbrough 8y ago
Flutter is closer, but it is too focused on being a framework. I'm thinking of a platform agnostic protocol with implementations in whatever language you like. Flutter is too many of things at once. If they basically ripped out each native platform's implementation and made it target the protocol I've described, then made their OOP framework target that protocol, then had the two interface using the protocol, that's what I'd like to see. That way you could write React Native on top of that new "Flutter Native Core".

mda 8y ago

Some of your points remind me of Flutter.

danpalmer 8y ago

While I admire your hopefulness that this could solve the problems, I don’t think it will for two reasons. 1. This is what the DOM is, to some extent. All those components on the page are “native” components, we have declarative markup for them (HTML) and we can script them. They are “rendered” by native code in browsers which are some of the most highly optimised codebases around. 2. The API design for JS and for higher performance systems is quite different. Take UITableViewDataSource for example, it does view prefetching, caching, object reuse, etc. As of iOS 12, this also feeds into the CPU throttling control to ensure enough compute is available on a cell fetch that it will meet the 8-16ms deadline so that no frames are dropped. This is just a totally different way of thinking about APIs and performance that the JS ecosystem is not designed to work with. It’s possible that a total rewrite of that stack could provide significant benefits, but I think that would be more from removing legacy cruft in the DOM, than from it being a paradigm shift.

lwansbrough 8y ago
Your UITableViewDataSource is a common problem with big JS applications, and has been solved to some extent with the same optimizations you've mentioned. However because JS and HTML is what it is, it has to be solved on a per-library basis (React Virtualized List, Vue Virtualized List, etc.) The problem remains that the JS renderer does not have access to those low level draw calls to optimize performance further. As for the DOM, again I think that while it's true that native HTML elements are highly optimized, many applications have much larger performance goals in mind and HTML/DOM has proven to be an ineffective system when trying to do very specific performance optimizations. Check out flipboard.com on your phone for an example of how much smoother a web app can feel when you bypass the DOM and let React take over the draw calls.
- stefanfisk 8y ago
  AFAIK all virtual lists break native searching in the browser, so sadly the problem is far from solved
  - evv 8y ago

giancarlostoro 8y ago

Wonder where Cordova falls under in this regard? https://cordova.apache.org/ https://cordova.apache.org/ I think Cordova could be leveraged at the very least.

Aissen 8y ago

It's funny because Netflix's Gibbon and YouTube's Cobalt already use some of those ideas. At least the latter is open source. Flutter is built with the same thinking. There will be some kind of convergence in the coming years.

hexmiles 8y ago
What are Gibbon and Cobalt? I tried google but with no success. Do you have any link?
- Aissen 8y ago
  Gibbon: - https://medium.com/netflix-techblog/crafting-a-high-performance-tv-user-interface-using-react-3350e5a6ad3b https://medium.com/netflix-techblog/crafting-a-high-performa... - https://medium.com/netflix-techblog/building-the-new-netflix-experience-for-tv-920d71d875de https://medium.com/netflix-techblog/building-the-new-netflix... - https://www.youtube.com/watch?v=eNC0mRYGWgc https://www.youtube.com/watch?v=eNC0mRYGWgc Cobalt: https://www.cobalt.dev/ https://www.cobalt.dev/
  - iaml 8y ago
    I find it funny that youtube made a cool new tech specifically to render UI on tv's and yet it works like shit on any tv I encountered.
    - Aissen 8y ago
      Most probably you're not using a cobalt-based YouTube app yet.
      - iaml 8y ago
        Is there a list of platforms that use it?
        Aissen 8y ago

jcelerier 8y ago

> we develop an intermediate standard for fundamental UI building blocks (divs/blocks/layers, images, video, etc.) + a layout engine, sort of like the DOM http://qmlbook.github.io/en/ch04/index.html http://qmlbook.github.io/en/ch04/index.html

amelius 8y ago

The problem is: the platform owners (Apple and Google) have absolutely no incentive to make this see the day of light. They make money selling apps, and want to bind developers to their platform.

lwansbrough 8y ago
In some respects I think they've already paved the way. An adapter could be created to target canvas or WebGL. As for APIs like file system access, I don't think there's anything wrong with the JS web APIs as they exist today. I'd love to see Apple actually push ahead with the implementation of more of these APIs, as well as PWAs, etc. but I know it's not at all in their interest.

jeremiep 8y ago

This assumes the web rendering context is the proper abstraction to build application UIs. Its not. Its too complex and object-oriented to be efficient. It was intended for documents and is a general pain to work with for applications. For one, there's a lot to be learned from immediate-mode UIs (I'm thinking Dear IMGUI, not Unity3D's GUI); ie state inside components generates complexity, which is a bad thing. Your views should be nothing more than a pure FSM transforming app state to UI directives. IMGUIs have a ton of tricks to simplify UI development. With these in mind, FP becomes a much more interesting paradigm than OOP for this. The same logic applies whether you render the entire UI 60 times a second or you cache the previous render to only do the minimum amount of updates. A good abstraction is one where features can emerge from its design. For example you rarely have to test your views when they're just a simple pure transform. You don't need an entire framework and command-patterns to implement undo when your app state is immutable and modelled as a sequence of actions. With each emergent feature you remove the need for tons of code and tooling; that yields agility. I still believe most web application are much more complex than they should be, especially on the UI level, regardless of the framework used. And until we solve that problem, I don't think the web stack is the way to go for native. We're just replacing one set of problems for another one. I strongly believe we shouldn't make things easier, but simpler.

lwansbrough 8y ago
I'm trying really hard to understand your argument. So you believe such a UI API should be immutable (whatever that means in this context), and modelled as a sequence of actions? I think that's fine, but probably not practical in reality. I don't see the point of having a debate about the merits of functional programming vs. OOP in the middle of your post. People are going to build tools around the API that will abstract away whatever semantics you feel are best in order to fit their own needs. So the goal is to build an API that's simple at it's core to implement and use. Inevitably someone actually has to go and code the adapter layer between the protocol receiver and the actual low level rendering layer (be it OpenGL, UIKit, etc.) and unless you build a system that's easy to understand on that end as well, you're gonna end up with an API no one actually likes or wants to use. I don't think it's unreasonable to expect that the low-level rendering engine (ie. an implementation in UIKit) maintains some level of UI state. That is - after all - where most of the optimizations will come from. And that's how UIKit, OpenGL and most other rendering systems work. Things generally aren't immutable because it's a.) not efficient, and b.) a nightmare to work with at the lowest levels. It seems to me the appropriate abstraction is build the protocol around a UI state tree, because that's what everyone else is already working with. This is incredibly easy to maintain on either end, and higher level libraries can easily remove the pain of UI state management (a la React, Vue, etc.)

basch 8y ago

have you heard of React Canvas? https://engineering.flipboard.com/2015/02/mobile-web https://engineering.flipboard.com/2015/02/mobile-web