6 ms·

I mean, I’m looking at this on mobile and desktop and just like a another poster mentioned, in relative terms, the damn thing is smoother than most web apps. W

I mean, I’m looking at this on mobile and desktop and just like a another poster mentioned, in relative terms, the damn thing is smoother than most web apps. What’s the end goal here, right? The web sucks at the moment and I’m open to all ideas.

naasking 5y ago
Smoother yes, but to be fair how much CPU and power is it using? That's invisible to you right now so appearances can be deceptive.
andrewmcwatters 5y ago
As soon as you want to do anything remotely as complex as the web does, like say arbitrary UI element sizes with scrollable overflow, you immediately have to do the same thing CSS 2.1 rasterizer engines do.
- runawaybottle 5y ago
  I think you are just dead wrong about this. I’ve played video games my whole life and they always always had more complicated UIs, akin to web apps. It’s always been a richer experience.
  - andrewmcwatters 5y ago
    I am the author of a CSS 2.1 rasterizer, as well as the author of GUI software for game engines. They have different ways they draw user-interfaces. CSS 2.1 and the associated modules and extensions to it that you would use today to build UIs are much more complicated to implement than game engine UI. If the opposite were true, you'd see more web browsers that weren't KHTML derivatives, but that's not the case.
    - runawaybottle 5y ago
      I am the author of a CSS 2.1 rasterizer Yikes, checkmate. I guess, the frustration expressed has more to do with applications shifting to the browser. However we are rendering now is just not acceptable for web apps. My gut said ‘fuck it, do what games do’. We need something different, and I’d be happy to hear your thoughts.
      - andrewmcwatters 5y ago
        I agree with you. I think the real issue is that web developers don't have any ability to control the compositor besides hinting it to move portions of documents into explicit backing layers. You do this today with CSS transform operations, because those operations are done in shaders today. That being said, I don't know if web developers should have that type of access, either. The slowness you experience is based primarily on not redrawing something once, but several times, as modern web browsers need to break down very large web pages into tiles, then repeat draw instructions per tile where intersections exist. If I recall correctly, WebKit actually has an option to let you see the tile boundaries and when layers invalidate. This is my best understanding of the architecture KHTML derivatives use today. Compositing is very slow. And layout algorithms are very, very slow. So you can either do some fun stuff that I think imgui does, like sending VBOs over which is very performant, because you can draw lots of elements but you can't at all do things that people care about like compositing (think blurs behind panels, or layer opacity, etc.), or you can composite layers, but this becomes very expensive and slow to do. If you simply said, hey redraw this portion of a web page into a single framebuffer, you'd cut down on draw time a little bit, I'm sure, but because you can't allocate framebuffers reliably at arbitrary sizes, or beyond the viewport size, you run into GUI software that has weird limitations like all of a sudden you notice that for some reason you can't resize windows beyond the bounds of your screen, etc. I think Valve actually did this somehow, but I'm not sure how they did it with their old VGUI code. Anyway, all of that being said, today's modern GUI architectures allow for two choices: fast and lot of elements with no compositing, or slow and really good looking GUIs. It turns out most people want the latter, but complain about it, because reasonably(!) it's not fast, and hardware keeps getting faster, but GUIs are one of the prime examples of us eating up all of that performance.
- kragen 5y ago
  Which same thing do you mean? I read your very interesting longer comment below, and you mentioned a lot of differences, and it's not clear which of them you mean arbitrary UI element sizes with scrollable overflow requires you to do. Do you mean that it requires you to use very slow layout algorithms? Maybe the problem there is that CSS wasn't designed to be efficiently implementable; I feel like CSS doesn't usually give me better layouts than LaTeX, which runs orders of magnitude faster than any CSS engine. Is there a specific reason that scrollable overflow (rather than visible or hidden, which TeX can do) inherently makes the problem much harder than what TeX does? Naively I'd think making the overflow scrollable wouldn't affect the computational complexity at all, but clearly I know a lot less about the space than you do. Are there other things in CSS that are very useful for UI layout but very difficult to implement with submillisecond timing guarantees? As I understand it, for example, iOS UI autolayout uses Cassowary, which uses the simplex algorithm and I think may at a better point in the expressiveness/performance tradeoff space than CSS. (Hilariously in this context, the original Cassowary paper https://constraints.cs.washington.edu/solvers/cassowary-tochi.pdf https://constraints.cs.washington.edu/solvers/cassowary-toch... has an overfull hbox.) But Cassowary is from 02002, and there have been a lot of advancements in constraint solving and optimization algorithms in the last 19 years, both in theory and in practice: • Z3 wasn't released until 02012; • SAT/SMT solvers in general have gotten astoundingly better (the SAT Competition started in 02002); • there have been enormous advances in conic optimization (overview in https://arxiv.org/abs/1709.08841 https://arxiv.org/abs/1709.08841), including open-source libraries like CVXOPT that are widely used (in the context of conic optimization, anyway); • for more general optimization, automatic differentiation has gone mainstream; and • there have been huge advances in gradient-descent variants including AdaGrad (02011), RMSprop (02014?), and Adam (02014). In general, gradient-descent solvers and the interior-point methods often used for conic optimization are anytime algorithms, so you can trade off responsiveness against stability/correctness. If your layout algorithm hasn't converged by the frame deadline, you can just draw the best layout it's come up with so far; it's guaranteed to have all the right stuff in it, just maybe overlapping or in the wrong place, probably only by a few pixels, and displaying something is often better than displaying nothing. The conic optimization algorithms can also guarantee that the layout they deliver is "feasible" in the sense of respecting all the hard constraints in the constraint set; it might just be suboptimal. (Gradient-descent algorithms don't really have hard constraints; they have to simulate them by adding heavy penalties to the loss function.) I don't know of anybody actually applying this stuff to UI layout; do you?

naasking 5y ago

Smoother yes, but to be fair how much CPU and power is it using? That's invisible to you right now so appearances can be deceptive.

andrewmcwatters 5y ago

As soon as you want to do anything remotely as complex as the web does, like say arbitrary UI element sizes with scrollable overflow, you immediately have to do the same thing CSS 2.1 rasterizer engines do.

runawaybottle 5y ago
I think you are just dead wrong about this. I’ve played video games my whole life and they always always had more complicated UIs, akin to web apps. It’s always been a richer experience.
- andrewmcwatters 5y ago
  I am the author of a CSS 2.1 rasterizer, as well as the author of GUI software for game engines. They have different ways they draw user-interfaces. CSS 2.1 and the associated modules and extensions to it that you would use today to build UIs are much more complicated to implement than game engine UI. If the opposite were true, you'd see more web browsers that weren't KHTML derivatives, but that's not the case.
  - runawaybottle 5y ago
    I am the author of a CSS 2.1 rasterizer Yikes, checkmate. I guess, the frustration expressed has more to do with applications shifting to the browser. However we are rendering now is just not acceptable for web apps. My gut said ‘fuck it, do what games do’. We need something different, and I’d be happy to hear your thoughts.
    - andrewmcwatters 5y ago
      I agree with you. I think the real issue is that web developers don't have any ability to control the compositor besides hinting it to move portions of documents into explicit backing layers. You do this today with CSS transform operations, because those operations are done in shaders today. That being said, I don't know if web developers should have that type of access, either. The slowness you experience is based primarily on not redrawing something once, but several times, as modern web browsers need to break down very large web pages into tiles, then repeat draw instructions per tile where intersections exist. If I recall correctly, WebKit actually has an option to let you see the tile boundaries and when layers invalidate. This is my best understanding of the architecture KHTML derivatives use today. Compositing is very slow. And layout algorithms are very, very slow. So you can either do some fun stuff that I think imgui does, like sending VBOs over which is very performant, because you can draw lots of elements but you can't at all do things that people care about like compositing (think blurs behind panels, or layer opacity, etc.), or you can composite layers, but this becomes very expensive and slow to do. If you simply said, hey redraw this portion of a web page into a single framebuffer, you'd cut down on draw time a little bit, I'm sure, but because you can't allocate framebuffers reliably at arbitrary sizes, or beyond the viewport size, you run into GUI software that has weird limitations like all of a sudden you notice that for some reason you can't resize windows beyond the bounds of your screen, etc. I think Valve actually did this somehow, but I'm not sure how they did it with their old VGUI code. Anyway, all of that being said, today's modern GUI architectures allow for two choices: fast and lot of elements with no compositing, or slow and really good looking GUIs. It turns out most people want the latter, but complain about it, because reasonably(!) it's not fast, and hardware keeps getting faster, but GUIs are one of the prime examples of us eating up all of that performance.
kragen 5y ago
Which same thing do you mean? I read your very interesting longer comment below, and you mentioned a lot of differences, and it's not clear which of them you mean arbitrary UI element sizes with scrollable overflow requires you to do. Do you mean that it requires you to use very slow layout algorithms? Maybe the problem there is that CSS wasn't designed to be efficiently implementable; I feel like CSS doesn't usually give me better layouts than LaTeX, which runs orders of magnitude faster than any CSS engine. Is there a specific reason that scrollable overflow (rather than visible or hidden, which TeX can do) inherently makes the problem much harder than what TeX does? Naively I'd think making the overflow scrollable wouldn't affect the computational complexity at all, but clearly I know a lot less about the space than you do. Are there other things in CSS that are very useful for UI layout but very difficult to implement with submillisecond timing guarantees? As I understand it, for example, iOS UI autolayout uses Cassowary, which uses the simplex algorithm and I think may at a better point in the expressiveness/performance tradeoff space than CSS. (Hilariously in this context, the original Cassowary paper https://constraints.cs.washington.edu/solvers/cassowary-tochi.pdf https://constraints.cs.washington.edu/solvers/cassowary-toch... has an overfull hbox.) But Cassowary is from 02002, and there have been a lot of advancements in constraint solving and optimization algorithms in the last 19 years, both in theory and in practice: • Z3 wasn't released until 02012; • SAT/SMT solvers in general have gotten astoundingly better (the SAT Competition started in 02002); • there have been enormous advances in conic optimization (overview in https://arxiv.org/abs/1709.08841 https://arxiv.org/abs/1709.08841), including open-source libraries like CVXOPT that are widely used (in the context of conic optimization, anyway); • for more general optimization, automatic differentiation has gone mainstream; and • there have been huge advances in gradient-descent variants including AdaGrad (02011), RMSprop (02014?), and Adam (02014). In general, gradient-descent solvers and the interior-point methods often used for conic optimization are anytime algorithms, so you can trade off responsiveness against stability/correctness. If your layout algorithm hasn't converged by the frame deadline, you can just draw the best layout it's come up with so far; it's guaranteed to have all the right stuff in it, just maybe overlapping or in the wrong place, probably only by a few pixels, and displaying something is often better than displaying nothing. The conic optimization algorithms can also guarantee that the layout they deliver is "feasible" in the sense of respecting all the hard constraints in the constraint set; it might just be suboptimal. (Gradient-descent algorithms don't really have hard constraints; they have to simulate them by adding heavy penalties to the loss function.) I don't know of anybody actually applying this stuff to UI layout; do you?