2 ms·

I would say the most labor intensive aspect is that indefinite element boundaries require you to pretty much have backing layers that are tiled, or if you're OK

I would say the most labor intensive aspect is that indefinite element boundaries require you to pretty much have backing layers that are tiled, or if you're OK with the trade off, because sometimes it is OK, to limit elements with scrollable panels to the viewport size, since that's generally what happens with framebuffers or rendertarget graphics APIs, I believe. You also don't always get to receive texture sizes based on your request, so you have to waste some space using power of 2 sizes sometimes. Yeah, layout algorithms add a lot of overhead and even the fastest ones today still add significant overhead that you have to account for versus manually placing elements by coordinates, scaled or not. I think TeX is faster generally speaking since it is concerned with fewer dimensions of layout. This is really what you want for a fast layout engine, but not what designers want for really rich UIs. It's hard for me to compare this without intuitively knowing it to be true, because I would need to measure the actual amount of layout CPU cycles, roughly, to get a sense of how efficient one over the other is. I'm not really sure about whether CSS inherently produces slower layouts. I wouldn't assume it would, but rather that specific portions of the box model are difficult or not possible to parallelize, and as a result become an obvious bottleneck. Scrollable overflows aren't inherently slow, although if you have internal target spaces that dynamically change in size very frequently, you would, as one would expect, be thrashing any allocated single framebuffers for that visible space if you took that strategy over tiled rendering. The best things about layouts in CSS is that CSS 2.1 is so old, everyone has been exposed to it a little bit, even if they can't formally implement it themselves. Developers generally have a little bit of a sense of what display properties do. That same oldness of CSS 2.1 unfortunately means that the algorithm for box layout hasn't been revisited by many people. If you took every person in the world who had actually implemented a significant portion of the box model rasterizer, we wouldn't easily fill a room. It's basically descriptive, and left as an exercise to the author to implement. People gave Microsoft a really hard time over this, understandably, but if you gave it a go yourself, you'd understand why. I'm unfortunately not intimately knowledgeable with Cassowary from an implementation point of view, but I have observed its design and would be very interested to see it placed side by side with a reference implementation of the box model layout, and flexbox in particular, but no one has created such an agnostic rasterizer platform to my knowledge other than browser authors themselves. I'd love to see more constraint solvers in the wild! We really need some variety out here. I believe some people in the WHATWG have been working on a standard to expose layout events to JavaScript, so maybe we'll have something a little bit more native than Cassowary implemented with JavaScript and `position: absolute` (which is totally an OK override to implement your own layout algorithm!). So while I am not aware of anyone applying this today, I do actively see the foundations making it possible. Anyway, basically game UIs use VBOs and no compositing, or viewport dimension constrained framebuffers and a lot of textures. Both of those strategies are nothing like the web, because the web uses tiled backing layers since you never know what a web developer is going to do. Are they going to create a 10000 pixel high rasterized document fragment? No idea. But if we don't tile it, it happens in software. That's slow, so no one does that anymore. The foundational difference in game UIs is usually in whether or not you're pushing a bunch of vertices and color data to the GPU alongside usually Freetype rasterized type, or a bunch of textures and color data alongside Freetype rasterized type. Yes, I know textures are accompanied by vertex data, too, but the general idea is that you're either reusing that texture data to do cool things like blurs, or you're not, and that's the gist of compositing, which is slow, but pretty. With browsers you make a bunch of usually 256 or 512(?) px wide and tall tiles, where geometry intersects you run some graphics commands that help draw that portion of the UI, and that's how your compositor works instead. I'm not sure what the industry has measured the ideal tile size to be these days, but I know that's on the mind of engineers, since you want to minimize redraw as much as possible.

kragen 5y ago
I'm so ignorant about modern graphics programming that I don't know what backing layers are or why they should be tiled. Googling doesn't help. Maybe they're offscreen pixel buffers you draw into? I guess VBOs are vertex buffer objects? How can a web developer create a rasterized document fragment at all, much less one ten thousand pixels high? Do you mean, like, creating a huge offscreen <canvas> and drawing stuff on it? (But in that case how would CSS layout be involved at all?) I think TeX typically takes in the ballpark of ten million machine instructions per page of text (say, 256-1024 words) to do the layout and produce the .dvi output. To guarantee reproducibility, it doesn't use floating-point. (It also doesn't use multithreading because when it was written in the 01980s multiprocessor computers were an exotic rarity, and threading libraries even rarer.) As a test, I downloaded the source files for https://arxiv.org/abs/1709.08841 https://arxiv.org/abs/1709.08841 and rendered them with valgrind --tool=cachegrind latex final_acc.tex, which took 2,440,969,752 instructions (in 1.1 seconds when run without valgrind), about half of them in startup stuff (loading TikZ, hyperref, amsmath, and so on) and the other half to render the 18 pages themselves, about 12000 words of text. That's about 70 million instructions per page or a hundred thousand instructions per word, including all the parsing and bibliographic database querying but none of the rasterizing --- TeX thinks of letterforms as just boxes and leaves the pixels to dvipdf or xdvi or whatever. (dvi2fax takes 1.7 seconds to convert the .dvi to G3 fax images.) I think this is an unusually slow document. I have the impression that this is about one to two orders of magnitude faster than doing CSS layout on common web pages, but usually the TeX results look better. But maybe I'm wrong about that? You say you think TeX is concerned with "fewer dimensions of layout". What kind of dimensions are you thinking about? Like, Z-order? My naive view of the pipeline is that you have some kind of "document" or "form" which specifies the things you want to display and how they relate to each other; then a layout algorithm (normally incremental) processes this document into a set of boxes, each of which has dimensions (x, y, w, h, layernum), like TeX in my example above; and then a rasterizer (like dvi2fax) uses these boxes to paint things like character glyphs and icons onto one or more layers (pixel buffers), at least the visible part of them, that then get alpha-composited together, maybe transformed using things like blurs and rotations and reflections and stuff. Am I suffering from some kind of basic misconception? Because that naive vision makes it really puzzling to think that blurring a background could affect the performance of the layout algorithm. I mean, rendering the blurred pixels themselves takes time, of course, both because computing the filter takes time and because you're rendering two layer pixels for every output pixel. But it doesn't affect the layout. Does it? I have the impression that, in my naive pipeline above, it's not the rasterizer or compositor that is the bottleneck in web pages, but the layout algorithm. I mean, I remember when scrolling text over a nonscrolling background image in a browser was slow, but that was a long time ago, and alpha-compositing together a screenful of two layers hardly seems like it should stretch the capabilities of even modern CPUs, much less modern GPUs; but, again, maybe I'm suffering from some basic misconceptions about what is involved. Isn't it the same problem as rendering a translucent HUD in a game UI while the 3-D world keeps getting rendered behind it? You don't have to actually iterate over pixels (or, you know, packed 128-bit vectors of pixels) for any images or letterforms that are outside the viewport, do you? I definitely do not mean to deprecate the difficulty of implementing CSS, much less implementing it efficiently; as you said elsewhere, if it were easy, we'd be seeing new CSS engines that weren't just forks of KHTML. But I suspect that CSS is hard to implement efficiently in large part because CSS is badly designed, and that a better design could simultaneously admit more efficient implementations and be more expressive. Cassowary might be an actual existing example of this.

kragen 5y ago

I'm so ignorant about modern graphics programming that I don't know what backing layers are or why they should be tiled. Googling doesn't help. Maybe they're offscreen pixel buffers you draw into? I guess VBOs are vertex buffer objects? How can a web developer create a rasterized document fragment at all, much less one ten thousand pixels high? Do you mean, like, creating a huge offscreen <canvas> and drawing stuff on it? (But in that case how would CSS layout be involved at all?) I think TeX typically takes in the ballpark of ten million machine instructions per page of text (say, 256-1024 words) to do the layout and produce the .dvi output. To guarantee reproducibility, it doesn't use floating-point. (It also doesn't use multithreading because when it was written in the 01980s multiprocessor computers were an exotic rarity, and threading libraries even rarer.) As a test, I downloaded the source files for https://arxiv.org/abs/1709.08841 https://arxiv.org/abs/1709.08841 and rendered them with valgrind --tool=cachegrind latex final_acc.tex, which took 2,440,969,752 instructions (in 1.1 seconds when run without valgrind), about half of them in startup stuff (loading TikZ, hyperref, amsmath, and so on) and the other half to render the 18 pages themselves, about 12000 words of text. That's about 70 million instructions per page or a hundred thousand instructions per word, including all the parsing and bibliographic database querying but none of the rasterizing --- TeX thinks of letterforms as just boxes and leaves the pixels to dvipdf or xdvi or whatever. (dvi2fax takes 1.7 seconds to convert the .dvi to G3 fax images.) I think this is an unusually slow document. I have the impression that this is about one to two orders of magnitude faster than doing CSS layout on common web pages, but usually the TeX results look better. But maybe I'm wrong about that? You say you think TeX is concerned with "fewer dimensions of layout". What kind of dimensions are you thinking about? Like, Z-order? My naive view of the pipeline is that you have some kind of "document" or "form" which specifies the things you want to display and how they relate to each other; then a layout algorithm (normally incremental) processes this document into a set of boxes, each of which has dimensions (x, y, w, h, layernum), like TeX in my example above; and then a rasterizer (like dvi2fax) uses these boxes to paint things like character glyphs and icons onto one or more layers (pixel buffers), at least the visible part of them, that then get alpha-composited together, maybe transformed using things like blurs and rotations and reflections and stuff. Am I suffering from some kind of basic misconception? Because that naive vision makes it really puzzling to think that blurring a background could affect the performance of the layout algorithm. I mean, rendering the blurred pixels themselves takes time, of course, both because computing the filter takes time and because you're rendering two layer pixels for every output pixel. But it doesn't affect the layout. Does it? I have the impression that, in my naive pipeline above, it's not the rasterizer or compositor that is the bottleneck in web pages, but the layout algorithm. I mean, I remember when scrolling text over a nonscrolling background image in a browser was slow, but that was a long time ago, and alpha-compositing together a screenful of two layers hardly seems like it should stretch the capabilities of even modern CPUs, much less modern GPUs; but, again, maybe I'm suffering from some basic misconceptions about what is involved. Isn't it the same problem as rendering a translucent HUD in a game UI while the 3-D world keeps getting rendered behind it? You don't have to actually iterate over pixels (or, you know, packed 128-bit vectors of pixels) for any images or letterforms that are outside the viewport, do you? I definitely do not mean to deprecate the difficulty of implementing CSS, much less implementing it efficiently; as you said elsewhere, if it were easy, we'd be seeing new CSS engines that weren't just forks of KHTML. But I suspect that CSS is hard to implement efficiently in large part because CSS is badly designed, and that a better design could simultaneously admit more efficient implementations and be more expressive. Cassowary might be an actual existing example of this.