4 ms·

It’s going to be a bloody mess for a long time, because the choices for how to handle resolution independence are all inherently filled with compromise. With f

by jchw 5y ago

It’s going to be a bloody mess for a long time, because the choices for how to handle resolution independence are all inherently filled with compromise. With font rendering, I think there is hope. Horizontal subpixel positioning with vertical hinting seems like a good tradeoff to me. Grid fitting vertically is not too jarring, and grid fitting horizontally to subpixels instead of pixels looks pretty good too, on low resolution displays. But it really is a son of a bitch elsewhere. For example, if you want a crisp 1px border on 96 dpi, you could specify it to be a 1px border at 96 dpi… but then what happens at 1.5x or 1.75x scale? From a purely logical position, the blurry line is actually the general case, and the integer scale case is actually an edge case. That desktop UIs aren’t blurry basically always is because we define them in terms of 96 DPI displays. It gets worse for APIs, because APIs that want to present a resolution-independent world will cause difficult to tolerate bugs. The VS Code terminal will often be blurry at non-integer scales because it is using HTML canvas. If the canvas width or height is not a multiple of the size of a CSS pixel, it will cause the internal buffer to be scaled horridly. The fix might be a new API that reveals true coordinates… very, very nasty. Apple’s solution was extreme: dump all font hacks, always render apps at 2x, then scale the whole framebuffer for different scale factors. It’s somewhat blurry, but avoids many ugly pitfalls in the common case, and makes apps simpler. Unfortunately, the rest of the world is just stuck with really bad scaling and more often blurring on 96 DPI displays, the worst of both worlds.

zozbot234 5y ago
> For example, if you want a crisp 1px border on 96 dpi, you could specify it to be a 1px border at 96 dpi… but then what happens at 1.5x or 1.75x scale? The border width should get snapped to the physical (sub-)pixel resolution as part of rendering. Typically, this should come with changes in contrast too, such that if a line is forced to become thinner it also gets drawn with higher contrast wrt. the surroundings, and vice versa. All of this stuff can be made to work. Also, if you are forced to render a canvas at a resampled resolution because existing APIs give you no other choice, at least do it right using a proper Lanczos-style resampling. This might end up with a quaint "watercolor" effect but guess what, that's a lot better than a blurry, eye-fatiguing mess.
- jchw 5y ago
  > All of this stuff can be made to work. We’ve tolerated a great degree of complexity just to make fonts look good at 96 DPI. Looks like we’re able to tolerate a bit more complexity to enable GPU rendering. However, many years into having high DPI displays, it’s not obvious people are willing to take the complexity to make low DPI and high DPI screens look good simultaneously. The thing is, with fonts, we already bear the burden of font rendering being complex because that was needed for 96 DPI displays. But, we won’t need much of this magic or complexity when a vast majority of people are using higher DPI displays, because at >200 PPI the difference between a blurry line and a sharp line is basically nil. That is obvious enough on Apple platforms, where many are perfectly happy with the scaling even though it uses 2x as a base for all scales. I think the future is simply pain. People want cleaner graphics pipelines, and only high DPI displays will get them anywhere.
  - hedora 5y ago
    Can't the font renderer render a straight line? Like sans serif "I”? The font rasterizer already exists (unless it is a bitmap font UI, but those aren't common any more). How does adding additional mechanisms make for a simpler or cleaner rendering pipeline?
    - jchw 5y ago
      The font rasterizer is a massive hack in modern UIs. Subpixel rendering is a serious pain in the ass. When you render text using subpixel rendering, you render the actual vectors at 3x the spatial resolution. But, not simply as if the vectors were 3x wider, because that would look too sharp: it needs to render as if there was 3x as many pixels, which is different. Then there’s compositing. Normal layers can be composited using alpha blending, assuming some sane format like premultiplied alpha RGBA. But not subpixel rendered text, because alpha blending the components will fuck up the subpixel rendering. And it goes on, because if you want to handle text like everything else, you need special cases for it to look right. Rotation? Need to render the vectors rotated; can’t rotate in raster. If you need to render to a surface then transform that surface, you’re SOL; it can’t go to rasters until the end. Normal surfaces can also be rendered at subpixel positions, and of course this does not work for surfaces containing text, because again, it will destroy the subpixel rendering. OK. So you can get rid of the subpixel rendering and render slightly blurrier glyphs instead. (R.I.P. anyone trying to tell hanzi/kanji apart.) It’s still going to murder legibility if you move it over by a subpixel value because text is already on the edge of readability at 96 DPI. I haven’t considered gamma correction, hinting, blending different colors, different blending modes, GPU acceleration, etc. because I simply don’t have the brain power to try to reconcile it all. It’s a nightmare. We already did some of this for text. Which is a herculean effort. We use a freakin virtual machine to power font hinting, and ugly, complex, slow special casing at many layers of already ridiculously complex vector graphics stacks (I mean if you disagree with that assessment, you may just be smarter than I am, but I have serious trouble following the Skia codebase and I doubt Cairo is really that much better.) And speaking of which, there only really seems to be a handful of them out there: there’s Skia, used by most web browsers; Cairo, used by GTK; Direct2D, in Windows; Whatever modern macOS uses that isn’t QuickDraw anymore; and I guess there’s Mozilla’s pathfinder, a promising Rust-based vector graphics engine that was built as part of Servo and seemingly mostly abandoned, much to the world’s detriment. This work is hard. It can be done, but it’s not something I think a single engineer can do, if you want to build one that competes with the big boys even disregarding a few things like performance. I’d love to be wrong, but I have a sinking feeling I’m not. Even text isn’t done being overcomplicated. As nyanpasu has mentioned above, some software have started implementing SDFs for font scaling. We do this because text legibility is really that important, whereas a line in the UI being slightly blurry for users on older screens is really just not that important. Some languages flat out can’t be read with crappy font rendering, and any of them will give you eyestrain if it’s ugly enough. As much as it sucks, a blurry border on a button doesn’t have an accessibility issue. And rendering at 1x and making the compositor upscale is not a great solution either because again, it’s already hard enough to read text in some languages; the added blurriness of scaling text and ruining subpixels is basically intolerable. These hacks aren’t free, and with high DPI displays, they’re not needed. There’s a reason Apple did what they did.
hedora 5y ago
I really don't understand what was wrong with the X11 approach. I had a high DPI monitor in 2001. I typed the DPI into /etc/XFree86.conf or whatever, and it all Just Worked (TM). Edit: I think modern web browsers implement ctrl-+ and ctrl-- the same way, except X11 apps kept separate directories of icons rendered for different DPIs, because 1GHz single core still seemed luxurious. Web browsers scale the bitmaps using some reasonable algorithm. Other than that, arbitrary zooms work with zero blur. For what it's worth, PostScript also got this right back in the 80s.
- jchw 5y ago
  Well, how much it Just Worked really depended on what you were doing and how. At a point, it all stopped Just Work-ing. Old old X11 apps used X11 drawing commands. These sucked, and nobody liked them. If you think you liked them, please show me your clean Xlib codebases for proof :P As far as I can recall, these still dealt with pixels, so clients were on the hook for dealing with scaling, though in theory it wasn’t too bad. They don’t really solve any of the pixel perfection issues that I am discussing, though. More modern apps (— early 2000s should be “modern” enough by X11 standards, but my memory is foggy and I’m too young to really be an expert here —) instead blit pixmaps sent over shmem, defeating both network transparency and the inherent “vector” nature of many of the old drawing commands. X11 didn’t really handle anything other than knowing the DPI (… that you told it …) At that point, up to GTK+2 and Qt 3, which is to say, even quite a while After 2001, you had at best limited scalability. If you had your CRT cranked up to around 150 PPI, everything was OK — you could get text scaling and the disparity wasn’t so bad. However, GTK+2 and Qt 3, and their ancestors, were not built with DPI independence. At best, they could adjust vector text sizes according to DPI and scaling preferences. Again, this looks OK for nvidia-xsettings and a modest PPI increase, but it’s absolutely terrible for anything more. Margins don’t adjust, padding doesn’t adjust, icon sizes don’t adjust, nothing. There’s no blur or jankiness because there’s no true scaling. (Just as a quick note, this is literally the reality of GIMP today, right now. It’s still on GTK+2, and so the best you can get is text scaling, or flat out nothing.) And that’s to say nothing about what happens if the DPI changes, which requires you to effectively restart everything. And that also doesn’t help people who have two different displays with different PPIs. The ever common case of the high DPI laptop with a cheap LCD plugged in. Have fun with that crap. Modern Linux can do better. The Wayland protocol comes with DPI negotiation that allows naive clients to get blurry upscaling, “simple” clients to pick a set of scales they can support and have the server adjust for whatever one they decide to render to, and advanced clients can render at any DPI, in response to the server advertising what DPI the current display is. With atomicity of configuration changes that allows a properly written client and server to never render an “intermediate” incorrect frame, and scaling that ensures that surfaces across multiple displays display at the correct DPI on all of them (albeit with either upscaling or downscaling on some of them.) And that still does absolutely nothing to solve the fact that pixel perfect layouts are inherently not perfectly “scalable.” Because truly scaling some vector drawing commands that just happen to be pixel perfect at one resolution will not always result in pixel perfect rendering in another. You would need code that compensates for the scaling. Old X11 apps did not do this. Of course I could be completely wrong and old X11 could’ve had some amazing DPI scaling technology that I somehow missed for decades. I don’t think so. My memory is that when I finally hooked up a high DPI display to Linux, I experienced tiny Skype, Pidgin (GAIM) with tiny icons and large text, and nvidia-xsettings with weird hinting/kerning. I’d like to move on from that kind of scaling. P.S.: PostScript doesn’t do anything magic either. Everyone’s graphics systems were PostScript inspired, and yet macOS wound up with the same DPI scaling conundrums as anyone else. Most people wouldn’t tolerate desktop apps as blurry as a PDF at 96 DPI.
divingdragon 5y ago
> The fix might be a new API that reveals true coordinates… We already have this with `ResizeObserver` using the `device-pixel-content-box` option. [1] [1]: https://developer.mozilla.org/en-US/docs/Web/API/ResizeObserver/observe https://developer.mozilla.org/en-US/docs/Web/API/ResizeObser...
- jchw 5y ago
  Pretty sure this API is brand new (last ~year or so) but I honestly had no idea it landed into the standards. Cool, although still, ugly :/

by jchw 5y ago

zozbot234 5y ago

> For example, if you want a crisp 1px border on 96 dpi, you could specify it to be a 1px border at 96 dpi… but then what happens at 1.5x or 1.75x scale? The border width should get snapped to the physical (sub-)pixel resolution as part of rendering. Typically, this should come with changes in contrast too, such that if a line is forced to become thinner it also gets drawn with higher contrast wrt. the surroundings, and vice versa. All of this stuff can be made to work. Also, if you are forced to render a canvas at a resampled resolution because existing APIs give you no other choice, at least do it right using a proper Lanczos-style resampling. This might end up with a quaint "watercolor" effect but guess what, that's a lot better than a blurry, eye-fatiguing mess.

jchw 5y ago
> All of this stuff can be made to work. We’ve tolerated a great degree of complexity just to make fonts look good at 96 DPI. Looks like we’re able to tolerate a bit more complexity to enable GPU rendering. However, many years into having high DPI displays, it’s not obvious people are willing to take the complexity to make low DPI and high DPI screens look good simultaneously. The thing is, with fonts, we already bear the burden of font rendering being complex because that was needed for 96 DPI displays. But, we won’t need much of this magic or complexity when a vast majority of people are using higher DPI displays, because at >200 PPI the difference between a blurry line and a sharp line is basically nil. That is obvious enough on Apple platforms, where many are perfectly happy with the scaling even though it uses 2x as a base for all scales. I think the future is simply pain. People want cleaner graphics pipelines, and only high DPI displays will get them anywhere.
- hedora 5y ago
  Can't the font renderer render a straight line? Like sans serif "I”? The font rasterizer already exists (unless it is a bitmap font UI, but those aren't common any more). How does adding additional mechanisms make for a simpler or cleaner rendering pipeline?
  - jchw 5y ago
    The font rasterizer is a massive hack in modern UIs. Subpixel rendering is a serious pain in the ass. When you render text using subpixel rendering, you render the actual vectors at 3x the spatial resolution. But, not simply as if the vectors were 3x wider, because that would look too sharp: it needs to render as if there was 3x as many pixels, which is different. Then there’s compositing. Normal layers can be composited using alpha blending, assuming some sane format like premultiplied alpha RGBA. But not subpixel rendered text, because alpha blending the components will fuck up the subpixel rendering. And it goes on, because if you want to handle text like everything else, you need special cases for it to look right. Rotation? Need to render the vectors rotated; can’t rotate in raster. If you need to render to a surface then transform that surface, you’re SOL; it can’t go to rasters until the end. Normal surfaces can also be rendered at subpixel positions, and of course this does not work for surfaces containing text, because again, it will destroy the subpixel rendering. OK. So you can get rid of the subpixel rendering and render slightly blurrier glyphs instead. (R.I.P. anyone trying to tell hanzi/kanji apart.) It’s still going to murder legibility if you move it over by a subpixel value because text is already on the edge of readability at 96 DPI. I haven’t considered gamma correction, hinting, blending different colors, different blending modes, GPU acceleration, etc. because I simply don’t have the brain power to try to reconcile it all. It’s a nightmare. We already did some of this for text. Which is a herculean effort. We use a freakin virtual machine to power font hinting, and ugly, complex, slow special casing at many layers of already ridiculously complex vector graphics stacks (I mean if you disagree with that assessment, you may just be smarter than I am, but I have serious trouble following the Skia codebase and I doubt Cairo is really that much better.) And speaking of which, there only really seems to be a handful of them out there: there’s Skia, used by most web browsers; Cairo, used by GTK; Direct2D, in Windows; Whatever modern macOS uses that isn’t QuickDraw anymore; and I guess there’s Mozilla’s pathfinder, a promising Rust-based vector graphics engine that was built as part of Servo and seemingly mostly abandoned, much to the world’s detriment. This work is hard. It can be done, but it’s not something I think a single engineer can do, if you want to build one that competes with the big boys even disregarding a few things like performance. I’d love to be wrong, but I have a sinking feeling I’m not. Even text isn’t done being overcomplicated. As nyanpasu has mentioned above, some software have started implementing SDFs for font scaling. We do this because text legibility is really that important, whereas a line in the UI being slightly blurry for users on older screens is really just not that important. Some languages flat out can’t be read with crappy font rendering, and any of them will give you eyestrain if it’s ugly enough. As much as it sucks, a blurry border on a button doesn’t have an accessibility issue. And rendering at 1x and making the compositor upscale is not a great solution either because again, it’s already hard enough to read text in some languages; the added blurriness of scaling text and ruining subpixels is basically intolerable. These hacks aren’t free, and with high DPI displays, they’re not needed. There’s a reason Apple did what they did.

hedora 5y ago

I really don't understand what was wrong with the X11 approach. I had a high DPI monitor in 2001. I typed the DPI into /etc/XFree86.conf or whatever, and it all Just Worked (TM). Edit: I think modern web browsers implement ctrl-+ and ctrl-- the same way, except X11 apps kept separate directories of icons rendered for different DPIs, because 1GHz single core still seemed luxurious. Web browsers scale the bitmaps using some reasonable algorithm. Other than that, arbitrary zooms work with zero blur. For what it's worth, PostScript also got this right back in the 80s.

jchw 5y ago
Well, how much it Just Worked really depended on what you were doing and how. At a point, it all stopped Just Work-ing. Old old X11 apps used X11 drawing commands. These sucked, and nobody liked them. If you think you liked them, please show me your clean Xlib codebases for proof :P As far as I can recall, these still dealt with pixels, so clients were on the hook for dealing with scaling, though in theory it wasn’t too bad. They don’t really solve any of the pixel perfection issues that I am discussing, though. More modern apps (— early 2000s should be “modern” enough by X11 standards, but my memory is foggy and I’m too young to really be an expert here —) instead blit pixmaps sent over shmem, defeating both network transparency and the inherent “vector” nature of many of the old drawing commands. X11 didn’t really handle anything other than knowing the DPI (… that you told it …) At that point, up to GTK+2 and Qt 3, which is to say, even quite a while After 2001, you had at best limited scalability. If you had your CRT cranked up to around 150 PPI, everything was OK — you could get text scaling and the disparity wasn’t so bad. However, GTK+2 and Qt 3, and their ancestors, were not built with DPI independence. At best, they could adjust vector text sizes according to DPI and scaling preferences. Again, this looks OK for nvidia-xsettings and a modest PPI increase, but it’s absolutely terrible for anything more. Margins don’t adjust, padding doesn’t adjust, icon sizes don’t adjust, nothing. There’s no blur or jankiness because there’s no true scaling. (Just as a quick note, this is literally the reality of GIMP today, right now. It’s still on GTK+2, and so the best you can get is text scaling, or flat out nothing.) And that’s to say nothing about what happens if the DPI changes, which requires you to effectively restart everything. And that also doesn’t help people who have two different displays with different PPIs. The ever common case of the high DPI laptop with a cheap LCD plugged in. Have fun with that crap. Modern Linux can do better. The Wayland protocol comes with DPI negotiation that allows naive clients to get blurry upscaling, “simple” clients to pick a set of scales they can support and have the server adjust for whatever one they decide to render to, and advanced clients can render at any DPI, in response to the server advertising what DPI the current display is. With atomicity of configuration changes that allows a properly written client and server to never render an “intermediate” incorrect frame, and scaling that ensures that surfaces across multiple displays display at the correct DPI on all of them (albeit with either upscaling or downscaling on some of them.) And that still does absolutely nothing to solve the fact that pixel perfect layouts are inherently not perfectly “scalable.” Because truly scaling some vector drawing commands that just happen to be pixel perfect at one resolution will not always result in pixel perfect rendering in another. You would need code that compensates for the scaling. Old X11 apps did not do this. Of course I could be completely wrong and old X11 could’ve had some amazing DPI scaling technology that I somehow missed for decades. I don’t think so. My memory is that when I finally hooked up a high DPI display to Linux, I experienced tiny Skype, Pidgin (GAIM) with tiny icons and large text, and nvidia-xsettings with weird hinting/kerning. I’d like to move on from that kind of scaling. P.S.: PostScript doesn’t do anything magic either. Everyone’s graphics systems were PostScript inspired, and yet macOS wound up with the same DPI scaling conundrums as anyone else. Most people wouldn’t tolerate desktop apps as blurry as a PDF at 96 DPI.

divingdragon 5y ago

> The fix might be a new API that reveals true coordinates… We already have this with `ResizeObserver` using the `device-pixel-content-box` option. [1] [1]: https://developer.mozilla.org/en-US/docs/Web/API/ResizeObserver/observe https://developer.mozilla.org/en-US/docs/Web/API/ResizeObser...

jchw 5y ago
Pretty sure this API is brand new (last ~year or so) but I honestly had no idea it landed into the standards. Cool, although still, ugly :/