9 ms·

HTML-in-Canvas

222 points by dannyobrien 1y ago

tantalor 1y ago
https://html2canvas.hertzen.com/ https://html2canvas.hertzen.com/
- Waterluvian 1y ago
  45kb gzipped is pretty beefy but incredibly small when you consider just what it takes to make this work today. If I understand correctly, it’s basically a DOM and CSS renderer.
- kizer 1y ago
  There's a bunch of CSS, etc. not supported by that. It would be great to have access to a native API to get bitmaps of the DOM.
  - masswerk 1y ago
    Well, a fingerprinting dream… Meaning, no way, just for the security aspect.
ttoinou 1y ago
Soon we’re gonna need Canvas rendering inside of HTML-in-Canvas
- bastawhiz 1y ago
  It should already work if the nested canvas uses the same approach. It's not cyclic, though. To make cyclic canvases work, you need to manually draw the parent canvas to a nested canvas.
  - koolala 1y ago
    Now your thinking with canvas.
hyperhello 1y ago
I see reliable rich text editing as more of a priority.
SeanAnderson 1y ago
I read the title and said "shut the fuck up, don't do that." but then I read the rationale and it's fair. It's true there is no layout engine inside canvas, and that is a pain, but I'm not sure it's such a pain as to invite this recursive hell.
- teaearlgraycold 1y ago
  You could disallow recursion.
  - deleted 1y ago
    [deleted]
- nine_k 1y ago
  One of the more senior engineers I worked with told me: "Every real-life data structure I encountered was tree-like". It would be easiest to just ask the browser to render a fragment of HTML onto a canvas, or onto some invisible bitmap, like you can with most other UI toolkits.
  - monster_truck 1y ago
    They would never do this because of fingerprinting, which is already the cause of most of the reasons we cannot 'just' do a lot of things, unfortunately. E: And the infamous other half: malware. A bit over a decade ago malware devs started using canvas to do things like hide fragments inside of bitmap data in seemingly harmless ads and then a second script would extract and assemble it to evade detection.
    - rezonant 1y ago
      The web platform can already do this, see SVG foreignObject elsewhere in the thread. The key is to have the proper bounds in place (cross origin resources, etc), and the infrastructure for that is already in place. This just removes the extra step of relying on SVG to accomplish rendering the HTML, adds a path for getting this content into the accessibility tree, and supporting input on the rendered elements.
mmastrac 1y ago
https://github.com/WICG/html-in-canvas/blob/main/security-privacy-questionnaire.md https://github.com/WICG/html-in-canvas/blob/main/security-pr... > TODO: Expand on fingerprinting risks
- rapnie 1y ago
  I searched for it in the text, but it is the missing one to add (and might cross-ref to "TODO: Don't be evil").
codelikeawolf 1y ago
I immediately got "Pimp My Ride" vibes. Yo dawg, I heard you like HTML so I put HTML inside the canvas inside the HTML.
BobbyTables2 1y ago
I suppose all we need to do now is to compile a browser into WASM and run that browser inside a canvas element of the main browser…
- sangeeth96 1y ago
  How about an entire OS with IE? https://copy.sh/v86/?profile=windows2000 https://copy.sh/v86/?profile=windows2000
- socalgal2 1y ago
  https://www.chromium.org/blink/blink-in-js/ https://www.chromium.org/blink/blink-in-js/
- alexhixon 1y ago
  https://trevorlinton.github.io https://trevorlinton.github.io
- xdennis 1y ago
  That's the premise of The Birth & Death of JavaScript[1]. [1]: https://www.destroyallsoftware.com/talks/the-birth-and-death-of-javascript https://www.destroyallsoftware.com/talks/the-birth-and-death...
- mkoubaa 1y ago
  And then, 'for sandboxing and security reasons', this will become mandatory
deadbabe 1y ago
true horror is finding every element of the HTML page your on has been rendered in a Canvas
- grimgrin 1y ago
  exactly, we need a standard https://github.com/WICG/html-in-canvas https://github.com/WICG/html-in-canvas :P
turnsout 1y ago
Sad to say, but the first thing I thought was "oh good, a new fingerprinting target."
- koolala 1y ago
  Can you give an example? If I used this to load HTML into a 3D environment every user could have the same virtual screen size.
  - turnsout 1y ago
    Every browser renders HTML slightly differently, so you could render some arbitrary HTML into the canvas and then read out the pixels.
    - koolala 1y ago
      I see what you mean. Isn't that already what the user agent tells you?
      - turnsout 1y ago
        Oh my sweet summer child. The game of cat and mouse is so much more vicious than that.
charcircuit 1y ago
This sounds really useful for being able to use standard web technology with webxr.
- socalgal2 1y ago
  I've never understood why they couldn't have just used the existing 3D CSS for WebXR. All the data is there, all they need to do is render the DOM from 2 POVs, one for each eye. They could even have had some standard to let it auto composite with WebGL.
ha1zum 1y ago
This could have a great utility, but for me HTML in canvas in HTML feels so cursed. To make it make sense in my opinion canvas should already be a first class format for web browsers, so it doesn't have to be inside a HTML. Then we would have a choice of HTML-first page with canvas elements in it, or a canvas-first page with HTML elements in it. But what do I know.
- Liron 1y ago
  What if you want an HTML-first page with a canvas in it, but then you realize you want some layout/styling for the text within the canvas? Seems unnecessary to propagate that situation up to the type of top-level page.
  - thrance 1y ago
    And then what if you realize you need a canvas-in-the-html-in-the-canvas? It's endless. Canvas-first makes sense, it's basically how it works everywhere outside of the web. Start with the smallest abstractions and build on them (html on canvas) rather than leave escape hatches to your big abstractions because they fail to cover every use cases (canvas in html).
    - sitkack 1y ago
      If you support the DOM and hitscan, then it doesn't matter. You can red pill Ouroboros yourself all day and not care. Every element a canvas, every raindrop an ocean.
- socalgal2 1y ago
  canvas first sites suck. They can't use any system services as it would all be a privacy issue. They can't use the system dictionary for correction since to do so they'd need the contents of the dictionary or at least a way to query user customized corrections. Similarly they can't offer the system level accessibility but end up having to roll their own in which case, every app that uses canvas has a completely different UI.
tinalovely101 1y ago
It seems to be impossible with html version.
ivanjermakov 1y ago
RIP accessibility
- progers7 1y ago
  Accessibility is a key reason for this proposal. Today, canvas accessibility is quite limited. This proposal enables the browser to know how accessible dom elements map to canvas pixels.
- koolala 1y ago
  html is the most accessible language. video games have to hardcode everything.
c-smile 1y ago
From the very beginning in Sciter an Image can be constructed in two ways at runtime ( other than just getting loaded image reference): 1. By painting on it using Canvas/Graphics API: new Graphics.Image(width, height, painter(graphics) [,initColor]); Where _painter_ is a function used for paining on the image surface using Canvas/Graphics reference. 2. By making snapshot of the existing DOM element: new Graphics.Image(width, height, element [,initColor]) Such images can be used in DOM, rendered by other Canvas/Graphics as also in WebGL as textures. See: https://docs.sciter.com/docs/Graphics/Image#constructor https://docs.sciter.com/docs/Graphics/Image#constructor
donatj 1y ago
I feel like this is bound to get kneecapped for the same reason as other canvas features, fingerprinting.
- cluckindan 1y ago
  Fingerprinting is not an issue of canvas drawing, it is an issue of canvas reading (ImageData).
cheeaun 1y ago
Kinda reminds me of Flipboard https://engineering.flipboard.com/2015/02/mobile-web https://engineering.flipboard.com/2015/02/mobile-web (2015)
wetpaws 1y ago
[dead]
talkingtab 1y ago
There is a real problem using canvas to replace HTML. Not all but most HTML. I have not found a good solution for the issue of doing something like MDX in canvas. I have tried SDF, looked at 2D canvas Text, Troika, MSDF. You can get text, it is just that laying it out is very difficult. React three drei has the ability to put HTML into the threejs ecosystem, but there are issues about CSS and text that make that impractical. For me the use case is very simple. I would like to take an MDX file and show it in a mesh. Laid out. Maybe I am missing something because I am new to the whole threejs thing, but I really tried. A good article about text https://css-tricks.com/techniques-for-rendering-text-with-webgl/ https://css-tricks.com/techniques-for-rendering-text-with-we... And an example from the above article: https://codesandbox.io/p/sandbox/css-tricks-msdf-text-fks8w https://codesandbox.io/p/sandbox/css-tricks-msdf-text-fks8w This shows it can be done, I gave up trying to reproduce it in React-three-fiber. Why? Personally, I think the use of 3D graphics produces an interface for users that is an order or magnitude better for users. The real question (and an interesting one to consider) is why are we still building HTML first websites?
- squidbeak 1y ago
  Why do we build HTML first websites? Because most websites serve documents, however much developers might wish they were applications.
nikeee 1y ago
Is that being made to make Flutter not being like Flash on the web?
- kbcool 1y ago
  I was searching for a comment like this. It sounds like a crazy workaround for Flutter's strange architectural choices
sampton 1y ago
Servo + wasm.
reactordev 1y ago
I support this, as odd as it is. There’s times when you’re needing something drawn but can easily reuse an html element from elsewhere. Previously you’d have to render that to a bitmap offscreen and then copy that to a full screen quad or draw it on the canvas. Up until recently, even if you tried to z-index elements with position absolute it would be visually overwritten by the canvas (I think this is mostly fixed though). I don’t know if this is the best solution but it’s better than previous hacks. IF you need to go that route. Basically html2canvas.
tombh 1y ago
Where does SVG's `foreignObject` fit into this? It seems that SVG supports all of thelproposal already? As is evidenced by projects like https://github.com/zumerlab/snapdom https://github.com/zumerlab/snapdom that can take "screenshots" of the webpage by copying the DOM with inlined styles into a `foreignObject` tag in an SVG. Then of course that SVG can be rendered to a canvas.
- progers7 1y ago
  This proposal is a lot like an easier way to draw foreign object into canvas. This proposal supports new features too, such as updating the canvas when the content changes, and interactivity.
russellbeattie 1y ago
How about a flag on the body element like: <body canvas="true"></body> This would make the entire visible page into a canvas-like drawing surface which also renders DOM elements as per usual. At some level there's a process which rasterizes the DOM - opening drawing APIs into that might be a better solution. It's sort of the same thing as HTML in canvas conceptually, but architecturally it makes DOM rendering and canvas rendering overlapping equals with awareness going both ways. E.g., a line drawn on the page will cause the DOM elements to reflow unless told to ignore it.
- pyrolistical 1y ago
 Screen readers all cried in unison
 - koolala 1y ago
 Why wouldn't they work normally. The HTML is still modeled as a ground truth.
- koolala 1y ago
 Would this mean a canvas could also automatically resize? This would be a new era for html composition that could surpass iframes.
 - flohofwoe 1y ago
 Isn't this already trivial? You just need to be aware that the framebuffer size of a canvas element is different from its dom element size, but you can easily glue the canvas framebuffer size to the element size by listening for resize events (this is for webgl and webgpu canvases, don't know about 2d canvas)
erichocean 1y ago
My Blossom library has a layout engine for Canvas [0] [0] https://github.com/erichocean/blossom https://github.com/erichocean/blossom
jonplackett 1y ago
Can it just be CSS on canvas - so you can do styling and layout nicely but not weirdness of an html sandwich?
tones411 1y ago
Next someone will want to render a canvas in the html in the canvas.
mircerlancerous 1y ago
Please correct me if I'm wrong, but I feel rendering html overtop of canvas solves this with vanilla just fine. Canvas is for rendering things you can't with html, and not replacement for the dom.
- flohofwoe 1y ago
  That only works if the html stuff is on top of everything that's rendered in the canvas, otherwise you need to add another canvas on top of the html (etc pp for each separate z-layer). IMHO this step finally starts to fix the "inverted api layer stack" in browsers. All browser rendering should build on top of a universal canvas api.
- esperent 1y ago
  Here's a simple example that's currently very hard to do and requires all kinds of hacky and unsatisfying workarounds: 1. A 3d model, say of a statue in a museum 2. Add annotations to the model drawing attention to specific features (especially if the annotations are not just a single word or number) If you want the annotations to be properly occluded by the model as you move the camera around, it's hard - you can't use HTML. If you do use HTML, you'll have to do complex calculations to make it match the correct place in the 3d scene, and it will always be a frame delayed, and occlusion is bad - usually just show or hide the entire HTML annotation based on the bounding box of the 3d model (I have seen better solutions but they took a ton of work). So you could use 3d text, maybe SDF, but now you've created a entire text rendering system without accessibility or anything like that. Also, if you want anything more than very simple annotations (for example, videos, lists, select menus, whatever) you either have to reinvent them or fall back HTML.
fullstackwife 1y ago
Text rendering on canvas is currently a huge pain, custom layouting code, lack of accessibility, so I'm happy to see this!
samwillis 1y ago
There is a lot valid concern on accessibility and abuse this could result in, but it think it's important to see the other side of the argument. There was a really good thread on Twitter a couple of days ago: > In light of recent Figma news, lemme reiterate that of all the goods that can happen to the web, 90% of them can't happen due to not having access to font rendering & metrics in JS https://x.com/_chenglou/status/1951481453046538493 https://x.com/_chenglou/status/1951481453046538493 And a few choice replies: > t’s kind of crazy that a platform specifically designed for presenting text doesn’t provide functionality to manipulate text at a detail level > Brute forcing text measurement in tldraw breaks my heart Love it or hate it, the web is a platform for application development, making this easer is only good for everyone. My argument on web APIs is what we should continue to go lower level, and so font and text metrics APIs for canvas would be awesome and an alternative to this. But I'm also a proponent of "using the platform" and for text layout, web engines are incredible, and very performant. Extending that capability to layout inside a canvas enables many awesome features. One that I've repeatedly gone back to over the years is paginated rich text editing. It's simply impossible to do with contenteditable in a product level way - one of the reasons Google docs has a custom layout engine. This proposal would enable full use of contenteditable for rich text, but with full page/print layout control. I hope it lands in the browsers.
- chrismorgan 1y ago
  > of all the goods that can happen to the web, 90% of them can't happen due to not having access to font rendering & metrics in JS I’d be interested to see a representative excerpt of this person’s “goods that can happen to the web”, because it sounds pretty ridiculous to me. Not much needs that stuff, and a lot of that stuff is exposed in JS these days, and a lot of the rest you can work around it without it being ruinous to performance. It’s also pretty irrelevant here (that is, about HTML-in-Canvas): allowing drawing HTML to canvas doesn’t shift the needle in these areas at all.
  - heresie-dabord 1y ago
    > excerpt of this person’s “goods that can happen to the web" 100% of my concern about the Web is about privacy and security... and why they don't happen.
troupo 1y ago
> As a result, <canvas>-based content suffers in accessibilty, internationalization, performance and quality. Ah yes. Because HTML is renowned for its performance and quality. Instead of pushing this idiocy they should add the things that canvas lacks instead
pacifika 1y ago
Yeah let’s break user control over text.
- flohofwoe 1y ago
  Improving accessibility for canvas rendered content is one of the first bullet points in the proposal, so I guess they thought of that little detail.
  - pacifika 1y ago
    This is just building a browser engine inside a browser engine, can’t see it happen. Also unless it has the same feature as the level of accessibility it has no it would be a step back. It would be a gargantuan job.
    - flohofwoe 1y ago
      It's not a browser engine in a browser engine, just making the already existing browser engine available in another context. I bet that at least 90% of the dom implementation code will be shared (since internally the dom is almost certainly rendered through the same renderer process that also runs webgl and webgpu.
rikroots 1y ago
Please excuse me for spamming this thread with examples of how my canvas library approaches these issues: > Use case: Styled, Laid Out Content in Canvas. There’s a strong need for better styled text support in Canvas. Examples include chart components (legend, axes, etc.), rich content boxes in creative tools, and in-game menus. Single line, unstyled text is relatively easy using the Canvas API. Multiline text is a world-of-pain. Styled text is a completely separate world-of-pain. Underlined text? Same! So that's gives us a problem space of world-of-pain-cubed. Don't talk to me about RTL text, vertical text, CJK punctuation, Thai text ignoring spaces as a word separator, heavily kerned fonts (staring at you, Arabic and Devangari), etc. Demo: https://scrawl-v8.rikweb.org.uk/demo/canvas-207.html https://scrawl-v8.rikweb.org.uk/demo/canvas-207.html This demo takes the following html markup and displays it in a truncated circle shape. The styling itself happens in CSS - see here: https://github.com/KaliedaRik/Scrawl-canvas/blob/v8/demo/canvas-207.html https://github.com/KaliedaRik/Scrawl-canvas/blob/v8/demo/can... const westernText = 'Lorem ipsum dolor sit amet, consectetur adipiscing élit, sed <s>do eius-mod</s> tempoj ynfigizqunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercit-ation ullamco laboris nisi ut aliquip ex ea "commodo" consequat. Duis (aute irure dölor) in reprehenderit în voluptate velit ©2024 esse <cillum> dolore eu fugiat nulla pariatur. Excepteur sint occaecat ¡cupidatat! non proident, sunt in culpa qui officiþa deserunt mollit anim id est laborum.'; As for the other things I don't want to talk about, see this other demo which attempts to overcome those issues: https://scrawl-v8.rikweb.org.uk/demo/canvas-206.html https://scrawl-v8.rikweb.org.uk/demo/canvas-206.html > Use case: Accessibility Improvements. There is currently no guarantee that the canvas fallback content currently used for <canvas> accessibility always matches the rendered content, and such fallback content can be hard to generate. With this API, elements drawn into the canvas bitmap will match their corresponding canvas fallback. I welcome and applaud this focus on making canvas text accessible. However it's not enough (in my highly opinionated opinion) to just reflect the text back into the DOM. People using screen readers probably don't need every number on the Y axis read out to them every time they navigate onto a chart, they probably just need to hear values as they keyboard navigate the chart. Demo: https://scrawl-v8.rikweb.org.uk/demo/modules-001.html https://scrawl-v8.rikweb.org.uk/demo/modules-001.html The canvas element is highly inaccessible - I've tried to detail all the issues that have to be addressed here (again, a highly opinionated take): https://scrawl-v8.rikweb.org.uk/docs/reference/sc-accessibility.html https://scrawl-v8.rikweb.org.uk/docs/reference/sc-accessibil... > Use case: Composing HTML Elements with Shaders. A limited set of CSS shaders, such as filter effects, are already available, but there is a desire to use general WebGL shaders with HTML. We already have a comprehensive set of filter effects available through SVG <filter> elements. They are, however, difficult to compose and have a tendancy to be computationally heavy. WebGL shaders can be fantastic, but face the (current) limit of how many WebGL canvas elements you can include on a page; they're also difficult to compose. For my library's filter engine, I took inspiration from the SVG approach. Details can be found here: https://scrawl-v8.rikweb.org.uk/docs/reference/sc-filter-engine.html https://scrawl-v8.rikweb.org.uk/docs/reference/sc-filter-eng... > Use case: HTML Rendering in a 3D Context. 3D aspects of sites and games need to render rich 2D content into surfaces within a 3D scene. HTML canvas elements are just DOM elements, and can be 3d-rotated like other elements. Interacting with 3d-rotated canvas elements is an interesting problem space. Classic rotating cube demo: https://scrawl-v8.rikweb.org.uk/demo/dom-008.html https://scrawl-v8.rikweb.org.uk/demo/dom-008.html Tracking the mouse over a 3d-rotated canvas element demo: https://scrawl-v8.rikweb.org.uk/demo/dom-013.html https://scrawl-v8.rikweb.org.uk/demo/dom-013.html
deleted 1y ago
[deleted]
wg0 1y ago
What we also need is a simple API that given a dom element, takes a screenshot of the subtree. Nothing such is available.
- formula1 1y ago
  A native version would be nice but this seems interesting https://html2canvas.hertzen.com/ https://html2canvas.hertzen.com/
Ono-Sendai 1y ago
I would love this. I have to do disgusting hacks to get an embedded browser window in my metaverse (https://substrata.info/ https://substrata.info/) that uses webgl. The disgusting hack is to render the browser window behind the main webgl canvas, and then punch a hole through the webgl canvas with zero alpha. Event handling (mouse, keyboard) is also a total pain.
TheRealPomax 1y ago
I pointed out text deficiencies in Canvas decades ago, only for Hixie to go "so put your text in a div and position the div over the canvas". I wonder if the working groups are still run by that attitude.
pbohun 1y ago
The whole point of canvas is to get away from the awful kludge that is HTML and CSS. I'd much rather see a new simple UI library that's developed for canvas.
kookamamie 1y ago
HTML with a canvas with HTML with a canvas…
j45 1y ago
This isn't bad, or good. Having this type of control, for certain use cases can be perfectly valid. It also feels Flash like. The javascriptists began on a journey 15 years ago to replace Flash. Things have gotten more complicated before becoming simpler, but maybe things will head in a direction soon. Flash itself was actionscript (ECMAScript) which is the same syntax as Javascript.

HTML-in-Canvas

222 points by dannyobrien 1y ago

tantalor 1y ago

https://html2canvas.hertzen.com/ https://html2canvas.hertzen.com/

Waterluvian 1y ago
45kb gzipped is pretty beefy but incredibly small when you consider just what it takes to make this work today. If I understand correctly, it’s basically a DOM and CSS renderer.
kizer 1y ago
There's a bunch of CSS, etc. not supported by that. It would be great to have access to a native API to get bitmaps of the DOM.
- masswerk 1y ago
  Well, a fingerprinting dream… Meaning, no way, just for the security aspect.

ttoinou 1y ago

Soon we’re gonna need Canvas rendering inside of HTML-in-Canvas

bastawhiz 1y ago
It should already work if the nested canvas uses the same approach. It's not cyclic, though. To make cyclic canvases work, you need to manually draw the parent canvas to a nested canvas.
- koolala 1y ago
  Now your thinking with canvas.

hyperhello 1y ago

I see reliable rich text editing as more of a priority.

SeanAnderson 1y ago

I read the title and said "shut the fuck up, don't do that." but then I read the rationale and it's fair. It's true there is no layout engine inside canvas, and that is a pain, but I'm not sure it's such a pain as to invite this recursive hell.

teaearlgraycold 1y ago
You could disallow recursion.
- deleted 1y ago
  [deleted]
nine_k 1y ago
One of the more senior engineers I worked with told me: "Every real-life data structure I encountered was tree-like". It would be easiest to just ask the browser to render a fragment of HTML onto a canvas, or onto some invisible bitmap, like you can with most other UI toolkits.
- monster_truck 1y ago
  They would never do this because of fingerprinting, which is already the cause of most of the reasons we cannot 'just' do a lot of things, unfortunately. E: And the infamous other half: malware. A bit over a decade ago malware devs started using canvas to do things like hide fragments inside of bitmap data in seemingly harmless ads and then a second script would extract and assemble it to evade detection.
  - rezonant 1y ago
    The web platform can already do this, see SVG foreignObject elsewhere in the thread. The key is to have the proper bounds in place (cross origin resources, etc), and the infrastructure for that is already in place. This just removes the extra step of relying on SVG to accomplish rendering the HTML, adds a path for getting this content into the accessibility tree, and supporting input on the rendered elements.

mmastrac 1y ago

https://github.com/WICG/html-in-canvas/blob/main/security-privacy-questionnaire.md https://github.com/WICG/html-in-canvas/blob/main/security-pr... > TODO: Expand on fingerprinting risks

rapnie 1y ago
I searched for it in the text, but it is the missing one to add (and might cross-ref to "TODO: Don't be evil").

codelikeawolf 1y ago

I immediately got "Pimp My Ride" vibes. Yo dawg, I heard you like HTML so I put HTML inside the canvas inside the HTML.

BobbyTables2 1y ago

I suppose all we need to do now is to compile a browser into WASM and run that browser inside a canvas element of the main browser…

sangeeth96 1y ago
How about an entire OS with IE? https://copy.sh/v86/?profile=windows2000 https://copy.sh/v86/?profile=windows2000
socalgal2 1y ago
https://www.chromium.org/blink/blink-in-js/ https://www.chromium.org/blink/blink-in-js/
alexhixon 1y ago
https://trevorlinton.github.io https://trevorlinton.github.io
xdennis 1y ago
That's the premise of The Birth & Death of JavaScript[1]. [1]: https://www.destroyallsoftware.com/talks/the-birth-and-death-of-javascript https://www.destroyallsoftware.com/talks/the-birth-and-death...
mkoubaa 1y ago
And then, 'for sandboxing and security reasons', this will become mandatory

deadbabe 1y ago

true horror is finding every element of the HTML page your on has been rendered in a Canvas

grimgrin 1y ago
exactly, we need a standard https://github.com/WICG/html-in-canvas https://github.com/WICG/html-in-canvas :P

turnsout 1y ago

Sad to say, but the first thing I thought was "oh good, a new fingerprinting target."

koolala 1y ago
Can you give an example? If I used this to load HTML into a 3D environment every user could have the same virtual screen size.
- turnsout 1y ago
  Every browser renders HTML slightly differently, so you could render some arbitrary HTML into the canvas and then read out the pixels.
  - koolala 1y ago
    I see what you mean. Isn't that already what the user agent tells you?
    - turnsout 1y ago
      Oh my sweet summer child. The game of cat and mouse is so much more vicious than that.

charcircuit 1y ago

This sounds really useful for being able to use standard web technology with webxr.

socalgal2 1y ago
I've never understood why they couldn't have just used the existing 3D CSS for WebXR. All the data is there, all they need to do is render the DOM from 2 POVs, one for each eye. They could even have had some standard to let it auto composite with WebGL.

ha1zum 1y ago

This could have a great utility, but for me HTML in canvas in HTML feels so cursed. To make it make sense in my opinion canvas should already be a first class format for web browsers, so it doesn't have to be inside a HTML. Then we would have a choice of HTML-first page with canvas elements in it, or a canvas-first page with HTML elements in it. But what do I know.

Liron 1y ago
What if you want an HTML-first page with a canvas in it, but then you realize you want some layout/styling for the text within the canvas? Seems unnecessary to propagate that situation up to the type of top-level page.
- thrance 1y ago
  And then what if you realize you need a canvas-in-the-html-in-the-canvas? It's endless. Canvas-first makes sense, it's basically how it works everywhere outside of the web. Start with the smallest abstractions and build on them (html on canvas) rather than leave escape hatches to your big abstractions because they fail to cover every use cases (canvas in html).
  - sitkack 1y ago
    If you support the DOM and hitscan, then it doesn't matter. You can red pill Ouroboros yourself all day and not care. Every element a canvas, every raindrop an ocean.
socalgal2 1y ago
canvas first sites suck. They can't use any system services as it would all be a privacy issue. They can't use the system dictionary for correction since to do so they'd need the contents of the dictionary or at least a way to query user customized corrections. Similarly they can't offer the system level accessibility but end up having to roll their own in which case, every app that uses canvas has a completely different UI.

tinalovely101 1y ago

It seems to be impossible with html version.

ivanjermakov 1y ago

RIP accessibility

progers7 1y ago
Accessibility is a key reason for this proposal. Today, canvas accessibility is quite limited. This proposal enables the browser to know how accessible dom elements map to canvas pixels.
koolala 1y ago
html is the most accessible language. video games have to hardcode everything.

c-smile 1y ago

From the very beginning in Sciter an Image can be constructed in two ways at runtime ( other than just getting loaded image reference): 1. By painting on it using Canvas/Graphics API: new Graphics.Image(width, height, painter(graphics) [,initColor]); Where _painter_ is a function used for paining on the image surface using Canvas/Graphics reference. 2. By making snapshot of the existing DOM element: new Graphics.Image(width, height, element [,initColor]) Such images can be used in DOM, rendered by other Canvas/Graphics as also in WebGL as textures. See: https://docs.sciter.com/docs/Graphics/Image#constructor https://docs.sciter.com/docs/Graphics/Image#constructor

donatj 1y ago

I feel like this is bound to get kneecapped for the same reason as other canvas features, fingerprinting.

cluckindan 1y ago
Fingerprinting is not an issue of canvas drawing, it is an issue of canvas reading (ImageData).

cheeaun 1y ago

Kinda reminds me of Flipboard https://engineering.flipboard.com/2015/02/mobile-web https://engineering.flipboard.com/2015/02/mobile-web (2015)

wetpaws 1y ago

[dead]

talkingtab 1y ago

There is a real problem using canvas to replace HTML. Not all but most HTML. I have not found a good solution for the issue of doing something like MDX in canvas. I have tried SDF, looked at 2D canvas Text, Troika, MSDF. You can get text, it is just that laying it out is very difficult. React three drei has the ability to put HTML into the threejs ecosystem, but there are issues about CSS and text that make that impractical. For me the use case is very simple. I would like to take an MDX file and show it in a mesh. Laid out. Maybe I am missing something because I am new to the whole threejs thing, but I really tried. A good article about text https://css-tricks.com/techniques-for-rendering-text-with-webgl/ https://css-tricks.com/techniques-for-rendering-text-with-we... And an example from the above article: https://codesandbox.io/p/sandbox/css-tricks-msdf-text-fks8w https://codesandbox.io/p/sandbox/css-tricks-msdf-text-fks8w This shows it can be done, I gave up trying to reproduce it in React-three-fiber. Why? Personally, I think the use of 3D graphics produces an interface for users that is an order or magnitude better for users. The real question (and an interesting one to consider) is why are we still building HTML first websites?

squidbeak 1y ago
Why do we build HTML first websites? Because most websites serve documents, however much developers might wish they were applications.

nikeee 1y ago

Is that being made to make Flutter not being like Flash on the web?

kbcool 1y ago
I was searching for a comment like this. It sounds like a crazy workaround for Flutter's strange architectural choices

sampton 1y ago

Servo + wasm.

reactordev 1y ago

I support this, as odd as it is. There’s times when you’re needing something drawn but can easily reuse an html element from elsewhere. Previously you’d have to render that to a bitmap offscreen and then copy that to a full screen quad or draw it on the canvas. Up until recently, even if you tried to z-index elements with position absolute it would be visually overwritten by the canvas (I think this is mostly fixed though). I don’t know if this is the best solution but it’s better than previous hacks. IF you need to go that route. Basically html2canvas.

tombh 1y ago

Where does SVG's `foreignObject` fit into this? It seems that SVG supports all of thelproposal already? As is evidenced by projects like https://github.com/zumerlab/snapdom https://github.com/zumerlab/snapdom that can take "screenshots" of the webpage by copying the DOM with inlined styles into a `foreignObject` tag in an SVG. Then of course that SVG can be rendered to a canvas.

progers7 1y ago
This proposal is a lot like an easier way to draw foreign object into canvas. This proposal supports new features too, such as updating the canvas when the content changes, and interactivity.

russellbeattie 1y ago

How about a flag on the body element like: <body canvas="true"></body> This would make the entire visible page into a canvas-like drawing surface which also renders DOM elements as per usual. At some level there's a process which rasterizes the DOM - opening drawing APIs into that might be a better solution. It's sort of the same thing as HTML in canvas conceptually, but architecturally it makes DOM rendering and canvas rendering overlapping equals with awareness going both ways. E.g., a line drawn on the page will cause the DOM elements to reflow unless told to ignore it.

pyrolistical 1y ago
Screen readers all cried in unison
- koolala 1y ago
  Why wouldn't they work normally. The HTML is still modeled as a ground truth.
koolala 1y ago
Would this mean a canvas could also automatically resize? This would be a new era for html composition that could surpass iframes.
- flohofwoe 1y ago
  Isn't this already trivial? You just need to be aware that the framebuffer size of a canvas element is different from its dom element size, but you can easily glue the canvas framebuffer size to the element size by listening for resize events (this is for webgl and webgpu canvases, don't know about 2d canvas)

erichocean 1y ago

My Blossom library has a layout engine for Canvas [0] [0] https://github.com/erichocean/blossom https://github.com/erichocean/blossom

jonplackett 1y ago

Can it just be CSS on canvas - so you can do styling and layout nicely but not weirdness of an html sandwich?

tones411 1y ago

Next someone will want to render a canvas in the html in the canvas.

mircerlancerous 1y ago

Please correct me if I'm wrong, but I feel rendering html overtop of canvas solves this with vanilla just fine. Canvas is for rendering things you can't with html, and not replacement for the dom.

flohofwoe 1y ago
That only works if the html stuff is on top of everything that's rendered in the canvas, otherwise you need to add another canvas on top of the html (etc pp for each separate z-layer). IMHO this step finally starts to fix the "inverted api layer stack" in browsers. All browser rendering should build on top of a universal canvas api.
esperent 1y ago
Here's a simple example that's currently very hard to do and requires all kinds of hacky and unsatisfying workarounds: 1. A 3d model, say of a statue in a museum 2. Add annotations to the model drawing attention to specific features (especially if the annotations are not just a single word or number) If you want the annotations to be properly occluded by the model as you move the camera around, it's hard - you can't use HTML. If you do use HTML, you'll have to do complex calculations to make it match the correct place in the 3d scene, and it will always be a frame delayed, and occlusion is bad - usually just show or hide the entire HTML annotation based on the bounding box of the 3d model (I have seen better solutions but they took a ton of work). So you could use 3d text, maybe SDF, but now you've created a entire text rendering system without accessibility or anything like that. Also, if you want anything more than very simple annotations (for example, videos, lists, select menus, whatever) you either have to reinvent them or fall back HTML.

fullstackwife 1y ago

Text rendering on canvas is currently a huge pain, custom layouting code, lack of accessibility, so I'm happy to see this!

samwillis 1y ago

There is a lot valid concern on accessibility and abuse this could result in, but it think it's important to see the other side of the argument. There was a really good thread on Twitter a couple of days ago: > In light of recent Figma news, lemme reiterate that of all the goods that can happen to the web, 90% of them can't happen due to not having access to font rendering & metrics in JS https://x.com/_chenglou/status/1951481453046538493 https://x.com/_chenglou/status/1951481453046538493 And a few choice replies: > t’s kind of crazy that a platform specifically designed for presenting text doesn’t provide functionality to manipulate text at a detail level > Brute forcing text measurement in tldraw breaks my heart Love it or hate it, the web is a platform for application development, making this easer is only good for everyone. My argument on web APIs is what we should continue to go lower level, and so font and text metrics APIs for canvas would be awesome and an alternative to this. But I'm also a proponent of "using the platform" and for text layout, web engines are incredible, and very performant. Extending that capability to layout inside a canvas enables many awesome features. One that I've repeatedly gone back to over the years is paginated rich text editing. It's simply impossible to do with contenteditable in a product level way - one of the reasons Google docs has a custom layout engine. This proposal would enable full use of contenteditable for rich text, but with full page/print layout control. I hope it lands in the browsers.

chrismorgan 1y ago
> of all the goods that can happen to the web, 90% of them can't happen due to not having access to font rendering & metrics in JS I’d be interested to see a representative excerpt of this person’s “goods that can happen to the web”, because it sounds pretty ridiculous to me. Not much needs that stuff, and a lot of that stuff is exposed in JS these days, and a lot of the rest you can work around it without it being ruinous to performance. It’s also pretty irrelevant here (that is, about HTML-in-Canvas): allowing drawing HTML to canvas doesn’t shift the needle in these areas at all.
- heresie-dabord 1y ago
  > excerpt of this person’s “goods that can happen to the web" 100% of my concern about the Web is about privacy and security... and why they don't happen.

troupo 1y ago

> As a result, <canvas>-based content suffers in accessibilty, internationalization, performance and quality. Ah yes. Because HTML is renowned for its performance and quality. Instead of pushing this idiocy they should add the things that canvas lacks instead

pacifika 1y ago

Yeah let’s break user control over text.

flohofwoe 1y ago
Improving accessibility for canvas rendered content is one of the first bullet points in the proposal, so I guess they thought of that little detail.
- pacifika 1y ago
  This is just building a browser engine inside a browser engine, can’t see it happen. Also unless it has the same feature as the level of accessibility it has no it would be a step back. It would be a gargantuan job.
  - flohofwoe 1y ago
    It's not a browser engine in a browser engine, just making the already existing browser engine available in another context. I bet that at least 90% of the dom implementation code will be shared (since internally the dom is almost certainly rendered through the same renderer process that also runs webgl and webgpu.

rikroots 1y ago

Please excuse me for spamming this thread with examples of how my canvas library approaches these issues: > Use case: Styled, Laid Out Content in Canvas. There’s a strong need for better styled text support in Canvas. Examples include chart components (legend, axes, etc.), rich content boxes in creative tools, and in-game menus. Single line, unstyled text is relatively easy using the Canvas API. Multiline text is a world-of-pain. Styled text is a completely separate world-of-pain. Underlined text? Same! So that's gives us a problem space of world-of-pain-cubed. Don't talk to me about RTL text, vertical text, CJK punctuation, Thai text ignoring spaces as a word separator, heavily kerned fonts (staring at you, Arabic and Devangari), etc. Demo: https://scrawl-v8.rikweb.org.uk/demo/canvas-207.html https://scrawl-v8.rikweb.org.uk/demo/canvas-207.html This demo takes the following html markup and displays it in a truncated circle shape. The styling itself happens in CSS - see here: https://github.com/KaliedaRik/Scrawl-canvas/blob/v8/demo/canvas-207.html https://github.com/KaliedaRik/Scrawl-canvas/blob/v8/demo/can... const westernText = 'Lorem ipsum dolor sit amet, consectetur adipiscing élit, sed <s>do eius-mod</s> tempoj ynfigizqunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercit-ation ullamco laboris nisi ut aliquip ex ea "commodo" consequat. Duis (aute irure dölor) in reprehenderit în voluptate velit ©2024 esse <cillum> dolore eu fugiat nulla pariatur. Excepteur sint occaecat ¡cupidatat! non proident, sunt in culpa qui officiþa deserunt mollit anim id est laborum.'; As for the other things I don't want to talk about, see this other demo which attempts to overcome those issues: https://scrawl-v8.rikweb.org.uk/demo/canvas-206.html https://scrawl-v8.rikweb.org.uk/demo/canvas-206.html > Use case: Accessibility Improvements. There is currently no guarantee that the canvas fallback content currently used for <canvas> accessibility always matches the rendered content, and such fallback content can be hard to generate. With this API, elements drawn into the canvas bitmap will match their corresponding canvas fallback. I welcome and applaud this focus on making canvas text accessible. However it's not enough (in my highly opinionated opinion) to just reflect the text back into the DOM. People using screen readers probably don't need every number on the Y axis read out to them every time they navigate onto a chart, they probably just need to hear values as they keyboard navigate the chart. Demo: https://scrawl-v8.rikweb.org.uk/demo/modules-001.html https://scrawl-v8.rikweb.org.uk/demo/modules-001.html The canvas element is highly inaccessible - I've tried to detail all the issues that have to be addressed here (again, a highly opinionated take): https://scrawl-v8.rikweb.org.uk/docs/reference/sc-accessibility.html https://scrawl-v8.rikweb.org.uk/docs/reference/sc-accessibil... > Use case: Composing HTML Elements with Shaders. A limited set of CSS shaders, such as filter effects, are already available, but there is a desire to use general WebGL shaders with HTML. We already have a comprehensive set of filter effects available through SVG <filter> elements. They are, however, difficult to compose and have a tendancy to be computationally heavy. WebGL shaders can be fantastic, but face the (current) limit of how many WebGL canvas elements you can include on a page; they're also difficult to compose. For my library's filter engine, I took inspiration from the SVG approach. Details can be found here: https://scrawl-v8.rikweb.org.uk/docs/reference/sc-filter-engine.html https://scrawl-v8.rikweb.org.uk/docs/reference/sc-filter-eng... > Use case: HTML Rendering in a 3D Context. 3D aspects of sites and games need to render rich 2D content into surfaces within a 3D scene. HTML canvas elements are just DOM elements, and can be 3d-rotated like other elements. Interacting with 3d-rotated canvas elements is an interesting problem space. Classic rotating cube demo: https://scrawl-v8.rikweb.org.uk/demo/dom-008.html https://scrawl-v8.rikweb.org.uk/demo/dom-008.html Tracking the mouse over a 3d-rotated canvas element demo: https://scrawl-v8.rikweb.org.uk/demo/dom-013.html https://scrawl-v8.rikweb.org.uk/demo/dom-013.html

deleted 1y ago

[deleted]

wg0 1y ago

What we also need is a simple API that given a dom element, takes a screenshot of the subtree. Nothing such is available.

formula1 1y ago
A native version would be nice but this seems interesting https://html2canvas.hertzen.com/ https://html2canvas.hertzen.com/

Ono-Sendai 1y ago

I would love this. I have to do disgusting hacks to get an embedded browser window in my metaverse (https://substrata.info/ https://substrata.info/) that uses webgl. The disgusting hack is to render the browser window behind the main webgl canvas, and then punch a hole through the webgl canvas with zero alpha. Event handling (mouse, keyboard) is also a total pain.

TheRealPomax 1y ago

I pointed out text deficiencies in Canvas decades ago, only for Hixie to go "so put your text in a div and position the div over the canvas". I wonder if the working groups are still run by that attitude.

pbohun 1y ago

The whole point of canvas is to get away from the awful kludge that is HTML and CSS. I'd much rather see a new simple UI library that's developed for canvas.

kookamamie 1y ago

HTML with a canvas with HTML with a canvas…

j45 1y ago

This isn't bad, or good. Having this type of control, for certain use cases can be perfectly valid. It also feels Flash like. The javascriptists began on a journey 15 years ago to replace Flash. Things have gotten more complicated before becoming simpler, but maybe things will head in a direction soon. Flash itself was actionscript (ECMAScript) which is the same syntax as Javascript.