4 ms·

I can imagine getting a large amount of flaming for this idea, but wouldn't the logical extreme just be to basically define a container for a JS canvas with emb

I can imagine getting a large amount of flaming for this idea, but wouldn't the logical extreme just be to basically define a container for a JS canvas with embeddable resources? In a more restricted form of course, to be sure that it will always render the same, and all compliant computers will have enough power to render it. You could do things like 480p training videos with ultra-crisp text, and snow/fire/etc effects. You'd probably have to prevent it from having network access or else you'd just be asking for the videos to rot over time, but you could give them an embedding API or even access to a few small bits of data, so you could have a YouTube video that changes with seasons and location. If you were strict about the no network access thing you could give access to all sensors, since the data would never leave the device. YouTube's already got some limited interactivity features, DVDs had them, why not all videos? It would mostly be a novelty, but you could also do really cool stuff like use Bluetooth beacons to skip around to certain parts for an interactive tour, although you might have to enforce downloading the whole thing before playing if you wanted 100% no privacy leaks.

csnover 4y ago
It seems like you are describing BD-J[0], or a network-restricted version of Adobe Flash? [0] https://en.wikipedia.org/wiki/BD-J https://en.wikipedia.org/wiki/BD-J
occamrazor 4y ago
The biggest problem with that idea is that it wouldn’t be possible to have hw decoders as efficient as the current ones. But there is definitely a trend towards more customizable decoders, with a larger set of primitive operations (DCT, motion estimation/compensation, convolution filters) and parameters. The codec space is slowly moving towards a model where the decoders are VM highly optimized for specific tasks.
matthberg 4y ago
That's an interesting idea. I reckon the complexity added with rendering vector graphics (text in particular, font rendering is notoriously difficult) outweighs the bandwidth savings, but still it seems like an area ripe for exploration. Canvas-like APIs might be too complicated to encode/decode efficiently, yet I suspect something closer to a Web/OpenGL fragment shader would be much more manageable (yet likely much worse for interactivity unfortunately). While that wouldn't quite mesh with the idea of vector-like text you proposed (without a heavy library or two thrown in), I suspect the engineering put into the existing graphics pipeline would make it a more feasible approach to augmented video. Looking at the stuff on Shadertoy.com and Inigo Quilez's work shows the capabilities of fragment shader based graphics, yet I suspect all of the magic would be in the details of the file format and encoding strategy. If anyone pokes around or explores a video/shader hybrid format let me know and post it on HN, I bet a bunch of people would be interested.
rikroots 4y ago
> ... but wouldn't the logical extreme just be to basically define a container for a JS canvas with embeddable resources? Here's my attempt at creating a grainy video effect in the browser using a media stream (from the device's camera - the page should ask for permission first). Very much an MVP; I'm already working on making the dithering algorithm more efficient. https://codepen.io/kaliedarik/pen/OJOaOZz https://codepen.io/kaliedarik/pen/OJOaOZz > YouTube's already got some limited interactivity features, DVDs had them, why not all videos? I don't know about videos, but again this is doable in the browser using the canvas. For instance, this example: https://scrawl-v8.rikweb.org.uk/demo/canvas-027.html https://scrawl-v8.rikweb.org.uk/demo/canvas-027.html - the swans, and both geese, are clickable (to their respective Wikipedia pages). The navigation around each goose is a clickable box, while the navigation on the swans uses a chroma-key effect to limit the clickable area.

csnover 4y ago

It seems like you are describing BD-J[0], or a network-restricted version of Adobe Flash? [0] https://en.wikipedia.org/wiki/BD-J https://en.wikipedia.org/wiki/BD-J

occamrazor 4y ago

The biggest problem with that idea is that it wouldn’t be possible to have hw decoders as efficient as the current ones. But there is definitely a trend towards more customizable decoders, with a larger set of primitive operations (DCT, motion estimation/compensation, convolution filters) and parameters. The codec space is slowly moving towards a model where the decoders are VM highly optimized for specific tasks.

matthberg 4y ago

That's an interesting idea. I reckon the complexity added with rendering vector graphics (text in particular, font rendering is notoriously difficult) outweighs the bandwidth savings, but still it seems like an area ripe for exploration. Canvas-like APIs might be too complicated to encode/decode efficiently, yet I suspect something closer to a Web/OpenGL fragment shader would be much more manageable (yet likely much worse for interactivity unfortunately). While that wouldn't quite mesh with the idea of vector-like text you proposed (without a heavy library or two thrown in), I suspect the engineering put into the existing graphics pipeline would make it a more feasible approach to augmented video. Looking at the stuff on Shadertoy.com and Inigo Quilez's work shows the capabilities of fragment shader based graphics, yet I suspect all of the magic would be in the details of the file format and encoding strategy. If anyone pokes around or explores a video/shader hybrid format let me know and post it on HN, I bet a bunch of people would be interested.

rikroots 4y ago

> ... but wouldn't the logical extreme just be to basically define a container for a JS canvas with embeddable resources? Here's my attempt at creating a grainy video effect in the browser using a media stream (from the device's camera - the page should ask for permission first). Very much an MVP; I'm already working on making the dithering algorithm more efficient. https://codepen.io/kaliedarik/pen/OJOaOZz https://codepen.io/kaliedarik/pen/OJOaOZz > YouTube's already got some limited interactivity features, DVDs had them, why not all videos? I don't know about videos, but again this is doable in the browser using the canvas. For instance, this example: https://scrawl-v8.rikweb.org.uk/demo/canvas-027.html https://scrawl-v8.rikweb.org.uk/demo/canvas-027.html - the swans, and both geese, are clickable (to their respective Wikipedia pages). The navigation around each goose is a clickable box, while the navigation on the swans uses a chroma-key effect to limit the clickable area.