That's an interesting idea. I reckon the complexity added with rendering vector graphics (text in particular, font rendering is notoriously difficult) outweighs

That's an interesting idea. I reckon the complexity added with rendering vector graphics (text in particular, font rendering is notoriously difficult) outweighs the bandwidth savings, but still it seems like an area ripe for exploration. Canvas-like APIs might be too complicated to encode/decode efficiently, yet I suspect something closer to a Web/OpenGL fragment shader would be much more manageable (yet likely much worse for interactivity unfortunately). While that wouldn't quite mesh with the idea of vector-like text you proposed (without a heavy library or two thrown in), I suspect the engineering put into the existing graphics pipeline would make it a more feasible approach to augmented video. Looking at the stuff on Shadertoy.com and Inigo Quilez's work shows the capabilities of fragment shader based graphics, yet I suspect all of the magic would be in the details of the file format and encoding strategy. If anyone pokes around or explores a video/shader hybrid format let me know and post it on HN, I bet a bunch of people would be interested.