3 ms·
I’ve pondered before the idea of a video codec that works like RAR, where the video embeds an arbitrary user-specified virtual machine that can be used to decod
by derefr 7y ago
I’ve pondered before the idea of a video codec that works like RAR, where the video embeds an arbitrary user-specified virtual machine that can be used to decode the video frames. (How is this not just a program binary? Because it still would have the semantics of a video stream, with no random access to frame data, only tape-head-like access.)
Seems like this would be perfect for videos that are just e.g. gameplay of games made of tiles+sprites: the video could just store one copy of the assets, and the frames could just be tile maps + sprite position information.
It would also work well for “videos” that are really just a single static image. Or videos that are visualizations of the audio stream: the VM could actually take the audio frames as input and output the respective video frame.
- saagarjha 7y agoI think Super Smash Bros records replays as a random seed and user input, then generates the video on the fly from that.
- recrof 7y agoalso, doom.
- derefr 7y agoSure, but the various machinima/demo file-formats of games are just application state-data formats, not video formats per se. The difference comes in what can decode them. An application state-data format can only be decoded by the original application, because necessary context—in this case, the game engine that translates user input to game-state and then to displayed frames, and also the library of visual assets the game uses to render those frames—is in the application, rather than in the video. A video format is self-contained, and usually not domain-specific. Many encoders and many decoders can be written to target a video format, and the decoders should not have to ship with an asset library (let alone a game engine) in order to properly render specific videos. A format like I'm talking about—one that doesn't know anything about application state, but does understand that it's compositing and placing a set of embedded assets each frame, rather than only knowing about pixels/gradels—seems like something generically useful to me. (Heck, we're close to support for such a format already, since many video players already understand the idea of compositing arbitrary stuff with placement instructions on the screen each frame, care of support for the https://en.wikipedia.org/wiki/SubStation_Alpha https://en.wikipedia.org/wiki/SubStation_Alpha subtitle format. That format is exactly the kind of "vector video" I'm talking about, except the only primitives it can position and style are text elements. Add RGBA-textured rectangles as another primitive type to it, and you'd get a video format!) And yes, I'm basically talking about the visual equivalent of a https://en.wikipedia.org/wiki/Module_file https://en.wikipedia.org/wiki/Module_file (embedded samples/synth patches + sequencing information); or, if you prefer another analogy, "what Flash movies are if you exclude the ability to execute ActionScript."