H.264, VP9, etc., are all macroblock-based DCT codecs with I-frames that contain a full still image (like a JPEG), and other kinds of frames that contain instru

H.264, VP9, etc., are all macroblock-based DCT codecs with I-frames that contain a full still image (like a JPEG), and other kinds of frames that contain instructions in terms of motion vectors of how those macroblocks move around. They also use colorspace transforms and color subsampling, so they intentionally sacrifice some color accuracy. But writing a data stream into a 2D still image in a way that can be decoded later is a solved problem, ie. 2D barcodes like DataMatrix, QR Code, and Microsoft Tag (which has up to 8 colors to further increase data density). These formats have built-in error correction that compensates for some missing blocks. However, we can tune the format to be closer to the video codec's internal structure, to make them play nicer together. For example, we can set each barcode block to be within 50% to 100% of pixel size of the video's macroblock, to make it more likely that the video codec can reuse the macroblock with motion vectors in a P/B-frame, instead of having to put more bits to it, or have it accidentally mangle it. Realistically, we can also increase our color palette, as we're not going to be scanning these barcodes in bad light conditions -- all we need to do is get the color mostly right. But the more we increase the palette, the less video codec can reuse blocks; so this is something we'll want to experiment with. The biggest problem for the barcode approach comes from the addition of the 3rd, temporal dimension. We can have each frame form its own independently scannable barcode, but doing so, we'll want to build in some temporal redundancy, ie. have a chunk of data, or error correction for said chunk, be present in more than one frame -- to protect against occasional frame drops, very inconvenient frame drops (like when you lose an I-frame and the video is grey- on green-blocky for several more frames), and offer some extra protection against "normal" decoding errors. By the way, there are existing implementations of this concept: [1] http://thruglassxfer.com/ http://thruglassxfer.com/ [2] Demo of above: https://www.youtube.com/watch?v=2_8GlFdlb0Y https://www.youtube.com/watch?v=2_8GlFdlb0Y [3] Same idea, some hackable code: https://github.com/Neohapsis/QRCode-Video-Data-Exfiltration https://github.com/Neohapsis/QRCode-Video-Data-Exfiltration