3 ms·
If the sending machine (that films my face) does the same decoding I know will be on the other end, then diffs the raw video with the decoded video and finally
by alkonaut 6y ago
If the sending machine (that films my face) does the same decoding I know will be on the other end, then diffs the raw video with the decoded video and finally sends both things, then the receiver should be able to always piece together a 100% reproduction of the actual video feed on my end. The transmitter can always predict exactly what the receiver will decode, so the correct amount of data to send is the amount of data that makes the receiver see what I want. If that means I send 0.1mbps key point data and 1mbps diff pixels then that’s what I want to send.
Is that how this works? Because it should be how it works...
Not sure if this is a viable way of compressing the video stream or if a transmission of the “diff” would be too costly. If this method would give 1/2 the original bandwidth I’d think it’s more impressive than cutting 3 orders of magnitude via “avatars”.
- ssivark 6y agoYou’re thinking along the right lines, but the challenge is that a raw diff will have the same number of pixels as the raw image, so no compression in bandwidth. So, how do we represent the diff/residue also with fewer numbers? At that point it’s the same as choosing better parameters within some clever encoding (be it pre-designed like JPEG or H.264 or learned via ML).
- alkonaut 6y agoI was thinking that subtracting the predicted image would give an image that has more zeroes and compresses better (much like dct+quantization for jpeg). After all, any time the neural network would predict an area of the image almost exactly, it can be omitted from the diff stream completely too.
- ssivark 6y agoThe diff will be low norm (for some suitable norm), but it needn’t be sparse in pixel space. It could be sparse in some other spaces (Eg: wavelet basis), but that’s right where the challenge lies — finding a basis where the data (and/or the residue) to be transmitted is sparse.
- kevincox 6y agoI suspect that this isn't feasible. Otherwise lossy video codecs wouldn't exist (or would be very niche). Because if you sent a perfect diff you would essentially have lossless compression again. So either your idea will revolutionize video compression or the "diff" would bring you back to the ballpark of lossless codecs.
- alkonaut 6y agoIt is a lossless codec yes, but I has a “base” image for the key frames (which are the most expensive) which can be transmitted as the key points. The question is how much that actually helps the end result vs how much it costs. As I said it would likely kill at lest nearly all the gains, but again - a small improvement in the state of the art for “normal” looking video would be more impressive than three orders of magnitude vs sending instructions to my head puppet.