4 ms·

Is this actually lossless - that is, the same pixels as the original are recovered, guaranteed? I'm surprised such guarantees can be made from a neural network.

tasty_freeze 7y ago
The way many compressors work is based on recent data, they try to predict immediately following data. The prediction doesn't have to be perfect; it just has to be good enough that only the difference between the prediction and the exact data needs to be encoded, and encoding that delta usually takes fewer bits than encoding the original data. The compression scheme here is similar. Transmit a low res version of an image, use a neural network to guess what a 2x size image would look like, then send just the delta to fix where the prediction was wrong. Then do it again until the final resolution image is reached. If the neural network is terrible, you'd still get a lossless image recovery, but the amount of data sent in deltas would be greater than just sending the image uncompressed.
- eximius 7y ago
  Ah, I understand! I wasn't aware that was how they worked!
jlebar 7y ago
The neural net predicts the upscaled image, then they add on the delta between that prediction and the desired output. No matter what the neural net predicts, you can always generate some delta.
- eximius 7y ago
  I was failing to understand the purpose of the delta. :)
  - magicalhippo 7y ago
    Note that this is similar to how for example MPEG does it with the intermediate frames and motion vectors. First it encodes a full frame using basically regular JPEG, then for the next frames it first does motion estimation by splitting the image into 8x8 blocks and then for each block it tries to find the position in the previous frame which best fits it. The difference in position is called the motion vector for that block. It can then take all the "best fit" block from the previous frame and use it to generate a prediction of the next frame. It then computes the difference between the prediction and the actual frame, and stores this difference along with the set of motion vectors used to generate the prediction image. If nothing much has changed, just camera moving a bit about, the difference between the prediction and the actual frame data is very small and can be easily compressed. Also, the range of the motion vectors is typically limited to +/-16 pixels, and you only need one per block, so they take up very little space.

tasty_freeze 7y ago

The way many compressors work is based on recent data, they try to predict immediately following data. The prediction doesn't have to be perfect; it just has to be good enough that only the difference between the prediction and the exact data needs to be encoded, and encoding that delta usually takes fewer bits than encoding the original data. The compression scheme here is similar. Transmit a low res version of an image, use a neural network to guess what a 2x size image would look like, then send just the delta to fix where the prediction was wrong. Then do it again until the final resolution image is reached. If the neural network is terrible, you'd still get a lossless image recovery, but the amount of data sent in deltas would be greater than just sending the image uncompressed.

eximius 7y ago
Ah, I understand! I wasn't aware that was how they worked!

jlebar 7y ago

The neural net predicts the upscaled image, then they add on the delta between that prediction and the desired output. No matter what the neural net predicts, you can always generate some delta.

eximius 7y ago
I was failing to understand the purpose of the delta. :)
- magicalhippo 7y ago
  Note that this is similar to how for example MPEG does it with the intermediate frames and motion vectors. First it encodes a full frame using basically regular JPEG, then for the next frames it first does motion estimation by splitting the image into 8x8 blocks and then for each block it tries to find the position in the previous frame which best fits it. The difference in position is called the motion vector for that block. It can then take all the "best fit" block from the previous frame and use it to generate a prediction of the next frame. It then computes the difference between the prediction and the actual frame, and stores this difference along with the set of motion vectors used to generate the prediction image. If nothing much has changed, just camera moving a bit about, the difference between the prediction and the actual frame data is very small and can be easily compressed. Also, the range of the motion vectors is typically limited to +/-16 pixels, and you only need one per block, so they take up very little space.