4 ms·

It's a really cool idea, but I don't know if this would ever be a practical method for image compression. First of all, you could never change the neural networ

by OskarS 7y ago

It's a really cool idea, but I don't know if this would ever be a practical method for image compression. First of all, you could never change the neural network without breaking the compression, so you can't ever "update" it. Like: what if you figure out a better network? Too bad! I mean, I guess you could, but then you need to to version the files and keep copies of all the networks you've ever used, but this gets messy quick. And speaking of storing the networks: I don't know that you would ever want to pay the memory hit that it would take to store the entire network in memory just to decompress images or video, nor the performance hit the decompression takes. The trade-off here is trading reduced drive space for massively increased RAM and CPU/GPU time. I don't know any case where you'd want to make that trade-off, at least not at this magnitude. Again though: it's an awesome idea. I just don't know that's ever going to be anything other than a cool ML curiosity.

daveguy 7y ago
I think the idea is the network is completely trained and encoded along with the image and delta data. A new network would just require retraining and storing that new network along with the image data. It doesn't use a global network for all compressions.
- codeflo 7y ago
  I don't think this would work, the size of the network would likely dominate the size of the compressed image.
  - mstade 7y ago
    Wouldn't the network be part of the decoder?
    - mastre_ 7y ago
      Yes, and this is why you couldn't update the network. Still, much like how various compression algos have "levels," this standard could be more open in this regard, adding new networks (sort of what others above refer to as versions) and the image could just specify which network it uses. Maybe have a central repo from where the decoder could pull a network it doesn't have (i.e. I make a site and encode all 1k images on it using my own network, pull the network to your browser once so you can decode all 1k images). And even support a special mode where the image explicitly includes the network to be used for decoding it along with image data (could make sense for a very large images, as well as for specialized/demonstrational/test purposes). All in all, a very interesting idea.
      - mstade 7y ago
        I wonder what the security implications of all this is, sounds dangerous to just run any old network. I suppose maybe if it's sandboxed enough with very strongly defined inputs and outputs then the worst that could happen is you get garbled imagery?
- stefs 7y ago
  they include the trained models under the "model weights" section. imagenet is ~20mb, openimages is ~17mb. now this might be prohibitive for images over the web, but it'd be interesting whether it might be applicable for images with huge resolutions for printing, where single images are are hundreds of megabytes
crazygringo 7y ago
For a standard network, you're right there would only be one version. So you just make sure it's very carefully put together. (If a massively better one comes along, then you just make it a new file format.) And as for performance/resources -- great point. But what about video, where the space/bandwidth improvements become drastically more important? Since h.264 and h.265 already has dedicated hardware, would it be reasonable to assume that a chip dedicated to this would handle it just fine? And that if you've already got hardware for video, then of course you'd just re-use it for still images?
- bryanrasmussen 7y ago
  >(If a massively better one comes along, then you just make it a new file format.) I guess you could have versioning of your file format, and some sort of organization that standardized it.
  - colejohnson66 7y ago
    Then you get the layperson who doesn’t understand that and asks why their version 42 .imgnet won’t open in a program only supporting up to 10 (but they don’t know their image is v42 and the program only supports v10). It’s easier to understand different formats more than different versions
    - brokenmachine 7y ago
      You'd think the program that only supports up to v10 should be able to emit an error saying that.
cbhl 7y ago
Even if it's not useful for general-purpose compression, it may still be useful in a more restricted domain. In text compression, Brotli can be found in Chrome with a dictionary that is tuned for HTTP traffic. And in audio compression, LPCnet is a research codec that used Wavenet (neural nets for speech synthesis) to compress speech to 1.6kb/s (prior discussion from 2019 at https://news.ycombinator.com/item?id=19520194 https://news.ycombinator.com/item?id=19520194).
tobr 7y ago
> First of all, you could never change the neural network without breaking the compression, so you can't ever "update" it. Like: what if you figure out a better network? Too bad! Isn’t this just a special version of a problem any type of compression will always have? There’s all kinds of ways you can imagine improving on a format like JPEG, but the reason it’s useful is because it’s locked down and widely supported.
- HelloNurse 7y ago
  Usual compression standards are mostly adaptive, estimating statistical models of input from implicit prior distributions (e.g. the probability of A followed by B begins at p(A)p(B)), reasonable assumptions (e.g. scanlines in an image follow the same distribution), small and fixed tables and rules (e.g. the PNG filters): not only a low volume of data, but data that can only change as a major change of algorithm. A neural network that models upscaling is, on the other hand, not only inconveniently big, but also completely explicit (inviting all sorts of tweaking and replacement) and adapted to a specific data set (further demanding specialized replacements for performance reasons). Among the applications that are able to store and process the neural network, which is no small feat, I don't think many would be able to amortize the cost of a tailored neural network over a large, fixed set of very homogeneous images. The imagenet64 model is over 21 MB: saving 21 MB over PNG size, at 4.29 vs 5.74 bpp (table 2a in the article), requires a set of more than 83 MB of perfectly imagenet64-like PNG images, which is a lot. Compressing with a custom upscaling model the image datasets used for neural network experiments, which are large and stable, is the most likely good application (with the side benefit of producing useful and interesting downscaled images for free in addition to compressing the originals).
contravariant 7y ago
If you've got a big enough image you can include the model parameters with the image.
ltbarcly3 7y ago
Why can't you update it? There could be a release of a new model every 6 months or something (although even that is probably too often, the incremental improvement due to statistical changes in the distribution of images being compressed isn't likely to change much over time), and you just keep a copy of all the old models (or lazily download them like msft foundation c++ library versions when you install an application). The models themselves aren't very large.
- zo1 7y ago
  Not just that, but you could take a page out of the "compression" book and treat the NN as a sort of dictionary in that it is part of the compressed payload. Maybe not the whole NN, but perhaps deltas from a reference implementation, assuming the network structure remains the same and/or similar.
- spuz 7y ago
  I don't know why this comment was downvoted - it's a legitimate question. One scenario I can picture is the Netflix app on your TV. Firstly, they create a neural network trained on the video data in their library and ship it to all their clients while they are idle. They could then stream very high-quality video at lower bandwidth than they currently use and, assuming decoding can be done quickly enough, provide a great experience for their users. Any updates to the neural network could be rolled out gradually and in the background.
  - devinplatt 7y ago
    Google used to do something called SDCH (Shared Dictionary Compression for HTTP), where a delta compression dictionary was downloaded to Chrome. The dictionary had to be updated from time to time to keep a good compression rate as the Google website changed over time. There was a whole protocol to handle verifying what dictionary the client had and such.
burntoutfire 7y ago
> I don't know that you would ever want to pay the memory hit that it would take to store the entire network in memory just to decompress images or video, nor the performance hit the decompression takes. The big memory load wouldn't neccesarily be a problem for the likes of Youtube and Netflix - they could just have dedicated machines which do nothing else but decoding. The performance penalty could be a killer though.

by OskarS 7y ago

daveguy 7y ago

I think the idea is the network is completely trained and encoded along with the image and delta data. A new network would just require retraining and storing that new network along with the image data. It doesn't use a global network for all compressions.

codeflo 7y ago
I don't think this would work, the size of the network would likely dominate the size of the compressed image.
- mstade 7y ago
  Wouldn't the network be part of the decoder?
  - mastre_ 7y ago
    Yes, and this is why you couldn't update the network. Still, much like how various compression algos have "levels," this standard could be more open in this regard, adding new networks (sort of what others above refer to as versions) and the image could just specify which network it uses. Maybe have a central repo from where the decoder could pull a network it doesn't have (i.e. I make a site and encode all 1k images on it using my own network, pull the network to your browser once so you can decode all 1k images). And even support a special mode where the image explicitly includes the network to be used for decoding it along with image data (could make sense for a very large images, as well as for specialized/demonstrational/test purposes). All in all, a very interesting idea.
    - mstade 7y ago
      I wonder what the security implications of all this is, sounds dangerous to just run any old network. I suppose maybe if it's sandboxed enough with very strongly defined inputs and outputs then the worst that could happen is you get garbled imagery?
stefs 7y ago
they include the trained models under the "model weights" section. imagenet is ~20mb, openimages is ~17mb. now this might be prohibitive for images over the web, but it'd be interesting whether it might be applicable for images with huge resolutions for printing, where single images are are hundreds of megabytes

crazygringo 7y ago

For a standard network, you're right there would only be one version. So you just make sure it's very carefully put together. (If a massively better one comes along, then you just make it a new file format.) And as for performance/resources -- great point. But what about video, where the space/bandwidth improvements become drastically more important? Since h.264 and h.265 already has dedicated hardware, would it be reasonable to assume that a chip dedicated to this would handle it just fine? And that if you've already got hardware for video, then of course you'd just re-use it for still images?

bryanrasmussen 7y ago
>(If a massively better one comes along, then you just make it a new file format.) I guess you could have versioning of your file format, and some sort of organization that standardized it.
- colejohnson66 7y ago
  Then you get the layperson who doesn’t understand that and asks why their version 42 .imgnet won’t open in a program only supporting up to 10 (but they don’t know their image is v42 and the program only supports v10). It’s easier to understand different formats more than different versions
  - brokenmachine 7y ago
    You'd think the program that only supports up to v10 should be able to emit an error saying that.

cbhl 7y ago

Even if it's not useful for general-purpose compression, it may still be useful in a more restricted domain. In text compression, Brotli can be found in Chrome with a dictionary that is tuned for HTTP traffic. And in audio compression, LPCnet is a research codec that used Wavenet (neural nets for speech synthesis) to compress speech to 1.6kb/s (prior discussion from 2019 at https://news.ycombinator.com/item?id=19520194 https://news.ycombinator.com/item?id=19520194).

tobr 7y ago

> First of all, you could never change the neural network without breaking the compression, so you can't ever "update" it. Like: what if you figure out a better network? Too bad! Isn’t this just a special version of a problem any type of compression will always have? There’s all kinds of ways you can imagine improving on a format like JPEG, but the reason it’s useful is because it’s locked down and widely supported.

HelloNurse 7y ago
Usual compression standards are mostly adaptive, estimating statistical models of input from implicit prior distributions (e.g. the probability of A followed by B begins at p(A)p(B)), reasonable assumptions (e.g. scanlines in an image follow the same distribution), small and fixed tables and rules (e.g. the PNG filters): not only a low volume of data, but data that can only change as a major change of algorithm. A neural network that models upscaling is, on the other hand, not only inconveniently big, but also completely explicit (inviting all sorts of tweaking and replacement) and adapted to a specific data set (further demanding specialized replacements for performance reasons). Among the applications that are able to store and process the neural network, which is no small feat, I don't think many would be able to amortize the cost of a tailored neural network over a large, fixed set of very homogeneous images. The imagenet64 model is over 21 MB: saving 21 MB over PNG size, at 4.29 vs 5.74 bpp (table 2a in the article), requires a set of more than 83 MB of perfectly imagenet64-like PNG images, which is a lot. Compressing with a custom upscaling model the image datasets used for neural network experiments, which are large and stable, is the most likely good application (with the side benefit of producing useful and interesting downscaled images for free in addition to compressing the originals).

contravariant 7y ago

If you've got a big enough image you can include the model parameters with the image.

ltbarcly3 7y ago

Why can't you update it? There could be a release of a new model every 6 months or something (although even that is probably too often, the incremental improvement due to statistical changes in the distribution of images being compressed isn't likely to change much over time), and you just keep a copy of all the old models (or lazily download them like msft foundation c++ library versions when you install an application). The models themselves aren't very large.

zo1 7y ago
Not just that, but you could take a page out of the "compression" book and treat the NN as a sort of dictionary in that it is part of the compressed payload. Maybe not the whole NN, but perhaps deltas from a reference implementation, assuming the network structure remains the same and/or similar.
spuz 7y ago
I don't know why this comment was downvoted - it's a legitimate question. One scenario I can picture is the Netflix app on your TV. Firstly, they create a neural network trained on the video data in their library and ship it to all their clients while they are idle. They could then stream very high-quality video at lower bandwidth than they currently use and, assuming decoding can be done quickly enough, provide a great experience for their users. Any updates to the neural network could be rolled out gradually and in the background.
- devinplatt 7y ago
  Google used to do something called SDCH (Shared Dictionary Compression for HTTP), where a delta compression dictionary was downloaded to Chrome. The dictionary had to be updated from time to time to keep a good compression rate as the Google website changed over time. There was a whole protocol to handle verifying what dictionary the client had and such.

burntoutfire 7y ago

> I don't know that you would ever want to pay the memory hit that it would take to store the entire network in memory just to decompress images or video, nor the performance hit the decompression takes. The big memory load wouldn't neccesarily be a problem for the likes of Youtube and Netflix - they could just have dedicated machines which do nothing else but decoding. The performance penalty could be a killer though.