6 ms·
Compressing Images with Neural Networks
- holoduke 3y agoHow much vram is needed? And computing power? To open a webpage you soon need 24gb and 2 seconds of 1000 watts energy to uncompress images. Bandwidth is reduced from 2mb to only 20kb.
- deleted 3y ago[deleted]
- guappa 3y ago> Bandwidth is reduced from 2mb to only 20kb. Plus the entire model, which comes with incorrect cache headers and must be redownloaded all the time.
- Dwedit 3y agoThere was an earlier article (Sep 20, 2022) about using the Stable Diffusion VAE to perform image compression. Uses the VAE to change from pixel space to latent space, dithers the latent space down to 256 colors, then when it's time to decompress it, it de-noises that. https://pub.towardsai.net/stable-diffusion-based-image-compresssion-6f1f0a399202?gi=7b867b7b4c10 https://pub.towardsai.net/stable-diffusion-based-image-compr... HN discussion: https://news.ycombinator.com/item?id=32907494 https://news.ycombinator.com/item?id=32907494
- dheera 3y agoI've done a bunch of experiments on my own on the Stable Diffusion VAE. Even when going down to 4-6 bits per latent space pixel the results are surprisingly good. It's also interesting what happens if you ablate individual channels; ablating channel 0 results in faithful color but shitty edges, ablating channel 2 results in shitty color but good edges, etc. The one thing it fails catastrophically on though is small text in images. The Stable Diffusion VAE is not designed to represent text faithfully. (It's possible to train a VAE that does slightly better at this, though.)
- StiffFreeze9 3y agoHow badly will its lossy-ness change critical things? In 2013, there were Xerox copiers with aggressive compression that changed numbers, https://www.theregister.com/2013/08/06/xerox_copier_flaw_means_dodgy_numbers_and_dangerous_designs/ https://www.theregister.com/2013/08/06/xerox_copier_flaw_mea...
- bluedino 3y agoIf I zoom all the way with my iPhone, the camera-assisting intelligence will mess up numbers too
- qrian 3y agoThe mentioned Xerox copier incident was not an OCR failure, but the copier actively changed the numbers in the original image due to its image compression algorithm.
- barfbagginus 3y agoHere's some of the context: www.dkriesel.com/blog/2013/0810_xerox_investigating_latest_mangling_test_findings Learn More: https://www.dkriesel.com/start?do=search&id=en%3Aperson&q=Xerox https://www.dkriesel.com/start?do=search&id=en%3Aperson&q=Xe... Brief: Xerox machines used template matching to recycle the scanned images of individual digits that recur in the document. In 2013, Kriesel discovered this procedure was faulty. Rationale: This method can create smaller PDFs, advantageous for customers that scan and archive numerical documents. Prior art: https://link.springer.com/chapter/10.1007/3-540-19036-8_22 https://link.springer.com/chapter/10.1007/3-540-19036-8_22 Tech Problem: Xerox's template matching procedure was not reliable, sometimes "papering over" a digit with the wrong digit! PR Problem: Xerox press releases initially claimed this issue did not happen in the factory default mode. Kriesel demonstrated this was not true, by replicating the issue in all of the factory default compression modes including the "normal" mode. He gave a 2015 FrOSCon talk, "Lies, damned lies and scans". Interesting work!
- lifthrasiir 3y agoAny lossy compressor changes the original image for better compression at expense of the perfect accuracy.
- rottc0dd 3y agoSomething similar by Fabrice Bellard: https://bellard.org/nncp/ https://bellard.org/nncp/
- p0w3n3d 3y agoSome people are fans of Metallica or Taylor Swift. I think Fabrice Bellard should get the same attention!
- p0w3n3d 3y agoAnd the same money for performance, of course
- skandium 3y agoIf you look at the winners of the Hutter prize, or especially the Large Text Compression Benchmark, then almost every approach uses some kind of machine learning approach for the adaptive probability model and then either arithmetic coding or rANS to losslessly encode it. This is intuitive, as the competition organisers say: compression is prediction.
- mbtwl 3y agoA first NN based image compression standard is currently being developed by JPEG. More information can be found here: https://jpeg.org/jpegai/documentation.html https://jpeg.org/jpegai/documentation.html Best overview you can probably get from “JPEG AI Overview Slides”
- calebm 3y agoAll learning is compression
- esafak 3y agoIt is not going to take off if it is not significantly better, and has browser support. WebP took off thanks to Chrome, while JPEG2000 floundered. If not native browser support, maybe the codec could be shipped by WASM or something? The interesting diagram to me is the last one, for computational cost, which shows the 10x penalty of the ML-based codecs.
- ufocia 3y agoBetter or cheaper, e.g. AV1?
- deleted 3y ago[deleted]
- dinkumthinkum 3y agoI think it is an interesting discussion, learning experience (no pun intended). I think this is more of a stop on a research project than a proposal; I could be wrong.
- dylan604 3y agoDid JPEG2000 really flounder? If your concept of it being a consumer facing product as a direct replacement for JPEG, then I could see being unsuccessful in that respect. However, JPEG2000 has found its place in the professional side of things.
- esafak 3y agoYes, I do mean broad- rather than niche adoption. I myself used J2K to archive film scans. One problem is that without broad adoption, support even in niche cases is precarious; the ecosystem is smaller. That makes the codec not safe for archiving, only for distribution. The strongest use case I see for this is streaming video, where the demand for compression is highest.
- actionfromafar 3y agoHuh, one more point for considering J2K for film scan archiving.
- amelius 3y agoHow do we know we don't get hands with 16 fingers?
- deleted 3y ago[deleted]
- deleted 3y ago[deleted]
- ogurechny 3y agoValid point. Conventional codecs draw things on screen that are not in the original, too, but we are used to low quality images and videos, and learned to ignore the block edges and smudges unconsciously. NN models “recover” much complex and plausible-looking features. It is possible that some future general purpose image compressor would do the same thing to small numbers lossy JBIG2 did.
- ufocia 3y agoHow do we know whether it's an image with 16 fingers or it just looks like 16 fingers to us? I looked at the bear example above and I could see how either the AI thought that there was an animal face embedded in the fur or we just see the face in the fur. We see all kinds of faces on toast even though neither the bread slicers nor the toasters intend to create them.
- jfdi 3y agoAnyone know of open models useful (and good quality) for going the other way? I.e., Input is a 800x600 jpg and output is 4k version.
- lsb 3y agoYou’re looking for what’s called upscaling, like with Stable Diffusion: https://huggingface.co/stabilityai/stable-diffusion-x4-upscaler https://huggingface.co/stabilityai/stable-diffusion-x4-upsca...
- cuuupid 3y agoThere are a bunch of great upscaler models although they tend to hallucinate a bit, I personally use magic-image-refiner: https://replicate.com/collections/super-resolution https://replicate.com/collections/super-resolution
- hansvm 3y agoI haven't explored the current SOTA recently, but super-resolution has been pretty good for a lot of tasks for few years at least. Probably just start with hugging-face [0] and try a few out, especially diffusion-based models. [0] https://huggingface.co/docs/diffusers/api/pipelines/stable_diffusion/upscale https://huggingface.co/docs/diffusers/api/pipelines/stable_d...
- godelski 3y agoLook for SuperResolution. These models will typically come as a GAN, Normalizing Flow (or Score, NODE), or more recently Diffusion (or SNODE) (or some combination!). The one you want will depend on your computational resources, how lossy you are willing to be, and your image domain (if you're unwilling to tune). Real time (>60fps) is typically going to be a GAN or flow. Make sure to test the models before you deploy. Nothing will be lossless doing superresolution but flows can get you lossless in compression.
- sitkack 3y agoOr else you get Ryan Gosling https://news.ycombinator.com/item?id=24196650 https://news.ycombinator.com/item?id=24196650