10 ms·

Faster neural networks straight from JPEG (2018)

253 points by Anon84 3y ago

naillo 3y ago
Finally someone who did this, I've always thought this was a low hanging fruit. I wonder if you could make interesting and quickly trained diffusion models with this trick.
- waldarbeiter 3y ago
  Note that it is from 2018. As someone here already mentioned there is a paper that applies the same idea to Vision Transformers published this year [1]. [1] https://openaccess.thecvf.com/content/CVPR2023/papers/Park_RGB_No_More_Minimally-Decoded_JPEG_Vision_Transformers_CVPR_2023_paper.pdf https://openaccess.thecvf.com/content/CVPR2023/papers/Park_R...
- liuliu 3y ago
  Sorta of? Latent diffusion models operate in a compressed latent space, which is just a richer / learnable representation than DCT.
goldemerald 3y ago
For those interested, a modern version (vision transformers) was just published this year at CVPR https://openaccess.thecvf.com/content/CVPR2023/html/Park_RGB_No_More_Minimally-Decoded_JPEG_Vision_Transformers_CVPR_2023_paper.html https://openaccess.thecvf.com/content/CVPR2023/html/Park_RGB...
- qwertox 3y ago
  Thank you. First published 2022-11-29 on arxiv [0] and updated one month ago. Interesting line: "With these two improvements -- ViT and data augmentation -- we show that our ViT-Ti model achieves up to 39.2% faster training and 17.9% faster inference with no accuracy loss compared to the RGB counterpart." [0] https://arxiv.org/abs/2211.16421v2 https://arxiv.org/abs/2211.16421v2
- w-m 3y ago
  Ha, I remember the poster from the conference, it was quite crowded when I passed by. This one seemed to have a big focus on data augmentation in the DCT space. I was asking myself (and the author) whether you couldn’t eke out a little more efficiency by trying to quantize your network similarly to the default JPEG quantization table. As I understood, currently all weights are quantized uniformly, which does not make sense when your inputs are heavily quantized, does it? Maybe I should dive a little deeper into the Uber paper, they were focusing a bit more in the quantization part. Sorry if I’m talking nonsense, this is absolutely not my area, but I found the topic captivating.
  - buildbot 3y ago
    There is some work on using JPEG style DCT blocks for quantization: https://dl.acm.org/doi/10.1109/ISCA45697.2020.00075 https://dl.acm.org/doi/10.1109/ISCA45697.2020.00075 (Disclaimer, not mine, but a friends work)
- m3affan 3y ago
  Transformers are the taking the cake in the Deep Learning community
- jcjohns 3y ago
  I'm one of the authors of this CVPR paper -- cool to see our work mentioned on HN! The Uber paper from 2018 is one that has been floating around in the back of my head for a while. Decoding DCT to RGB is essentially an 8x8 stride 8 convolution -- it seems wasteful to perform this operation on CPU for data loading, then immediately pass the resulting decoded RGB into convolution layers that probably learn similar filters as those used during DCT decoding anyway. Compared to the earlier Uber paper, our CVPR paper makes two big advances: (1) Cleaner architecture: The Uber paper uses a CNN, while we use a ViT. It's kind of awkward to modify an existing CNN architecture to accept DCT instead of RGB since the grayscale data is 8x lower resolution than RGB, and the color information is 16x lower than RGB. With a CNN, you need to add extra layers to deal with the downsampled input, and use some kind of fusion mechanism to fuse the luma/chroma data of different resolution. With a ViT it's very straightforward to accept DCT input; you only need to change the patch embedding layer, and the body of the network is unchanged. (2) Data augmentation: The original Uber paper only showed speedup during inference. During training they need to perform data augmentation, so convert DCT to RGB, augment in RGB, then convert back to DCT to feed the augmented data to the model. This means that their approach will be slower during training vs an RGB model. In our paper we show to to perform all standard image augmentations directly in DCT, so we can get speedups during both training and inference. Happy to answer any questions about the project!
  - inoop 3y ago
    > Decoding DCT to RGB is essentially an 8x8 stride 8 convolution -- it seems wasteful to perform this operation on CPU for data loading Then why not do it on the GPU? Feels like exactly the sort of thing it was designed to do. Or alternatively, use nvjpeg?
    - jcjohns 3y ago
      This makes sense in theory, but is hard to get working in practice. We tried using nvjpeg to do JPEG decoding on GPU as a additional baseline, but using it as a drop-in replacement to a standard training pipeline gives huge slowdowns for a few reasons: (1) Batching: nvjpeg isn't batched; you need to decode one at a time in a loop. This is slow but could in principle be improved with a better GPU decoder. (2) Concurrent data loading / model execution: In a standard training pipeline, the CPU is loading and augmenting data on CPU for the next batch in parallel with the model running forward / backward on the current batch. Using the GPU for decoding blocks it from running the model concurrently. If you were careful I think you could probably find a way to interleave JPEG decoding and model execution on the GPU, but it's not straightforward. Just naively swapping out to use nvjpeg in a standard PyTorch training pipeline gives very bad performance. (3) Data augmentation: If you do DCT -> RGB decoding on the GPU, then you have to think about how and where to do data augmentation. You can augment in DCT either on CPU or on GPU; however DCT augmentation tends to be more expensive than RGB augmentation (especially for resize operations), so if you are already going through the trouble of decoding to RGB then it's probably much cheaper to augment in RGB. If you augment in RGB on GPU, then you are blocking parallel model execution for both JPEG decoding and augmentation, and problem (2) gets even worse. If you do RGB augmentation on CPU, you end up with and extra GPU -> CPU -> GPU round trip on every model iteration which again reduces performance.
GaggiX 3y ago
Add "(2018)"
- readthenotes1 3y ago
  Back when Uber was thinking it could make self driving taxis
  - londons_explore 3y ago
    Uber now has a deal with Waymo - so in effect they are still trying to do self-driving taxi's, but now via a complex business relationship.
    - capableweb 3y ago
      Ah, just like I "make hamburgers" when I go through the McDonals drive-through, although it's via a business transaction.
      - ianlevesque 3y ago
        God this website is cynical. Licensing technology from another company to commercialize it is completely legitimate.
        wpietri 3y ago
        That is legitimate, but that's not the point here. The point is Uber's hubris. A hubris very useful in pumping up its stock price ahead of an IPO. If they had quietly planned to license it from the get-go, nobody would have mentioned it.
        ianlevesque 3y ago
        Uber literally killed people trying to develop their own first. Tesla continues to do so. I wish everyone would license Waymo instead.
kunalgupta 3y ago
see also: https://twitter.com/goodside/status/1679358632431853568?s=46&t=YuSD72_CTtyLX4szIUa5JQ https://twitter.com/goodside/status/1679358632431853568?s=46...
- deleted 3y ago
  [deleted]
bob1029 3y ago
> Accuracy gains are due primarily to the specific use of a DCT representation, which turns out to work curiously well for image classification. It would seem quantization is a useful tool for any sort of NN-style application. If the expected output is intended to be human-like, why not feed it information that a typical human could not distinguish from a lossless representation? Seems like a simple game of expectations and information theory.
- kridsdale3 3y ago
  That's kind of the key theory behind why JPEG (and other lossy encodings) work at all. A perfect being would see a JPEG next to a PNG or TIFF and find the first repugnantly error-ridden. But we tend to ignore high-frequency data's specifics most of the time, so it psychologically works. I often wonder though, what do my cat and dog hear when I'm playing compressed music? Does it sounds like a muddy phone call to them?
  - jdiff 3y ago
    No, audio compression doesn't filter out high frequencies, that's just what computer audio as a whole does. And I don't think there's enough of those high frequency components in what humans typically record for a cat or dog to notice the difference. As far as compression, the tricks that work on us should work on them.
    - LordDragonfang 3y ago
      >MP3 compression works by reducing (or approximating) the accuracy of certain components of sound that are considered (by psychoacoustic analysis) to be beyond the hearing capabilities of most humans. -via Wikipedia This holds true for most other audio compression as well. Now, it's true that max recording frequency is bounded by sample rate via the Nyquist theorem, but that doesn't mean we're incapable of recording at higher fidelity - we just don't bother most of the time, because on consumer hardware it's going to be filtered out eventually anyway (or just not reproduced well enough, due to low-quality physical hardware). Recording studios will regularly produce masters that far exceed that normal hearing range though.
quantumstar4k 3y ago
[dead]
a-dub 3y ago
i guess an interesting question is: can you coax a network into learning a better perceptual compression transform than the dcts in jpeg?
rullelito 3y ago
I've heard of this long before 2018, was this really seen as novel in 2018?
wodenokoto 3y ago
That seems just in line with an earlier front page submission that used knn and gzipped text for categorization.
watersb 3y ago
IIRC, “neural network” style systems design started in the realm of computer vision with “Perceptrons”: https://en.wikipedia.org/wiki/Perceptrons_(book)Perceptrons https://en.wikipedia.org/wiki/Perceptrons_(book)Perceptrons = https://g.co/kgs/8Un4eW https://g.co/kgs/8Un4eW Makes sense that image processing would be a good fit in some cases.
shoulderfake 3y ago
[dead]
mochomocha 3y ago
I was doing the same thing at Netflix around the same time as a 20% research project. Training GANs end2end directly in JPEG coeffs space (and then rebuild a JPEG from the generated coeffs using libjpeg to get an image). The pitch was that it not only worked, but you could get fast training by representing each JPEG block as a dense + sparse vector (dense for the low DCT coeffs, sparse for the high ones since they're ~all zeros) and using a neural network library with fast ops on sparse data. Training on pixels is inefficient. Why have your first layers of CNNs relearn what's already smartly encoded in the JPEG bits in the first place before it's blown into a bloated height x width x 3 float matrix?
- corysama 3y ago
  As an AI armchair quarterback, I've always held the opinion that the image ML space has a counter-productive bias towards not pre-processing images stemming from a mixture of test purity and academic hubris. "Look what this method can learn from raw data completely independently!" makes for a nice paper. So, they stick with sRGB inputs rather than doing basic classic transforms like converting to YUV420. Everyone learns from the papers, so that's assumed to be the standard practice.
  - whimsicalism 3y ago
    In deep ML, people are pretty familiar with the bitter lesson and don't want to waste time on this.
    - Llamamoe 3y ago
      The bitter lesson is about not trying to encode impossible-to-formalize conceptual knowledge, not avoiding data efficiency and the need to scale the model up to ever higher parameter counts. If we followed this logic, we'd be training LLMs on character-level UTF-32 and just letting it figure everything out by itself, while needing two orders of magnitude bigger contexts and parameter counts.
      - whimsicalism 3y ago
kevmo314 3y ago
A lot of generative audio works very similar these days: it's much faster to predict and generate a codebook than a raw waveform.
waynecochran 3y ago
Since convolution in the spatial domain is simply multiplication in the frequency domain it seems that this would be wayyy faster.
namjh 3y ago
Might be related: https://arxiv.org/abs/2211.16421 https://arxiv.org/abs/2211.16421 It's a paper about directly learning via JPEG encodings which works well with visusal transformers' patch mechanism.
ulrikrasmussen 3y ago
The quantization used for JPEG is optimized to throw away information in the frequency space that doesn't matter much to human perception, but I wonder if that is also optimal for training neutral networks? Also, as far as I know, the human eye doesn't process images in blocks. I also wonder how blockless encoders such as JPEG 2000 would fare in this approach.
shreyshnaccount 3y ago
The next wave of ai magic will come from domain expertise.
jacobgorm 3y ago
I’ve tried this idea in the past and found it not very useful in practice. It breaks when you want to add image augmentation during training, and JPEG is anyway a pretty lousy format for storing training samples if you care about saving space or retaining image quality.

253 points by Anon84 3y ago

Finally someone who did this, I've always thought this was a low hanging fruit. I wonder if you could make interesting and quickly trained diffusion models with this trick.

waldarbeiter 3y ago
Note that it is from 2018. As someone here already mentioned there is a paper that applies the same idea to Vision Transformers published this year [1]. [1] https://openaccess.thecvf.com/content/CVPR2023/papers/Park_RGB_No_More_Minimally-Decoded_JPEG_Vision_Transformers_CVPR_2023_paper.pdf https://openaccess.thecvf.com/content/CVPR2023/papers/Park_R...
liuliu 3y ago
Sorta of? Latent diffusion models operate in a compressed latent space, which is just a richer / learnable representation than DCT.

goldemerald 3y ago

For those interested, a modern version (vision transformers) was just published this year at CVPR https://openaccess.thecvf.com/content/CVPR2023/html/Park_RGB_No_More_Minimally-Decoded_JPEG_Vision_Transformers_CVPR_2023_paper.html https://openaccess.thecvf.com/content/CVPR2023/html/Park_RGB...

qwertox 3y ago
Thank you. First published 2022-11-29 on arxiv [0] and updated one month ago. Interesting line: "With these two improvements -- ViT and data augmentation -- we show that our ViT-Ti model achieves up to 39.2% faster training and 17.9% faster inference with no accuracy loss compared to the RGB counterpart." [0] https://arxiv.org/abs/2211.16421v2 https://arxiv.org/abs/2211.16421v2
w-m 3y ago
Ha, I remember the poster from the conference, it was quite crowded when I passed by. This one seemed to have a big focus on data augmentation in the DCT space. I was asking myself (and the author) whether you couldn’t eke out a little more efficiency by trying to quantize your network similarly to the default JPEG quantization table. As I understood, currently all weights are quantized uniformly, which does not make sense when your inputs are heavily quantized, does it? Maybe I should dive a little deeper into the Uber paper, they were focusing a bit more in the quantization part. Sorry if I’m talking nonsense, this is absolutely not my area, but I found the topic captivating.
- buildbot 3y ago
  There is some work on using JPEG style DCT blocks for quantization: https://dl.acm.org/doi/10.1109/ISCA45697.2020.00075 https://dl.acm.org/doi/10.1109/ISCA45697.2020.00075 (Disclaimer, not mine, but a friends work)
m3affan 3y ago
Transformers are the taking the cake in the Deep Learning community
jcjohns 3y ago
I'm one of the authors of this CVPR paper -- cool to see our work mentioned on HN! The Uber paper from 2018 is one that has been floating around in the back of my head for a while. Decoding DCT to RGB is essentially an 8x8 stride 8 convolution -- it seems wasteful to perform this operation on CPU for data loading, then immediately pass the resulting decoded RGB into convolution layers that probably learn similar filters as those used during DCT decoding anyway. Compared to the earlier Uber paper, our CVPR paper makes two big advances: (1) Cleaner architecture: The Uber paper uses a CNN, while we use a ViT. It's kind of awkward to modify an existing CNN architecture to accept DCT instead of RGB since the grayscale data is 8x lower resolution than RGB, and the color information is 16x lower than RGB. With a CNN, you need to add extra layers to deal with the downsampled input, and use some kind of fusion mechanism to fuse the luma/chroma data of different resolution. With a ViT it's very straightforward to accept DCT input; you only need to change the patch embedding layer, and the body of the network is unchanged. (2) Data augmentation: The original Uber paper only showed speedup during inference. During training they need to perform data augmentation, so convert DCT to RGB, augment in RGB, then convert back to DCT to feed the augmented data to the model. This means that their approach will be slower during training vs an RGB model. In our paper we show to to perform all standard image augmentations directly in DCT, so we can get speedups during both training and inference. Happy to answer any questions about the project!
- inoop 3y ago
  > Decoding DCT to RGB is essentially an 8x8 stride 8 convolution -- it seems wasteful to perform this operation on CPU for data loading Then why not do it on the GPU? Feels like exactly the sort of thing it was designed to do. Or alternatively, use nvjpeg?
  - jcjohns 3y ago
    This makes sense in theory, but is hard to get working in practice. We tried using nvjpeg to do JPEG decoding on GPU as a additional baseline, but using it as a drop-in replacement to a standard training pipeline gives huge slowdowns for a few reasons: (1) Batching: nvjpeg isn't batched; you need to decode one at a time in a loop. This is slow but could in principle be improved with a better GPU decoder. (2) Concurrent data loading / model execution: In a standard training pipeline, the CPU is loading and augmenting data on CPU for the next batch in parallel with the model running forward / backward on the current batch. Using the GPU for decoding blocks it from running the model concurrently. If you were careful I think you could probably find a way to interleave JPEG decoding and model execution on the GPU, but it's not straightforward. Just naively swapping out to use nvjpeg in a standard PyTorch training pipeline gives very bad performance. (3) Data augmentation: If you do DCT -> RGB decoding on the GPU, then you have to think about how and where to do data augmentation. You can augment in DCT either on CPU or on GPU; however DCT augmentation tends to be more expensive than RGB augmentation (especially for resize operations), so if you are already going through the trouble of decoding to RGB then it's probably much cheaper to augment in RGB. If you augment in RGB on GPU, then you are blocking parallel model execution for both JPEG decoding and augmentation, and problem (2) gets even worse. If you do RGB augmentation on CPU, you end up with and extra GPU -> CPU -> GPU round trip on every model iteration which again reduces performance.

GaggiX 3y ago

Add "(2018)"

readthenotes1 3y ago
Back when Uber was thinking it could make self driving taxis
- londons_explore 3y ago
  Uber now has a deal with Waymo - so in effect they are still trying to do self-driving taxi's, but now via a complex business relationship.
  - capableweb 3y ago
    Ah, just like I "make hamburgers" when I go through the McDonals drive-through, although it's via a business transaction.
    - ianlevesque 3y ago
      God this website is cynical. Licensing technology from another company to commercialize it is completely legitimate.
      - wpietri 3y ago
        That is legitimate, but that's not the point here. The point is Uber's hubris. A hubris very useful in pumping up its stock price ahead of an IPO. If they had quietly planned to license it from the get-go, nobody would have mentioned it.
        ianlevesque 3y ago
        Uber literally killed people trying to develop their own first. Tesla continues to do so. I wish everyone would license Waymo instead.

kunalgupta 3y ago

see also: https://twitter.com/goodside/status/1679358632431853568?s=46&t=YuSD72_CTtyLX4szIUa5JQ https://twitter.com/goodside/status/1679358632431853568?s=46...

deleted 3y ago
[deleted]

bob1029 3y ago

> Accuracy gains are due primarily to the specific use of a DCT representation, which turns out to work curiously well for image classification. It would seem quantization is a useful tool for any sort of NN-style application. If the expected output is intended to be human-like, why not feed it information that a typical human could not distinguish from a lossless representation? Seems like a simple game of expectations and information theory.

kridsdale3 3y ago
That's kind of the key theory behind why JPEG (and other lossy encodings) work at all. A perfect being would see a JPEG next to a PNG or TIFF and find the first repugnantly error-ridden. But we tend to ignore high-frequency data's specifics most of the time, so it psychologically works. I often wonder though, what do my cat and dog hear when I'm playing compressed music? Does it sounds like a muddy phone call to them?
- jdiff 3y ago
  No, audio compression doesn't filter out high frequencies, that's just what computer audio as a whole does. And I don't think there's enough of those high frequency components in what humans typically record for a cat or dog to notice the difference. As far as compression, the tricks that work on us should work on them.
  - LordDragonfang 3y ago
    >MP3 compression works by reducing (or approximating) the accuracy of certain components of sound that are considered (by psychoacoustic analysis) to be beyond the hearing capabilities of most humans. -via Wikipedia This holds true for most other audio compression as well. Now, it's true that max recording frequency is bounded by sample rate via the Nyquist theorem, but that doesn't mean we're incapable of recording at higher fidelity - we just don't bother most of the time, because on consumer hardware it's going to be filtered out eventually anyway (or just not reproduced well enough, due to low-quality physical hardware). Recording studios will regularly produce masters that far exceed that normal hearing range though.

quantumstar4k 3y ago

[dead]

a-dub 3y ago

i guess an interesting question is: can you coax a network into learning a better perceptual compression transform than the dcts in jpeg?

rullelito 3y ago

I've heard of this long before 2018, was this really seen as novel in 2018?

wodenokoto 3y ago

That seems just in line with an earlier front page submission that used knn and gzipped text for categorization.

watersb 3y ago

IIRC, “neural network” style systems design started in the realm of computer vision with “Perceptrons”: https://en.wikipedia.org/wiki/Perceptrons_(book)Perceptrons https://en.wikipedia.org/wiki/Perceptrons_(book)Perceptrons = https://g.co/kgs/8Un4eW https://g.co/kgs/8Un4eW Makes sense that image processing would be a good fit in some cases.

shoulderfake 3y ago

[dead]

mochomocha 3y ago

I was doing the same thing at Netflix around the same time as a 20% research project. Training GANs end2end directly in JPEG coeffs space (and then rebuild a JPEG from the generated coeffs using libjpeg to get an image). The pitch was that it not only worked, but you could get fast training by representing each JPEG block as a dense + sparse vector (dense for the low DCT coeffs, sparse for the high ones since they're ~all zeros) and using a neural network library with fast ops on sparse data. Training on pixels is inefficient. Why have your first layers of CNNs relearn what's already smartly encoded in the JPEG bits in the first place before it's blown into a bloated height x width x 3 float matrix?

corysama 3y ago
As an AI armchair quarterback, I've always held the opinion that the image ML space has a counter-productive bias towards not pre-processing images stemming from a mixture of test purity and academic hubris. "Look what this method can learn from raw data completely independently!" makes for a nice paper. So, they stick with sRGB inputs rather than doing basic classic transforms like converting to YUV420. Everyone learns from the papers, so that's assumed to be the standard practice.
- whimsicalism 3y ago
  In deep ML, people are pretty familiar with the bitter lesson and don't want to waste time on this.
  - Llamamoe 3y ago
    The bitter lesson is about not trying to encode impossible-to-formalize conceptual knowledge, not avoiding data efficiency and the need to scale the model up to ever higher parameter counts. If we followed this logic, we'd be training LLMs on character-level UTF-32 and just letting it figure everything out by itself, while needing two orders of magnitude bigger contexts and parameter counts.
    - whimsicalism 3y ago

kevmo314 3y ago

A lot of generative audio works very similar these days: it's much faster to predict and generate a codebook than a raw waveform.

waynecochran 3y ago

Since convolution in the spatial domain is simply multiplication in the frequency domain it seems that this would be wayyy faster.

namjh 3y ago

Might be related: https://arxiv.org/abs/2211.16421 https://arxiv.org/abs/2211.16421 It's a paper about directly learning via JPEG encodings which works well with visusal transformers' patch mechanism.

ulrikrasmussen 3y ago

The quantization used for JPEG is optimized to throw away information in the frequency space that doesn't matter much to human perception, but I wonder if that is also optimal for training neutral networks? Also, as far as I know, the human eye doesn't process images in blocks. I also wonder how blockless encoders such as JPEG 2000 would fare in this approach.

shreyshnaccount 3y ago

The next wave of ai magic will come from domain expertise.

jacobgorm 3y ago

I’ve tried this idea in the past and found it not very useful in practice. It breaks when you want to add image augmentation during training, and JPEG is anyway a pretty lousy format for storing training samples if you care about saving space or retaining image quality.