6 ms·

The Speed of VITs and CNNs

74 points by jxmorris12 1y ago

GaggiX 1y ago
>text in photos, phone screens, diagrams and charts, 448px² is enough Not in the graph you provided as an example.
- yorwba 1y ago
  It has this note at the bottom: "Note that I chose an unusually long chart to exemplify an extreme case of aspect ratio stretching. Still, 512px² is enough. This is two_col_40643 from ChartQA validation set. Original resolution: 800x1556." But yeah, ultimately which resolution you need depends on the image content, and if you need to squeeze out every bit of accuracy, processing at the original resolution is unavoidable.
- zamadatix 1y ago
  It's enough, especially if you select one of the sharper options like Lanczos, but 512px is sure a lot easier for a human.
ninamoss 1y ago
Really appreciated the post, very insightful. We also use VITs for some of our models and find that between model compilation and hyperparameter tuning we are able to get sub second evaluation of images on commodity hardware while maintaining a high precision and recall.
John7878781 1y ago
In the Twitter thread the article mentions, LeCun makes his claim only for "high-resolution" images and the article assumes 1024x1024 to fall under this category. To me, 1024x1024 is not "high-resolution." This assumption is flawed imo I currently use convnext for image classification at a size of 4096x2048 (definitely counts as "high-resolution"). For my use case, it would never be practical to use VITs for this. I can't downscale the resolution because extremely fine details need to be preserved. I don't think LeCun's comment was a "knee-jerk reaction" as the article claims.
- hedgehog 1y ago
  LeCun's technical assessments have borne out over a lot of years. The likely next step in scaling vision transformers is to treat the image as a MIP pyramid and use the transformer to adaptively sample out of that. Requires RL to train (tricky) but it would decouple compute footprint from input size.
  - tbalsam 1y ago
    As someone who has worked in computer vision ML for nearly a decade, this sounds like a terrible idea. You don't need RL remotely for this usecase. Image resolution pyramids are pretty normal tho and handling them well/efficiently is the big thing. Using RL for this would be like trying to use graphene to make a computer screen because it's new and flashy and everyone's talking about it. RL is inherently very sample inefficient, and is there to approximate when you don't have certain defined informative components, which we do have in computer vision in spades. Crossentropy losses (and the like) are (generally, IME/IMO) what RL losses try to approximate, only on a much larger (and more poorly-defined) scale. Please mark speculation as such -- I've seen people see confident statements like this and spend a lot of time/manhours on it (because it seems plausible). It is not a bad idea from a creativity standpoint, but practically is most certainly not the way to go about it. (That being said, you can try for dynamic sparsity stuff, it has some painful tradeoffs that generally don't scale but no way in Illinois do you need RL for that)
    - hedgehog 1y ago
      SPECULATION ALERT! I think there's reasonable motivation though. In the last few years there has been a steady drip of papers in the general area, at least insofar as they use vision transformers and image pyramids, and work on applying RL to object detection goes back before that. IoU and the general way SSD and YOLO descendants are set up is kind of wacky so I don't think it's much of a stretch to try to both 1) avoid attending to or materializing most of the pyramid, and 2) go directly to feature proposals without worrying about box anchors or grid cells or any of that. Now with that context if you still think it's a terrible idea, well, you're probably more current than I am.
kookamamie 1y ago
> You don't need very high resolution Yes, you do. Also, 1024x1024 is not high resolution. An example is segmenting basic 1920x1080 (FHD) video in 60 Hz formats.
- CHY872 1y ago
  The article basically argues: You would expect to get similarly good results with subsampling in practice. E.g. no need to process at 1920x1080 when you can do 960x540. Separately, you can break down many problems into smaller tiles and get similar quality results without the compute overheads of a high res ViT.
- dimatura 1y ago
  Yeah, the article was painting with a bit too of a broad stroke IMO, though they did briefly acknowledge "special exceptions" such as satellite or medical imagery. It's very application-dependent. That said, in my experience beginners do often overestimate how much image resolution is needed for a given task for some reason. I often find myself asking to retry their experiments with a lower resolution. There's a surprising amount of information in 128x128 or even smaller images.
  - magicalhippo 1y ago
    I have a vivid memory of playing Rise of the Triad[1] against my buddy over serial cable. As most PC games from back then, it used mode 13h[2], so 320x200 resolution with a 256 color palette. I have the distinct memory of firing a rocket at him from far away because I thought that one pixel had the wrong color, and killing him to his great frustration. Good times. You can play the shareware portion of the game here[3] to get an idea. [1]: https://en.wikipedia.org/wiki/Rise_of_the_Triad https://en.wikipedia.org/wiki/Rise_of_the_Triad [2]: https://en.wikipedia.org/wiki/Mode_13h https://en.wikipedia.org/wiki/Mode_13h [3]: https://www.dosgames.com/game/rise-of-the-triad/ https://www.dosgames.com/game/rise-of-the-triad/
jacobgorm 1y ago
A nice feature of CNNs is that you can change the resolution at inference time without retraining. For instance, when the user plugs in a camera with a different aspect or decides to the change the orientation of his phone from landscape to portrait. It is not clear to me if VITs can support aspect or resolution changes without any retraining?
- lava_pidgeon 1y ago
  Can you elaborate? In my experience it is the opposite: CNNs are highly depend on the input tensor shapes thus resolution change need even an architectional change. While resolution changes in ViT lead to more tokens, a ViT model can handle that (for image classification e.g. you always take the CLS token, Segmentation maps and similar task have the same output as in the input).

74 points by jxmorris12 1y ago

GaggiX 1y ago

>text in photos, phone screens, diagrams and charts, 448px² is enough Not in the graph you provided as an example.

yorwba 1y ago
It has this note at the bottom: "Note that I chose an unusually long chart to exemplify an extreme case of aspect ratio stretching. Still, 512px² is enough. This is two_col_40643 from ChartQA validation set. Original resolution: 800x1556." But yeah, ultimately which resolution you need depends on the image content, and if you need to squeeze out every bit of accuracy, processing at the original resolution is unavoidable.
zamadatix 1y ago
It's enough, especially if you select one of the sharper options like Lanczos, but 512px is sure a lot easier for a human.

ninamoss 1y ago

Really appreciated the post, very insightful. We also use VITs for some of our models and find that between model compilation and hyperparameter tuning we are able to get sub second evaluation of images on commodity hardware while maintaining a high precision and recall.

John7878781 1y ago

In the Twitter thread the article mentions, LeCun makes his claim only for "high-resolution" images and the article assumes 1024x1024 to fall under this category. To me, 1024x1024 is not "high-resolution." This assumption is flawed imo I currently use convnext for image classification at a size of 4096x2048 (definitely counts as "high-resolution"). For my use case, it would never be practical to use VITs for this. I can't downscale the resolution because extremely fine details need to be preserved. I don't think LeCun's comment was a "knee-jerk reaction" as the article claims.

hedgehog 1y ago
LeCun's technical assessments have borne out over a lot of years. The likely next step in scaling vision transformers is to treat the image as a MIP pyramid and use the transformer to adaptively sample out of that. Requires RL to train (tricky) but it would decouple compute footprint from input size.
- tbalsam 1y ago
  As someone who has worked in computer vision ML for nearly a decade, this sounds like a terrible idea. You don't need RL remotely for this usecase. Image resolution pyramids are pretty normal tho and handling them well/efficiently is the big thing. Using RL for this would be like trying to use graphene to make a computer screen because it's new and flashy and everyone's talking about it. RL is inherently very sample inefficient, and is there to approximate when you don't have certain defined informative components, which we do have in computer vision in spades. Crossentropy losses (and the like) are (generally, IME/IMO) what RL losses try to approximate, only on a much larger (and more poorly-defined) scale. Please mark speculation as such -- I've seen people see confident statements like this and spend a lot of time/manhours on it (because it seems plausible). It is not a bad idea from a creativity standpoint, but practically is most certainly not the way to go about it. (That being said, you can try for dynamic sparsity stuff, it has some painful tradeoffs that generally don't scale but no way in Illinois do you need RL for that)
  - hedgehog 1y ago
    SPECULATION ALERT! I think there's reasonable motivation though. In the last few years there has been a steady drip of papers in the general area, at least insofar as they use vision transformers and image pyramids, and work on applying RL to object detection goes back before that. IoU and the general way SSD and YOLO descendants are set up is kind of wacky so I don't think it's much of a stretch to try to both 1) avoid attending to or materializing most of the pyramid, and 2) go directly to feature proposals without worrying about box anchors or grid cells or any of that. Now with that context if you still think it's a terrible idea, well, you're probably more current than I am.

kookamamie 1y ago

> You don't need very high resolution Yes, you do. Also, 1024x1024 is not high resolution. An example is segmenting basic 1920x1080 (FHD) video in 60 Hz formats.

CHY872 1y ago
The article basically argues: You would expect to get similarly good results with subsampling in practice. E.g. no need to process at 1920x1080 when you can do 960x540. Separately, you can break down many problems into smaller tiles and get similar quality results without the compute overheads of a high res ViT.
dimatura 1y ago
Yeah, the article was painting with a bit too of a broad stroke IMO, though they did briefly acknowledge "special exceptions" such as satellite or medical imagery. It's very application-dependent. That said, in my experience beginners do often overestimate how much image resolution is needed for a given task for some reason. I often find myself asking to retry their experiments with a lower resolution. There's a surprising amount of information in 128x128 or even smaller images.
- magicalhippo 1y ago
  I have a vivid memory of playing Rise of the Triad[1] against my buddy over serial cable. As most PC games from back then, it used mode 13h[2], so 320x200 resolution with a 256 color palette. I have the distinct memory of firing a rocket at him from far away because I thought that one pixel had the wrong color, and killing him to his great frustration. Good times. You can play the shareware portion of the game here[3] to get an idea. [1]: https://en.wikipedia.org/wiki/Rise_of_the_Triad https://en.wikipedia.org/wiki/Rise_of_the_Triad [2]: https://en.wikipedia.org/wiki/Mode_13h https://en.wikipedia.org/wiki/Mode_13h [3]: https://www.dosgames.com/game/rise-of-the-triad/ https://www.dosgames.com/game/rise-of-the-triad/

jacobgorm 1y ago

A nice feature of CNNs is that you can change the resolution at inference time without retraining. For instance, when the user plugs in a camera with a different aspect or decides to the change the orientation of his phone from landscape to portrait. It is not clear to me if VITs can support aspect or resolution changes without any retraining?

lava_pidgeon 1y ago
Can you elaborate? In my experience it is the opposite: CNNs are highly depend on the input tensor shapes thus resolution change need even an architectional change. While resolution changes in ViT lead to more tokens, a ViT model can handle that (for image classification e.g. you always take the CLS token, Segmentation maps and similar task have the same output as in the input).