3 ms·

In the Twitter thread the article mentions, LeCun makes his claim only for "high-resolution" images and the article assumes 1024x1024 to fall under this categor

In the Twitter thread the article mentions, LeCun makes his claim only for "high-resolution" images and the article assumes 1024x1024 to fall under this category. To me, 1024x1024 is not "high-resolution." This assumption is flawed imo I currently use convnext for image classification at a size of 4096x2048 (definitely counts as "high-resolution"). For my use case, it would never be practical to use VITs for this. I can't downscale the resolution because extremely fine details need to be preserved. I don't think LeCun's comment was a "knee-jerk reaction" as the article claims.

hedgehog 1y ago
LeCun's technical assessments have borne out over a lot of years. The likely next step in scaling vision transformers is to treat the image as a MIP pyramid and use the transformer to adaptively sample out of that. Requires RL to train (tricky) but it would decouple compute footprint from input size.
- tbalsam 1y ago
  As someone who has worked in computer vision ML for nearly a decade, this sounds like a terrible idea. You don't need RL remotely for this usecase. Image resolution pyramids are pretty normal tho and handling them well/efficiently is the big thing. Using RL for this would be like trying to use graphene to make a computer screen because it's new and flashy and everyone's talking about it. RL is inherently very sample inefficient, and is there to approximate when you don't have certain defined informative components, which we do have in computer vision in spades. Crossentropy losses (and the like) are (generally, IME/IMO) what RL losses try to approximate, only on a much larger (and more poorly-defined) scale. Please mark speculation as such -- I've seen people see confident statements like this and spend a lot of time/manhours on it (because it seems plausible). It is not a bad idea from a creativity standpoint, but practically is most certainly not the way to go about it. (That being said, you can try for dynamic sparsity stuff, it has some painful tradeoffs that generally don't scale but no way in Illinois do you need RL for that)
  - hedgehog 1y ago
    SPECULATION ALERT! I think there's reasonable motivation though. In the last few years there has been a steady drip of papers in the general area, at least insofar as they use vision transformers and image pyramids, and work on applying RL to object detection goes back before that. IoU and the general way SSD and YOLO descendants are set up is kind of wacky so I don't think it's much of a stretch to try to both 1) avoid attending to or materializing most of the pyramid, and 2) go directly to feature proposals without worrying about box anchors or grid cells or any of that. Now with that context if you still think it's a terrible idea, well, you're probably more current than I am.
    - tbalsam 1y ago
      Not bad frustrations at all. That said -- IoU is how the final box scores are calculated, that doesn't change how you do feature aggregation, this will happen in basically any technique you use. Modern SSD/YOLO-style detectors use efficient feature pyramids, you need that to know where to propose where things are in the image. This sounds a lot like going back to the old school object detection techniques which end up being more inefficient in general, generally very compute inefficient.
- dimatura 1y ago
  There's been a huge amount of work on image transformers since the original VIT. A lot of it has explored different schemes to slice up the image in tokens, and I've definitely seen some of it using a multiresolution pyramid. Not sure about the RL part - after all, the higher/low-res levels of the pyramid would add less tokens than the base/high-res level, so it doesn't seem that necessary. But given the sheer volume of work out there I can bet someone has explored this idea or something pretty close to it already.
djoldman 1y ago
Interesting. Can you run your images through a segment model first and then only classify interesting boxes?
lairv 1y ago
Curious what kind of classification problems requires full 4096x2048 images, couldn't you feed multiple 512x512 overlapping crops instead?
threeducks 1y ago
ConvNeXT's architecture contains an AdaptiveAvgPool2d layer: https://github.com/pytorch/vision/blob/5f03dc524bdb7529bb4f2e84d2d8c237233fc62a/torchvision/models/convnext.py#L172 https://github.com/pytorch/vision/blob/5f03dc524bdb7529bb4f2... This means that you can split your image into tiles, process each tile individually, average the results, apply a final classification layer to the average and get exactly the same result. For reference, see the demonstration below. You could of course do exactly the same thing with a vision transformer instead of a convolutional neural network. That being said, architecture is wildly overemphasized in my opinion. Data is everything. import torch, torchvision.models device = torch.device("cuda" if torch.cuda.is_available() else "cpu") model = torchvision.models.convnext_small() model.to(device) tile_size, image_size = 32, 224 # note that 32 divides 224 evenly image = torch.randn((1, 3, image_size, image_size), device=device) # Process image as usual x_expected = model(image) # Process image as tiles (using for-loops for educational purposes; should use .view and .permute instead for performance) features = [ model.features(image[:, :, y:y + tile_size, x:x + tile_size]) for y in range(0, image_size, tile_size) for x in range(0, image_size, tile_size)] x = model.classifier(sum(features) / len(features)) print(f"Mean squared error: {(x - x_expected).pow(2).mean().item():.20f}")
- tbalsam 1y ago
  As someone who's done a fair bit of architecture work -- both are important! Making it either or is a very silly thing, both are the limiting factor for the other and there are no two ways about it. Also, for classification, MaxPooling is often far superior, you can learn an average smoothing filter in your convolutions beforehand in a data-dependent manner so that Nyquist sampling stuff is properly preserved. Also, please do smoothed crossentropy for image class stuff (generally speaking, unless maybe data is hilariously large), MSE won't nearly cut it! But that being said, adaptive stuff certainly is great when doing classification. Something to note is that batching does become an issue at a certain point -- as well as certain other fine-grained details if you're simply going to average it all down to one single vector (IIUC).
  - threeducks 1y ago

hedgehog 1y ago

LeCun's technical assessments have borne out over a lot of years. The likely next step in scaling vision transformers is to treat the image as a MIP pyramid and use the transformer to adaptively sample out of that. Requires RL to train (tricky) but it would decouple compute footprint from input size.

tbalsam 1y ago
As someone who has worked in computer vision ML for nearly a decade, this sounds like a terrible idea. You don't need RL remotely for this usecase. Image resolution pyramids are pretty normal tho and handling them well/efficiently is the big thing. Using RL for this would be like trying to use graphene to make a computer screen because it's new and flashy and everyone's talking about it. RL is inherently very sample inefficient, and is there to approximate when you don't have certain defined informative components, which we do have in computer vision in spades. Crossentropy losses (and the like) are (generally, IME/IMO) what RL losses try to approximate, only on a much larger (and more poorly-defined) scale. Please mark speculation as such -- I've seen people see confident statements like this and spend a lot of time/manhours on it (because it seems plausible). It is not a bad idea from a creativity standpoint, but practically is most certainly not the way to go about it. (That being said, you can try for dynamic sparsity stuff, it has some painful tradeoffs that generally don't scale but no way in Illinois do you need RL for that)
- hedgehog 1y ago
  SPECULATION ALERT! I think there's reasonable motivation though. In the last few years there has been a steady drip of papers in the general area, at least insofar as they use vision transformers and image pyramids, and work on applying RL to object detection goes back before that. IoU and the general way SSD and YOLO descendants are set up is kind of wacky so I don't think it's much of a stretch to try to both 1) avoid attending to or materializing most of the pyramid, and 2) go directly to feature proposals without worrying about box anchors or grid cells or any of that. Now with that context if you still think it's a terrible idea, well, you're probably more current than I am.
  - tbalsam 1y ago
    Not bad frustrations at all. That said -- IoU is how the final box scores are calculated, that doesn't change how you do feature aggregation, this will happen in basically any technique you use. Modern SSD/YOLO-style detectors use efficient feature pyramids, you need that to know where to propose where things are in the image. This sounds a lot like going back to the old school object detection techniques which end up being more inefficient in general, generally very compute inefficient.
dimatura 1y ago
There's been a huge amount of work on image transformers since the original VIT. A lot of it has explored different schemes to slice up the image in tokens, and I've definitely seen some of it using a multiresolution pyramid. Not sure about the RL part - after all, the higher/low-res levels of the pyramid would add less tokens than the base/high-res level, so it doesn't seem that necessary. But given the sheer volume of work out there I can bet someone has explored this idea or something pretty close to it already.

djoldman 1y ago

Interesting. Can you run your images through a segment model first and then only classify interesting boxes?

lairv 1y ago

Curious what kind of classification problems requires full 4096x2048 images, couldn't you feed multiple 512x512 overlapping crops instead?

threeducks 1y ago

ConvNeXT's architecture contains an AdaptiveAvgPool2d layer: https://github.com/pytorch/vision/blob/5f03dc524bdb7529bb4f2e84d2d8c237233fc62a/torchvision/models/convnext.py#L172 https://github.com/pytorch/vision/blob/5f03dc524bdb7529bb4f2... This means that you can split your image into tiles, process each tile individually, average the results, apply a final classification layer to the average and get exactly the same result. For reference, see the demonstration below. You could of course do exactly the same thing with a vision transformer instead of a convolutional neural network. That being said, architecture is wildly overemphasized in my opinion. Data is everything. import torch, torchvision.models device = torch.device("cuda" if torch.cuda.is_available() else "cpu") model = torchvision.models.convnext_small() model.to(device) tile_size, image_size = 32, 224 # note that 32 divides 224 evenly image = torch.randn((1, 3, image_size, image_size), device=device) # Process image as usual x_expected = model(image) # Process image as tiles (using for-loops for educational purposes; should use .view and .permute instead for performance) features = [ model.features(image[:, :, y:y + tile_size, x:x + tile_size]) for y in range(0, image_size, tile_size) for x in range(0, image_size, tile_size)] x = model.classifier(sum(features) / len(features)) print(f"Mean squared error: {(x - x_expected).pow(2).mean().item():.20f}")

tbalsam 1y ago
As someone who's done a fair bit of architecture work -- both are important! Making it either or is a very silly thing, both are the limiting factor for the other and there are no two ways about it. Also, for classification, MaxPooling is often far superior, you can learn an average smoothing filter in your convolutions beforehand in a data-dependent manner so that Nyquist sampling stuff is properly preserved. Also, please do smoothed crossentropy for image class stuff (generally speaking, unless maybe data is hilariously large), MSE won't nearly cut it! But that being said, adaptive stuff certainly is great when doing classification. Something to note is that batching does become an issue at a certain point -- as well as certain other fine-grained details if you're simply going to average it all down to one single vector (IIUC).
- threeducks 1y ago