I wiiiiish we could do stride 2. I have tried many schemes to get it to work, it would be a dream. The best I could guess would be a fused conv+max operator. It

I wiiiiish we could do stride 2. I have tried many schemes to get it to work, it would be a dream. The best I could guess would be a fused conv+max operator. It drops a very unreasonable amount with anything else (including 5x5, stride 2). Something about the sparse activations I think is playing very nicely with the high learning rate. If it's not indexing, I'm not sure what quite it is. It's the backwards pass that is slow (all in GPU of course), like super slow. If I was able to crack out Triton I would just make a bitmask, nearest neighbor expand the incoming gradients and multiply against the bitmask. On the GPU, gather-like ops have been slow in Pytorch from what I've seen, but your comment is making me scratch my head as to what that could be now. Hm.... very interesting. Thanks for letting me know. Very curious. I'll have to take a look at that.