3 ms·
given the focus on transformers in recent years, modernizing and adapting CNNs to hardware advances seems like an under-researched area. it's interesting and so
by panabee 4y ago
given the focus on transformers in recent years, modernizing and adapting CNNs to hardware advances seems like an under-researched area. it's interesting and something we're exploring.
FAIR did great work with this paper.
- macrolocal 4y agoNb. There are plenty of specialized architectures for convolutions: Nervana, Habana, Groq, etc.
- panabee 4y agothanks for sharing
- thatsadude 4y agoYou should see MobileOne and FastVIT from Apple
- panabee 4y agothanks, will do!
- godelski 4y agoThis is still being pursued. Ross Wightmann's timm[0,1] package (now on Hugging Face) has done a lot of this. There's also a V2 of ConvNext[2]. Ross does write about this a lot on Twitter fwiw. I should also mention that there are still many transformer based networks that still beat convs. So there probably won't be a resurgence in convs until someone can show that there's a really strong reason for them. They have some advantages but they also might not be flexible enough for the long range tasks in segmentation and detection. But maybe they are. FAIR definitely did great work with ConvNext, and I do hope to see more. There always needs to be people pushing unpopular paradigms. [0] https://github.com/huggingface/pytorch-image-models https://github.com/huggingface/pytorch-image-models [1] https://arxiv.org/abs/2110.00476 https://arxiv.org/abs/2110.00476 [2] https://arxiv.org/abs/2301.00808 https://arxiv.org/abs/2301.00808
- panabee 4y agoagreed. breakthroughs by definition upend conventional wisdom. transformers currently represent conventional wisdom. maybe transformers are indeed better than CNNs, but tech history is full of cases where primitive technologies take magical leaps when married to powerful hardware and intelligent system design. low probability at this stage, especially given the impressive long-range dependencies and generalizability of transformers, but let's see what happens. besides wightmann, who else do you recommend following for CNNs?
- godelski 4y ago> who else do you recommend following for CNNs? Hard to say tbh. Saining Xie was last on both convnext papers. I do know Trever Darrel always has interesting things going on in his lab. But I can't say anyone consistently though. I've done a few architecture papers but my focus is on generative modeling.
- panabee 4y agomy interest in your research is now piqued, trying to find papers now. :) re your comment on chatgpt and citations, my sincere belief is that one of the more enduring benefits of LLMs, once researchers can eliminate/reduce hallucinations, will be to expose baseless claims and flimsy logic. this won't eliminate all misinformation and misleading arguments, of course, but it will increase friction for bad actors and help good actors identify specious reasoning. an automated BS detector if you will -- not unlike a noisy car alarm that deters theft.
- panabee 4y agothinking about this further, it seems difficult for CNNs to overcome the unified modeling benefits afforded by transformers. what do you think?
- godelski 4y agoI'll go a little against Sutton here, though I mostly agree with him. Sometimes restriction in architecture has benefits. Not all dimensions are equal.