3 ms·

agreed. breakthroughs by definition upend conventional wisdom. transformers currently represent conventional wisdom. maybe transformers are indeed better tha

agreed. breakthroughs by definition upend conventional wisdom. transformers currently represent conventional wisdom. maybe transformers are indeed better than CNNs, but tech history is full of cases where primitive technologies take magical leaps when married to powerful hardware and intelligent system design. low probability at this stage, especially given the impressive long-range dependencies and generalizability of transformers, but let's see what happens. besides wightmann, who else do you recommend following for CNNs?

godelski 4y ago
> who else do you recommend following for CNNs? Hard to say tbh. Saining Xie was last on both convnext papers. I do know Trever Darrel always has interesting things going on in his lab. But I can't say anyone consistently though. I've done a few architecture papers but my focus is on generative modeling.
- panabee 4y ago
  my interest in your research is now piqued, trying to find papers now. :) re your comment on chatgpt and citations, my sincere belief is that one of the more enduring benefits of LLMs, once researchers can eliminate/reduce hallucinations, will be to expose baseless claims and flimsy logic. this won't eliminate all misinformation and misleading arguments, of course, but it will increase friction for bad actors and help good actors identify specious reasoning. an automated BS detector if you will -- not unlike a noisy car alarm that deters theft.
- panabee 4y ago
  thinking about this further, it seems difficult for CNNs to overcome the unified modeling benefits afforded by transformers. what do you think?
  - godelski 4y ago
    I'll go a little against Sutton here, though I mostly agree with him. Sometimes restriction in architecture has benefits. Not all dimensions are equal.
    - panabee 3y ago
      dino v2 reflipped me, reasoning that text supervision artificially constrains image learning: https://arxiv.org/abs/2304.07193 https://arxiv.org/abs/2304.07193 if unified modeling isn't necessary, or even hurtful, a key advantage of transformers goes away.

godelski 4y ago

> who else do you recommend following for CNNs? Hard to say tbh. Saining Xie was last on both convnext papers. I do know Trever Darrel always has interesting things going on in his lab. But I can't say anyone consistently though. I've done a few architecture papers but my focus is on generative modeling.

panabee 4y ago
my interest in your research is now piqued, trying to find papers now. :) re your comment on chatgpt and citations, my sincere belief is that one of the more enduring benefits of LLMs, once researchers can eliminate/reduce hallucinations, will be to expose baseless claims and flimsy logic. this won't eliminate all misinformation and misleading arguments, of course, but it will increase friction for bad actors and help good actors identify specious reasoning. an automated BS detector if you will -- not unlike a noisy car alarm that deters theft.
panabee 4y ago
thinking about this further, it seems difficult for CNNs to overcome the unified modeling benefits afforded by transformers. what do you think?
- godelski 4y ago
  I'll go a little against Sutton here, though I mostly agree with him. Sometimes restriction in architecture has benefits. Not all dimensions are equal.
  - panabee 3y ago
    dino v2 reflipped me, reasoning that text supervision artificially constrains image learning: https://arxiv.org/abs/2304.07193 https://arxiv.org/abs/2304.07193 if unified modeling isn't necessary, or even hurtful, a key advantage of transformers goes away.