5 ms·

Wait, are you saying SoTA NN research hasnt evolved from hardcoding a bunch of layer structures and sizes? I'm kind of shocked. I thought there would be more d

Wait, are you saying SoTA NN research hasnt evolved from hardcoding a bunch of layer structures and sizes? I'm kind of shocked. I thought there would be more dynamism by now and I stopped dabbling in like 2018.

astrange 2y ago
The innovation is that everything is just one standardized structure now (transformer models) and you make it bigger if you feel like you need that. There's still some room for experimenting if you care about memory/power efficiency, like MoE models, but they're not as well understood yet.
- aDyslecticCrow 2y ago
  There are too many papers throwing transformers on everything without thinking. Transformers are amazing for language but kinda mid on everything else. CS researchers tend to jump on trends really hard, so it will probably go back to normal again soon.
  - imtringued 2y ago
    I don't know what you mean by amazing for language. Almost everything is built on transformers nowadays. Image segmentation uses transformers. Text to speech uses transformers. Voice recognition uses transformers. There are robotics transformers that take image inputs and output motion sequences. Transformers are inherently multi-modal. They handle whatever you throw at them, it's just that language tends to be a very common input or output.
  - Hugsun 2y ago
    That is not true. Transformers are being applied all over because they work better than what was used before in so many cases.
pshc 2y ago
My wild guess is that adjusting the shape before each step is not worth the speed hit. Uniform structures make GPUs go brrrrr
- astrange 2y ago
  It's also easier to train and in particular easier to parallelize.
delusional 2y ago
The innovation is the amount of resources people are willing to spend right now. From looking at the research code it's clear that the whole field is basically doing a (somewhat) guided search in the entire space of possible layer permutations. There seems to be no rhyme or reason, no scientific insight, no analysis. They just try a million different permutations, and whatever scores the highest on the benchmarks gets published.
- moffkalast 2y ago
  Well it took evolution 4 billion years of testing out random permutations that resulted in a pretty good local maximum, so there is hope for us yet.
  - greenchair 2y ago
    [flagged]
    - stefs 2y ago
      Or maybe a watch made by a blind guy instead of a bridge?
      - moffkalast 2y ago
        Would that make it more of a hear? Since obviously the guy can't watch.
        ben_w 2y ago
        I can't see the flagged comment, but I'm fairly sure @stefs is making a reference to this: https://en.wikipedia.org/wiki/The_Blind_Watchmaker https://en.wikipedia.org/wiki/The_Blind_Watchmaker
        stefs 2y ago
        yes. the flagged comment wanted to sell moffkalast a bridge, w/o further explanation. i interpreted it as him saying that the human brain isn't governed by evolution and calling moffkalast naive.
curious_cat_163 2y ago
There is a tick-tock between searching the dominant NN architectures (tick) and optimizing for accuracy, compute and inference latency and throughput (tock). This particular (tock) is still playing out. The next (tick) does not feel imminent and will likely depend on when we discover the limits of the transformers when it comes to solving for long tail of use-cases. My $0.02.
- rdedev 2y ago
  My wish is they would move on to the next phase. The whole deal with SSMs look really good. But looking for better architects is countered with "a regular architecture with more parameters are doing better. What's the point of this"
  - tysam_and 2y ago
    Heyo! Have been doing this for a while. SSMs certainly are flashy (most popular topics-of-the-year are), and it would be nice to see if they hit a point of competitive performance with transformers (and if they stand the test of time!) There are certainly tradeoffs to both, the general transformer motif scales very well on a number of axis, so that may be the dominant algorithm for a while to come, though almost certainly it will change and evolve as time goes along (and who knows? something else may come along as well <3 :')))) ).
  - throwawaymaths 2y ago
    There's something about a transformer being at its core based on a differentiable hash table data structure that makes them special. I think it's dominance is not going to substantially change any time soon. Dont you know, the solution to all leetcode interviews is a hash table?
  - curious_cat_163 2y ago
    IMO, SSMs are an optimization. They don't represent enough of a fundamental departure from the kinds of things Transformers can _do_. So, while I like the idea of saving on the energy costs, I speculate that such saving can be obtained with other optimizations while staying with transformer blocks. Hence, the motivation to change is a bit of an uphill here. I would love to hear counter-arguments to this view. :) Furthermore, I think a replacement will require that we _understand_ what the current crop of models are doing mechanically. Some of it was motivated in [1]. [1] https://openaipublic.blob.core.windows.net/neuron-explainer/paper/index.html#sec-discussion https://openaipublic.blob.core.windows.net/neuron-explainer/...
dauertewigkeit 2y ago
There are things like NAS (neural architectural search) but all you are doing is just growing the search space and making the optimization problem much harder. Typically you do the architectural optimization by hand, using heuristics and past experiments as guidance.
Mehdi2277 2y ago
I’ve occasionally worked with more dynamic models (tree structured decoding). They are generally not a good fit for trying to max gpu thoroughput. A lot of magic of transformers and large language models is about pushing gpu as much we can and simpler static model architecture that trains faster can train on much more data. So until the hardware allows for comparable (say with 2-4x) thoroughput of samples per second I expect model architecture to mostly be static for most effective models and dynamic architectures to be an interesting side area.
aDyslecticCrow 2y ago
The only thing that has changed since 2018 is the most popular network structure to play with. The code looks the same as always; python notebooks where someone manually calculated the size of each hard-coded layer to make it fit.
- galaxyLogic 2y ago
  > someone manually calculated the size of each hard-coded layer I wonder shouldn't AI be the best tool to optimize itself?
  - octonion137 2y ago
    In theory yes, but unfortunately AI hasn't been invented yet
    - psychoslave 2y ago
      I don't know, shouldn't the AI then be trapped at evaluating all possible AI implementations? And since it will face the halting problem, it won't discriminate the very best one, though it will probably be able to return the best one given a capped amount of resources that is reachable through exhaustion in its space. It won't necessarily be better than what can be provided by human beings given an equivalent amount of resources.
danielmarkbruce 2y ago
People would love to have dynamism. It's a cost thing.

astrange 2y ago

The innovation is that everything is just one standardized structure now (transformer models) and you make it bigger if you feel like you need that. There's still some room for experimenting if you care about memory/power efficiency, like MoE models, but they're not as well understood yet.

aDyslecticCrow 2y ago
There are too many papers throwing transformers on everything without thinking. Transformers are amazing for language but kinda mid on everything else. CS researchers tend to jump on trends really hard, so it will probably go back to normal again soon.
- imtringued 2y ago
  I don't know what you mean by amazing for language. Almost everything is built on transformers nowadays. Image segmentation uses transformers. Text to speech uses transformers. Voice recognition uses transformers. There are robotics transformers that take image inputs and output motion sequences. Transformers are inherently multi-modal. They handle whatever you throw at them, it's just that language tends to be a very common input or output.
- Hugsun 2y ago
  That is not true. Transformers are being applied all over because they work better than what was used before in so many cases.

pshc 2y ago

My wild guess is that adjusting the shape before each step is not worth the speed hit. Uniform structures make GPUs go brrrrr

astrange 2y ago
It's also easier to train and in particular easier to parallelize.

delusional 2y ago

The innovation is the amount of resources people are willing to spend right now. From looking at the research code it's clear that the whole field is basically doing a (somewhat) guided search in the entire space of possible layer permutations. There seems to be no rhyme or reason, no scientific insight, no analysis. They just try a million different permutations, and whatever scores the highest on the benchmarks gets published.

moffkalast 2y ago
Well it took evolution 4 billion years of testing out random permutations that resulted in a pretty good local maximum, so there is hope for us yet.
- greenchair 2y ago
  [flagged]
  - stefs 2y ago
    Or maybe a watch made by a blind guy instead of a bridge?
    - moffkalast 2y ago
      Would that make it more of a hear? Since obviously the guy can't watch.
      - ben_w 2y ago
        I can't see the flagged comment, but I'm fairly sure @stefs is making a reference to this: https://en.wikipedia.org/wiki/The_Blind_Watchmaker https://en.wikipedia.org/wiki/The_Blind_Watchmaker
        stefs 2y ago
        yes. the flagged comment wanted to sell moffkalast a bridge, w/o further explanation. i interpreted it as him saying that the human brain isn't governed by evolution and calling moffkalast naive.

curious_cat_163 2y ago

There is a tick-tock between searching the dominant NN architectures (tick) and optimizing for accuracy, compute and inference latency and throughput (tock). This particular (tock) is still playing out. The next (tick) does not feel imminent and will likely depend on when we discover the limits of the transformers when it comes to solving for long tail of use-cases. My $0.02.

rdedev 2y ago
My wish is they would move on to the next phase. The whole deal with SSMs look really good. But looking for better architects is countered with "a regular architecture with more parameters are doing better. What's the point of this"
- tysam_and 2y ago
  Heyo! Have been doing this for a while. SSMs certainly are flashy (most popular topics-of-the-year are), and it would be nice to see if they hit a point of competitive performance with transformers (and if they stand the test of time!) There are certainly tradeoffs to both, the general transformer motif scales very well on a number of axis, so that may be the dominant algorithm for a while to come, though almost certainly it will change and evolve as time goes along (and who knows? something else may come along as well <3 :')))) ).
- throwawaymaths 2y ago
  There's something about a transformer being at its core based on a differentiable hash table data structure that makes them special. I think it's dominance is not going to substantially change any time soon. Dont you know, the solution to all leetcode interviews is a hash table?
- curious_cat_163 2y ago
  IMO, SSMs are an optimization. They don't represent enough of a fundamental departure from the kinds of things Transformers can _do_. So, while I like the idea of saving on the energy costs, I speculate that such saving can be obtained with other optimizations while staying with transformer blocks. Hence, the motivation to change is a bit of an uphill here. I would love to hear counter-arguments to this view. :) Furthermore, I think a replacement will require that we _understand_ what the current crop of models are doing mechanically. Some of it was motivated in [1]. [1] https://openaipublic.blob.core.windows.net/neuron-explainer/paper/index.html#sec-discussion https://openaipublic.blob.core.windows.net/neuron-explainer/...

dauertewigkeit 2y ago

There are things like NAS (neural architectural search) but all you are doing is just growing the search space and making the optimization problem much harder. Typically you do the architectural optimization by hand, using heuristics and past experiments as guidance.

Mehdi2277 2y ago

I’ve occasionally worked with more dynamic models (tree structured decoding). They are generally not a good fit for trying to max gpu thoroughput. A lot of magic of transformers and large language models is about pushing gpu as much we can and simpler static model architecture that trains faster can train on much more data. So until the hardware allows for comparable (say with 2-4x) thoroughput of samples per second I expect model architecture to mostly be static for most effective models and dynamic architectures to be an interesting side area.

aDyslecticCrow 2y ago

The only thing that has changed since 2018 is the most popular network structure to play with. The code looks the same as always; python notebooks where someone manually calculated the size of each hard-coded layer to make it fit.

galaxyLogic 2y ago
> someone manually calculated the size of each hard-coded layer I wonder shouldn't AI be the best tool to optimize itself?
- octonion137 2y ago
  In theory yes, but unfortunately AI hasn't been invented yet
  - psychoslave 2y ago
    I don't know, shouldn't the AI then be trapped at evaluating all possible AI implementations? And since it will face the halting problem, it won't discriminate the very best one, though it will probably be able to return the best one given a capped amount of resources that is reachable through exhaustion in its space. It won't necessarily be better than what can be provided by human beings given an equivalent amount of resources.

danielmarkbruce 2y ago

People would love to have dynamism. It's a cost thing.