3 ms·

Pressing [X] to doubt. There are many alternatives to the good old transformers: RWKV, Mamba, etc. Yet here we are, still using transformers (actually, just t

Pressing [X] to doubt. There are many alternatives to the good old transformers: RWKV, Mamba, etc. Yet here we are, still using transformers (actually, just the decoder part). Is it because the industry has so much inertia to pick up new methods? I doubt it because there's $BILLIONS in this market and everyone wants a piece of the AI cake, so it doesn't make sense to ignore promising methods. Why, then, we barely see any non-transformer production-ready LLM these days?

solidasparagus 2y ago
It’s going to take time. I can’t speak to the actual quality of mamba other than to say the authors are extraordinary and should be taken seriously. But training a large model requires a huge amount of capital so the biggest runs are designed around risk minimization. And remember, many of the decision makers of these runs made are in their positions by doing transformer-centric work. The true value of mamba is still unclear to me with very long context techniques being effective for transformers.
- inciampati 2y ago
  To be frank, the long-context techniques that you're describing are still extremely limited in the length of context to consider, only a million-token order, and extremely expensive to apply.
  - imtringued 2y ago
    And yet quadratic attention is still unavoidable. Anything that lets you have sub quadratic attention is going to have an accuracy Vs performance tradeoff.
    - inciampati 2y ago
      No, you can make multiple passes across the data set. There's no reason that you have to use quadratic attention. I point to the shape of our own brains, which are limited in size. Yet, through multiple passes over information and random access, which we guide, we're able to process very large systems.
Buttons840 2y ago
I believe the attention mechanism we use now was introduced in 2014 by Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio in their paper titled "Neural Machine Translation by Jointly Learning to Align and Translate." 2014. It took almost a decade for the potential of this technique to be realized and come to the attention (heh) of most developers. I don't know what researchers are doing with Mamba and RWKV, but we should let them cook.
hansvm 2y ago
> I doubt it because there's $BILLIONS in this market and everyone wants a piece of the AI cake, so it doesn't make sense to ignore promising methods. I also doubt this result. The "why have $BILLIONS not already invested" question is interesting in its own right though. Generally, the literature on the theoretical bounds of swarm optimization is pertinent. Those $BILLIONS aren't being invested by a single omniscient entity, so they're subject to interesting constraints. As one of many explanations, fragmentation is common. If $BILLIONS are split between greedy, mostly non-interacting entities (e.g., competing companies each trying to replace the transformer in a bounded number of hours and dollars while securing their market dominance), you expect, probabilistically, for each of them to converge on the same strateg(y/ies), especially if the "best" alternatives are obvious or globally known for some reason (e.g., some solutions intuitively feel "natural" or your researchers publish early results or you have employee movement between companies or whatever). Riskier strategies won't be touched, and you'll have $BILLIONS spent duplicating the same most likely alternatives when $MILLIONS would have sufficed. The normal counterpoint is that a few big players dominate the spending, and they would have higher internal coordination. Interestingly though, they don't usually, except when that coordination would tend to enforce the same strategies smaller competition are pursuing. How often do you hear about stories like the misaligned Google+ integrations resulting in employee bonuses for poor customer experiences vs a forward-thinking executive actively devoting funds to a meaningful number of competing solutions? Approximately never. It's career suicide if you fail and depend on other people for your position, you _are_ actually more likely to outdo the competition with your increased resources if you just lean into the "best" alternatives, and for a whole host of reasons very few executives (except for people with real power) will coordinate a more comprehensive strategy, certainly not one orthogonal to the competition's just for the sake of allocating the global $BILLIONS more efficiently. Separately (going back to the merits of the preprint), I'll probably read the full thing later, but a few points stuck out as suspicious on an initial skim. Notably, they seem to mix linear transformations in different domains. E.g., `xa` is linear in both `x` and `a`, and `vx` is linear in both `v` and `x`, but `xax` is _not_ linear in `x`, even if you try to "prove" that idea with `v = xa`. Linearity in `v` isn't enough to make the composition linear in `x`. A lot of their results seem to rely on eliminating those "redundant" computations, even though the things they're replacing with linear computations are actually higher order polynomials. On an initial skim, the other "novel" ideas also don't seem well grounded. Their experimental results are decent. That could mean a lot of things (normally that the authors made more errors in their competitors' than in their own work), but it's probably worth looking into for a few hours despite my other complaints.
marcinzm 2y ago
> Why, then, we barely see any non-transformer production-ready LLM these days? Because having a 5% better non-transformer model doesn't help you if as a result you can't use the 10% improvements people publish that only apply to transformers. Very quickly you'll be 5% worse than those who stuck with transformers, and have wasted a ton of time and money.

solidasparagus 2y ago

It’s going to take time. I can’t speak to the actual quality of mamba other than to say the authors are extraordinary and should be taken seriously. But training a large model requires a huge amount of capital so the biggest runs are designed around risk minimization. And remember, many of the decision makers of these runs made are in their positions by doing transformer-centric work. The true value of mamba is still unclear to me with very long context techniques being effective for transformers.

inciampati 2y ago
To be frank, the long-context techniques that you're describing are still extremely limited in the length of context to consider, only a million-token order, and extremely expensive to apply.
- imtringued 2y ago
  And yet quadratic attention is still unavoidable. Anything that lets you have sub quadratic attention is going to have an accuracy Vs performance tradeoff.
  - inciampati 2y ago
    No, you can make multiple passes across the data set. There's no reason that you have to use quadratic attention. I point to the shape of our own brains, which are limited in size. Yet, through multiple passes over information and random access, which we guide, we're able to process very large systems.

Buttons840 2y ago

I believe the attention mechanism we use now was introduced in 2014 by Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio in their paper titled "Neural Machine Translation by Jointly Learning to Align and Translate." 2014. It took almost a decade for the potential of this technique to be realized and come to the attention (heh) of most developers. I don't know what researchers are doing with Mamba and RWKV, but we should let them cook.

hansvm 2y ago

> I doubt it because there's $BILLIONS in this market and everyone wants a piece of the AI cake, so it doesn't make sense to ignore promising methods. I also doubt this result. The "why have $BILLIONS not already invested" question is interesting in its own right though. Generally, the literature on the theoretical bounds of swarm optimization is pertinent. Those $BILLIONS aren't being invested by a single omniscient entity, so they're subject to interesting constraints. As one of many explanations, fragmentation is common. If $BILLIONS are split between greedy, mostly non-interacting entities (e.g., competing companies each trying to replace the transformer in a bounded number of hours and dollars while securing their market dominance), you expect, probabilistically, for each of them to converge on the same strateg(y/ies), especially if the "best" alternatives are obvious or globally known for some reason (e.g., some solutions intuitively feel "natural" or your researchers publish early results or you have employee movement between companies or whatever). Riskier strategies won't be touched, and you'll have $BILLIONS spent duplicating the same most likely alternatives when $MILLIONS would have sufficed. The normal counterpoint is that a few big players dominate the spending, and they would have higher internal coordination. Interestingly though, they don't usually, except when that coordination would tend to enforce the same strategies smaller competition are pursuing. How often do you hear about stories like the misaligned Google+ integrations resulting in employee bonuses for poor customer experiences vs a forward-thinking executive actively devoting funds to a meaningful number of competing solutions? Approximately never. It's career suicide if you fail and depend on other people for your position, you _are_ actually more likely to outdo the competition with your increased resources if you just lean into the "best" alternatives, and for a whole host of reasons very few executives (except for people with real power) will coordinate a more comprehensive strategy, certainly not one orthogonal to the competition's just for the sake of allocating the global $BILLIONS more efficiently. Separately (going back to the merits of the preprint), I'll probably read the full thing later, but a few points stuck out as suspicious on an initial skim. Notably, they seem to mix linear transformations in different domains. E.g., `xa` is linear in both `x` and `a`, and `vx` is linear in both `v` and `x`, but `xax` is _not_ linear in `x`, even if you try to "prove" that idea with `v = xa`. Linearity in `v` isn't enough to make the composition linear in `x`. A lot of their results seem to rely on eliminating those "redundant" computations, even though the things they're replacing with linear computations are actually higher order polynomials. On an initial skim, the other "novel" ideas also don't seem well grounded. Their experimental results are decent. That could mean a lot of things (normally that the authors made more errors in their competitors' than in their own work), but it's probably worth looking into for a few hours despite my other complaints.

marcinzm 2y ago

> Why, then, we barely see any non-transformer production-ready LLM these days? Because having a 5% better non-transformer model doesn't help you if as a result you can't use the 10% improvements people publish that only apply to transformers. Very quickly you'll be 5% worse than those who stuck with transformers, and have wasted a ton of time and money.