4 ms·

I wonder if the transformer will remain the de facto arch for LLMs in a couple years. We already have Mamba, RWKV, etc. which are huge improvements over transfo

I wonder if the transformer will remain the de facto arch for LLMs in a couple years. We already have Mamba, RWKV, etc. which are huge improvements over transformer in terms of memory requirement and speed. I wonder why there's still so much work (and hacks and workarounds) on transformer to make it go faster and consume less VRAM whereas the same effort could be spent on other architectures that solve some of those problems fundamentally.

uoaei 3y ago
> why there's still so much work (and hacks and workarounds) on transformer to make it go faster and consume less VRAM whereas the same effort could be spent on other architectures Probably because the pace of development is now no longer determined by ML researchers but by CS engineers. ML is "easy to pick up" until you have to understand the reasons certain architectures work or don't for different applications.
cyanydeez 3y ago
technical debt
__loam 3y ago
The best technology doesn't necessarily win. First to market matters, especially when models on the scale of gpt-4 cost 100mln to train.
- evilantnie 3y ago
  "best" is subjective, but I do think the industry will eventually converge onto an architecture that is significantly more cost effective than current state of the art. Regardless of who is first to market, everyone is incentivized to continue down this path with their research on improving LLM performance.
- realusername 3y ago
  While the first mover advantage exists, I don't think it's a sufficient moat in the long term. OpenAI's moat is the high cost of training as you mentioned but this might be obsolete in a few papers down the road. They themselves realized this by trying to turn it into a platform but I don't think it's enough.
andy99 3y ago
The speed of transformers is memory bandwidth limited. I think that it's possible to speed them way up with different chip architectures, and I know there are lots of people working on this e.g. https://etched.ai https://etched.ai making asics and https://untether.ai https://untether.ai making chips will co-located memory and compute cells. So unless some architecture really starts beating transformers badly on language tasks, I think the speed problem is going to disappear as silicon architectures adapt.
- pavelstoev 3y ago
  There is still so much performance to gain by applying software-based optimizations for superior performance of transformer LLMs on GPUs. From ~ 30% GPU utilization to over 90% is possible. I wrote a white paper on this, page 5 and 6: http://tinyurl.com/uujr3z4f http://tinyurl.com/uujr3z4f
- tome 3y ago
  Yup, the company I work for has the world's fastest LLM appliance, based on a hardware design co-locating memory and compute. It requires compilation innovation as well in a sort of hardware-software codesign. We don't see transformer models as being an impediment in terms of memory requirements or speed in the near future. [I would be happy to say more but I just got a top-level comment flag killed, presumably because they though I was advertising, so I won't mention the company name.]
  - tome 3y ago
    Link to HN discussion about the appliance: https://news.ycombinator.com/item?id=38739199 https://news.ycombinator.com/item?id=38739199
  - andy99 3y ago
    Are you guys hiring? I'm actually looking for a job in this industry.
    - tome 3y ago
      Yes we are! (I put links in my profile because I'm worried if link here or mention the name I'll be flag killed again.)
beoberha 3y ago
There’s hacks and workarounds because billions of dollars have been poured into transformers by the biggest tech companies in the world. Your assumption that the same effort could be given to MAMBA or RWKV isn’t necessarily true. I’m sure there’s research arms looking to see if they scale to the level OpenAI or Google have their transformers at now, but they’re still very much in their infancy. This all ignores the risk of them not panning out and the lost time not focusing some energy on transformers.
taneq 3y ago
The current best-in-class performance is from transformer models so of course research is focusing on this. You're creating a false dichotomy between "work on transformers" and "work on something new and fundamentally better." All of the current work on optimizing transformer architectures is exploring and better understanding the space, and so it IS leading us to better architectures. This is the most fertile ground for new insights that will outperform the current approaches. New technologies aren't generally just created out of whole cloth by some genius, they're build up of layers of incremental improvements, each of which is modest in its own right but which when taken together are groundbreaking.
logicchains 3y ago
Theoretically transformers are fundamentally more powerful than stuff like Mamba and RWKV because they don't "forget"; the newest token can attend perfectly to the oldest token if it wants. Mamba and RWKV on the other hand compress the old state, based on information available at the time of compression. If it turns out the new token wants some information from the old token that during the compression was excluded due to the compression thinking it wouldn't be needed, the new token has no way to access it.
- nickpsecurity 3y ago
  I didn’t know those alternatives had those weaknesses. I did have a whole list of alternatives to check out. Having already looked at some, do you know any transformer alternatives that don’t forget things but were still sub-quadratic? What rabbit holes are worth digging deeper into?
  - deepnotderp 3y ago
    Doesn't sub-quadratic attention fundamentally imply forgetting things?
- evnc 3y ago
  This is fair -- the newest token can attend perfectly to the oldest token, within the context window. but also, on a broader scale, if a transformer model is presented with a long input that does not fit in its context (e.g.: you are building a chatbot, and you have a very long chat history), it must "compress" or "forget" some of that information (e.g.: repeatedly summarizing historical messages, dropping them and appending the summary at the beginning of the input). Mamba/RWKV/other "recurrent" architectures, can theoretically operate on unbounded input lengths; they "forget" information from earlier tokens over time, but is that not comparable to what a transformer must do with input lengths greater than their context window?
- D0TheMath 3y ago
  The jury very much still seems out on this. Computationally speaking, I believe Mamba is Turing complete, while transformers aren't (they can't do loops), so technically Mamba is more expressive. But of course, the question is always whether it ends up with lower loss.

uoaei 3y ago

> why there's still so much work (and hacks and workarounds) on transformer to make it go faster and consume less VRAM whereas the same effort could be spent on other architectures Probably because the pace of development is now no longer determined by ML researchers but by CS engineers. ML is "easy to pick up" until you have to understand the reasons certain architectures work or don't for different applications.

cyanydeez 3y ago

technical debt

__loam 3y ago

The best technology doesn't necessarily win. First to market matters, especially when models on the scale of gpt-4 cost 100mln to train.

evilantnie 3y ago
"best" is subjective, but I do think the industry will eventually converge onto an architecture that is significantly more cost effective than current state of the art. Regardless of who is first to market, everyone is incentivized to continue down this path with their research on improving LLM performance.
realusername 3y ago
While the first mover advantage exists, I don't think it's a sufficient moat in the long term. OpenAI's moat is the high cost of training as you mentioned but this might be obsolete in a few papers down the road. They themselves realized this by trying to turn it into a platform but I don't think it's enough.

andy99 3y ago

The speed of transformers is memory bandwidth limited. I think that it's possible to speed them way up with different chip architectures, and I know there are lots of people working on this e.g. https://etched.ai https://etched.ai making asics and https://untether.ai https://untether.ai making chips will co-located memory and compute cells. So unless some architecture really starts beating transformers badly on language tasks, I think the speed problem is going to disappear as silicon architectures adapt.

pavelstoev 3y ago
There is still so much performance to gain by applying software-based optimizations for superior performance of transformer LLMs on GPUs. From ~ 30% GPU utilization to over 90% is possible. I wrote a white paper on this, page 5 and 6: http://tinyurl.com/uujr3z4f http://tinyurl.com/uujr3z4f
tome 3y ago
Yup, the company I work for has the world's fastest LLM appliance, based on a hardware design co-locating memory and compute. It requires compilation innovation as well in a sort of hardware-software codesign. We don't see transformer models as being an impediment in terms of memory requirements or speed in the near future. [I would be happy to say more but I just got a top-level comment flag killed, presumably because they though I was advertising, so I won't mention the company name.]
- tome 3y ago
  Link to HN discussion about the appliance: https://news.ycombinator.com/item?id=38739199 https://news.ycombinator.com/item?id=38739199
- andy99 3y ago
  Are you guys hiring? I'm actually looking for a job in this industry.
  - tome 3y ago
    Yes we are! (I put links in my profile because I'm worried if link here or mention the name I'll be flag killed again.)

beoberha 3y ago

There’s hacks and workarounds because billions of dollars have been poured into transformers by the biggest tech companies in the world. Your assumption that the same effort could be given to MAMBA or RWKV isn’t necessarily true. I’m sure there’s research arms looking to see if they scale to the level OpenAI or Google have their transformers at now, but they’re still very much in their infancy. This all ignores the risk of them not panning out and the lost time not focusing some energy on transformers.

taneq 3y ago

The current best-in-class performance is from transformer models so of course research is focusing on this. You're creating a false dichotomy between "work on transformers" and "work on something new and fundamentally better." All of the current work on optimizing transformer architectures is exploring and better understanding the space, and so it IS leading us to better architectures. This is the most fertile ground for new insights that will outperform the current approaches. New technologies aren't generally just created out of whole cloth by some genius, they're build up of layers of incremental improvements, each of which is modest in its own right but which when taken together are groundbreaking.

logicchains 3y ago

Theoretically transformers are fundamentally more powerful than stuff like Mamba and RWKV because they don't "forget"; the newest token can attend perfectly to the oldest token if it wants. Mamba and RWKV on the other hand compress the old state, based on information available at the time of compression. If it turns out the new token wants some information from the old token that during the compression was excluded due to the compression thinking it wouldn't be needed, the new token has no way to access it.

nickpsecurity 3y ago
I didn’t know those alternatives had those weaknesses. I did have a whole list of alternatives to check out. Having already looked at some, do you know any transformer alternatives that don’t forget things but were still sub-quadratic? What rabbit holes are worth digging deeper into?
- deepnotderp 3y ago
  Doesn't sub-quadratic attention fundamentally imply forgetting things?
evnc 3y ago
This is fair -- the newest token can attend perfectly to the oldest token, within the context window. but also, on a broader scale, if a transformer model is presented with a long input that does not fit in its context (e.g.: you are building a chatbot, and you have a very long chat history), it must "compress" or "forget" some of that information (e.g.: repeatedly summarizing historical messages, dropping them and appending the summary at the beginning of the input). Mamba/RWKV/other "recurrent" architectures, can theoretically operate on unbounded input lengths; they "forget" information from earlier tokens over time, but is that not comparable to what a transformer must do with input lengths greater than their context window?
D0TheMath 3y ago
The jury very much still seems out on this. Computationally speaking, I believe Mamba is Turing complete, while transformers aren't (they can't do loops), so technically Mamba is more expressive. But of course, the question is always whether it ends up with lower loss.