4 ms·
But unless it can directly or indirectly support current transformer architectures (with their huge parameter counts), it's going to critically struggle with ad
by dontwearitout 3y ago
But unless it can directly or indirectly support current transformer architectures (with their huge parameter counts), it's going to critically struggle with adoption.
- bob1029 3y agoWhy can't we develop different architectures?
- dunefox 3y agoWell, show me one.
- Philpax 3y agohttps://www.rwkv.com/ https://www.rwkv.com/ and notably its semi-adaptation to SNNs https://github.com/ridgerchu/SpikeGPT https://github.com/ridgerchu/SpikeGPT
- qeternity 3y agoYou do realize that Transformers were designed around the hardware that exists. It's not as if we said wow, all you need is attention, and then Nvidia built some GPUs for it.
- dontwearitout 3y agoWe absolutely can and will in time, but for better or worse LLMs and large transformer models are highly effective and are entrenching themselves. Any competing hardware architecture that cannot port LLMs will have to do a lot of bootstrapping, and will have to have some extremely compelling reasons for people to choose it. Cutting energy costs by an order of magnitude could be enough motivation, but you'll still need to demonstrate competitive model performance.