3 ms·
Why can't we develop different architectures?
by bob1029 3y ago
Why can't we develop different architectures?
- dunefox 3y agoWell, show me one.
- Philpax 3y agohttps://www.rwkv.com/ https://www.rwkv.com/ and notably its semi-adaptation to SNNs https://github.com/ridgerchu/SpikeGPT https://github.com/ridgerchu/SpikeGPT
- qeternity 3y agoYou do realize that Transformers were designed around the hardware that exists. It's not as if we said wow, all you need is attention, and then Nvidia built some GPUs for it.
- dontwearitout 3y agoWe absolutely can and will in time, but for better or worse LLMs and large transformer models are highly effective and are entrenching themselves. Any competing hardware architecture that cannot port LLMs will have to do a lot of bootstrapping, and will have to have some extremely compelling reasons for people to choose it. Cutting energy costs by an order of magnitude could be enough motivation, but you'll still need to demonstrate competitive model performance.