3 ms·
"can model up to 17 million parameters" So about about 1000x away from being useful? Most current models are billions not millions.
by jacknews 3y ago
"can model up to 17 million parameters"
So about about 1000x away from being useful? Most current models are billions not millions.
- bob1029 3y agoNeuromorphic hardware is more about action potentials per unit time than raw parameter count.
- samstave 3y agoELI5 Please
- bob1029 3y agoI'd start with the wikipedia article on spiking neural networks.
- dontwearitout 3y agoBut unless it can directly or indirectly support current transformer architectures (with their huge parameter counts), it's going to critically struggle with adoption.
- bob1029 3y agoWhy can't we develop different architectures?
- dunefox 3y agoWell, show me one.
- Philpax 3y agohttps://www.rwkv.com/ https://www.rwkv.com/ and notably its semi-adaptation to SNNs https://github.com/ridgerchu/SpikeGPT https://github.com/ridgerchu/SpikeGPT
- qeternity 3y agoYou do realize that Transformers were designed around the hardware that exists. It's not as if we said wow, all you need is attention, and then Nvidia built some GPUs for it.
- dontwearitout 3y agoWe absolutely can and will in time, but for better or worse LLMs and large transformer models are highly effective and are entrenching themselves. Any competing hardware architecture that cannot port LLMs will have to do a lot of bootstrapping, and will have to have some extremely compelling reasons for people to choose it. Cutting energy costs by an order of magnitude could be enough motivation, but you'll still need to demonstrate competitive model performance.