3 ms·
> Spiking networks require lots of storage, but a lot, lot less compute. One way or another we need a 1000x increase in efficiency to be able to run these mode
by visarga 5y ago
> Spiking networks require lots of storage, but a lot, lot less compute.
One way or another we need a 1000x increase in efficiency to be able to run these models on edge hardware with full privacy and outside the control of the big corporations.
Funny that Gary Marcus is pleading on Twitter to get Dall-E 2 access in order to formulate his response. He isn't getting access yet. https://twitter.com/GaryMarcus/status/1513215530366234625 https://twitter.com/GaryMarcus/status/1513215530366234625
That kind of gate-keeping is possible because the costs of training and inferencing these models is too high today.
- prox 5y agoWhat’s the current problem with control here? Outside of the loop layman here.
- teruakohatu 5y agoThese transformer models are so huge, they require extremely expensive and specialist hardware beyond what enthusiasts and even many academica access to. There is no chance in the near future consumers or Edge devices will be able to run these models locally, data is going to have to be fed back into the cloud.
- prox 5y agoThanks for replying! I had no idea there were models this large. Feels a bit like going back to the mainframe age.
- robbedpeter 5y agoSmaller models with better performance are beginning to arrive. Things like RETRO, better training data, longer training time, and scale optimization will have these models on phones and desktops doing crazy things in the near future.
- teruakohatu 5y agoThey are but performance is decreased. In many cases transformers are encoding vast amounts of training data within the insane number of parameters.