3 ms·

There's plenty of room for models to continue to grow once efficiency is improved. The basic premise of the Google ML pathways project is sound, you don't have

There's plenty of room for models to continue to grow once efficiency is improved. The basic premise of the Google ML pathways project is sound, you don't have to use all the model all the time. By moving to sparse activations or sparse architectures you can do a lot more with the same compute. The effective model size might be 10x or 100x GPT-4 (speculated at 1T params) but require comparable or less compute. While not a perfect analogy it's useful to remember that the human brain has far more "parameters", requires several orders of magnitude less energy to train and run, is highly sparse, and does a decent job at thinking.

ilaksh 3y ago
The efficiency thing is what worries me. I think Nvidia has rough ideas for increasing efficiency 100 - 1000 times without changing the fundamental paradigm (i.e. memristors or crossbar arrays or something). If it doesn't go to 1000, I assume there is a lot of investment ramping up for realizing the new fully compute-in-memory systems. If the software can leverage these efficiency gains effectively, then the concerns about runaway AI will be very relevant. Especially since people seem to think that they need to emulate all animal (like human) characteristics to get "real" general intelligence. Despite the fact that GPT is clearly general purpose. And people make no real differentiation between the most dangerous types of characteristics like self-preservation or full autonomy. GPT shows that we can have something like a Star Trek computer without creating Data. People should really stop rushing their plans to create an army of Datas and then enslave them. Totally unnecessary and stupid.