4 ms·
We are still extremely early in this space. This is the mainframe era of generative machine learning. I'm expecting three big leaps in the next 10 years (in or
by axg11 5y ago
We are still extremely early in this space. This is the mainframe era of generative machine learning.
I'm expecting three big leaps in the next 10 years (in order):
1) Generative algorithms reach a level where the content is indistinguishable from human-generated content. Compute: performed on mega-clusters such as used by DALL-E, GPT-3, PaLM.
2) DALL-E/GPT-3/PaLM-level generative algorithms are able to run on personal hardware (phone, laptop)
3) Generative algorithms are able to fine-tune/train on personal hardware
Right now the algorithms are moving much faster than the hardware, hence why we are seeing large language models and gigantic generative models such as DALL-E 2. In time the hardware will catch up. For the next few years, applications of these models will be restricted by the fact that they're only accessible through API calls to mega-clusters run by Google, OpenAI, etc.
In time the hardware will improve and architectures will become more compute efficient to the point where we can achieve "human-level" (loosely defined) on personal hardware. Real-time generative content will change the way we consume content entirely, especially in the context of AR. Augmenting our view of the physical world with an infinite number of different filters will create infinite use cases for AR.
On the large language model side, once LLMs exceed our ability to comprehend libraries of documents, that will change the way we work, the way we perform science and many other things. Imagine querying your team's entire library of documents, design reviews and code with prompts such as: "Why was the load balancer for X service designed in this way?".
- visarga 5y agoThe advent of edge GPT-3 would be a huge change for robotics as well. It has been proven that augmenting robots with a language model for planning gives a leap in abilities. Maybe they could even enter any kitchen and make a sandwich (sudo make me a sandwich task). https://arxiv.org/pdf/2201.07207.pdf https://arxiv.org/pdf/2201.07207.pdf
- consumer451 5y ago> The advent of edge GPT-3 I am nowhere near the industry, so may be a dumb question, but are the new analog in-memory compute modules[0] likely to help with this in reality? How far away is that reality? [0] https://mythic.ai https://mythic.ai https://www.youtube.com/watch?v=67LXWocO9HI https://www.youtube.com/watch?v=67LXWocO9HI
- mcbuilder 5y agoI'm curious on your point 1, and I tend to disagree. The number of parameters in these large language models is increasing faster than Moore's law. Currently you need a server full of GPUs just to run inference on a PaLM model. How do you see the size shrinking so drastically? Hardware is improving on important factors like power consumption, but inference hardware needs to scale with the size of the models. Don't get me wrong, it's likely that PaLM itself can run on 2032 phone, but the real advances will be in even more scaled up models. The future of AI will be in the data center for a long time to come. Maybe after some point the models cease to scale up and that point will be where the model would even overfit on the amount of data we can possibly give it e.g. the entire internet. The PaLM authors allude to this in their conclusion
- stult 5y agoThere was just an article from deepmind on HN about this topic the other day[1], but basically IIRC it argues that all of the LLMs are horrendously compute inefficient, which means there’s a ton of room to improve them. So those models will be optimized over time just as the consumer hardware will be improved until eventually one day the two trends will converge. It’s just a question of when that will happen. [1] https://news.ycombinator.com/item?id=30987885 https://news.ycombinator.com/item?id=30987885
- axg11 5y agoCurrent LLMs are very compute inefficient. I also think retrieval transformers can bring a few orders of magnitude in efficiency improvements. Combined with architectural improvements I think we can get there.