5 ms·
Upon reading this, my immediate thought is: It's only a matter of time before we have robots powered by large models pretrained to "predict the next token" acr
by cs702 3y ago
Upon reading this, my immediate thought is:
It's only a matter of time before we have robots powered by large models pretrained to "predict the next token" across a bunch of different sensory modalities -- sight, sound, smell, touch, taste, etc. in a variety of artificial and natural settings, including social-interaction settings. Learning to read, learning to talk, learning to interact with the physical world, and so on -- all of it could very well be built upon the simple idea of learning to "predict the next token."
We live in interesting times.
- catchnear4321 3y agoacross a bunch of different gradients. senses will be the next step, humans grok that good and easy. once multiple gradients are considered, non-sensory gradients are going to be next. this is all a bunch of gobbledygook until it isn’t.
- raidicy 3y agoThese sentiments are pretty close to my own. I read a paper that claimed that llms are General Pattern Machines and could be used to complete small games in gym environments. It seems to me that if these things really are General Pattern Machines all we have to do is figure out a way to represent any data as a pattern and try and predict the next step in the pattern right? The multi token[1] project which allows you to take any type of data and turn it into a token it's pretty interesting and seems like it's going in this direction. I would really like to see a framework where you can take any modality of any type turn it into a series of tokens and just cram it into a language model and effectively turning into a multimodal model with almost no effort. [0]https://general-pattern-machines.github.io/ https://general-pattern-machines.github.io/ [1] https://github.com/sshh12/multi_token https://github.com/sshh12/multi_token
- sshh12 3y agoYeah this is exactly what I was thinking when building multi_token! Just bottlenecked so far by my own GPU, datasets, and freetime
- raidicy 3y agoThank you thank you thank you! I think it's a wonderful repo it's a shame I haven't been able to dig deeper into it. I starred your project as soon as I saw it!
- kelseyfrog 3y agoIf they're looking for a name, The Glass Bead Game isn't a bad start.
- texas2toss 3y agoProbably my favorite “smart” book of all time
- famouswaffles 3y agoOh yeah. The exciting thing is that is pretty low hanging fruit (at least for the common modalities) What would a Large Language Model that can manipulate audio-visual data as expertly as it can manipulate text look like ? This is beyond just Text to Speech or Captioning and Image Q&A. I think we'll find out very soon.
- jononor 3y agoYeah, I think this is now in the "we know it can be done, and how to do it". And also som I think we might have audio2audio editing on the level of Stable Diffusion within a year or two. Based on recent progress with AudioLM etc. For short clips, we will probably get video2video editing in the same timeframe. While it is computationally more challenging, it might end up being here before audio editing - because video is so popular in todays social media and marketing landscape. Usable joint video and audio editing (that is coherent) will probably take longer.
- gwern 3y agoThat pretty much already exists. Look at DeepMind's Gato: all tasks and modalities are simply sequences of tokens, everything from 'predict English text' to 'predict VAE image token sequences' to 'predict robotic arms commands and movements IRL'.
- jerpint 3y agoGATO was heavily biased towards tasks in a simple simulator, but didn’t exhibit emergent behaviours
- cs702 3y agoAh, yes, I'd forgotten about Gato. Thank you for reminding me. There's so much research activity that the Gato paper feels as if it was published eons ago. There's only so much I can retain in my puny little human mind at once! In any case, I'm not sure Gato qualifies as a "large" model with 1.2B parameters -- it's kinda right below the threshold at which it could or would start exhibiting emergent behaviors. Maybe a new Gato with 10's or 100's of billions of parameters operating in the physical world?
- gwern 3y agoYes. Gato was a good proof-of-concept that the Decision Transformer approach of 'just model literally everything as a sequence' scales well and doesn't exhibit some sort of catastrophic interference and can successfully imitation-learn from all the expert datasets, and a bit of transfer. But they need to push it at least another OOM or 2 to show major transfer, some emergences, and ideally do both from-scratch learning and additional learning on many new tasks. We continue to wait. :( I hope it didn't all get rolled up into Gemini and become a state secret they'll never publish on again, or lost in the shuffle in the chaos of the DeepMind/Brain merger/liquidation.
- cs702 3y ago> ...lost in the shuffle in the chaos of the DeepMind/Brain merger/liquidation That's the most likely explanation, in my view.
- ldjkfkdsjnv 3y agoYeah its going to happen. We will see and speak with intelligent machines
- jefft255 3y agoI did that a few years back: https://arxiv.org/abs/2011.11751 https://arxiv.org/abs/2011.11751
- cs702 3y agoWith a large model? How many parameters? See my other comment here: https://news.ycombinator.com/item?id=38536178 https://news.ycombinator.com/item?id=38536178
- jefft255 3y agoA couple millions IIRC. Nothing "large" compared to modern transformer models.
- deleted 3y ago[deleted]
- cs702 3y agoThanks for getting back to me. That's what I thought. The magic seems to start happening in the low billions of parameters -- and I say "seems" b/c there's no consensus as to whether it's really truly magic! In any case, it's a shame that most of the human brainpower capable of improving SotA AI doesn't have access to large-scale resources.
- dottedmag 3y agoOne can argue there are ~8bln of these robots already roaming the Earth.
- ww520 3y agoLet me guess. Someone will train a LLM on the stock prices to predict the stock market. It might work as well as how human has predicted the market.
- astrange 3y agoPast performance is not a guarantee of future results.
- Kelkonosemmel 3y agoYepp. I think a robot who can easily follow your commands should be doable in 10-20 years. I plan to buy a farm when I have the money and I'm pretty sure while I will/want to do a lot of hands on renovation and sculpting (park, etc) long term some type of robot should be good and affordable enough to take over when I'm too old.