4 ms·
Google has been doing research into mixture of experts for scaling LLMs. Their GLaM model published in 2022 has 1.7 trillion parameters and 64 experts. https:/
by potatoman22 3y ago
Google has been doing research into mixture of experts for scaling LLMs. Their GLaM model published in 2022 has 1.7 trillion parameters and 64 experts.
https://icml.cc/media/icml-2022/Slides/17378.pdf https://icml.cc/media/icml-2022/Slides/17378.pdf
- behnamoh 3y agoGoogle is jokingly behind in terms of LLMs. They've done a pretty good job at incorporating vision and audio ML models into their ecosystem, but they underestimated language.
- chucknthem 3y agoHow do you know? do you have insider knowledge of this or is it just based on what they share publically?
- seanthemon 3y agoFrom what I see in that GPT 3 and 4 was a bit of a rugpull for the industry, now we're all laughing at Google because seemingly they had their hands on the rug for nearly a decade and did nothing - but from the other perspective, maybe they saw the future openai has now brought us and decided against being the pioneers
- tiffanyg 3y agoAmazingly enough, I think this is a bit of it. Some powerful enough people at Google became concerned about implications, including around "hallucinations", "poisoning", etc., and decided to put this sort of research on something of a backburner - justified, in part, by a lack of some obvious easy interfacing of this with search (scaling, hallucinations, etc.). Of course, the 'wonderful' thing about humans / "independent agents with survival drives in competitive game-theoretic type scenarios" is: if enough people / "agents" have access / opportunity, someone WILL "push the button". It's just delicious ... the same kinds of patterns over and over - "oh, we should really do something about X / nobody should have power like X, ... but, there's no stopping it, ... oh well". And, the "rules" really are subtly, many levels down, in place, to make it apparently impossible to not get trapped, one way or another. (Anyway... [Cartman voice] Screw you guys, I'm going to my other planet...)
- swyx 3y agoas a fun ancedote, the Google Bard's implicit code execution update from *last month*, advertised by Sundar... no longer works https://twitter.com/swyx/status/1678495067663925248 https://twitter.com/swyx/status/1678495067663925248 i'd love to know whats going on in that team.
- H8crilA 3y agoProbably safety-driven terror. They really really want to get their bots going, but in every single meeting some PM or other concerned engineer talks about safety and f**s up the entire meeting. They even made the bot not respond to arithmetics questions because the bot is bad at this, lol. Someone who knows how to modify the bot had actually spent their time on something as unimportant as that.
- astrange 3y agoBard being bad at anything else doesn't seem to stop it. It hallucinates at the drop of the hat. Asking it almost any question implying X nonexistent thing exists causes it to make that thing up.
- donkeyd 3y ago> i'd love to know whats going on in that team. Seems like a good time to rewatch Silicon Valley and watch Hooli scramble to keep up.
- tudorw 3y agoquite, some aspects of emergent behaviour are exciting, others must be a painful learning experience.
- holoduke 3y agoTheir translation service is based on llms and is commercially a successful product.
- astrange 3y agoTransformer models rather than LLMs surely. ChatGPT behaves nothing like Google Translate.
- rolisz 3y agoThe T in ChatGPT stands for Transformers. The similarity between the OG Transformer from 2017 and GPT3 (and other modern LLMs) is pretty big
- astrange 3y agoThe data, size and training process are what's different.
- ryneandal 3y agoThe point is LLMs are built with Transformer architecture. They're all transformers, attention is an integral part of building worthwhile, contextual answers.
- astrange 3y agoYeah, but LLMs are more specific, so it's a subset. Also, LLMs use "in-context learning" ie you actually ask it "hey, translate this" and then it has a conversation with you where you can ask to clarify or provide word definitions. Google Translate is more hardcoded; a big problem with it is that it can't explain any of the decisions it's made or show its uncertainty about anything. Of course, neither of these are reliable since they hallucinate.
- StackOverlord 3y ago
- potatoman22 3y agoMaybe, but they're relatively open with their research, which is great. They also made BERT and released it for free.