4 ms·
weird title, note that the tweet said "so yes, GPT4 is *technically* 10x the size of GPT3, and all the small circle big circle memes from January were actually.
by andreyk 3y ago
weird title, note that the tweet said "so yes, GPT4 is *technically* 10x the size of GPT3, and all the small circle big circle memes from January were actually... in the ballpark?"
It's really 8 models that are 220B, which is not the same as one model that is 1.7T params. There have been 1T+ models via mixtures of experts for a while now.
Note also the follow up tweet: "since MoE is So Hot Right Now, GLaM might be the paper to pay attention to. Google already has a 1.2T model with 64 experts, while Microsoft Bing’s modes are different mixes accordingly"
There is also this linked tweet https://twitter.com/LiamFedus/status/1536791574612303872 https://twitter.com/LiamFedus/status/1536791574612303872 -
"They are all related to Switch-Transformers and MoE. Of the 3 people on Twitter, 2 joined OpenAI. Could be related, could be unrelated"
Which links to this tweet:
"Today we're releasing all Switch Transformer models in T5X/JAX, including the 1.6T param Switch-C and the 395B param Switch-XXL models. Pleased to have these open-sourced!"
Anyway... remember not to just read the headlines, they can be misleading.
- ignoramous 3y agoMore Efficient In-Context Learning with Generalist Language Model (GLaM) on the Google blog (2021): https://archive.is/cuyW0 https://archive.is/cuyW0 Btw, MoE is one of the 5 other ways Google thinks LLMs can be more efficient: https://archive.is/2XMvh https://archive.is/2XMvh
- swyx 3y ago(OP here) - yeah i know, but i also know how AI twitter works so I put both the headline and the caveats. i always hope to elevate the level of discourse by raising the relevant facts to those at my level/a little bit behind me in terms of understanding. think theres always a fine balance between getting deep/technical/precise and getting attention and you have to thread the needle in a way that feels authentic to you to do this "job" ultimately my goal is to Learn in Public and demonstrate to experts that spending their time teaching/sharing with me is a good use of time because i will augment/amplify/simplify their message. (pls give the full podcast a listen/read/watch, George went deep on tinygrad/tinybox/tinycorp and there's lots there he IS the authority on, and people are overly fixated on the GPT4 rumor https://www.latent.space/p/geohot#details https://www.latent.space/p/geohot#details )
- primordialsoup 3y agoMakes it worse, since you have this understanding and still went with this explanation that it's 1.7 trillion patients
- mrtranscendence 3y ago1.7 trillion patients? We're going to need more hospital beds ...
- varelse 3y ago[dead]
- behnamoh 3y agoFor a moment I thought you were Andrej Karapathy!
- devinprater 3y agoOh, is that how Bing's image descriptions work? If you upload an image and ask for a description, it'll give a really good one. But if you ask it where it got it from, it says it got it from another algorithm. So is it using another expert AI to describe, and then feed it into GPT4 for conversion into language?
- albrewer 3y agoPretty sure this is how the human brain works. If we add models that can transform information into updates to existing models, then add a model that can process streams of information and determine which models to update with the information (and can also apply that process to itself), we start to approach a primitive brain, no?
- baja_blast 3y ago> Pretty sure this is how the human brain works How we thought it works in the 1960's yes.
- Infinity315 3y agoHow do we think it works currently?
- deleted 3y ago[deleted]
- irthomasthomas 3y agoTitle should be: "GPT4 is 220B" Factual, and surprising to many.