26 ms·
GPT4 is 8 x 220B params = 1.7T params
- sylware 3y agoIs this still orders of magnitude smaller than a human brain? How many? Based on current human neurons/synapses knowledge?
- birriel 3y ago2 orders of magnitude smaller, assuming 100T synaptic connections in the human brain.
- ada1981 3y agoIt’s remarkable. I’m curious how long until people are just using training brains in a jar to compute.
- quickthrower2 3y agoThey already are, for the hyperhyperparameter choices.
- mlashcorp 3y agoWell, not in a jar ... Right?
- zxcnmbzxcmb 3y agoHow would you know that you're not in a jar? :-)
- quickthrower2 3y agoOpen plan noisy offices are almost that.
- elzbardico 3y agoNot yet, but in our last call we assured our stakeholders that we are very close to that.
- jimmySixDOF 3y agoIn the podcast they talk about 20 Peta FLOPS as the human brain equivalent for measuring comparison.
- baq 3y agoAnd how big is the memory and language part…?
- justrealist 3y agoWe really have no idea how to directly compare the two. Also, vast portions of the human brain are dedicated to the visual cortex, smelling, breathing, muscle control... things which have value to us but which don't contribute to knowledge work when evaluating how many parameters it would take to replace human knowledge work.
- vbezhenar 3y agoIt's really interesting that human organism requires so much computational power to support live.
- gwoolhurme 3y agoI think it's even more interesting that the required amount of energy to do that high computational work isn't that high. Evolution has been working on it for a long time, and some things are really inefficient but overall it does an OK job at making squishy machines.
- tedivm 3y agoThe human brain uses roughly 20 watts, which is really a remarkably low number. https://psychology.stackexchange.com/a/12386 https://psychology.stackexchange.com/a/12386
- interlinked 3y agoThat's 20% of total energy consumption.
- loh 3y agoI had a good chuckle at "squishy machines". That's a really interesting way to think about it. It makes me wonder if, some day, we will be able to build "squishy machines" of our own, capable of outperforming silicon while using a tiny fraction of the energy.
- WithinReason 3y ago
- light_hue_1 3y agoWe have no idea how to estimate the computational capacity of the brain at the moment. We can make silly estimates like saying that 1 human neuron is equivalent to something in an artificial network. But this is definitely wrong, biological neurons are far more complex than this. The big problem is that we don't understand the locus of computation in the brain. What is the thing performing the meaningful unit of computation in a neuron? And what is a neuron really equivalent to? The ranges are massive. Some people say that computation is some high level property of the neuron as a whole, so they think each neuron is equivalent to just a few logic gates. These people would say that the brain has a capacity of about 1 petaFLOP/s. https://lips.cs.princeton.edu/what-is-the-computational-capacity-of-the-brain/ https://lips.cs.princeton.edu/what-is-the-computational-capa... Then there are people who think every Na, K, and Ca ion channel performs meaningful computation. They would say the brain has a capacity of 1 zettaFLOP/s. https://arxiv.org/pdf/2009.10615.pdf https://arxiv.org/pdf/2009.10615.pdf Then there are computational researchers who just want to approximate what a neuron does. Their results say that neurons are more like whole 4-8 layer artificial networks. This would place the brain well somewhere in the yottaFLOP/s range https://www.quantamagazine.org/how-computationally-complex-is-a-single-neuron-20210902/ https://www.quantamagazine.org/how-computationally-complex-i... And we're learning more about how complex neurons are all the time. No one thinks the picture above is accurate in any way. Then there are the extremists who think that there is something non-classical about our brains. That neurons individually or areas of the brain as a whole exploit some form of quantum computation. If they're right, we're not even remotely on the trajectory to matching brains, and very likely nothing we're doing today will ever pay off in that sense. Almost no one believes them. Let's say the brain is in the zettaFLOP/s range. That's 10^21 FLOP/s. Training GPT-3 took 10^23 FLOPS total over 34 days. 34 days has 2937600 seconds. 10^23/10^7 is about 10^16 FLOP/s. So by this back of the envelope computation the brain has about 4 orders of magnitude more capacity, or 1000x. This makes a lot of sense, they're using a pettaFLOP/s supercomputer basically which we already knew. We'll have zettaFLOP/s supercomputers soon, yottaFLOP/s, people are worried we're going to hit some fundamental physical limits before we get there. All of this is a simplification and there are problems with every one of these estimates. But, in some sense none of it means anything at all. You can have an extremely efficient algorithm that runs 1 million times faster than an extremely inefficient algorithm. Machines and brains do not run the same "software", the same algorithms. So comparing their hardware directly doesn't say anything at all.
- bluedevilzn 3y agoI wouldn’t trust anything geohot says. He doesn’t have access to any inside information.
- valianteffort 3y agoHe doesn't strike me as the type of person to lie (except when trolling). His reputation is solid enough that I'm sure he's had discussions with people in the space.
- zoklet-enjoyer 3y agoHe's a crypto scammer. Look up cheap eth
- valianteffort 3y agoI mean that was marketed as a memecoin from the beginning. More so than doge even.
- pixelpoet 3y agoHe added some code to give himself tons of it and didn't report it to anyone, dismissed it as a joke or something when it was found and then disappeared after hyping the project. Sound familiar? To me he joins the ranks of the most basic shady crypto types.
- imtringued 3y agoYou mean he has become Elon Musk?
- lannisterstark 3y ago>his reputation is solid Eh, is it? Not sure if I consider him an authority on anything anymore. https://www.reddit.com/r/ProgrammerHumor/comments/z2y8i0/from_this_this_is_my_mission_an_i_will_accomplish/ https://www.reddit.com/r/ProgrammerHumor/comments/z2y8i0/fro... >This is the interview. Build this feature. You don't get source access. Link the GitHub and license it MIT. is akin to "Build this for free, license it MIT so I can use it without any issues, and oh, btw, I dont have authority to hire you, teehee."
- SheinhardtWigCo 3y agoGPT-4 is 1.7T params in the same way that an AMD Ryzen 9 7950X is 72 GHz.
- eyegor 3y agoYou might be surprised to learn that dell "hpc engineers" (or maybe HPE?) have attempted to sell me hardware using that exact logic to compare servers.
- lillecarl 3y agoVMware still uses ghz for displaying load and compute capacity on your servers.
- DaiPlusPlus 3y agoWell, they're not wrong... Given a highly-parallelizable numerics job, and all other things being equal, a 20 core 2.0Ghz machine will run the job in the same time as a 40 core 1.0Ghz machine. ...though usually we'd just give it in FLOPS - though FLOPS doesn't describe non-float operation performance. Probably just a sales/marketing person who learned a new technical word and wants to sound knowledgable to the customer but it back-fires.
- zarzavat 3y agoAt the risk of torturing this analogy beyond recognition, it’s more like having an 8 core CPU but with a weak 150W PSU such that you can only run one core at a time.
- SecretDreams 3y agoAnalogy sufficiently tortured
- bee_rider 3y agoIs each core specialized to some particular type of task or something like that? Which would make this less silly of an idea.
- ramraj07 3y agoIf it’s trivial then why does every other competitor suck at replicating it? Is it possible this is just a case of sour grapes that this intellectual is annoyed they’re not at the driving wheel of the coolest thing anymore?
- typon 3y agoI don't think he thinks its trivial - just that its not some revolutionary new architecture. It's an immense engineering effort rather than a fundamental breakthrough.
- MuffinFlavored 3y ago> GPT-4: 8 x 220B experts trained with different data/task distributions and 16-iter inference. There was a post on HackerNews the other day about a 13B open source model. Any 220B open source models? Why or why not? I wonder what the 8 categories were. I wonder what goes into identifying tokens and then trying to guess which category/model you should look up. What if tokens go between two models, how do the models route between each other?
- jerpint 3y agoI think it’s just an ensemble of models, so you do some kind of pooling/majority vote on your output tokens
- heliophobicdude 3y agoWould this be before or after inference? Is there some sort of a delegation based on the matter?
- cfn 3y agoIf it is output tokens then it is after the inference.
- swyx 3y agosame. i wish i had asked george instead of nodding along like an idiot. he probably wouldnt know but at least he’d speculate in interesting ways.
- fancyfredbot 3y agoIt was a great interview, thank you.
- Tepix 3y ago220B open source models wouldn't be as useful for most users. You need two RTX 3090 24GB cards already to run inference with a 65B model that is 4bit quantized. Going beyond that (already expensive) hardware is out of reach for the average hobbyist developer.
- eightysixfour 3y agoI find it interesting that geohot says it is what you do “when you are out of ideas,” I can’t help but think that having multiple blended models is what makes GPT-4 seem like it has more “emergent” behavior than earlier models.
- jerpint 3y agoOpenAI has always stuck to “simpler” approaches that scale. It’s the best bet to make and it’s paid off time and time again for them. I don’t think it’s so much “they’re out of ideas”, it’s just an idea that scales well
- H8crilA 3y agoYeah, and who says that this is all that they're currently working on? They likely found this to work better, launched, and did other things in the meantime.
- jerpint 3y agoI really wonder if it is the case that the image processing is simply more tokens appended to the sequence. Would make the most sense from an architecture perspective, training must be a whole other ballgame of alchemy though
- benob 3y agoProbably. Check the kosmos-1 paper from Microsoft that appeared a few days before GPT4 was released: https://arxiv.org/abs/2302.14045 https://arxiv.org/abs/2302.14045
- it_citizen 3y ago« We can’t really make models bigger than 220B parameters » Can someone explains why?
- sp332 3y agoIt doesn't fit in VRAM.
- hesdeadjim 3y agoI’ve been a bit surprised that Nvidia hasn’t gone to extreme lengths to fit 1tb of memory on a card just for this reason.
- andrewstuart2 3y agohttps://nvidianews.nvidia.com/news/nvidia-announces-dgx-gh200-ai-supercomputer https://nvidianews.nvidia.com/news/nvidia-announces-dgx-gh20... I think they _are_ going pretty extreme now.
- samplatt 3y agoOfftopic, but as a VR gamer that article just made me very sad. I was really hoping to see NVidia produce some decent cards in the near future, but looks like their main revenue is really going to be gargantuan number-crunchers. They'll likely only keep increasing the VRAM of gaming cards by arbitrarily-small numbers once every few years :-(
- deleted 3y ago[deleted]
- speedgoose 3y agoYes, the future of VR gaming looks closer to the Sony Playstation or even the Apple Vision than NVIDIA's products.
- Tepix 3y ago
- andreyk 3y agoweird title, note that the tweet said "so yes, GPT4 is *technically* 10x the size of GPT3, and all the small circle big circle memes from January were actually... in the ballpark?" It's really 8 models that are 220B, which is not the same as one model that is 1.7T params. There have been 1T+ models via mixtures of experts for a while now. Note also the follow up tweet: "since MoE is So Hot Right Now, GLaM might be the paper to pay attention to. Google already has a 1.2T model with 64 experts, while Microsoft Bing’s modes are different mixes accordingly" There is also this linked tweet https://twitter.com/LiamFedus/status/1536791574612303872 https://twitter.com/LiamFedus/status/1536791574612303872 - "They are all related to Switch-Transformers and MoE. Of the 3 people on Twitter, 2 joined OpenAI. Could be related, could be unrelated" Which links to this tweet: "Today we're releasing all Switch Transformer models in T5X/JAX, including the 1.6T param Switch-C and the 395B param Switch-XXL models. Pleased to have these open-sourced!" Anyway... remember not to just read the headlines, they can be misleading.
- ignoramous 3y agoMore Efficient In-Context Learning with Generalist Language Model (GLaM) on the Google blog (2021): https://archive.is/cuyW0 https://archive.is/cuyW0 Btw, MoE is one of the 5 other ways Google thinks LLMs can be more efficient: https://archive.is/2XMvh https://archive.is/2XMvh
- swyx 3y ago(OP here) - yeah i know, but i also know how AI twitter works so I put both the headline and the caveats. i always hope to elevate the level of discourse by raising the relevant facts to those at my level/a little bit behind me in terms of understanding. think theres always a fine balance between getting deep/technical/precise and getting attention and you have to thread the needle in a way that feels authentic to you to do this "job" ultimately my goal is to Learn in Public and demonstrate to experts that spending their time teaching/sharing with me is a good use of time because i will augment/amplify/simplify their message. (pls give the full podcast a listen/read/watch, George went deep on tinygrad/tinybox/tinycorp and there's lots there he IS the authority on, and people are overly fixated on the GPT4 rumor https://www.latent.space/p/geohot#details https://www.latent.space/p/geohot#details )
- andy_xor_andrew 3y agoAre the models specifically trained to be experts in certain domains? Or the models are all trained on the same corpus, but just queried with different parameters? Is this functionally the same as beam search? Do they select the best output on a token-by-token basis, or do they let each model stream to completion and then pick the best final output?
- pizza 3y agoDemocracy of descendant models that have been trained separately by partitioning the identified clusters with strong capabilities from an ancestor model, so, in effect, they are modular, and can be learned to be combined competitively.
- deely3 3y agoHeh. I understand all these words separately. Btw, to which of the question of parent comment this is an answer?
- omeze 3y agoAncestor model = pre-trained model that used a large diverse corpus Descendant models = models fine tuned from an ancestor on one particular domain, e.g. by partitioning your training data by subject or source Democracy = some weighted mix of the descendant models is used to find the next token
- radq 3y agoIf it's similar to the switch transformer architecture [1], which I suspect it is, then the models are all trained on the same corpus and the routing model learns automatically which experts to route to. It's orthogonal to beam search - the benefit of the architecture is that it allows sparse inference. [1] https://arxiv.org/pdf/2101.03961.pdf https://arxiv.org/pdf/2101.03961.pdf
- mrfinn 3y agoSo in layman's terms does this mean that on top of big base of knowledge (?) they trained 8 different 220B models and each model specialized in different areas, in practice like an 8 units "brain"? PS. Thinking now how human brain does something similar as our brain is split in two parts and each one specialize in different tasks.
- lyu07282 3y agofull podcast here: https://www.latent.space/p/geohot https://www.latent.space/p/geohot
- swyx 3y agothanks :) posted it on HN but got no traction, definitely underperformed my expectations. my only explanation is i think i screwed up the title :/ i somewhat knew that people were going to obsess over the gpt4 tidbit, but hey ultimately George is just relaying secondhand info, and I wish more people focused on the tinycorp/tinybox/tinygrad story. I'm going to try again tomorrow to tell the story better.
- esperent 3y agoSo if this is true - which is a big if since this looks like speculation rather than real information - could this work with even smaller models? For example, what about 20 x 65B = 1.3T params? Or 100 x 13B = 1.3T params? Hell, what about 5000 x 13B params? Thousands of small highly specialized models, with maybe one small "categorization" model as the first pass?
- janalsncm 3y agoWell at the end of the day you’ll also need a model for ranking the candidates, which becomes harder as the number of candidates grows. And the mean quality of any one candidate response will drop as the model size decreases, as will the max quality.
- cypress66 3y agohttps://arxiv.org/abs/2112.06905 https://arxiv.org/abs/2112.06905
- janalsncm 3y agoAt a minimum he glossed over the multimodal capabilities of GPT-4. If they use the same set of tokens, it’s unclear how this doesn’t pollute text training data. If they use separate tokens, the model size should be bigger.
- ShamelessC 3y agoI assume they just intersperse and/or use a special <IMAGE> token to prime the models appropriately. This has been done before by others training on e.g. HTML with <img> tags replaced with image tokens from a vision model.
- chriskanan 3y agoThis is the same strategy used by Palm-E and Mini-GPT4. They have a projection layer that transforms the vision embeddings into the token embedding space.
- Havoc 3y agoInteresting. And makes sense. Eg I could see one of the eight being very close focussed and trained on GitHub like data. Could help it stay on tasks too
- swyx 3y agoi'd love any and all informed speculation on what the 8 models could be. it seems fascinating that we don't know.
- LarsDu88 3y agoThat's on the order of 25 4090 GPUs to run inference. Not a crazy number by any means. We will see consumer robots running that by the end of the decade, mark my words.
- swyx 3y agoi believe that's George's goal with tinybox
- jerrygenser 3y agoIt's a lot of electricity to power it and heat that gets generated... and electricity to cool it from the heat
- hardware2win 3y ago25 GPUs? A lot of electricity?
- londons_explore 3y agoLanguage models have tiny inputs (words), and tiny outputs (words), and no tight latency requirements (inference takes seconds). That makes them perfect for running where compute is cheap (ie. next to a hydrothermal plant in iceland) and querying them over the internet.
- ChatGTP 3y agoHow are. 254090 GPUs supposed to run inside a robot ?
- deleted 3y ago[deleted]
- fnands 3y agoThey mean 25 x RTX 4090 GPUs. 4090 is a model number
- 3y ago
- letitgo12345 3y agoWhat's 16 iter inference?
- coolspot 3y ago16 iterations of inference. Like each of 8 models is processed 2 times.
- reportgunner 3y agoSad that there is a grammar error lining the top of the video.
- swyx 3y agoliterally unwatchable
- deleted 3y ago[deleted]
- jiggywiggy 3y agoI often hear the idea of digital is faster then biology. This seems mostly derived from small math computations. Yet it seems the current form of large language computations is much much slower then our biology. Making it even larger will be necessary to come closer to human levels but the speed? If this is the path to GI, the computational levels need to be very High and very centralized. Are there ways to improve this in its current implementation other the cache & more hardware?
- yreg 3y agoFor all we know these models should be compressible to much smaller sizes.
- smif 3y agoThis is probably not the path to GI. First we would need a precise scientific formalism to accurately describe intelligence, which currently does not exist. Second, it may or may not end up being tied to consciousness, and there's a thing called the hard problem of consciousness, that possibly might not be solvable. It might end up being the kind of thing where if you want to accurately model consciousness, you would need a computer the size of the universe, and it's gotta run for like 13.8 billion years or so, but that's just my own pure speculation - I don't think anybody even has a clue on where to start tackling this problem. This is not to discourage progress in the field. I'm personally very curious to see where all this will lead. However, if I had to place a bet, it wouldn't be on GI coming any time soon.
- blharr 3y agoTo accurately model consciousness it seems like we'd only need a computer the size of our brains, we'd just need to be as efficient as it.
- smif 3y agoSeems like it, doesn't it? I'd be curious to see if and how we can get there. What I was highlighting are some serious challenges along the way that might end up leading us to insights about why it might be harder than we think, or why there may be factors that we aren't considering. It's very easy to say "brain is made of fundamental particles and forces, all we have to do is create a similar configuration or a model of them," but it's precisely in the task of understanding the higher order patterns of those fundamental particles and forces where we seem to run into some serious challenges that as of yet remain unaddressed. The AI/ML way of approaching this is more of a top-down approach, where we just sort of ignore the fact that we don't understand how our own brains/minds work and just try to build something kind of like it in the folksy sense. I'm not discouraging that approach, but I'm very curious to see where it will lead us.
- donkeyboy 3y agoI think this could be fake. He says it’s an MoE model, but then explains that it’s actually a blended ensemble. Anyone else have thoughts on that?
- bigyikes 3y agoI like how geohot takes these concepts that seem impossibly complex to an outsider (mixture of models, multi modality) and discusses them so casually that they seem accessible to anyone. The flippancy is refreshing.
- refulgentis 3y agoThere's another layer to that sort of...erasure. You'll see it in a few years.
- refulgentis 3y agoWe're sourcing off Geohot? Yikes.
- generalizations 3y agoReminds me of this: https://en.wikipedia.org/wiki/Society_of_Mind https://en.wikipedia.org/wiki/Society_of_Mind > In his book of the same name, Minsky constructs a model of human intelligence step by step, built up from the interactions of simple parts called agents, which are themselves mindless. He describes the postulated interactions as constituting a "society of mind", hence the title.
- chriskanan 3y agoThis seems pretty consistent with what Sam Altman has said in past interviews regarding the end of continuously increasing scale and having multiple smaller specialist models: https://finance.yahoo.com/news/openai-sam-altman-says-giant-164924270.html https://finance.yahoo.com/news/openai-sam-altman-says-giant-...