13 ms·
DeepMind’s New Language Model, Chinchilla
- hwers 5y agoCan't wait for DeepMind to take a stab at outcompeting dall-e.
- isaacfrond 5y agoThey trained over 400 language models ranging from 70 million to over 16 billion parameters on 5 to 500 billion tokens while staying under a given compute budget. The results are modelled, and they pick the best one. Turns out the having a bit fewer tokens improves performance.
- gbasin 5y agoThank you :)
- alpineidyll3 5y agoIf these things get put on specialized hardware for inference with much lower energy costs, the world will never be the same.
- lobstey 5y agothe biggest problem first of all might be the memory requirements given so many parameters. It couldn't be as cheap as a high end computer in the foreseeable future.
- f38zf5vdt 5y agoThere is probably a space-time trade off that needs to be explored in this space. It might be possible to preload the some of the most likely tokens to be selected next into the cache and/or RAM. These are glorified auto-complete algorithms that are poorly understood, as DeepMind's optimizations appear to show. For the English language, it is probable that there are only so many possible grammatically correct selections for the next token, for example.
- visarga 5y agoGlorified autocomplete? Autocomplete can guess the next word .. sometimes, GPT-3 goes hundreds of words ahead. On generic topics it can be hard to distinguish from human text. And it can't cache tokens because all tokens are evaluated in the context of all the other tokens, so they don't have the same representations when they reoccur at different positions.
- f38zf5vdt 5y agoThey're evaluated in the context of the last 2^n many tokens, for many models it is 1024, 2048, or 4096 tokens as a scanning window. The tokens (words and sometimes punctuation) are represented by integer values, so the last 2^n many tokens would certainly qualify for storage in a cache. Then next token selection only has so many possible assignable selections in any given language model because of grammatical limitations. This is only one such optimization, there could also be optimizations around the likelihood of certain words to be used given the presence of certain previous tokens, and so on. But, yes, tokens are chosen one word as a time based on the previous content, similar to earlier auto-completion algorithms.
- priansh 5y agoI’ve been saying this for years, language models are the ML equivalent of the billionaire space race, it’s just a bunch of orgs with unlimited funding spending millions of dollars on compute to get more parameters than their rivals. It could be decades before we start to see them scale down or make meaningful optimizations. This paper is a good start but I’d be willing to bet everyone will ignore it and continue breaking the bank. Can you say that about any other task in ML? When Inceptionv3 came out I was able to run the model pretty comfortable on a 1060. Even pix2pix and most GANs fit comfortably in commercial compute, and the top of the line massive models can still run inference on a 3090. It’s so unbelievably ironic that one of the major points Transformers aimed to solve when introduced was the compute inefficiency of recurrent networks, and it’s devolved into “how many TPUs can daddy afford” instead.
- native_samples 5y ago
- deleted 5y ago[deleted]
- hwers 5y agoImagine any diffusion-style text-to-image model on specialized ASIC hardware.
- astrange 5y agoThat’s what an ANE/TPU is. If you mean putting the model weights into gates directly, it’d be useless because users would get bored of the model as soon as they figured out what its style looked like. Also, large models can memorize their training data so eventually you’ll get it to output something copyrighted.
- alpineidyll3 5y agoThese models are definitely entering the space where no one could ever get bored of them, and many styles can be generated.
- galcerte 5y agoI have to ask, why call it that? I had a chuckle once I saw the name.
- binarymax 5y agoLarge language models have a (recent) history of silly names. BERT, BART, ELMO, RoBERTa, BIGBIRD, PaLM, Megatron etc. Might as well go full nonsense.
- omarhaneef 5y agoTrue. I will add that it is customary to justify it by demonstrating it is some sort of acronym or contraction.
- yeetsfromhellL2 5y agoIt's a recursive, selective acronym C CH CHI CHIN CHINC CHINCH CHINCHI CHINCHIL CHINCHILL ==> CHINCHILLA HINCHILLA INCHILLA NCHILLA CHILLA HILLA ILLA LLA LA A
- omarhaneef 5y agoI know what recursive means, I know what selective means, I know what an acronym is, and I think I see the pattern in that picture, but when I put it all together I am lost. Alternatively, is this a joke and the "recursive, selective acronym" can be used to justify any word?
- veonik 5y agoA AR ARB ARBI ARBIT ARBITR ARBITRA ARBITRAR ==> ARBITRARY RBITRARY BITRARY ITRARY TRARY RARY ARY RY Y Yup, seems it works for any word.
- ShamelessC 5y agoSeems the link is down. Found a decent synopsis/discussion on lesswrong. https://www.lesswrong.com/posts/midXmMb2Xg37F2Kgn/new-scaling-laws-for-large-language-models https://www.lesswrong.com/posts/midXmMb2Xg37F2Kgn/new-scalin... > On March 29th, DeepMind published a paper, "Training Compute-Optimal Large Language Models", that shows that essentially everyone -- OpenAI, DeepMind, Microsoft, etc. -- has been training large language models with a deeply suboptimal use of compute. > Following the new scaling laws that they propose for the optimal use of compute, DeepMind trains a new, 70-billion parameter model that outperforms much larger language models, including the 175-billion parameter GPT-3 and DeepMind's own 270-billion parameter "Gopher".
- eigenvalue 5y agoAccording to the LessWrong post, the smaller model trained on more data performs better on most of the tasks, but it’s worse on “college level math” questions. I wonder why that is. Is it because the extra capacity of the larger model was used to basically memorize theorems? Or is it because the extra “brain power” let it model the math better? Oddly, one of the tasks that the smaller most outperformed the larger model on is “high school level math”! Very counterintuitive, and I am curious if there are any big takeaways lurking in that disparity.
- deleted 5y ago[deleted]
- adamsmith143 5y agoProbably right. Most people dump on these language models for this reason but it would be absurd for a HS student to have to re-derive the quadratic equation every time they worked on an Algebra problem so naturally you memorize it. Why should it be any different for a language model?
- eutectic 5y agoI never memorized the quadratic formula, and I did OK.
- mangoham 5y agoCached version since the original is down (I'm assuming it's down due to load issues and not due to the author taking it down). https://webcache.googleusercontent.com/search?q=cache:PLSLy9MT5eMJ:https://www.marktechpost.com/2022/04/09/check-out-this-deepminds-new-language-model-chinchilla-70b-parameters-which-significantly-outperforms-gopher-280b-and-gpt-3-175b-on-a-large-range-of-downstream-evaluation-tasks/+&cd=1&hl=en&ct=clnk&gl=se https://webcache.googleusercontent.com/search?q=cache:PLSLy9...
- ianbutler 5y agoThis is exciting if only because as we discover more compute optimal models that out perform the behemoths that have been state of the art it opens up the ability for smaller independent groups to train and release their own versions, more fully democratizing AI. Looking forward to a group like Eluther or Hugging Face releasing a version of this.
- adamsmith143 5y ago>This is exciting if only because as we discover more compute optimal models that out perform the behemoths that have been state of the art it opens up the ability for smaller independent groups to train and release their own versions, more fully democratizing AI. I think I support this in principle but it seems like the scaling curves keep going so it's easier to just make larger models with more data. >Looking forward to a group like Eluther or Hugging Face releasing a version of this Both of those groups have access to dozens if not hundreds of Cloud GPUs, I'd hardly call them small. It would be impossible to replicate these models as say an independent researcher or even in an academic research group outside of maybe Stanford/Berkeley/MIT/etc. and I'd even doubt their ability to replicate models like this based purely on Cost alone.
- ianbutler 5y agoSmall is relative -- and to Google, Facebook and Microsoft they're positively tiny. Perfect is the enemy of good or some such and I think this is a move in the right direction even if I can't personally train this on my 3090.
- g051051 5y agoIs there a good reference as to what a "parameter" is in this context? I've looked a few times, but the explanations don't make any sense to me.
- guipsp 5y agoYou can think of a parameter as a number you can tweak while training. This network has 70B such numbers.
- sirk390 5y agoAnd if every parameter is one byte, the minimum, it will take at least 70gb to save or share this model. So it's still way to big to package directly in a app.
- cshimmin 5y agoFrom the paper, they are using bfloat16, so I guess two bytes. But distributing and "packaging into an app" are not at all of practical interest for these kinds of models. You (a consumer) would interact via some API service, with the model running on a hardware-accelerated compute cloud. In any case, during training (where the model is run in possibly large batches), and even during inference, the size of the parameters is completely dwarfed by the intermediate tensor representations.
- brrrrrm 5y ago> even during inference, the size of the parameters is completely dwarfed by the intermediate tensor representations What makes you say this?
- cshimmin 5y agoIt's especially true for models that do some kind of weight sharing, which is very common (CNNs, RNNs, transformers, etc). For a concrete example, consider a layer from an image convolutional network, which maps from a 3-dim colorspace to a 128-dim feature space. Assuming a 5x5 kernel that's about 10k parameters. However, after applying this layer, you go from having an (B,H,W,3) tensor to a (B,H-4,W-4,128) tensor, where H,W are the height and width of the image, and B is the number of images in the batch. If you're working with even moderately high resolution images, the memory required for these intermediate tensors at each layer is much larger than the parameters. Something similar applies for RNNs (same weights applied at each element of a sequence), GNNs and transformers (same weights applied at each pair of data).
- mrfusion 5y agoDoes this imply we will run out of data to keep up with larger model sizes? Is there much more data out there than what they’re already using?
- adamsmith143 5y agoProbably not an issue just yet, think of how much data is generated by Twitter on a daily basis for example.
- teraflop 5y agoOn the other hand, consider the difficulty of taking massive amounts of data from the modern web and filtering out the subset that was actually generated by humans, rather than previous generations of language models.
- adamsmith143 5y agoDefinitely an interesting future problem. I'm sure OpenAI and others are thinking about it but I don't think these models are ubiquitous enough to have much impact just yet.
- axg11 5y agoSome estimates: - 500M tweets per day - 30 words/tokens per tweet - 40% of all tweets thrown away due to being duplicate/spam/bots = 9B tokens generated per day
- zarzavat 5y agoIf you want to teach your kid to learn English, and they came back to you and said "Dad/mum, I finished reading the entire internet but I still don't understand English fully", would you say "OK son, now go and stare at the Twitter firehouse until you grok perfect English" ? It's clear that these models have orders of magnitude too much data already. It somewhat reminds me of the proposals for larger and larger colliders in the hopes of seeing new physics that is always one collider in the future.
- mark_l_watson 5y agoThe design of the original Transformer model in the Attention is all You Need paper was predicated on efficiency (all layers the same size, combining word/token embedding with position in the input stream harmonic embedding). It is good to see improvements!
- sirk390 5y agoIs outperforming GPT-3 still a good reference? It seems there are many models outperforming GPT-3 in the superglue benchmark: https://super.gluebenchmark.com/leaderboard/ https://super.gluebenchmark.com/leaderboard/ GPT-3 is in position #21, with 71.8% score. The best model is at 91.2%. Note the human baseline in #6 with 89.8%
- changoplatanero 5y agoAren’t most of the models at the top not suitable for text generation? That’s what makes gpt different from Bert
- colordrops 5y agoWhat are the models at the top used for? Excuse my ignorance.
- redredrobot 5y agoIt's a good reference because people are familiar with GPT-3. The paper mostly compares Chinchilla to LaMDA, Jurassic, Gopher, MT-NLG, and GPT-3. In the broader tech industry and even to a certain extent within the AI field, GPT-3 is the only one that most people know by name.
- screye 5y agoNote that this isn't an apples-to-apples comparison. The GPT-3 position is for a few-shot use-case that has not been trained for this particular task. When fine-tuned, GPT-3 would be expected to perform a lot better. Lastly, GPT-3 is currently operating on the text-002 models, and the 3rd version of GPT-3 is generally the one considered current. These benchmarks are for the original GPT3 model.
- wiz21c 5y agoI understand I can query such a model, one query at a time. But are there way to query these models with several queries in a row such that the N+1-th query benefit from the knowledge that was used to answer the N first questions ? Basically, following a conversation. For example, youtube subtitles can badly translate some terms but if "it" had in mind the overall subject of the video, then it'd probably pick the correct word...
- ritwikgupta 5y agoOff-topic to Chinchilla, but relevant to the source site: MarkTechPost consistently borderline plagiarizes articles and shares them on their website as "paper summaries". They copy-paste from the source material and change some of the wording around as to appear original. My work, as well as other work from Berkeley AI Research, has been posted in this manner on their site. This seems highly unethical, and I'm surprised how they continue to operate.
- brrrrrm 5y agoThanks for the heads up! In that case, I'd prefer not to share this link with peers. Do you have an alternative source with similar high-level content to share?
- lstamour 5y agoTough to say. Technically https://arxiv.org/pdf/2203.15556.pdf https://arxiv.org/pdf/2203.15556.pdf has the same content, it just isn’t highlighted the same way.
- ghgxbjfvnkj 5y ago
- georgehill 5y agoOP here - Thanks for sharing. I wasn't aware of this but despite this behavior, they are getting 600k visits. https://www.similarweb.com/website/marktechpost.com/#overview https://www.similarweb.com/website/marktechpost.com/#overvie...
- boplicity 5y agoFill out a DMCA notice: https://abuse.cloudflare.com/ https://abuse.cloudflare.com/ Cloudflare will forward it to their host, I believe, who will then ask that they remove the infringing material, or provide a counter claim.
- 5y ago
- narrator 5y agoI'd love to take a language model, load it up, and train it on everything I write in online learning mode. Does one need some massive hardware to do online learning with these models instead of just running the distilled final models?
- deleted 5y ago[deleted]