19 ms·
Megatron-Turing NLG 530B, the World’s Largest Generative Language Model
- xnx 5y agoThis reminds me a little bit of the early 2000's where search engines would list the number of indexed pages on their homepage. For language models, does large = good? I'm guessing the quality of the corpus matters as much.
- bpiche 5y agoI'm not sure it moves the needle on NLU/classification tasks very much, compared to models with many fewer parameters. But it does seem to make the NLG better, which is what Microsoft seems obsessed with lately.
- thunderbird120 5y ago>For language models, does large = good? The short answer is yes, the long answer is it's complicated. You could actually think of these models as a type of indexer because, at their heart, what they are doing is memorizing the training data and storing it in such a way that incomplete samples can be used as keys to extract complete samples. The magic happens because the models themselves (even the 100+ billion parameter ones) are nowhere near complex enough to actually store all of these possible key value pairs. Instead, the model has to compress its representation of the data which leads to generalization. Larger models can model more complexities which leads to better performance as long as your training dataset is sufficiently large and varied.
- ewheeler 5y agoMaybe? The Scaling Hypothesis[1] suggests that greater capabilities of intelligence may emerge from scaling up 'scalable architectures' to large sizes. GPT-3 exhibits 'meta-learning' capabilities that GPT-2 did not (like learning how to sum numbers)--probably just because its a 100x larger version of GPT-2. [1] https://www.gwern.net/Scaling-hypothesis https://www.gwern.net/Scaling-hypothesis
- petters 5y agoTraining data has 0.339T tokens, less than the number of training parameters. A model like that could store all of the training text with 100B+ parameters left for computation.
- knuthsat 5y agoFor some reason this issue with model having insane amounts of weights but training data being small is not something that is an issue for modern NNs.
- lucidrains 5y agohttps://arxiv.org/abs/2109.02355 https://arxiv.org/abs/2109.02355
- thewarrior 5y agoMaybe that’s what it’s doing under the hood.
- visarga 5y agoBut then you try to predict the next token on a completely unseen piece of the corpus and fail miserably if all you do is store the training data.
- canjobear 5y agoParameters have been sufficient to memorize the training data for a while now. The fact that neural networks still generalize in this setting is a big mystery that is under active investigation.
- tgv 5y agoA single weight can’t encode an individual word, but the ratio looks close to overfitting too me too.
- petters 5y agoIf 16-bit floating-point numbers are used, it can presumably encode all tokens. In theory. It would not be very easy to work with.
- bane 5y agoWhat's really interesting is that these models are using some non-trivial portion of all easily accessible human writing -- yet humans learn language really well with significantly less input data. What's missing in the field to replicate human performance in learning?
- dlkf 5y agoHumans use language to accomplish tasks in their environment - establishing relationships, making deals, coaxing others, etc. By contrast, all neural language models do is predict the next word as a function of the previous word. So far, these language models have nothing at all to do with language learning. They're only valuable insofar as they advance downstream engineering tasks like machine translation.
- whimsicalism 5y agohttps://arxiv.org/pdf/1802.10217.pdf https://arxiv.org/pdf/1802.10217.pdf This is the paper I love to link in response to these sort of objections.
- YeGoblynQueenne 5y agoHuman priors are a feature, not a bug. The reason deep neural networks need so much data to train them (and still do not handle languag nearly as well as humans) is precisely that humans have "background knowledge", things that we already know and don't have to learn all over again from scratch. We bring this knowledge to bear in our ability to learn and understand language. Deep neural networks on the other hand have only a very, very limited ability to represent and therefore use background knowledge- and so they have to make up for it with astounding amounts of data. The OP's criticism is valid. Being forced to learn everything end-to-end, from scratch, is a severe limitation.
- bane 5y agoI think what's very interesting is that most of the answers to my question sort of boil down (if you squint a little at the answers) to "this is hard, but general AI will make it easy".
- trash3 5y agoHow has the previous largest model, gpt3, generateda value? How much better is this model at those tasks?
- sanxiyn 5y agoGPT-3 powers GitHub Copilot, so it generated some value.
- bpiche 5y agoWonder how much compute it would cost to train this thing, if you weren't Nvidia..
- cs702 5y agoSo we now have models with 0.5 trillion parameters, each the weight of a connection in a neural network. Trillion-parameter models are surely within reach in the near term -- and that's only within two orders of magnitude of the number of synapses in the human brain, which is in the hundreds of trillions, give or take. To paraphrase the popular saying, a trillion here, a trillion there, and pretty soon you're talking really big numbers. I know the figures are not comparable apples-to-apples, but still, I find myself in awe looking at how far we've come in just the last few years, to the point that we're realistically contemplating the possibility of seeing dense neural networks with hundreds of trillions of parameters used for real-world applications in our lifetime. We sure live in interesting times.
- rajnathani 5y agoIf retrieval based NLP [0] becomes a thing, then trillion plus parameters models will likely be less of a thing; as very likely, most of these tens to hundreds of billions of parameters are likely over-fitting (better word: memorized) on training data [text corpus] as seen in the case of GPT-3. [0] https://ai.stanford.edu/blog/retrieval-based-NLP/ https://ai.stanford.edu/blog/retrieval-based-NLP/
- cs702 5y agoYes, self-attention mechanisms are dense associative memories, so it might be possible to replace them in many cases with simpler storage mechanisms. Still, I would count the required storage space as part of a model's parameter size -- e.g., a model consisting of 1 trillion values in RAM and 99 trillion values in storage consists of... 100 trillion values.
- LuisMondragon 5y agoMy issue with this kind of reasoning is the comparison and reference to the human brain. The potential and reach of AI transcends the brain. We never had to master the "mystery" of how birds fly to invent aviation. It was never necessary to compare the number of turbine revolutions of early airplanes to the number of an eagle's feathers. Maybe birds were an inspiration or a metaphor, but thankfully aviation has not been limited to the means of propulsion of the beautiful yet humble pigeon. The potential of aviation has taken us into space exploration and massive international travel. I don't know where AI will take us, but I don't think it will be constrained by this temporary organ called 'human brain'.
- rustc 5y agoHas there been any update on the legality of using this kind of model? Is it ok to just crawl the web, take any content you want, train a model and sell access to the model like OpenAI/GPT-3/GitHub Copilot?
- wyldfire 5y agoFor the most part, everything that's not barred by law is "legal." Does this use constitute copyright infringement (if it were trained on copyrighted material)? IMO no, but it depends very much on the use of the model. Copilot is especially interesting because instead of being used for simple inference the model is being used to author new works that might aspire to also be copyrighted. Are those new works derivative works? Perhaps. We consider art and science produced by humans to be inspired in part by that which they've been exposed to before. If the model hasn't been overfitted, it should generalize its 'knowledge' sufficiently that it's 'similar' to our intelligence. Humans can commit copyright infringement when they recall and author content so specifically as to be a derived work. In any case: my opinion matters for naught. The only 'update' you'd get that matters is from a court producing a ruling. Legal journals might chime in but their opinion isn't binding. Theoretically there could be legislation to clarify but that's probably a really, really, really long way off. Certainly some of the training looks to be content that's not copyrighted or no longer copyrighted, btw.
- savant_penguin 5y agoReally cool! I'd love to see a table comparing the results against the other gigantic models (I know could Google the other results and merge them together but no thanks)
- captn3m0 5y agoInteresting that books3 and The Pile are among the largest corpus used for training - both with copyright concerns.
- robbedpeter 5y agoDo you want to be the reason we can't have nice things? Please don't post things like this.
- captn3m0 5y agoThey’re using and citing it, better than whatever GPT-3 did.
- posharma 5y agoThis is great. Now, how do we inference these models economically? It appears there's some kind of competition to train larger and larger models, but the inferencing side of the story seems to be neglected?
- buffington 5y agoWhen you say "inference", do you mean "interface", or is "inference" an ML term I'm not familiar with?
- sanity31415 5y agoFirst you train a model then you use it, "inference" is a fancy word for using the model.
- igorkraw 5y agoIt's a ML term, inference basically means using the probability model you learned to draw "inferences" about a piece of data. In this context, it means giving the language model some context and using some method (either arg max sampling or something more sophisticated like beam search) to do what amounts to statistical auto implemention on it. As you might imagine, doing this with 530 GB of data at speed is quite energy intensive, even though there are things you can do to compress the model (distillation, pruning, compression/discretization) and specialised inference hardware. Technically there is some very specific meaning to inference vs. prediction, but it's been heavily overloaded with meaning by now
- dwohnitmok 5y agoModel inference is actually comparatively very cheap. If you have the resources to train a model, you most definitely have the resources to run it.
- shock-value 5y agoDoes that hold as the workload scales up? E.g. could this or similar models be used as part of a general-purpose search engine whereby (at least) one inference is completed per unique search? Aside from computation, I know these models consume an intense amount of memory -- would that scale horizontally easily / economically? Would it need to?
- macrolime 5y agoWill anyone outside of Nvidia be able to access it? GPT-3 at least has an API.
- thenightcrawler 5y agohoping so!
- sonic-boom 5y agoSame!
- saurkt 5y ago(Team member of this project) Just a clarification, both Microsoft and Nvidia have ownership of this model. Here is the Microsoft version of same announcement. https://www.microsoft.com/en-us/research/blog/using-deepspeed-and-megatron-to-train-megatron-turing-nlg-530b-the-worlds-largest-and-most-powerful-generative-language-model/ https://www.microsoft.com/en-us/research/blog/using-deepspee...
- miket 5y agohttps://en.wikipedia.org/wiki/Wu_Dao https://en.wikipedia.org/wiki/Wu_Dao
- canjobear 5y agoWhat's the perplexity?
- sonic-boom 5y agoAny idea if they’ll release an API similar to GPT-3? It’s great that larger and larger models are trained but without enabling access to the trained models developers are left out from the progress…
- jowday 5y agoI hope they don’t release an API the way they released an API for GPT-3.
- sanxiyn 5y agoWhy? What do you do not like about GPT-3 API?
- nullc 5y agoCan't fine tune it. Can't use it on private data. Pay per symbol. High latency access. Inconsistent performance. No internet disconnected use. Can't guarentee repeatable results. Can't (easily) replace the sampling with alternatives that have different behavior (e.g. for using it for compression).
- RhysU 5y agoSo, uh, what do the seed-to-seed variance studies look like on a network of this size? Surely someone trained 100 to see the distribution. ducks
- putlake 5y agoChina's WuDao model had 1.75T parameters and Google's Switch transformer had 1.6T. How is this the world's largest then?
- moochi 5y agoall these models are over-hyped. we are nowhere close to AGI until we can come up with a reasonable definition for consciousness
- ultra_nick 5y agoIt's possible consciousness isn't a real thing. Humans might just be big neural networks that predict the actions most likely to result in survival.
- bobm_kite9 5y agoI guess I'm interested to see if this performs qualitatively better than GPT-3, given how many more parameters it has. However, I think this is really a dead-end: throwing more hardware at this is just going to generate better-sounding nonsense. Yes, we are learning the "model" of the English language - which words go with which others, but successively larger transformer models don't really expose much more about the nature of intelligent conversation. I think we need a better algorithm now.
- jamesbriggs 5y agoAlso agree with this, it's almost like a marketing ploy - especially from OpenAI. They produce awesome stuff but things with GPT can get silly sometimes, like when they wouldn't release the larger versions because 'they were too powerful', and in the end you ask it how many eyes my foot has and it says seven... It producing interesting results, but doesn't really progress the field - although who knows, maybe skynet is actually a 100T parameter transformer
- simonh 5y agoListening to conversations and learning how they flow is only one aspect of language learning. What these things are missing is the interactive part. Next generation systems need to be able to form hypotheses about what appropriate responses should be, try out various responses and then see what the results are. They can only learn so much from passive consumption of training sets, so I agree this approach is going to hit a wall of diminishing returns.