17 ms·
Pathways Language Model (PaLM): Scaling to 540B parameters
- amai 5y agoFrom the paper: "In this work, we continue the scaling line of language modeling improvements and train a 540 billion parameter, densely activated, autoregressive Transformer on 780 billion tokens of high-quality text." The number of parameters is nearly the number of text tokens. So aren't they simply overfitting a lot?
- deleted 5y ago[deleted]
- lukasb 5y agoDoes anyone know what the units are on the "performance improvement over SOTA" chart?
- lukasb 5y agoTurns out it's a composite of "normalized task-specific metrics", details in the paper. Shrug. Numbers go up!
- r-zip 5y agoI was wondering the same. Without better y-axis labeling, it's not that informative of a graphic.
- whymauri 5y agoPoetic that the top post right now is (partially) about how science communication over-simplifying figures results in a popular misunderstanding of science, leading readers to believe that conducting research is easier than it actually is.
- EvgeniyZh 5y agoIf you're talking about fig. 4, then it's some units scaled so that random performance is 0 and perfect performance is 100 (depending on task it may be accuracy or something else). Since the models are so large, good benchmarks are diverse, and different tasks require different metrics.
- WithinReason 5y agoBased on their 3rd figure, it would take an approximately 100x larger model (and more data) to surpass the performance of the best humans
- danuker 5y agoI placed a transparent plastic ruler on the screen to come to the same conclusion, then I saw your comment.
- WithinReason 5y agoYour methodology is much more sophisticated than mine
- sib 5y agoMy dad, who worked on jet engine production many decades ago, would refer to MIL SPEC EYEBALL Mk I. (I think he was kidding...)
- drusepth 5y agoIts performance on answers to chained inference questions (on page 38 of https://storage.googleapis.com/pathways-language-model/PaLM-paper.pdf https://storage.googleapis.com/pathways-language-model/PaLM-...) has already surpassed the performance of this human.
- DantesKite 5y agoIs there even enough text for that? Wild. Although after a while, maybe you can generate some.
- alphabetting 5y agothe joke explanations on page 38 of the full paper linked here are blowing my mind. it's crazy how far language models have come https://storage.googleapis.com/pathways-language-model/PaLM-paper.pdf https://storage.googleapis.com/pathways-language-model/PaLM-...
- WithinReason 5y agoI like how it appears that it had to convert from imperial to metric before it could make an inference: 300 miles per hour is about 480 km/h. This is about the speed of a commercial airplane. [...]
- sib 5y agoAlso, that's a very slow commercial airplane. (Unless talking about an old turboprop?)
- theincredulousk 5y agoHaven't looked further, but I'm wondering about that. Is that the result of training to be able to explain that specific joke, or is it generalized? In the past these things have been misleading. Some impressive capability ends up being far more narrow than implied, so it's kind of like just storing information and retrieving it with extra steps.
- whimsicalism 5y agoFrom the example, it seems hard to imagine that it has been trained to explain this specific joke. I understand language model skepticism is very big on HN, but this is impressive.
- mjburgess 5y agoHow much of human written history can be compressed and aproximately stored in 504Bn parameters? It seems to me bascially certain that no compressed representation of text can be an understanding of langugae, so necessarily, any statistical algorithm here is always using coincidental tricks. That it takes 500bn parameters to do it, i think, is a clue that we dont even really need. Words mean what we do with them -- you need to be here in the world with us, to understand what we mean. There is nothing in the patterns of our usage of words which provides their semantics, so the whole field of distributional analysis precludes this superstition. You cannot, by mere statistical analysis of patterns in mere text, understand the nature of the world. But it is precisely this we communicate in text. We succeed because we are both in the world, not because "w" occuring before "d" somehow communicates anything. Apparent correlations in text are meaningful to us, because we created them, and we have their semantics. The system must by is nature be a mere remebering.
- sidcool 5y agoWhat do 540 billion parameters mean in this case?
- minimaxir 5y ago540B float32 values in the model. (although since this model was trained via TPUs, likely bfloat16s instead)
- cubefox 5y agoDo those floats encode the weights of synapses? One synapse per float?
- whimsicalism 5y agoThey encode weights. Not sure what a "synapse" is though.
- nynx 5y agoMachine learning models are much more complicated than perceptrons now.
- yobbo 5y agoNetwork weights would take up around 2TB. It also means at least 4TB of TPU/GPU memory required for training, or at least 50 Nvidia A100s.
- endisneigh 5y agoThe model is insane, but could this realistically be used in production?
- gk1 5y agoWhy not? I'm curious if you're picturing any specific roadblocks in mind. OpenAI makes their large models available through an API, removing any issues with model hosting and operations.
- minimaxir 5y agoLatency, mostly. The GPT-3 APIs were very slow on release, and even with the current APIs it still takes a couple seconds to get results from the 175B model.
- motoboi 5y agoYes. You don’t need a model in ram memory, NVME disks are fine.
- ekelsen 5y agoThat would have very slow inference latency if you had to read the model off disk for every token.
- 1024core 5y ago540B parameters means ~1TB of floating bytes (assuming BFLOAT16). Quadruple that for other associated stuff, and you'd need a machine with 4TB of RAM.
- endisneigh 5y agoright - and even if you did run happen to have a machine with 4TB of ram - what type of latency would you have on a single machine running this as a service? how many machines would you need for google translate performance? doesn't seem like you can run this as a service, yet.
- gk1 5y agoIs there an equivalent to Moore’s Law for language models? It feels like every week an even bigger (and supposedly better) model is announced.
- visarga 5y agoScaling Laws for Neural Language Models - https://arxiv.org/abs/2001.08361 https://arxiv.org/abs/2001.08361
- lucidrains 5y agorevised scaling laws https://arxiv.org/abs/2203.15556 https://arxiv.org/abs/2203.15556 https://www.lesswrong.com/posts/midXmMb2Xg37F2Kgn/new-scaling-laws-for-large-language-models https://www.lesswrong.com/posts/midXmMb2Xg37F2Kgn/new-scalin...
- nickvincent 5y agoCrazy impressive! A question about the training data: anyone familiar with this line of work know what social media platforms the "conversation" data component of the training set came from? There's a datasheet that points to prior work https://arxiv.org/abs/2001.09977 https://arxiv.org/abs/2001.09977, which sounds like it could be reddit, HN, or a similar platform?
- dgreensp 5y agoWhen can we try using it?? :)
- gjstein 5y agoWould love an answer on this too. It would be even better not just to try using this, but also be able to run it locally, something that has been impossible for GPT-3.
- arkano 5y agoDoes it look like it would be possible to run locally?
- whimsicalism 5y agoThis is not something that will be possible to run locally. If you had 1 bit per parameter (not realistic), it would still take ~100 GB of RAM just to load into memory.
- The_rationalist 5y agoYou could technically dynamically offload the RAM overload to disk but this would probably be too slow?
- whimsicalism 5y agoI mean, theoretically if you can get the model weights onto disk then you should be able to do the computation - but it might takes days or months on commodity hardware. It would also require creating a system that can do this and I doubt there is much demand.
- ausbah 5y agoi wonder if pruning and other methods that reduce size drastically while not compromising on performance would be possible
- visarga 5y agoInteresting that they used Chain of Thought Prompting[1] for improved reasoning so soon after its publication. Also related to DeepMind AlphaCode which generates code and filters results by unit tests, while Chain of Thought Prompting filters by checking for correct answer at the end. Seems like language models can generate more training data for language models in an iterative manner. [1] https://arxiv.org/abs/2201.11903 https://arxiv.org/abs/2201.11903
- make3 5y agoThe chain of thought paper is from Google, so they've known about it internally for a while potentially
- nullc 5y agoThe general technique is pretty obvious, I discussed and demonstrated it in some HN comments with GPT2 and GPT3 a couple times in the last couple years, and suggested some speculative extensions (which might be totally unworkable, unfortunately these networks are too big for me to attempt to train to try it out) https://news.ycombinator.com/item?id=24005638 https://news.ycombinator.com/item?id=24005638
- gwern 5y agoIn fact, people had already shown it working with GPT-3 before you wrote your comment: https://twitter.com/kleptid/status/1284069270603866113 https://twitter.com/kleptid/status/1284069270603866113 https://twitter.com/kleptid/status/1284098635689611264 https://twitter.com/kleptid/status/1284098635689611264 Seeing how much smarter it could be with dialogue was very exciting back then, when people were still super-skeptical. The followup work has also brought out a lot of interesting points: why didn't anyone get that working with GPT-2, and why wouldn't your GPT-2 suggestion have worked? Because inner-monologue capabilities seem to only emerge at some point past 100b-parameters (and/or equivalent level of compute), furnishing one of the most striking examples of emergent capability-spikes in large NNs. GPT-2 is just way too small, and if you had tried, you would've concluded inner-monologue doesn't work. It doesn't work, and it keeps on not working... until suddenly it does work.
- NiceElephant 5y agoI wonder if AI is a technology that will move from "local producers" to a more centralized setup, where everybody just buys it as a service, because it becomes too complicated to operate it by yourself. What are examples in history where this has happened before? The production of light, heat and movement comes to mind, that, with the invention of electricity, moved from people's homes and businesses to (nuclear) power plants, which can only be operated by a fairly large team of specialists. Anybody has other examples?
- eunos 5y agoHosting from your owning servers on home, localized data centers to global cloud companies.
- NiceElephant 5y agoYeah, this kinda goes in the same direction, but in this case, as well as for example agriculture, I feel it is mostly for convenience. You could still do it at home if you wanted to, in contrast to operating a nuclear power plant. I thought chip-making might be another example, but I'm not sure that was ever decentralized in its early days.
- castratikron 5y agoWill we see an intelligence too cheap to meter? https://www.atlanticcouncil.org/blogs/energysource/is-power-ever-too-cheap-to-meter/ https://www.atlanticcouncil.org/blogs/energysource/is-power-...
- napoleon_thepig 5y agoThis is kind of already happening with services like Google cloud translation.
- The_rationalist 5y agonot a single super large language model has beaten the state of the art in the key NLP tasks (POS tag, dep tag, coreference, wsd, ner, etc) They are always only used for higher level tasks, which is tragic.
- oofbey 5y agoWhy is that tragic? Classic NLP tasks are IMHO kinda pointless. Nobody _actually_ cares about parse trees, etc. These things were useful when that was the best we could do with ML, because they allowed us to accomplish genuinely-useful NLP tasks by writing code that uses things like parse trees, NER, etc. But why bother with parse trees and junk like that if you can just get the model to answer the question you actually care about?
- The_rationalist 5y agoif you can just get the model to answer the question you actually care about? LOL your level of optimism is over 9000. Language models are shit, they regularly utter nonsense and have an impressive number of limitations (e.g. no continual learning). A neuro-symbolic system on the other hand can be incrementally improved upon, has continual learning/a memory, and is interpretable.
- oofbey 5y agoYou selling buggy whips? Good luck with that.
- hashingroll 5y agoI would not discount NLP tasks just yet. In practice, they are still used to solve problems like spellcheck, autocomplete and even words and text rewrites.
- oofbey 5y agoI'd argue that every one of those tasks is better solved with a neural LM. The fact that people still use traditional techniques does not mean those old techniques are better. It just means those tasks haven't caught up to the modern era yet. Or maybe it's because MLM are too expensive to run. But that's hardly an argument that MLM should solve traditional NLP tasks.
- londons_explore 5y agoThis huge language model was trained 'from scratch' - ie. before the first batch of data went into the training process, the model state was simply initialized using random noise. I believe we are near the end of that. As models get more and more expensive to train, we'll see future huge models being 'seeded' with weights from previous models. Eventually nation-state levels of effort will be used to further train such networks to then distribute results to industry to use. A whole industry will be built around licensing 'seeds' to build ML models on - you'll have to pay fees to all the 'ancestors' of models you use.
- aceon48 5y agoWould that even work? Using the pretrained model, wouldn’t you then be in the “local” optimum? Although from a transfer learning perspective, chopping off last layers and retraining… You are prob right
- deleted 5y ago[deleted]
- londons_explore 5y agoPlease Google.... Please include in your papers non-cherry-picked sample outputs! And explicitly say that they aren't cherry picked. I understand that there is a chance that the output could be offensive/illegal. If necessary you can censor a few outputs, but make clear in the paper you've done that. It's better to do that than just show us the best picked outputs and pretend all outputs are as good.
- imranq 5y agoExactly, it's so easy to try out thousands of examples and then just show the best one.
- modeless 5y agoThis is why Google built TPUs. This alone justifies the whole program by itself. This level of natural language understanding, once it is harnessed for applications and made efficient enough for wide use, is going to revolutionize literally everything Google does. Owning the chips that can do this is incredibly valuable and companies that are stuck purchasing or renting whatever Nvidia makes are going to be at a disadvantage.
- IIAOPSW 5y agoI honestly think we will one day look back and see this as taking a hammer to a screw. Sure it will work, but its a horribly brute-force way of doing it. The brain almost certainly processes language in a more elegant way that, once understood, certainly won't take up such raw computational power. The brainwaves associated with a wakeful state (and thus generating and parsing language) happen around 40 Hz. Our machines are running in the GHz region. We are missing some fundamental insight. At the risk of ending up front and center in the hall of prediction hubris, here is my bold prediction: By 2040 it will be possible to do what these TPU's are doing today on your consumer grade computing device, but it won't be because of a dramatic increase in the computing resources available to such devices.
- voz_ 5y agoBut brains are trained from millions of generations across the lifetime of the earth (or from when life started, if you ignore the environment and context needed to start life). You are conflating the training with the inference.
- andbberger 5y agoat least in the visual cortex, synaptic weights are not genetically encoded in humans. they are in drosophila though
- whimsicalism 5y agoSource? I am skeptical that we even can know that because so much can be implicitly encoded.
- mountainriver 5y agoGoogle for all their flaws really is building the future of AI. This is incredibly impressive and makes me think we are relatively close to GAI
- qeternity 5y ago> and makes me think we are relatively close to GAI We’re not. This is undeniably impressive, and likely has huge addressable markets. But we’re still not remotely close to AGI.
- hackinthebochs 5y agoWhy do you think we're not? It's weird how people don't find it weird to just give assertions like this without any supporting argument.
- whimsicalism 5y agoThere are just so many unsolved problems. Taking sequences of actions, having a general model of the world beyond just language, memory, etc. are all large ones off the top of my head.
- hackinthebochs 5y agoSure, there are problems that remain. But how can we say there are "so many" of them? We've seen large leaps in capabilities from architectures based on transformers. We've yet to see the limit of transformers as the scaling curves haven't plateaued. It seems to me that we don't know if we're two innovations or two thousand away from AGI. What we do know is that progress has been faster than anyone would have predicted 10 years ago.
- whimsicalism 5y agoLook, I'm actually a massive optimist on this topic and think we can get very far with large language models. Memory I think will actually not be that difficult to solve, optimizing for long series of actions will be a hard one though, as will making an AI that can reason in a representationally invariant and transferrable way (ie. something that can reason just as well about data in the form of images as text). I'm not saying we're not seeing continuous improvements on performance on benchmarks from scaling up - we absolutely are. But "the end of the benchmarks" is not AGI - even if we have a tool AI/oracle AI that can perform at a superhuman level on benchmark tasks (which I think is absolutely doable within the current paradigm), we do not have a general AI.
- gibsonf1 5y agoWhy scale up a technology that has resulted in no understanding whatsoever - only statistical matching of word groups like a super parrot? Words are simply the data we humans use to convert to a model of space/time for computation as we read them. The model we build in our minds is conceptually linked with little resemblance to those very few input words. You could dim the lights of the earth running the largest computer ever made with the largest training set ever assembled and still, you would be no closer to understanding or anything genuinely useful for us humans.
- pingeroo 5y agoWhat do you mean by "understanding"? Even if it doesn't "think" exactly how humans do, does that really matter?
- gibsonf1 5y agoIt doesn't grasp the world at all, there is no intelligence. It's just statistics about word use.
- 1024core 5y agoPlease explain the "joke explanations" to me, using statistical methods.
- yldedly 5y agoTake 100k joke and explanation tuples as training. Pattern match input joke to training jokes. Edit amalgamation of most structuraly similar training jokes to match content of input joke. Edit corresponding training explanation with same substitutions. Output explanation. Repeat with 100 input jokes, pick the best one.
- 1024core 5y agoBut that is not what they did. Please read the paper.
- ilrwbwrkhv 5y agoAll these approaches are incorrect. They are trying to brute force. What we need are better nets.
- AlexanderTheGr8 5y agoAbout 12 pages just for references...goes to show the insane speed of DL research.
- telotortium 5y agoI've always had this question, so I figured I'd ask it here: these large models are trained on enormous quantities of text - here, 780B tokens, which corresponds to around 1-2 TiB of plaintext scraped from all over the web, using their tokenization scheme. How careful are researchers about making sure their test datasets don't end up in the model? How specifically do they ensure that this doesn't happen? EDIT: It looks like they specifically address in the section "Dataset Contamination" on page 35 of the paper - it appears the performance on the clean subsets of certain tasks, "clean" because that data was confirmed not to be in the training data, is similar to the performance on those tasks when all the questions, including those in the training data, are included. I'd need to have an ML research comment to know if this a valid way to control for the effects of memorization.
- Inufu 5y agoOn many natural language tasks there can be significant overlap, making it difficult to judge performance. That's why I like more complex code generation tasks such the dataset we used for AlphaCode.
- telotortium 5y agoApparently there is a loose convention to exclude any documents from ML training corpora which contain a certain UUID - submitted at https://news.ycombinator.com/item?id=30927569 https://news.ycombinator.com/item?id=30927569.
- telotortium 5y agoTwo basic questions about the performance of the model: 1. What's the maximum context size (don't know the right term for this) that the model can consider at once? For example, for GPT-3 I believe the limit is around 2000 tokens (where each token corresponds to a few characters or perhaps a word in a dictionary)? 2. What is the inference latency - how long does it take to get a result from this model? Given that they don't have any demos linked to in the blog post that I can see, seems like it's long (like multiple seconds, maybe even a minute?).
- EvgeniyZh 5y agoSequence length is 2048 (page 10). Guesswork based on page 65 about compute says that single inference of 2048 tokens is about 1 petaflops, which may be not that bad if your commutinacation is fast (seconds?)
- telotortium 5y agoAccording to https://cloud.google.com/tpu https://cloud.google.com/tpu, each individual TPUv3 has 420 Teraflops, and TPUv4 is supposed to double that performance, so if that guess is correct, it should take a few seconds to do inference. Quite impressive really
- WhitneyLand 5y agoI didn’t see mentioned any papers published on PaLM, nor any open access program. It all sounds impressive but is there a way to sort out substance from the hype in terms of advancing the state of the art?
- mordymoop 5y agoFull paper: https://arxiv.org/abs/2203.15556 https://arxiv.org/abs/2203.15556
- EvgeniyZh 5y agoIt's a different paper
- FeepingCreature 5y agoActual paper: https://storage.googleapis.com/pathways-language-model/PaLM-paper.pdf https://storage.googleapis.com/pathways-language-model/PaLM-...
- wiz21c 5y agoTwo questions : 1/ How much CO2 was emitted to train this ? How much to query it ? 2/ Does Google use that in its search engine ? I'd be very happy to have that kind of AI helping me to find the information I need !
- cubefox 5y ago1/ "Using 378.5W measured system power per TPU v4 chip, this leads to a total effective emissions of 271.43 tCO2e. To put this in perspective, total emissions of a direct round trip of a single passenger jet between San Francisco and New York (JFK) is estimated to be 180 tCO2e (Patterson et al., 2021), and total emissions for GPT-3 are estimated to be 552 tCO2e (Patterson et al., 2021). All of the energy use and emissions for PaLM training and the experiments described in this paper are compensated with renewable energy sources (Sustainability, 2022)."