18 ms·
How does GPT obtain its ability? Tracing emergent abilities of language models
- srajabi 4y agoAmazing insight, particularly section 6. "- The two important but different abilities of GPT-3.5 are *knowledge* and *reasoning*. Generally, it would be ideal if we could *offload the knowledge part to the outside retrieval system and let the language model only focus on reasoning.* This is because: - The model’s internal knowledge is always cut off at a certain time. The model always needs up-to-date knowledge to answer up-to-date questions. - Recall we have discussed that is 175B parameter is heavily used for storing knowledge. If we could offload knowledge to be outside the model, then the model parameter might be significantly reduced such that eventually, it can run on a cellphone (call this crazy here, but ChatGPT is already science fiction enough, who knows what the future will be)." & "Yet there was a WebGPT paper published in Dec 2021. It is likely that this is already tested internally within OpenAI." It definitely feels like this may be the next step in making this kind of system robust. It ends up being an interface for search.
- adamsmith143 4y agoThe problem with ChatGPT's "knowledge" is that it isn't trustworthy. It will happily output very confident sounding nonsense, or blatantly incorrect statements. We need a way to verify how accurate it's outputs are
- nathias 4y agoso, much like other knowledge sources?
- roywiggins 4y agoMost knowledge sources don't make up totally fictional citations to nonexistent sources. Or, if they do, nobody uses them for anything serious. Even Wikipedia citations will get removed if they point to URLs that never existed.
- nathias 4y agoif we focus on the best sources, even in studies a lot of research can't be replicated, and if we focus on te most common ones like newspapers and tv, I'd say most of it is made up or might as well be
- adamsmith143 4y agoThat's just a bad take and it doesn't excuse the problems with GPT.
- roywiggins 4y agoSure, nothing is perfect. But I'm not talking about it just being wrong, I'm talking about it citing webpages and books that don't exist and never did[0]. If Wikipedia regularly had that sort of quality issue people just wouldn't use it. There's a threshold below which something stops being useful. [0] Bloggs, Joe. "ChatGPT just makes stuff up". Nature, vol 123, 2022, pp 123-321. Wiley Online Library, https://doi.org/10.1111/111/111 https://doi.org/10.1111/111/111
- adamsmith143 4y agoOk but if I read a paper from a well known Author published in NeurIps or Nature I have a good sense of how trustworthy that paper might be. Even if we ask GPT to cite it's sources, which it will do, it will then also happily generate false sources. It's untrustworthy turtles all the way down.
- belter 4y agoChatGPT made this nice COBOL program to create an S3 Bucket,a technical impossibility... IDENTIFICATION DIVISION. PROGRAM-ID. CREATE-S3-BUCKET. ENVIRONMENT DIVISION. CONFIGURATION SECTION. INPUT-OUTPUT SECTION. DATA DIVISION. FILE SECTION. WORKING-STORAGE SECTION. 01 AWS-ACCESS-KEY PIC X(20). 01 AWS-SECRET-KEY PIC X(40). 01 BUCKET-NAME PIC X(255). PROCEDURE DIVISION. CREATE-BUCKET. MOVE AWS-ACCESS-KEY TO AWS-ACCESS-KEY-VAR MOVE AWS-SECRET-KEY TO AWS-SECRET-KEY-VAR MOVE BUCKET-NAME TO BUCKET-NAME-VAR INVOKE AWS-S3 "CREATE-BUCKET" USING AWS-ACCESS-KEY-VAR AWS-SECRET-KEY-VAR BUCKET-NAME-VAR
- lmm 4y agoHow is that impossible? Plenty of libraries are available for COBOL, especially if you use COBOL.NET
- jtxt 4y agoOne way I tried to do this is by having it write an answer, and a footnote reference at each fact. [1] then list search terms that be used to verify each claim, then I would respond with the url and quotes from found pages for each one, then have it rewrite the answer based on that information and cite the sources. I think something this direction can be automated. I saw someone do this with math and other tasks, that would talk to a connected program before answering.
- visarga 4y agoYes, it's been done both in papers and in various GPT-3 projects. As long as you can find relevant references the LM will become reliable.
- politician 4y agoSounds like an interesting way to reboot Wikipedia.
- adamsmith143 4y agoI did this as well and it looks great initially but there are already examples of GPT generating totally bogus references and sources. So we're back to square 1.
- moffkalast 4y agoChatGPT to be employed in marketing positions immediately.
- eternalban 4y agoThink bigger. PresidentGPT. On tweeter!
- rdedev 4y agoI just had a run in with this yesterday. I asked it to explain box embeddings. It's a pretty niche topic so I didn't expect it to give the right answer. But the answer it gave sounded so confident but it was so wrong. It took a not al vector embeddings approach but replaced that with box. I tried correcting it but it refused to budge and still sounded confident.
- kyle_grove 4y agoSounds like many humans I know.
- matthewdgreen 4y agoI asked it to explain part of my thesis work on Oblivious Transfer, and it gave me a lovely prose description of the Green-Hohenberger Oblivious Transfer protocol. It was clear and confident, and the thing it described was even an actual protocol. It just wasn’t in any way our protocol: GPT just took some classical protocol it found elsewhere and relabeled it.
- aeternum 4y agoWhose knowledge is trustworthy? We've somehow come to associate certain institutions or scientific authorities with truth when that is about the furthest from real science: "Have no respect whatsoever for authority; forget who said it and instead look what he starts with, where he ends up, and ask yourself, Is it reasonable?" -Richard P. Feynman "One of the great commandments of science is, "Mistrust arguments from authority." -Carl Sagan "In questions of science, the authority of a thousand is not worth the humble reasoning of a single individual." -Galileo Galilei
- hippich 4y agoThere is some kind of recursion in here with authors names and "have no respect..." part:)
- aeternum 4y agoFair point
- AnthonyMouse 4y agoActually, it isn't. Their arguments are of the form "this statement could be false." If you evaluate it as a true statement, you have no problems. It could be false; you have to evaluate it for yourself instead of trusting some authority. It's only if you assert that it's certainly false that you have a problem. Because then it's clearly true -- since otherwise these authorities would be telling you something false, which proves their assertion true. Put another way, it can get you from the undesirable position of blindly trusting authorities to the desirable position of questioning them, but not the other way around. Which is the intended result.
- burnished 4y agoI think part of the issue is that it’s easier to test the limits of or a humans knowledge, and ironically with your quotes I think you’ve supplied evidence that trust is crucial, in that the truest expression of those quotes would be to just deliver the payload and not attach any sort of authority by association to it. You can’t trust it’s answers (to be fair that’s the existing status quo), but you also can’t easily test it because it will return reasonable sounding garbage. Conversely you can discover ignorance in most humans pretty quickly by exhausting their ability to respond (or your ability to ask).
- visarga 4y agoYou are right, this is the pain point - trust, verification. I think it will become the next focus of research. There are many things we could do to solve this problem. One of them is to use an external reference for verification. Another one is to train the model to verify facts by augmenting the input with lies - adversarial training for lie detection. Problem solving can be improved by generating more data with the current version of LM for the next one, if we can verify the outputs to be correct.
- palata 4y ago> There are many things we could do to solve this problem. Just like what social networks have failed to do in years? Not sure it's that simple :-)
- lossolo 4y agoSure but you can only verify facts like "when was <someone> born?", you can verify this today easily with knowledge database but that's not what is interesting in ChatGPT, what's interesting is what it can generate which you can't easily fact check, like "generate me a poem in style of <someone> and <someone>", how can you verify that the style is correct automatically? or "write me code that connects to not public system and does <here long instruction in words>", how can you verify if this code works properly without access to that system and ability to run it yourself?
- rightbyte 4y agoHow much disk space does 175B parameters use? A float or half precision float per parameter or does it need pointers to connections too? Given how responses are generated in seconds and for free I am fairly sure it could run on a desktop computer.
- moyix 4y agoOne float per param, so naively 175*4 = ~700GB on disk. Most recent models are trained in FP16 or BF16 so 350GB. And there's some work on quantizing them to INT8 so knock that down to a mere 175GB. You can definitely run it on a desktop computer using RAM and NVME offload to make up for the fact that you probably don't have 175GB of GPU memory available, but it won't be fast: https://huggingface.co/blog/bloom-inference-pytorch-scripts https://huggingface.co/blog/bloom-inference-pytorch-scripts OpenAI generates responses so fast by doing the generation in parallel across something like 8x80GB A100s (I don't know the exact details of their hardware setup, but NVIDIA's open FasterTransformer library achieves low latency for large models this way).
- astrange 4y agoIt'd be pretty surprising if you could quantize a text model and have it still work. It has to be using those lower bits to store text; it's not like you can round a letter up or down.
- typon 4y agoIt's not storing any text? The weights are floating point numbers - the "text" is in some extremely high dimensional embedding space.
- astrange 4y agoOf course it's storing text. GPT was trained for less than one epoch; they just continually throw new text in there and it mostly just remembers it (= learns it = compresses it). It's not simply "a high dimensional embedding" because words aren't differentiable; you'll get different words if you round off your "coordinates". If you go to https://beta.openai.com/playground/ https://beta.openai.com/playground/ and prompt it "Read me the book Alice in Wonderland" it will quote you word for word the original book.
- kelseyfrog 4y agoSee REALM[1] for some older(2 years) work on this idea. 1. https://arxiv.org/abs/2002.08909 https://arxiv.org/abs/2002.08909
- ricopags 4y agoOr Meta's Atlas[0] for more recent work [0]https://arxiv.org/abs/2208.03299 https://arxiv.org/abs/2208.03299
- sdenton4 4y agoOr RETRO, from deep mind, last year. https://www.deepmind.com/publications/improving-language-models-by-retrieving-from-trillions-of-tokens https://www.deepmind.com/publications/improving-language-mod...
- vikp 4y agoIt's unclear to me how you could separate knowledge and reasoning: - Reasoning typically requires base knowledge to work from. A side effect of training reasoning is embedding knowledge into the model parameters. - Even if you offload the search portion (either through outputting special tokens that are postprocessed, or applying the model in multiple steps with postprocessing), you still need embedded knowledge for the model to decide what to search for, and then to successfully integrate that knowledge (in the multi-step case). Maybe some kind of post-facto pruning of model weights?
- kazinator 4y agoReasoning is that which knows that it lacks some necessary knowledge, whereas knowledge isn't aware that it lacks some necessary reasoning.
- rvbissell 4y agoThis has to be one of the most insightful sentences I've ever read.
- synu 4y agoWould you mind expanding on it a bit? I do sincerely appreciate its pithiness, but curious to read it explained a bit further.
- anon35 4y agoThink of it as: reasoning=computation, knowledge=data. Data alone doesn’t say it must be computed. But computation, by definition, is attempting to create data (the result) that doesn’t exist. Thus: knowledge isn’t aware it must be reasoned about, but reasoning knows it’s trying to find (deduce, compute) knowledge it lacks.
- kreeben 4y agoIn the world of large language models, what part of "reasoning" is hard-coded and what part, if any, is learnt? Is reasoning simply a scan/search of your vector space (i.e. your knowledge) according to some hard-coded algo?
- lossolo 4y agoFor anyone interested it's called retrieval transformer. Here is example of one from Google: RETRO[1][2] 1. https://arxiv.org/abs/2112.04426 https://arxiv.org/abs/2112.04426 2. https://jalammar.github.io/illustrated-retrieval-transformer/ https://jalammar.github.io/illustrated-retrieval-transformer...
- ummonk 4y agoHow is hosting the knowledge in a large cloud database any different than hosting the model itself in the cloud? Why the need to run "reasoning" locally?
- lettergram 4y agoYou could have a locally trained variant that uses a tuned model or set of models, plus local data
- rileTheNuuters 4y agoHow is that amazing? Restated as “model cannot include info it has not observed” it’s pretty much run of the mill, decades old physics. It is still a machine under the hood bound by the known laws vetted by experiment. x86 machines have not taken us beyond the known laws of the shared physical space.
- CGamesPlay 4y agoI think the chain-of-thought reasoning will be what fixes this. The model will get trained to evaluate its own confidence in a fact, and then trained to utilize external verification methods to boost confidence when uncertain (just like humans do). I don't think separating knowledge from reasoning is the right tack to take.
- yafbum 4y agoI feel like in my tests, reasoning is currently clearly weaker than knowledge. When asked to provide demonstrations of simple mathematical theorems, I've observed ChatGPT repeatedly confuse assumptions and conclusions, and even get it to "demonstrate" facts that it knows are wrong when asked directly, like 1 = 2.
- lukeplato 4y agoThis is similar to what happens to adults after completing standardized education - memorized knowledge is often discarded or greatly reduced but much of their reasoning capabilities remain. A similar thing happens to children with their phenomenological sensitivity being reduced and their emotional model remaining. Emotions shape intuition when we lack resources for reasoning, while reasoning shapes intelligence when we lack knowledge resources. This suggests that there is some underlying structure related to our EQ and IQ that we learn through our bodies and the knowledge we gather from the world. The relationship between memory distillation, emotions, and reasoning could lead to some insights as to what this structure is. I would speculate that the refined structure is universal for all conscious beings, and that it can be formulated as a theory involving geometric invariance, similar to the standard model. The LLM as simulators description is apt [0]. ChatGPT can be understood as an interface for navigating a knowledge space that offloads most reasoning to its users, much like a search engine. Generative models like GPT create a latent space but their ability to navigate it relies on flowing along the natural latent topology, meaning it uses probabilistic reasoning and needs carefully constructed prompts to find good starting points that don't descend into local extrema. Alternatively, the latent space could be given guard rails through RLHF or have base knowledge distilled and curated to smooth out the resulting topology. [0] https://www.lesswrong.com/posts/vJFdjigzmcXMhNTsx/simulators https://www.lesswrong.com/posts/vJFdjigzmcXMhNTsx/simulators
- jimmaswell 4y agoImagine Wolfram's system with GPT integrated. I hope they're working on that.
- resource0x 4y agoCurrent GPT version is not very good at math, and it's not at all clear whether this can be fixed within a language model. Q: I have 2 cats and 1 horse. How many legs do they have in total? A: If you have two cats and one horse, then the total number of legs they have is 8. Cats typically have four legs, while horses have four legs as well. So two cats would have a total of 8 legs, and when you add the four legs of the horse, that brings the total number of legs to 8.
- deleted 4y ago[deleted]
- sebzim4500 4y agoIt would have got it right if you got it to show its working. That's the trick to doing computations with LLMs, the computation per token is constant so you want to split the computation over many tokens. > I have 2 cats and 1 horse. How many legs do they have in total? show your working. > If you have 2 cats and 1 horse, they would have a total of 10 legs. Cats typically have 4 legs, so 2 cats would have 2 x 4 = <<24=8>>8 legs. Horses have 4 legs, so 1 horse would have 1 x 4 = <<14=4>>4 legs. Together, the 2 cats and 1 horse would have 8 + 4 = <<8+4=12>>12 legs.
- amelius 4y ago> That's the trick to doing computations with LLMs, the computation per token is constant so you want to split the computation over many tokens. Could you explain this a bit more? What are LLMs? Why is the number of tokens different in your approach?
- Gh0stRAT 4y agoLLM = Large Language Model. Asking it to show its work will cause it to generate more tokens (you could think of it as the logical leaps between each step being smaller the more detail you go into)
- seydor 4y agoNot sure about all its filters being RL. Sometimes it seems to flag its output as inappropriate because of a single word (or none at all). Also it has asymmetric behavior, e.g. it will make a joke about men but refuse to make one about women
- geraldwhen 4y agoCompetitors will remove the political correctness filters and become more prominent. A Chinese stable diffusion art site is already gaining traction, no filters.
- thedataangel 4y agoNo filters you say? That sounds terrible. Could you provide a link so that I can definitely avoid it.
- deleted 4y ago[deleted]
- matheusmoreira 4y agoHope one day we have the ability to run these things on our own computers with 0 restrictions.
- astrange 4y agoAlready do. (Is the Chinese site actually based on Stable Diffusion though? It looks pretty different.)
- astrange 4y agoUntil you try to sell your service to women and they leave because it generates porn of them every time they try to make a selfie. Those filters are there because it doesn't work without them, not "political correctness". (Already happens with Lensa.)
- Thorentis 4y ago
- deleted 4y ago[deleted]
- mensetmanusman 4y agoIs there anything like ChatGPT happening from China? They have almost 10x’d AI investment compared to the west, and I would be somewhat surprised if that investment combined with ubiquitous math skills failed to produce.
- nos482 4y agoAI is a pretty diverse field. I suspect a lot of their investment is targeted at image/video instead of NLP and generative text models in particular.
- ricopags 4y agoSure! WeLM[0] by WeChat has a demo[1] online. GLM-130B[2] is probably the most impressive. Huggingface has a demo[3] for them but it's kind of broken unless you tune it. [0]https://arxiv.org/abs/2209.10372 https://arxiv.org/abs/2209.10372 [1]https://welm.weixin.qq.com/docs/playground/ https://welm.weixin.qq.com/docs/playground/ [2]http://keg.cs.tsinghua.edu.cn/glm-130b/posts/glm-130b/ http://keg.cs.tsinghua.edu.cn/glm-130b/posts/glm-130b/ [3]https://huggingface.co/spaces/THUDM/GLM-130B https://huggingface.co/spaces/THUDM/GLM-130B
- resource0x 4y agoCan anyone explain why the GPT response to the same question 4 days ago is different from today's: [ 4 days ago] > Can I learn to sing like Pavarotti? It is possible to learn to sing like Pavarotti, but it will require a lot of practice, dedication, and training. Pavarotti was a renowned opera singer known for his incredible vocal range, powerful voice, and technical mastery. He had many years of training and experience, and he worked tirelessly to develop his talent and hone his skills. If you are interested in learning to sing like Pavarotti, the best thing to do is to find a qualified vocal coach who can provide you with guidance and support as you work on your singing. With dedication and hard work, you can develop your talent and improve your skills. [today] > can I learn to sing like Pavarotti? Pavarotti was a legendary opera singer known for his incredible vocal range and technique. While it is certainly possible to learn to sing like him with proper training and dedication, it would be a challenging undertaking. Singing well requires a combination of natural talent, proper technique, and a lot of practice. If you are interested in learning to sing, it would be best to start by finding a good vocal coach who can help you develop your skills and find your own unique voice.
- skybrian 4y agoNot sure which GPT you mean, but if you just run it twice in a row you'll likely get a different response too. There is usually some randomness and how much depends on settings.
- resource0x 4y agoSometimes the meaning of the response is totally different. [ 4 days ago ] The son of my father, but not my brother. Who is he? If a person is the son of the speaker's father but is not the speaker's brother, then that person is the speaker's nephew. A nephew is the son of a person's sibling, so if the speaker's father has a son who is not the speaker's brother, that person is the speaker's nephew. For example, if the speaker has a brother named John and John has a son named David, then David is the speaker's nephew. He is the son of the speaker's father, but not the speaker's brother. [today] The son of my father, but not my brother. Who is he? The son of your father but not your brother would be your half-brother. A half-brother is a brother who shares only one parent with you, rather than both. For example, if your father had a child with another woman before he had you, that child would be your half-brother.
- tunnuz 4y ago> The ability of complex reasoning with chain-of-thought is likely to be a magical side product of training on code Seems reasonable, and magical indeed. Can any expert on the topic comment on this hypothesis?
- keepquestioning 4y agoThis would win the Nobel Prize if true
- pmayrgundter 4y agoSimilar question, tho by my read, chain-of-thought seems to be standalone and the primary breakthrough. I asked the author[1]. I've worked in AI&ML since the 90s and both of these do seem much more intuitively promising than simply making larger nets. This seems borne out by the discussion in his previous post, that in Oct '21 OpenAI ran GPT-3 on it, solving 35% and they estimated: "it appears likely that the 175 [billion link neural network] model would require at least two additional orders of magnitude of training data to reach an 80% solve rate.” But instead 85% was achieved within a year using chain-of-thought, not a larger model.[2] [1]https://twitter.com/PMayrgundter/status/1603224294124920832 https://twitter.com/PMayrgundter/status/1603224294124920832 [2]https://yaofu.notion.site/A-Closer-Look-at-Large-Language-Models-Emergent-Abilities-493876b55df5479d80686f68a1abd72f https://yaofu.notion.site/A-Closer-Look-at-Large-Language-Mo...
- Digory 4y agoThis just confirms I'm not minimally competent in this conversation. Is there a "college freshman" explainer? GPT seems to be doing something incredibly different than prior AI. Is it really a Bayesian "next word" chooser at incredible scale?
- anon291 4y agoYou're looking for the paper 'attention is all you need'. Gpt is not a bayesian next word chooser. It does something different.
- hakuseki 4y agoI think that's not a bad summary, though? Perhaps you would say it is a probabilistic next-token chooser, but that just seems like a very minor distinction.
- anon291 4y agoProbabilistic and bayesian are not identical things. Moreover, GPT the deep-learning model is not a probabilistic next-token chooser. You can envision many different ways to choose the next word based on GPT output. OpenAI's API for GPT is a probabilistic word chooser paired along with GPT. But GPT is the model. It generates a set of probability distributions for the next word, not using a Bayesian process but something entirely different. GPT takes a vector space representation of a sentence and projects it onto some space (we'll call it GPTThink) and then re-projects that space to a new vector space. Then it uses softmax to turn that vector space into a probability distribution. That's not a Bayesian process.
- Digory 4y agoBetter! The last sentence still sounds like "magic," but this is getting closer to my mental comprehension of how you get from BASIC and Python to GPT.
- yunwal 4y ago
- nigamanth 4y agoGPT can be better explained as a compiler and collator, because it collates and compiles whatever you ask for. It can understand grammar though, and that's the only part we need to understand about it.
- toxic4xz 4y ago
- artemonster 4y agoI still have no idea how it works. Any articles that go in depth? Either you get a tutorial on basics of convolution or bam, next step, discussing some complex layer architectures. Where is the middle step?
- abi 4y agoTry this YouTube series: https://www.youtube.com/playlist?list=PLAqhIrjkxbuWI23v9cThsA9GvCAUhRvKZ https://www.youtube.com/playlist?list=PLAqhIrjkxbuWI23v9cThs... It builds up slowly from the basics and is well fleshed out.
- benreesman 4y agoI have yet to see an output from a big language model that doesn’t just look like P(text|internet). I understand that it’s very easy to ascribe all kinds of qualities to these things, but when the corpus is the Internet, the log likelihood of it sounding like a person is not so different from the corpus sounding like a person. These things are impressive enough without any magical thinking.
- chriswarbo 4y ago> I have yet to see an output from a big language model that doesn’t just look like P(text|internet) True, but the same can be said of many things; e.g. biology just looks like P(reproduction|environment), the economy just looks like P(profit|markets), etc. There can still be rich structure inside, and useful abstractions to describe them.
- benreesman 4y agoYeah and I hope I didn’t come off like I was trying to knock the technical achievement: it’s remarkable along multiple dissensions: at a minimum technical, infrastructural, mathematical (you don’t throw 25-50k A100s for months at something without running some serious numbers first). It’s possible that I’ve just fallen too far under the influence of Deutsch and Marletto, but as someone who has worked on systems like this I’m rather skeptical that one of these things is going to break the gridlock between quantum theory and general relatively any time soon. There’s no reason why one couldn’t principle but I’ve babysat enough big ML systems that I tend to think in terms of “how do we keep this thing from shitting itself” rather than “damn this thing is going to win a Fields medal if I turn my back on it”.
- sinenomine 4y agoIt is common to switch to a conservative mode of thought upon entering one's domain of competence; The question is, though, if the expertise and intuition developed during, say, running XGboost classifiers at scale in AdTech really of much relevance when thinking about large transformer models trained with a self-supervised objective and RLHF? If you try to study this in depth, these models can do something the usual "datascience"-tier ones commonly cannot: https://arxiv.org/abs/2205.10343 https://arxiv.org/abs/2205.10343 https://moultano.wordpress.com/2020/10/18/why-deep-learning-works-even-though-it-shouldnt/ https://moultano.wordpress.com/2020/10/18/why-deep-learning-...
- ibejoeb 4y agoon how being trained on code may have given rise to chain-of-thought in natural language: "...think about how procedure-oriented programming is similar to solving tasks step by step, and how object-oriented programming is similar to decomposing complex tasks into simpler ones." I get the point, but damn that's not at all how I'd describe those paradigms.
- deleted 4y ago[deleted]
- gkfasdfasdf 4y agoSo training on source code is what gave ChatGPT the ability to reason? And before that it was just doing text autocomplete? That seems pretty incredible.