4 ms·

> I meant that nobody knows. I know what you mean, and I'm saying you're wrong. > This is a basic problem of machine learning (explainability) and we're nowhe

> I meant that nobody knows. I know what you mean, and I'm saying you're wrong. > This is a basic problem of machine learning (explainability) and we're nowhere understanding how or why LLMs have the emergent capabilities that they do (we're just getting started with that research) Sorry, which emergent capabilities are you talking about? You're trying to slip in that there even are emergent capabilities, when in fact there's literally nothing unexpected which has emerged. Given how LLMs work, literally everything they do is exactly what we'd expect them to do. The problem of explainability is twofold: 1) Connecting the input data with the output data, so a human can see what input data produced the output data. This is actually pretty easy to do naively, but it's hard to do performantly. 2) Presenting that data in a way that is meaningful to a non-technical user. Note that both of these are just about being able to audit which input data produced which output data. Nothing here is about how the output data was produced from the input data: we understand that pretty clearly, it's just that the process isn't reversible. > Gradient descent training is just a little more efficient method of essentially permuting program code at random until you get something that passes all cases. Doesn't mean you understand the final program at all. There are some things we can't understand about the final program, but there are still pretty understandable bounds on what the final program can do. For example, it cannot produce something from nothing: any output the program produces must strictly be a combination of its inputs. From this we can tell that no program produced is capable of originality. Likewise, I'm not even sure what argument you could possibly make that it would be capable of preference: any preferences you might perceive are obviously representative of the preferences of the humans who created the input data. As I've said elsewhere, just because you use a program to take the average of billions of integers which you can't possibly understand all of, doesn't mean you don't understand what the "average()" function does. Obviously LLMs are a lot more complex than an average, but they aren't beyond human understanding.

spion 4y ago
> Sorry, which emergent capabilities are you talking about? You're trying to slip in that there even are emergent capabilities, when in fact there's literally nothing unexpected which has emerged. Given how LLMs work, literally everything they do is exactly what we'd expect them to do. https://openreview.net/pdf?id=yzkSU5zdwD https://openreview.net/pdf?id=yzkSU5zdwD FWIW, none of the capabilities of LLMs are expected, but the above are not expected in the sense that they suddenly emerge beyond a certain scale in a way that couldn't be extrapolated by looking at the smaller models. > For example, it cannot produce something from nothing: any output the program produces must strictly be a combination of its inputs. From this we can tell that no program produced is capable of originality. Can you explain what you mean by "strictly a combination of its inputs"? I've personally implemented a few (non-language) NN prediction models and they definitely extrapolate, someitmes in quite funny (but still essentially accurate) ways when given previously unseen (and ridiculous) inputs.
- kerkeslager 4y ago
  > https://openreview.net/pdf?id=yzkSU5zdwD https://openreview.net/pdf?id=yzkSU5zdwD That article defines "emergent" as: "An ability is emergent if it is not present in smaller models but is present in larger models." That doesn't make it unexpected, it just makes it a product of scale. The article isn't claiming what you think it's claiming. The article is, frankly, not very interesting. You change the inputs and get different outputs? What a surprise! > Can you explain what you mean by "strictly a combination of its inputs"? I'm saying it can't do anything with data it doesn't have. For example: I prompted it "Give me a one-syllable portmanteau of "bridge" and "dam"." and it returned ""Bridam" is a one-syllable portmanteau of "bridge" and "dam".". It doesn't understand portmanteaus and it doesn't understand syllables, it just sees that when people in its dataset talk about portmanteaus in this pattern, they mash together a word. It has the letters, so it gets right that "bridam" is a portmanteau of "bridge" and "dam", but it can't comply with the "one-syllable" aspect. If you ask it for the number of syllables in a word that is in its dataset, it usually gets it right, because people talk about how many syllables are in words. In fact, if you ask it how many syllables are in the word "bridam" it correctly says two, because the pattern is close enough to what's in its dataset. But you can immediately ask it to create a one-syllable portmanteau and it will again return "bridam", because it's not holding a connected thought, it's just continuing to match the pattern in its data. It simply doesn't have actual syllable data. You'll even get responses such as, "A one-syllable portmanteau of "bridge" and "dam" could be "bridam" (pronounced as "brih-dam")." Even as it produces a syllable breakdown with two syllables due to its pattern-matching, it still produces a two syllable portmanteau while claiming it's one syllable. A human child, if they know what a portmanteau is, can easily come up with a few options such as "bram", or "dadge". > I've personally implemented a few (non-language) NN prediction models and they definitely extrapolate, someitmes in quite funny (but still essentially accurate) ways when given previously unseen (and ridiculous) inputs. The form of extrapolation these systems are capable of isn't so much extrapolation, as matching that the response pattern should be longer and trying to fill it in with existing data. It's a very specific kind of extrapolation and again, not unexpected. EDIT: If you want to see the portmanteau thing in action, it's easy to see how the pattern-matching is applying to this: me: Give me a one-syllable portmanteau of "cap" and "plant"? ChatGPT: Caplant me: Give me a one-syllable portmanteau of "bat" and "tar" ChatGPT: Batar me: Give me a one-syllable portmanteau of "fear" and "red" ChatGPT: Ferred Pick pretty much any two one-syllable words where the first ends with the same letter as the second begins with, and it will go for a two-syllable portmanteau.
zestyping 4y ago
Here's what it means to understand how something works. If you can accurately predict what _change_ you need to make to its workings in order to achieve a desired _change_ in the resulting behaviour, then you know how it works. The more reliably and precisely you can predict the impact of various changes, the better your understanding. I know how a bicycle works. I can prove that I have a very basic level of understanding, for example, by showing you that if I want the bicycle to go faster, I can pedal harder; if I want the bicycle to slow down, I can squeeze the brakes. I can prove a greater level of understanding by showing you that I can increase efficiency by increasing the tire pressure, or showing you that it's impossible to make a left turn (https://www.youtube.com/watch?v=llRkf1fnNDM https://www.youtube.com/watch?v=llRkf1fnNDM) unless I first start by turning right. That's what it means to explain how a bicycle works. (Notice that saying the bicycle is made of atoms does not help you do any of this.) I don't think you can show me that kind of understanding of large language models. Which is to say, I don't think you can accurately predict what changes you need to make to the internal structure of an LLM to cause specific changes to the interesting high-level ("emergent") behaviours that people are seeing. That's what is meant by "nobody knows."
- kerkeslager 4y ago
  > Here's what it means to understand how something works. If you can accurately predict what _change_ you need to make to its workings in order to achieve a desired _change_ in the resulting behaviour, then you know how it works. The more reliably and precisely you can predict the impact of various changes, the better your understanding. For example, one might decide that you don't want your chat engine to return pornography. So you train your model to reject inappropriate requests. They did that. I'm not sure why you think that what you're saying doesn't apply to LLMs. > I don't think you can show me that kind of understanding of large language models. See above. > Which is to say, I don't think you can accurately predict what changes you need to make to the internal structure of an LLM to cause specific changes to the interesting high-level ("emergent") behaviours that people are seeing. What emergent behaviors? There's literally nothing I've seen any of these LLMs do that I would not expect from the inputs and design of the system. The behaviors aren't "emergent", they're exactly what we'd expect.

spion 4y ago

> Sorry, which emergent capabilities are you talking about? You're trying to slip in that there even are emergent capabilities, when in fact there's literally nothing unexpected which has emerged. Given how LLMs work, literally everything they do is exactly what we'd expect them to do. https://openreview.net/pdf?id=yzkSU5zdwD https://openreview.net/pdf?id=yzkSU5zdwD FWIW, none of the capabilities of LLMs are expected, but the above are not expected in the sense that they suddenly emerge beyond a certain scale in a way that couldn't be extrapolated by looking at the smaller models. > For example, it cannot produce something from nothing: any output the program produces must strictly be a combination of its inputs. From this we can tell that no program produced is capable of originality. Can you explain what you mean by "strictly a combination of its inputs"? I've personally implemented a few (non-language) NN prediction models and they definitely extrapolate, someitmes in quite funny (but still essentially accurate) ways when given previously unseen (and ridiculous) inputs.

kerkeslager 4y ago
> https://openreview.net/pdf?id=yzkSU5zdwD https://openreview.net/pdf?id=yzkSU5zdwD That article defines "emergent" as: "An ability is emergent if it is not present in smaller models but is present in larger models." That doesn't make it unexpected, it just makes it a product of scale. The article isn't claiming what you think it's claiming. The article is, frankly, not very interesting. You change the inputs and get different outputs? What a surprise! > Can you explain what you mean by "strictly a combination of its inputs"? I'm saying it can't do anything with data it doesn't have. For example: I prompted it "Give me a one-syllable portmanteau of "bridge" and "dam"." and it returned ""Bridam" is a one-syllable portmanteau of "bridge" and "dam".". It doesn't understand portmanteaus and it doesn't understand syllables, it just sees that when people in its dataset talk about portmanteaus in this pattern, they mash together a word. It has the letters, so it gets right that "bridam" is a portmanteau of "bridge" and "dam", but it can't comply with the "one-syllable" aspect. If you ask it for the number of syllables in a word that is in its dataset, it usually gets it right, because people talk about how many syllables are in words. In fact, if you ask it how many syllables are in the word "bridam" it correctly says two, because the pattern is close enough to what's in its dataset. But you can immediately ask it to create a one-syllable portmanteau and it will again return "bridam", because it's not holding a connected thought, it's just continuing to match the pattern in its data. It simply doesn't have actual syllable data. You'll even get responses such as, "A one-syllable portmanteau of "bridge" and "dam" could be "bridam" (pronounced as "brih-dam")." Even as it produces a syllable breakdown with two syllables due to its pattern-matching, it still produces a two syllable portmanteau while claiming it's one syllable. A human child, if they know what a portmanteau is, can easily come up with a few options such as "bram", or "dadge". > I've personally implemented a few (non-language) NN prediction models and they definitely extrapolate, someitmes in quite funny (but still essentially accurate) ways when given previously unseen (and ridiculous) inputs. The form of extrapolation these systems are capable of isn't so much extrapolation, as matching that the response pattern should be longer and trying to fill it in with existing data. It's a very specific kind of extrapolation and again, not unexpected. EDIT: If you want to see the portmanteau thing in action, it's easy to see how the pattern-matching is applying to this: me: Give me a one-syllable portmanteau of "cap" and "plant"? ChatGPT: Caplant me: Give me a one-syllable portmanteau of "bat" and "tar" ChatGPT: Batar me: Give me a one-syllable portmanteau of "fear" and "red" ChatGPT: Ferred Pick pretty much any two one-syllable words where the first ends with the same letter as the second begins with, and it will go for a two-syllable portmanteau.

zestyping 4y ago

Here's what it means to understand how something works. If you can accurately predict what _change_ you need to make to its workings in order to achieve a desired _change_ in the resulting behaviour, then you know how it works. The more reliably and precisely you can predict the impact of various changes, the better your understanding. I know how a bicycle works. I can prove that I have a very basic level of understanding, for example, by showing you that if I want the bicycle to go faster, I can pedal harder; if I want the bicycle to slow down, I can squeeze the brakes. I can prove a greater level of understanding by showing you that I can increase efficiency by increasing the tire pressure, or showing you that it's impossible to make a left turn (https://www.youtube.com/watch?v=llRkf1fnNDM https://www.youtube.com/watch?v=llRkf1fnNDM) unless I first start by turning right. That's what it means to explain how a bicycle works. (Notice that saying the bicycle is made of atoms does not help you do any of this.) I don't think you can show me that kind of understanding of large language models. Which is to say, I don't think you can accurately predict what changes you need to make to the internal structure of an LLM to cause specific changes to the interesting high-level ("emergent") behaviours that people are seeing. That's what is meant by "nobody knows."

kerkeslager 4y ago
> Here's what it means to understand how something works. If you can accurately predict what _change_ you need to make to its workings in order to achieve a desired _change_ in the resulting behaviour, then you know how it works. The more reliably and precisely you can predict the impact of various changes, the better your understanding. For example, one might decide that you don't want your chat engine to return pornography. So you train your model to reject inappropriate requests. They did that. I'm not sure why you think that what you're saying doesn't apply to LLMs. > I don't think you can show me that kind of understanding of large language models. See above. > Which is to say, I don't think you can accurately predict what changes you need to make to the internal structure of an LLM to cause specific changes to the interesting high-level ("emergent") behaviours that people are seeing. What emergent behaviors? There's literally nothing I've seen any of these LLMs do that I would not expect from the inputs and design of the system. The behaviors aren't "emergent", they're exactly what we'd expect.