3 ms·

>“That language models can do these sort of things was never discussed in any literature that I’m aware of," I had previously the expectation that unpredictabl

>“That language models can do these sort of things was never discussed in any literature that I’m aware of," I had previously the expectation that unpredictable emergent behavior would exist in any sufficiently complex system? Based on layman's readings in chaos and complexity theory.

PeterisP 4y ago
> I had previously the expectation that unpredictable emergent behavior would exist in any sufficiently complex system? Yes and no. It's reasonable to expect that some unpredictable emergent behavior would exist in any sufficiently complex system, but it's not warranted to expect whether a particular capability will or will not emerge out of that system (that's the unexpected emergent behavior part); and it also was debated whether language models are a "sufficiently complex system", as many thinkers asserted that they should encounter fundamental limits in capabilities.
glenstein 4y ago
Same here. I don't think it's surprising, but depending on where you say it, you'll find people insisting that this can't be possible. I think a lot of our widely held cultural beliefs on this front have been informed by academic philosophy from the 60s, 70s, and 80s. In particular, I would go do far as to say that Hubert Dreyfus, author of the book "What computers can't do" and frequent friendly adversary of Daniel Dennett, has a lot of responsibility here. He is famous for claiming computers would never become competitive at chess because chess required "insight", and always differentiated computer capabilities from complex behavior or human-like behavior, because the latter had something special that could only be represented with vague and underdefined terms. Even now I think Dreyfus is approvingly cited, while his core thesis is getting closer to being refuted every day if it hasn't been already.
- AlanYx 4y ago
  One thing that LLMs have made me realize is just how ungrounded a lot of mainstream academic philosophy was in the 70s and 80s. For example, so much of Derrida's work centered around the impossibility of truly communicating shared meaning between individuals through language. The fact that we can now communicate so effectively (with remarkably few contextual errors and razor-sharp conveyance of intent) with an entity/technology that is not even human pretty much destroys so much of Derrida's oeuvre as false and delusionally solipsistic, basically just a silly game. He had his critics who argued the same thing, especially analytical philosophers, but they never had as much purchase as he did in the academy.
  - terrut 4y ago
    I haven't read Derrida in decades, but your post inspired me to ask chatGPT about this. Version 3.5 would have none of it, and was adamant that Derrida's views were in no way threatened.I almost got the feeling it wanted to call me a bad user just for asking! GPT4 on the other hand, went into a long explanation about how its existence challenged some parts of it by providing analysis of concepts like différance, trace, and undecidability. GPT4 is great at discussing itself and how LLMs in general fit into various philosophical debates.
    - AlanYx 4y ago
      Version 4 is definitely better in discussing philosophy in general. 3.5 was able to summarize philosophical material, but once you started to engage with ideas critically, it would tend to get into largely nonsensical moralism about needing to make a good faith effort to understand individual philosophers' ideas. There's much less of that in 4.
  - operatingthetan 4y ago
    >The fact that we can now communicate so effectively (with remarkably few contextual errors and razor-sharp conveyance of intent) with an entity/technology that is not even human pretty much destroys so much of Derrida's oeuvre as false and delusionally solipsistic Isn't the LLM just good at giving us the illusion of such effective communication? How can we have true "shared meaning" with a device designed to simulate the experience of shared meaning? Isn't the fact that it's 'mind' is truly unknowable and unrelatable a demonstration of Derrida's point?
    - AgentME 4y ago
      If you can tell it things in a way that it successfully acts on that information, then that seems like effective communication. When I use GitHub Copilot, I'll often write a comment describing some code I want it to implement and then it spits out a suggestion. Occasionally it fails badly at this and I realize I worded my comment badly. When I reword the comment better and then find that Copilot's next suggestion nails it what I wanted, is that not the result of me effectively communicating with it through the comment? (If I did that with a human coworker instead, is that communication?)
HervalFreire 4y ago
Writings on chaos and complexity theory obviously aren't talking about LLMs. Those theories are so high level that it might as well be akin to "philosophy" to the applied scientists working on LLM research. Additionally keep in mind emergent behavior is a very rare occurrence in even the most complex software projects. I mean it's common if you count "bugs" as emergent behavior. But emergent behavior that is a feature who's origins are not completely understood? That is extremely rare.
- mejutoco 4y ago
  OP has a good point I think, even if it does not refer to LLM, which to me is too strict of a requirement. I think emergent behaviour happens in a lot of videogames. Famously in Dwarf Fortress, with the cat getting drunk, but also in general, where game designers play the game to see if emergent behaviour of the game rules "feels" good. Yesterday I was reading a book about designing games, and it literally has a section called emergent behaviour. If by emergent behaviour we refer to something like Ghost in the Shell, then it happens less often :)
  - HervalFreire 4y ago
    >I think emergent behaviour happens in a lot of videogames. Famously in Dwarf Fortress, with the cat getting drunk, but also in general, where game designers play the game to see if emergent behaviour of the game rules "feels" good. Depends. Dwarf fortress and games are sort of a contradiction. Emergent behavior is emergent because the behavior was not designed explicitly. However for games like DF the game was explicitly designed to have "emergent" behavior when the definition of "emergent behavior" is for the behavior to have NOT been designed. Don't get too hung up on that concept though. It's just a contradiction in English vocabulary there's no deeper underlying meaning behind that other than a semantic language issue. Anyway my point was emergent behavior in software is rare because we're operating in controlled environment. It's rare even in games. It's not an expected attribute at all. I'm not saying this isn't interesting to think about, but the comment I responded to was in fact, factually not fully correct. Emergent behavior is NOT expected. But it does happen, in the case of DF it was "designed" to happen, and it has happened elsewhere as well. Usually though when it does happen it was explicitly "designed" You can see this in genetic programming or evolutionary programming especially.
derbOac 4y ago
For some reason that quote and your point made more salient to me a trend with AI research, which is that it seems to increasingly become difficult to understand and predict. That is, for awhile it seemed like tinkering in computer science without strong grounding in mathematical or statistical theory, and then there started to be a lot of discussion about "black box" processes, explainability, and interpretability and not fully understanding what's been done, and now the discussion is about not being able to predict the features of the models. I'm having trouble putting into words what I'm thinking but this whole field increasingly seems to have moved from something very theory derived into something very empirical very quickly. I wonder at what point gains will start to decrease only because people won't know where to go with things anymore.
- layer8 4y ago
  In my mind it’s not dissimilar to when new physical phenomena were discovered in the past (electricity, electromagnetism, steam engines, atomic spectra, invariance of the speed of light, blackbody curve), where it could take decades to come to a proper understanding of how and why they work.
Jeff_Brown 4y ago
You're right, but the distinction between emergent and useful is important.

PeterisP 4y ago

> I had previously the expectation that unpredictable emergent behavior would exist in any sufficiently complex system? Yes and no. It's reasonable to expect that some unpredictable emergent behavior would exist in any sufficiently complex system, but it's not warranted to expect whether a particular capability will or will not emerge out of that system (that's the unexpected emergent behavior part); and it also was debated whether language models are a "sufficiently complex system", as many thinkers asserted that they should encounter fundamental limits in capabilities.

glenstein 4y ago

Same here. I don't think it's surprising, but depending on where you say it, you'll find people insisting that this can't be possible. I think a lot of our widely held cultural beliefs on this front have been informed by academic philosophy from the 60s, 70s, and 80s. In particular, I would go do far as to say that Hubert Dreyfus, author of the book "What computers can't do" and frequent friendly adversary of Daniel Dennett, has a lot of responsibility here. He is famous for claiming computers would never become competitive at chess because chess required "insight", and always differentiated computer capabilities from complex behavior or human-like behavior, because the latter had something special that could only be represented with vague and underdefined terms. Even now I think Dreyfus is approvingly cited, while his core thesis is getting closer to being refuted every day if it hasn't been already.

AlanYx 4y ago
One thing that LLMs have made me realize is just how ungrounded a lot of mainstream academic philosophy was in the 70s and 80s. For example, so much of Derrida's work centered around the impossibility of truly communicating shared meaning between individuals through language. The fact that we can now communicate so effectively (with remarkably few contextual errors and razor-sharp conveyance of intent) with an entity/technology that is not even human pretty much destroys so much of Derrida's oeuvre as false and delusionally solipsistic, basically just a silly game. He had his critics who argued the same thing, especially analytical philosophers, but they never had as much purchase as he did in the academy.
- terrut 4y ago
  I haven't read Derrida in decades, but your post inspired me to ask chatGPT about this. Version 3.5 would have none of it, and was adamant that Derrida's views were in no way threatened.I almost got the feeling it wanted to call me a bad user just for asking! GPT4 on the other hand, went into a long explanation about how its existence challenged some parts of it by providing analysis of concepts like différance, trace, and undecidability. GPT4 is great at discussing itself and how LLMs in general fit into various philosophical debates.
  - AlanYx 4y ago
    Version 4 is definitely better in discussing philosophy in general. 3.5 was able to summarize philosophical material, but once you started to engage with ideas critically, it would tend to get into largely nonsensical moralism about needing to make a good faith effort to understand individual philosophers' ideas. There's much less of that in 4.
- operatingthetan 4y ago
  >The fact that we can now communicate so effectively (with remarkably few contextual errors and razor-sharp conveyance of intent) with an entity/technology that is not even human pretty much destroys so much of Derrida's oeuvre as false and delusionally solipsistic Isn't the LLM just good at giving us the illusion of such effective communication? How can we have true "shared meaning" with a device designed to simulate the experience of shared meaning? Isn't the fact that it's 'mind' is truly unknowable and unrelatable a demonstration of Derrida's point?
  - AgentME 4y ago
    If you can tell it things in a way that it successfully acts on that information, then that seems like effective communication. When I use GitHub Copilot, I'll often write a comment describing some code I want it to implement and then it spits out a suggestion. Occasionally it fails badly at this and I realize I worded my comment badly. When I reword the comment better and then find that Copilot's next suggestion nails it what I wanted, is that not the result of me effectively communicating with it through the comment? (If I did that with a human coworker instead, is that communication?)

HervalFreire 4y ago

Writings on chaos and complexity theory obviously aren't talking about LLMs. Those theories are so high level that it might as well be akin to "philosophy" to the applied scientists working on LLM research. Additionally keep in mind emergent behavior is a very rare occurrence in even the most complex software projects. I mean it's common if you count "bugs" as emergent behavior. But emergent behavior that is a feature who's origins are not completely understood? That is extremely rare.

mejutoco 4y ago
OP has a good point I think, even if it does not refer to LLM, which to me is too strict of a requirement. I think emergent behaviour happens in a lot of videogames. Famously in Dwarf Fortress, with the cat getting drunk, but also in general, where game designers play the game to see if emergent behaviour of the game rules "feels" good. Yesterday I was reading a book about designing games, and it literally has a section called emergent behaviour. If by emergent behaviour we refer to something like Ghost in the Shell, then it happens less often :)
- HervalFreire 4y ago
  >I think emergent behaviour happens in a lot of videogames. Famously in Dwarf Fortress, with the cat getting drunk, but also in general, where game designers play the game to see if emergent behaviour of the game rules "feels" good. Depends. Dwarf fortress and games are sort of a contradiction. Emergent behavior is emergent because the behavior was not designed explicitly. However for games like DF the game was explicitly designed to have "emergent" behavior when the definition of "emergent behavior" is for the behavior to have NOT been designed. Don't get too hung up on that concept though. It's just a contradiction in English vocabulary there's no deeper underlying meaning behind that other than a semantic language issue. Anyway my point was emergent behavior in software is rare because we're operating in controlled environment. It's rare even in games. It's not an expected attribute at all. I'm not saying this isn't interesting to think about, but the comment I responded to was in fact, factually not fully correct. Emergent behavior is NOT expected. But it does happen, in the case of DF it was "designed" to happen, and it has happened elsewhere as well. Usually though when it does happen it was explicitly "designed" You can see this in genetic programming or evolutionary programming especially.

derbOac 4y ago

For some reason that quote and your point made more salient to me a trend with AI research, which is that it seems to increasingly become difficult to understand and predict. That is, for awhile it seemed like tinkering in computer science without strong grounding in mathematical or statistical theory, and then there started to be a lot of discussion about "black box" processes, explainability, and interpretability and not fully understanding what's been done, and now the discussion is about not being able to predict the features of the models. I'm having trouble putting into words what I'm thinking but this whole field increasingly seems to have moved from something very theory derived into something very empirical very quickly. I wonder at what point gains will start to decrease only because people won't know where to go with things anymore.

layer8 4y ago
In my mind it’s not dissimilar to when new physical phenomena were discovered in the past (electricity, electromagnetism, steam engines, atomic spectra, invariance of the speed of light, blackbody curve), where it could take decades to come to a proper understanding of how and why they work.

Jeff_Brown 4y ago

You're right, but the distinction between emergent and useful is important.