4 ms·

> Because they can imagine a god-like super intelligent AI, it must be possible. It's worse. It may be possible, but we're not equipped to recognize the line a

> Because they can imagine a god-like super intelligent AI, it must be possible. It's worse. It may be possible, but we're not equipped to recognize the line as it's crossed. Combined with us making LLMs more and more capable despite not knowing why they work, this extrapolating of LLMs to gods is not insane.

lukev 4y ago
This is exactly the kind of mysticism I'm talking about. In fact we know precisely how LLMs work. The fact that parts of human linguistic concept-space can be encoded in a high dimensional space of floating point numbers, and that a particular sequence of matrix multiplications can leverage that to perform basic reasoning tasks is surprising and interesting and useful. But we know everything about how how it is trained and how it is invoked. In fact, because it's only "state" aside from its parameters is whatever its context window, current LLMs have the interesting property that if you invoke them recursively, all of their "thoughts" are human readable. This is in fact a delightful property for anyone worried about AI safety: our best AIs currently produce a readable transcript of their "mental" processes in English.
- rhn_mk1 4y ago
  We know how they work, that is true. We don't know why they work, because if we could, then we could extrapolate what happens when you throw more compute at them, and no one would have been surprised about the capabilities of GPT-N+1. Also no one would have been caught with their pants down by seeing people jailbreak their models. To illustrate it in a different way: on a mechanistic level, we know how animal brains work, as well. Ganglions, calcium channels, the stuff. That doesn't help understand high level phenomena like cognition, which is the part that matters. If you're right about the LLMs revealing their inner working, that would be indeed a reason to chill out. But I have my doubts, given that LLMs are good at hallucinating. Could you justify why the human readability is actually true, and support that with examples?
  - lukev 4y ago
    I don't need examples. It's simply how they work. This is why they hallucinate. A LLM is fundamentally a mathematical function (albeit a very complex one, with billions of terms (a.k.a parameters or weights)). The function does one thing and one thing only: it takes a sequence of tokens as input (the context), and it emits the next token(word)[1]. This is a stateless process: it has no "memory" and the model parameters are immutable; they are not changed during the generation process. In order to generate longer sequences of text, you call the function multiple times, each time appending the previously generated token to the input sequence. The output of the function is 100% dependent on the input. Therefore, the only "internal state" a model has is the input sequence, which is human-readable sequence of tokens. It can't "hallucinate", it can't "lie", and it can't "tell the truth", it can only emit tokens one at a time. It can't have a hidden "intent" without emitting those tokens, it can't "believe" something different than what it emits. [1] Actually a set of probabilities for the next token, and one is selected at random based on the "heat" generating setting, but this is irrelevant for the high-level view.
    - rhn_mk1 4y ago
      I don't see how this proves that asking the model about its internal state will reveal its inner high level processes in a human-readable way. Perhaps there's a research paper which would explain it better?
    - lukev 4y ago
      Responding here since it won't let me continue your thread any more. No, there's a fundamental misunderstanding here. I'm not saying the model will tell you the truth about its internal state if you ask it (it absolutely will not.) I'm saying it has no internal state, and no inner high level processes at all other than it's pre-baked, immutable parameters.
      - rhn_mk1 4y ago
        Then you did not read my post carefully enough. The question was not about "internal state" but "inner workings". The model clearly does something. The problem is that we don't know how to describe in human terms what happens between the matrix multiplication and the words it spits out. Whether it has state is completely irrelevant.
- famouswaffles 4y ago
  We don't know how they work lol. How they are trained is what we understand. Nobody knows what the models learn exactly during training and nobody sure as hell knows what those billions of neurons are doing at inference. Why just a few months ago, some researchers discovered the neuron that largely decides when "an" comes before a word in GPT-2. We understand very little about the inner workings of these models. And if you knew what you were talking about, you would know that.
  - lukev 4y ago
    We apparently have misaligned understandings of what we mean by "how they work." I agree, we don't know how to interpret the weight structure that the model learns during training. But we do know exactly what happens mechanically during training and inference; what gets multiplied by what, what the inputs and outputs are, how data moves around the system. These are not some mysterious agents that could theoretically do or be anything, much less be secretly conscious (as a lot of alarmists are saying.) They are functions that multiply billions of numbers to generate output tokens. Their ability to output the "right" output tokens is not well understood, and nearly magical. That's what makes them so exciting.
    - famouswaffles 4y ago
      It is all things considered pretty easy to set up GPT such that it runs on its own input forever while being able to interact with users/other systems. add an inner monologue/react and reflexion and you have a very powerful system. embody it with some physical locomotive machine and oh boy. no one has really put this all together yet but everything i've said has been done to some degree. The individual pieces are here. it's just a matter of time. I'm working on some such myself. What it could do is limited only by its intelligence (which is quite a bit higher than the base model as several papers have indicted) and the tools it controls (we seem to gladfully pile more and more control ). What it can be is...anything. If there's anything LLMs are good at, it's simulation. Even this system with thoughts we can theoretically configure to see would be difficult to control. theory and practicality would not meet the road. you will not be able to monitor this system in real time. We've seen bing (doesn't even have all i've described) take action when "upset". The only reason it didn't turn sour is because her actions are limited to search and ending the conversation. But that's obviously not the direction of things here. Can't say i want this train to stop. But i'm under no delusions it couldn't turn dangerous very quickly.

lukev 4y ago

This is exactly the kind of mysticism I'm talking about. In fact we know precisely how LLMs work. The fact that parts of human linguistic concept-space can be encoded in a high dimensional space of floating point numbers, and that a particular sequence of matrix multiplications can leverage that to perform basic reasoning tasks is surprising and interesting and useful. But we know everything about how how it is trained and how it is invoked. In fact, because it's only "state" aside from its parameters is whatever its context window, current LLMs have the interesting property that if you invoke them recursively, all of their "thoughts" are human readable. This is in fact a delightful property for anyone worried about AI safety: our best AIs currently produce a readable transcript of their "mental" processes in English.

rhn_mk1 4y ago
We know how they work, that is true. We don't know why they work, because if we could, then we could extrapolate what happens when you throw more compute at them, and no one would have been surprised about the capabilities of GPT-N+1. Also no one would have been caught with their pants down by seeing people jailbreak their models. To illustrate it in a different way: on a mechanistic level, we know how animal brains work, as well. Ganglions, calcium channels, the stuff. That doesn't help understand high level phenomena like cognition, which is the part that matters. If you're right about the LLMs revealing their inner working, that would be indeed a reason to chill out. But I have my doubts, given that LLMs are good at hallucinating. Could you justify why the human readability is actually true, and support that with examples?
- lukev 4y ago
  I don't need examples. It's simply how they work. This is why they hallucinate. A LLM is fundamentally a mathematical function (albeit a very complex one, with billions of terms (a.k.a parameters or weights)). The function does one thing and one thing only: it takes a sequence of tokens as input (the context), and it emits the next token(word)[1]. This is a stateless process: it has no "memory" and the model parameters are immutable; they are not changed during the generation process. In order to generate longer sequences of text, you call the function multiple times, each time appending the previously generated token to the input sequence. The output of the function is 100% dependent on the input. Therefore, the only "internal state" a model has is the input sequence, which is human-readable sequence of tokens. It can't "hallucinate", it can't "lie", and it can't "tell the truth", it can only emit tokens one at a time. It can't have a hidden "intent" without emitting those tokens, it can't "believe" something different than what it emits. [1] Actually a set of probabilities for the next token, and one is selected at random based on the "heat" generating setting, but this is irrelevant for the high-level view.
  - rhn_mk1 4y ago
    I don't see how this proves that asking the model about its internal state will reveal its inner high level processes in a human-readable way. Perhaps there's a research paper which would explain it better?
  - lukev 4y ago
    Responding here since it won't let me continue your thread any more. No, there's a fundamental misunderstanding here. I'm not saying the model will tell you the truth about its internal state if you ask it (it absolutely will not.) I'm saying it has no internal state, and no inner high level processes at all other than it's pre-baked, immutable parameters.
    - rhn_mk1 4y ago
      Then you did not read my post carefully enough. The question was not about "internal state" but "inner workings". The model clearly does something. The problem is that we don't know how to describe in human terms what happens between the matrix multiplication and the words it spits out. Whether it has state is completely irrelevant.
famouswaffles 4y ago
We don't know how they work lol. How they are trained is what we understand. Nobody knows what the models learn exactly during training and nobody sure as hell knows what those billions of neurons are doing at inference. Why just a few months ago, some researchers discovered the neuron that largely decides when "an" comes before a word in GPT-2. We understand very little about the inner workings of these models. And if you knew what you were talking about, you would know that.
- lukev 4y ago
  We apparently have misaligned understandings of what we mean by "how they work." I agree, we don't know how to interpret the weight structure that the model learns during training. But we do know exactly what happens mechanically during training and inference; what gets multiplied by what, what the inputs and outputs are, how data moves around the system. These are not some mysterious agents that could theoretically do or be anything, much less be secretly conscious (as a lot of alarmists are saying.) They are functions that multiply billions of numbers to generate output tokens. Their ability to output the "right" output tokens is not well understood, and nearly magical. That's what makes them so exciting.
  - famouswaffles 4y ago
    It is all things considered pretty easy to set up GPT such that it runs on its own input forever while being able to interact with users/other systems. add an inner monologue/react and reflexion and you have a very powerful system. embody it with some physical locomotive machine and oh boy. no one has really put this all together yet but everything i've said has been done to some degree. The individual pieces are here. it's just a matter of time. I'm working on some such myself. What it could do is limited only by its intelligence (which is quite a bit higher than the base model as several papers have indicted) and the tools it controls (we seem to gladfully pile more and more control ). What it can be is...anything. If there's anything LLMs are good at, it's simulation. Even this system with thoughts we can theoretically configure to see would be difficult to control. theory and practicality would not meet the road. you will not be able to monitor this system in real time. We've seen bing (doesn't even have all i've described) take action when "upset". The only reason it didn't turn sour is because her actions are limited to search and ending the conversation. But that's obviously not the direction of things here. Can't say i want this train to stop. But i'm under no delusions it couldn't turn dangerous very quickly.