5 ms·

For some reason, it's still amazing to me that the model creators means of controlling the model are just prompts as well. This just feels like a significant t

For some reason, it's still amazing to me that the model creators means of controlling the model are just prompts as well. This just feels like a significant threshold. Not saying this makes it AGI (obviously its not AGI), but it feels like it makes it something. Imagine if you created a web api and the only way you could modify the responses to the different endpoints are not from editing the code but by sending a request to the api.

clysm 1y ago
No, it’s not a threshold. It’s just how the tech works. It’s a next letter guesser. Put in a different set of letters to start, and it’ll guess the next letters differently.
- Trasmatta 1y ago
  I think we need to start moving away from this explanation, because the truth is more complex. Anthropic's own research showed that Claude does actually "plan ahead", beyond the next token. https://www.anthropic.com/research/tracing-thoughts-language-model https://www.anthropic.com/research/tracing-thoughts-language... > Instead, we found that Claude plans ahead. Before starting the second line, it began "thinking" of potential on-topic words that would rhyme with "grab it". Then, with these plans in mind, it writes a line to end with the planned word.
  - cmiles74 1y ago
    It reads to me like they compare the output of different prompts and somehow reach the conclusion that Claude is generating more than one token and "planning" ahead. They leave out how this works. My guess is that they have Claude generate a set of candidate outputs and the Claude chooses the "best" candidate and returns that. I agree this improves the usefulness of the output but I don't think this is a fundamentally different thing from "guessing the next token". UPDATE: I read the paper and I was being overly generous. It's still just guessing the next token as it always has. This "multi-hop reasoning" is really just another way of talking about the relationships between tokens.
    - Trasmatta 1y ago
      That's not the methodology they used. They're actually inspecting Claude's internal state and suppression certain concepts, or replacing them with others. The paper goes into more detail. The "planning" happens further in advance than "the next token".
      - cmiles74 1y ago
        Okay, I read the paper. I see what they are saying but I strongly disagree that the model is "thinking". They have highlighted that relationships between words is complicated, which we already knew. They also point out that some words are related to other words which are related to other words which, again, we already knew. Lastly they used their model (not Claude) to change the weights associated with some words, thus changing the output to meet their predictions, which I agree is very interesting. Interpreting the relationship between words as "multi-hop reasoning" is more about changing the words we use to talk about things and less about fundamental changes in the way LLMs work. It's still doing the same thing it did two years ago (although much faster and better). It's guessing the next token.
        Trasmatta 1y ago
        I said "planning ahead", not "thinking". It's clearly doing more than only predicting the very next token.
    - therealpygon 1y ago
      They have written multiple papers on the subject, so there isn’t much need for you to guess incorrectly what they did.
  - ceh123 1y ago
    I'm not sure if this really says the truth is more complex? It is still doing next-token prediction, but it's prediction method is sufficiently complicated in terms of conditional probabilities that it recognizes that if you need to rhyme, you need to get to some future state, which then impacts the probabilities of the intermediate states. At least in my view it's still inherently a next-token predictor, just with really good conditional probability understandings.
    - jermaustin1 1y ago
      But then so are we? We are just predicting the next word we are saying, are we not? Even when you add thoughts behind it (sure some people think differently - be it without an inner monologue, or be it just in colors and sounds and shapes, etc), but that "reasoning" is still going into the act of coming up with the next word we are speaking/writing.
      - thomastjeffery 1y ago
        We are really only what we understand ourselves to be? We must have a pretty great understanding of that thing we can't explain then.
      - hadlock 1y ago
        Humans and LLMs are built differently, it seems disingenuous to think we both use the same methods to arrive at the same general conclusion. I can inherently understand some proofs of pythagorean's theorem but an LLM might apply different ones for various reasons. But the output/result is still the same. If a next token generator run in parallel can generate a performant relational database that doesn't directly imply I am also a next token generator.
        wetpaws 1y ago
        [dead]
      - spookie 1y ago
        This type of response always irks me. It shows that we, computer scientists, think of ourselves as experts on anything. Even though biological machines are well outside our expertise. We should stop repeating things we don't understand.
  - dontlikeyoueith 1y ago
    > Anthropic's own research showed that Claude does actually "plan ahead", beyond the next token. For a very vacuous sense of "plan ahead", sure. By that logic, a basic Markov-chain with beam search plans ahead too.
tpm 1y ago
To me it feels like an unsolved challenge. Sure there is finetuning and various post-training stuff but it still feels like there should be a tool to directly change some behavior, like editing a binary with a hex editor. There are many efforts to do that and I'm hopeful we will get there eventually.
- Chabsff 1y ago
  I've been bearish of these efforts over the years, and remain so. In my more cynical moments, I even entertain the thought that it's mostly a means to delay aggressive regulatory oversight by way of empty promises. Time and time again, opaque end-to-end models keep outperforming any attempt to enforce structure, which is needed to _some_ degree to achieve this in non-prompting manners. And in a vague intuitive way, that makes sense. The whole point of training-based AI is to achieve stuff you can't practically from a pure algorithmic approach. Edit: before the pedants lash out. Yes, model structure matters. I'm oversimplifying here.
WJW 1y ago
Its creators can 100% "change the code" though. That is called "training" in the context of LLMs and choosing which data to include in the training set is a vital part of the process. The system prompt is just postprocessing. Now of course you and me can't change the training set, but that's because we're just users.
- thunky 1y ago
  Yeah they can "change the code" like that, like someone can change the api code. But the key point is that they're choosing to change the behavior without changing the code, because it's possible and presumably more efficient to do it that way, which is not possible to do with an api.
lxgr 1y ago
Or even more dramatically, imagine C compilers were written in C :)
- jsnider3 1y ago
  I only got half a sentence into "well-actually"ing you before I got the joke.
jbentley1 1y ago
This isn't exactly correct, it is a combination of training and system prompt. You could train the system prompt into the model. This could be as simple as running the model with the system prompt, then training on those outputs until it had internalized the instructions. The downside is that it will become slightly less powerful, it is expensive, and if you want to change something you have to do it all over again. This is a little more confusing with Anthropic's naming scheme, so I'm going to describe OpenAI instead. There is GPT-whatever the models, and then there is ChatGPT the user facing product. They want ChatGPT to use the same models as are available via API, but they don't want the API to have all the behavior of ChatGPT. Hence, a system prompt. If you do use the API you will notice that there is a lot of behavior that is in fact trained in. The propensity to use em dashes, respond in Markdown, give helpful responses, etc.
- IX-103 1y ago
  You can't just train with the negative examples showing filtered content, as that could lead to poor generalization. You'd need to supplement with samples from the training set to prevent catastrophic forgetting. Otherwise it's like taking slices out of someone's brain until they can't recite a poem. Yes, at the end they can't recite a poem, but who knows what else they can no longer do. The positive examples from training essentially tell you what slices you need to put back to keep it functional.
sanderjd 1y ago
I think it reflects the technology's fundamental immaturity, despite how much growth and success it has already had.
- Mahn 1y ago
  At its core what it really reflects is that the technology is a blackbox that wasn't "programmed" but rather "emerged". In this context, this is the best we can do to fine tune behavior without retraining it.
- james-bcn 1y ago
  Agreed. It seems incredibly inefficient to me.
jcims 1y ago
And we get to learn all of the same lessons we've learned about mixing code and data. Yay!
- EvanAnderson 1y ago
  That's what I was thinking, too. It would do some good for the people implementing this stuff to read about in-band signaling and blue boxes, for example.
  - int_19h 1y ago
    They are well aware of it, which is why there's a distinction between "system" and "user" messages, for example. The problem is that, at the end of the day, it's still a single NN processing everything. You can train it to make this distinction, but by their very nature the outcome is still probabilistic. This is similar to how you as a human cannot avoid being influenced (one way or another, however subtly) by any text that you encounter, simply by virtue of having read it.
HarHarVeryFunny 1y ago
Well, it is something - a language model, and this is just a stark reminder of that. It's predicting next word based on the input, and the only way to steer the prediction is therefore to tweak the input. In terms of feels, this feels to me more like pushing on a string.
morsecodist 1y ago
For me it's the opposite. We don't really have a reliable way of getting the models to do what we want or even to measure if they are doing what we want.
- spaceywilly 1y ago
  Yeah it’s kind of like we have invented a car that drives around wildly in any direction, and we are trying to steer it by putting up guard rails to get it to go where we want. What we need is to invent the steering wheel and brake pedals, which I’m sure smart people are working on. We’re just at a very early point with this technology, which I think people tend to forget.

clysm 1y ago

No, it’s not a threshold. It’s just how the tech works. It’s a next letter guesser. Put in a different set of letters to start, and it’ll guess the next letters differently.

Trasmatta 1y ago
I think we need to start moving away from this explanation, because the truth is more complex. Anthropic's own research showed that Claude does actually "plan ahead", beyond the next token. https://www.anthropic.com/research/tracing-thoughts-language-model https://www.anthropic.com/research/tracing-thoughts-language... > Instead, we found that Claude plans ahead. Before starting the second line, it began "thinking" of potential on-topic words that would rhyme with "grab it". Then, with these plans in mind, it writes a line to end with the planned word.
- cmiles74 1y ago
  It reads to me like they compare the output of different prompts and somehow reach the conclusion that Claude is generating more than one token and "planning" ahead. They leave out how this works. My guess is that they have Claude generate a set of candidate outputs and the Claude chooses the "best" candidate and returns that. I agree this improves the usefulness of the output but I don't think this is a fundamentally different thing from "guessing the next token". UPDATE: I read the paper and I was being overly generous. It's still just guessing the next token as it always has. This "multi-hop reasoning" is really just another way of talking about the relationships between tokens.
  - Trasmatta 1y ago
    That's not the methodology they used. They're actually inspecting Claude's internal state and suppression certain concepts, or replacing them with others. The paper goes into more detail. The "planning" happens further in advance than "the next token".
    - cmiles74 1y ago
      Okay, I read the paper. I see what they are saying but I strongly disagree that the model is "thinking". They have highlighted that relationships between words is complicated, which we already knew. They also point out that some words are related to other words which are related to other words which, again, we already knew. Lastly they used their model (not Claude) to change the weights associated with some words, thus changing the output to meet their predictions, which I agree is very interesting. Interpreting the relationship between words as "multi-hop reasoning" is more about changing the words we use to talk about things and less about fundamental changes in the way LLMs work. It's still doing the same thing it did two years ago (although much faster and better). It's guessing the next token.
      - Trasmatta 1y ago
        I said "planning ahead", not "thinking". It's clearly doing more than only predicting the very next token.
  - therealpygon 1y ago
    They have written multiple papers on the subject, so there isn’t much need for you to guess incorrectly what they did.
- ceh123 1y ago
  I'm not sure if this really says the truth is more complex? It is still doing next-token prediction, but it's prediction method is sufficiently complicated in terms of conditional probabilities that it recognizes that if you need to rhyme, you need to get to some future state, which then impacts the probabilities of the intermediate states. At least in my view it's still inherently a next-token predictor, just with really good conditional probability understandings.
  - jermaustin1 1y ago
    But then so are we? We are just predicting the next word we are saying, are we not? Even when you add thoughts behind it (sure some people think differently - be it without an inner monologue, or be it just in colors and sounds and shapes, etc), but that "reasoning" is still going into the act of coming up with the next word we are speaking/writing.
    - thomastjeffery 1y ago
      We are really only what we understand ourselves to be? We must have a pretty great understanding of that thing we can't explain then.
    - hadlock 1y ago
      Humans and LLMs are built differently, it seems disingenuous to think we both use the same methods to arrive at the same general conclusion. I can inherently understand some proofs of pythagorean's theorem but an LLM might apply different ones for various reasons. But the output/result is still the same. If a next token generator run in parallel can generate a performant relational database that doesn't directly imply I am also a next token generator.
      - wetpaws 1y ago
        [dead]
    - spookie 1y ago
      This type of response always irks me. It shows that we, computer scientists, think of ourselves as experts on anything. Even though biological machines are well outside our expertise. We should stop repeating things we don't understand.
- dontlikeyoueith 1y ago
  > Anthropic's own research showed that Claude does actually "plan ahead", beyond the next token. For a very vacuous sense of "plan ahead", sure. By that logic, a basic Markov-chain with beam search plans ahead too.

tpm 1y ago

To me it feels like an unsolved challenge. Sure there is finetuning and various post-training stuff but it still feels like there should be a tool to directly change some behavior, like editing a binary with a hex editor. There are many efforts to do that and I'm hopeful we will get there eventually.

Chabsff 1y ago
I've been bearish of these efforts over the years, and remain so. In my more cynical moments, I even entertain the thought that it's mostly a means to delay aggressive regulatory oversight by way of empty promises. Time and time again, opaque end-to-end models keep outperforming any attempt to enforce structure, which is needed to _some_ degree to achieve this in non-prompting manners. And in a vague intuitive way, that makes sense. The whole point of training-based AI is to achieve stuff you can't practically from a pure algorithmic approach. Edit: before the pedants lash out. Yes, model structure matters. I'm oversimplifying here.

WJW 1y ago

Its creators can 100% "change the code" though. That is called "training" in the context of LLMs and choosing which data to include in the training set is a vital part of the process. The system prompt is just postprocessing. Now of course you and me can't change the training set, but that's because we're just users.

thunky 1y ago
Yeah they can "change the code" like that, like someone can change the api code. But the key point is that they're choosing to change the behavior without changing the code, because it's possible and presumably more efficient to do it that way, which is not possible to do with an api.

lxgr 1y ago

Or even more dramatically, imagine C compilers were written in C :)

jsnider3 1y ago
I only got half a sentence into "well-actually"ing you before I got the joke.

jbentley1 1y ago

This isn't exactly correct, it is a combination of training and system prompt. You could train the system prompt into the model. This could be as simple as running the model with the system prompt, then training on those outputs until it had internalized the instructions. The downside is that it will become slightly less powerful, it is expensive, and if you want to change something you have to do it all over again. This is a little more confusing with Anthropic's naming scheme, so I'm going to describe OpenAI instead. There is GPT-whatever the models, and then there is ChatGPT the user facing product. They want ChatGPT to use the same models as are available via API, but they don't want the API to have all the behavior of ChatGPT. Hence, a system prompt. If you do use the API you will notice that there is a lot of behavior that is in fact trained in. The propensity to use em dashes, respond in Markdown, give helpful responses, etc.

IX-103 1y ago
You can't just train with the negative examples showing filtered content, as that could lead to poor generalization. You'd need to supplement with samples from the training set to prevent catastrophic forgetting. Otherwise it's like taking slices out of someone's brain until they can't recite a poem. Yes, at the end they can't recite a poem, but who knows what else they can no longer do. The positive examples from training essentially tell you what slices you need to put back to keep it functional.

sanderjd 1y ago

I think it reflects the technology's fundamental immaturity, despite how much growth and success it has already had.

Mahn 1y ago
At its core what it really reflects is that the technology is a blackbox that wasn't "programmed" but rather "emerged". In this context, this is the best we can do to fine tune behavior without retraining it.
james-bcn 1y ago
Agreed. It seems incredibly inefficient to me.

jcims 1y ago

And we get to learn all of the same lessons we've learned about mixing code and data. Yay!

EvanAnderson 1y ago
That's what I was thinking, too. It would do some good for the people implementing this stuff to read about in-band signaling and blue boxes, for example.
- int_19h 1y ago
  They are well aware of it, which is why there's a distinction between "system" and "user" messages, for example. The problem is that, at the end of the day, it's still a single NN processing everything. You can train it to make this distinction, but by their very nature the outcome is still probabilistic. This is similar to how you as a human cannot avoid being influenced (one way or another, however subtly) by any text that you encounter, simply by virtue of having read it.

HarHarVeryFunny 1y ago

Well, it is something - a language model, and this is just a stark reminder of that. It's predicting next word based on the input, and the only way to steer the prediction is therefore to tweak the input. In terms of feels, this feels to me more like pushing on a string.

morsecodist 1y ago

For me it's the opposite. We don't really have a reliable way of getting the models to do what we want or even to measure if they are doing what we want.

spaceywilly 1y ago
Yeah it’s kind of like we have invented a car that drives around wildly in any direction, and we are trying to steer it by putting up guard rails to get it to go where we want. What we need is to invent the steering wheel and brake pedals, which I’m sure smart people are working on. We’re just at a very early point with this technology, which I think people tend to forget.