5 ms·

> The goal of character training is to make Claude begin to have more nuanced, richer traits like curiosity, open-mindedness, and thoughtfulness. You can't mak

> The goal of character training is to make Claude begin to have more nuanced, richer traits like curiosity, open-mindedness, and thoughtfulness. You can't make a transformer-based language model "have" curiosity. Real curiosity is an innate trait in intelligent animals that promotes learning and exploration by increasing focus and stick-to-it-ness when the situation is interesting/unexplored (i.e. not well predicted - surprising). An LLM could be trained to fake curiosity in same way that ELIZA did ("Tell me more about your fear of prison showers"), but fundamentally it deals in words and statistics, not facts and memories, so any poorly-predicted surprise signal will be surface level such as "that was an unusual word pattern" or "you haven't mentioned that before in the input context". Even if fake curiosity ("interest") makes the model seem more conversationally engaging, maybe helps exploration by prompting the use to delve deeper into a subject, it's not going to help learning since LLMs are pre-trained and cannot learn by experience (except ephemerally within context).

mckirk 2y ago
I think you _can_ make an LLM 'have' curiosity, for all practical intents and purposes. I'm e.g. thinking of the 'have the LLM research a topic' task, such as the 'come up with suitable search terms, search the web, summarize the article, then think of potential next questions' cycle implemented by Perplexity, for example. I'm pretty sure the results would vary noticeably between an LLM that was trained to be 'curious' (i.e., follow more unusual trains of thought) versus not, and the differences would probably compound the more freedom you give the LLM, for example by giving it more iterations of the 'formulate questions, search, summarize' loop.
- HarHarVeryFunny 2y ago
  The problem is how can "follow more unusual trains of thought" apply to a language model ? Sure it can selectively attend to certain parts of the input and generate based on that, but what is the internal signal for "unusual" ? Any selective focus is also going to appear like groundhog day since the model's weights are fixed and what was "unusual" today will continue to be "unusual" even after it's been exposed to it for the 1000th time!
  - mckirk 2y ago
    That's a good point. Thinking about this a bit, it might be a bit late actually to start to guide an LLM towards curiosity only at the fine-tuning stage, since this 'exploring unusual-trains-of-thoughts' is precisely what the LLM _isn't_ learning during training, where it sees (basically by definition) a ton of 'usual trains-of-thoughts'. Maybe you'd have to explicitly model 'surprise' during training, to get the LLM to try to fit precisely those examples better that don't really fit its already learned model (which would require the network to reserve some capacity for creativity/curiosity, which it otherwise might not do, because it's not necessary to model _most_ of what it sees). But then you enter the territory of 'if you open your mind too much, your brain might fall out', and could end up accidentally training QAnonGPT, and that you definitely don't want... So maybe this way of 'hoping the LLM builds up enough creative intelligence during training, which can then be guided during fine-tuning' is the best we can do at the moment.
renegade-otter 2y ago
Curiosity is inherently proactive. An LLM, fundamentally, is a file. It's a FILE. It just sits there until you ask for output. Saying anything more about that process is just anthropomorphizing. No one ever accused the Google index of having "curiosity", but the idea is basically the same - you give it a query and it gives you back a response. Then it just sits idle until the next query.
- powerapple 2y ago
  proactive can be: we take infinite inputs at a resolution, having eyes, ears, all sensors constantly "asking" for output
- swid 2y ago
  I disagree on the premise that computers are able to simulate many different things; it would be just as easy to say that the universe cannot have curiosity; that the universe is just things playing out according to some preset rules. But of course, people exist within the universe, and while our brains do function according to those rules; likely all rules that can be expressed with math and statistics; we do have curiosity. You can look at the level of abstraction of the universe, and you will not find subjective experience or choice, yet the “platform” of the universe demonstrably allows for that. When I see arguments like yours and the parent’s, I cannot help but think the arguments would seem to apply just as well to an accurate simulation of the universe, which shows the argument must be flawed. You are a file in the universe, loaded into memory and being run in parallel with my file. If you believe physics can be expressed through math and humans have subjective experienced, then the right kind of simulation can also have these things. Of course any simulation can be represented digitally and saved to disk.
  - HarHarVeryFunny 2y ago
    > it would be just as easy to say that the universe cannot have curiosity; that the universe is just things playing out according to some preset rules. Behavior/dynamics depends on structure, and structure is scale dependent. At the scale of galaxies and stars, the universe has a relatively simple structure, with relatively boring dynamics mostly governed by gravity and nuclear fusion. When you zoom down to the level of dynamics on a planet full of life, or of a living entity such as a human, then things get a lot more interesting since at those scales there is far more structure causing a much greater variety of dynamics. It's the structure of an LLM vs the structure of a human brain that makes the difference.
zeroonetwothree 2y ago
Right. Curiosity only exists if you have a proper model of the world, which LLMs do not
nxpnsv 2y ago
They aren't open-minded or thoughtful either. But they can act that way. So yeah it will be fake curiousness, but is that really a problem?
- HarHarVeryFunny 2y ago
  That depends if your goal is just to have an LLM follow a conversational style imitating curiosity (in order to appear more engaging or friendly/whatever), or whether you actually want it to BE curious about things it doesn't know, as the basis for directed learning.
JumpCrisscross 2y ago
> An LLM could be trained to fake curiosity in same way that ELIZA did Or, let’s be real, the way a lot of people do.
Me1000 2y ago
This arguments is made every singe time a new LLM article gets posted and I'm not sure it's really adding anything to the conversation. Everyone understands language models are not human, there's no need to add a philosophical argument because you don't like the word choice. "Characters" in any other context are also by definition not curious. They're not open-minded or thoughtful either. Characters in a book are not people, they don't have thoughts, they are whatever the author put on paper. Yet we still use these words to describe them, as if the characters' consciousness and motivations existed beyond the paper on which they're described. It becomes really hard to talk about these traits without using words like curiosity and thoughtfulness. No one thinks the characters in a fictional book are real people.
- HarHarVeryFunny 2y ago
  I'm not sure that "these are just characters, your expectations are too high" is totally valid here, especially as Anthropic are claiming their LLM progression is on the path to AGI. Any true human-level (or even rat-level for that matter) AGI would need to have actual innate traits such as curiosity to drive lifelong exploration and learning. That seems a pretty minimal bar to meet to claim that something is AGI. I suppose in context of an article about giving Claude a character (having it play a character) then we need to interpret "having a trait" as "playing a character having a trait", because it certainly is very different from actually having it.
  - pinneycolton 2y ago
    Some humans who are not trained that curiosity produces beneficial outcomes are often not all that curious. It is difficult to know what is truly innate vs. the result of conditioning. It is also difficult to know what is the result of an absence of opposing conditioning. If I was raised in a poor area with high crime and little economic opportunity, my conditioning would be quite different. How would that have changed my behavior? Could curiosity get me in more trouble than it's worth? Could I observe that and adjust my own behavior as a result? We are all, to some extent, a product of our environment (training data). I wasn't raised in that type of area, but does that mean my own intellectual curiosity is more innate? Or does it mean it is less innate? I could argue that both ways.

mckirk 2y ago

I think you _can_ make an LLM 'have' curiosity, for all practical intents and purposes. I'm e.g. thinking of the 'have the LLM research a topic' task, such as the 'come up with suitable search terms, search the web, summarize the article, then think of potential next questions' cycle implemented by Perplexity, for example. I'm pretty sure the results would vary noticeably between an LLM that was trained to be 'curious' (i.e., follow more unusual trains of thought) versus not, and the differences would probably compound the more freedom you give the LLM, for example by giving it more iterations of the 'formulate questions, search, summarize' loop.

HarHarVeryFunny 2y ago
The problem is how can "follow more unusual trains of thought" apply to a language model ? Sure it can selectively attend to certain parts of the input and generate based on that, but what is the internal signal for "unusual" ? Any selective focus is also going to appear like groundhog day since the model's weights are fixed and what was "unusual" today will continue to be "unusual" even after it's been exposed to it for the 1000th time!
- mckirk 2y ago
  That's a good point. Thinking about this a bit, it might be a bit late actually to start to guide an LLM towards curiosity only at the fine-tuning stage, since this 'exploring unusual-trains-of-thoughts' is precisely what the LLM _isn't_ learning during training, where it sees (basically by definition) a ton of 'usual trains-of-thoughts'. Maybe you'd have to explicitly model 'surprise' during training, to get the LLM to try to fit precisely those examples better that don't really fit its already learned model (which would require the network to reserve some capacity for creativity/curiosity, which it otherwise might not do, because it's not necessary to model _most_ of what it sees). But then you enter the territory of 'if you open your mind too much, your brain might fall out', and could end up accidentally training QAnonGPT, and that you definitely don't want... So maybe this way of 'hoping the LLM builds up enough creative intelligence during training, which can then be guided during fine-tuning' is the best we can do at the moment.

renegade-otter 2y ago

Curiosity is inherently proactive. An LLM, fundamentally, is a file. It's a FILE. It just sits there until you ask for output. Saying anything more about that process is just anthropomorphizing. No one ever accused the Google index of having "curiosity", but the idea is basically the same - you give it a query and it gives you back a response. Then it just sits idle until the next query.

powerapple 2y ago
proactive can be: we take infinite inputs at a resolution, having eyes, ears, all sensors constantly "asking" for output
swid 2y ago
I disagree on the premise that computers are able to simulate many different things; it would be just as easy to say that the universe cannot have curiosity; that the universe is just things playing out according to some preset rules. But of course, people exist within the universe, and while our brains do function according to those rules; likely all rules that can be expressed with math and statistics; we do have curiosity. You can look at the level of abstraction of the universe, and you will not find subjective experience or choice, yet the “platform” of the universe demonstrably allows for that. When I see arguments like yours and the parent’s, I cannot help but think the arguments would seem to apply just as well to an accurate simulation of the universe, which shows the argument must be flawed. You are a file in the universe, loaded into memory and being run in parallel with my file. If you believe physics can be expressed through math and humans have subjective experienced, then the right kind of simulation can also have these things. Of course any simulation can be represented digitally and saved to disk.
- HarHarVeryFunny 2y ago
  > it would be just as easy to say that the universe cannot have curiosity; that the universe is just things playing out according to some preset rules. Behavior/dynamics depends on structure, and structure is scale dependent. At the scale of galaxies and stars, the universe has a relatively simple structure, with relatively boring dynamics mostly governed by gravity and nuclear fusion. When you zoom down to the level of dynamics on a planet full of life, or of a living entity such as a human, then things get a lot more interesting since at those scales there is far more structure causing a much greater variety of dynamics. It's the structure of an LLM vs the structure of a human brain that makes the difference.

zeroonetwothree 2y ago

Right. Curiosity only exists if you have a proper model of the world, which LLMs do not

nxpnsv 2y ago

They aren't open-minded or thoughtful either. But they can act that way. So yeah it will be fake curiousness, but is that really a problem?

HarHarVeryFunny 2y ago
That depends if your goal is just to have an LLM follow a conversational style imitating curiosity (in order to appear more engaging or friendly/whatever), or whether you actually want it to BE curious about things it doesn't know, as the basis for directed learning.

JumpCrisscross 2y ago

> An LLM could be trained to fake curiosity in same way that ELIZA did Or, let’s be real, the way a lot of people do.

Me1000 2y ago

This arguments is made every singe time a new LLM article gets posted and I'm not sure it's really adding anything to the conversation. Everyone understands language models are not human, there's no need to add a philosophical argument because you don't like the word choice. "Characters" in any other context are also by definition not curious. They're not open-minded or thoughtful either. Characters in a book are not people, they don't have thoughts, they are whatever the author put on paper. Yet we still use these words to describe them, as if the characters' consciousness and motivations existed beyond the paper on which they're described. It becomes really hard to talk about these traits without using words like curiosity and thoughtfulness. No one thinks the characters in a fictional book are real people.

HarHarVeryFunny 2y ago
I'm not sure that "these are just characters, your expectations are too high" is totally valid here, especially as Anthropic are claiming their LLM progression is on the path to AGI. Any true human-level (or even rat-level for that matter) AGI would need to have actual innate traits such as curiosity to drive lifelong exploration and learning. That seems a pretty minimal bar to meet to claim that something is AGI. I suppose in context of an article about giving Claude a character (having it play a character) then we need to interpret "having a trait" as "playing a character having a trait", because it certainly is very different from actually having it.
- pinneycolton 2y ago
  Some humans who are not trained that curiosity produces beneficial outcomes are often not all that curious. It is difficult to know what is truly innate vs. the result of conditioning. It is also difficult to know what is the result of an absence of opposing conditioning. If I was raised in a poor area with high crime and little economic opportunity, my conditioning would be quite different. How would that have changed my behavior? Could curiosity get me in more trouble than it's worth? Could I observe that and adjust my own behavior as a result? We are all, to some extent, a product of our environment (training data). I wasn't raised in that type of area, but does that mean my own intellectual curiosity is more innate? Or does it mean it is less innate? I could argue that both ways.