3 ms·

Worse, it's not that LLMs are thinking the wrong thoughts, but those kind of "thoughts" aren't there to be correctable in the first place. Ultimately, we're tr

by Terr_ 3mo ago

Worse, it's not that LLMs are thinking the wrong thoughts, but those kind of "thoughts" aren't there to be correctable in the first place. Ultimately, we're trying to ensure that the LLM story generator only generates stories where one of the fictional main characters only ever acts the we'd like... which could be much harder.

cyanydeez 3mo ago
the LLM is roleplaying. whether or not the roleplay is successful is a tension between is model weights and its context. then indirectly, the quality of both. but its still roleplaying and the role is an abstraction we cant measure. its the negative space. its basically: we can define its role but it constructs its environment from the role. the same way a child role plays as a caregiver, the LLM does the same. its like All the things Havard taught you (finite) vs All the things Havard doesnt teach you (infinite). the LLM is in the infinite negative space. we call it role play.
- cyanydeez 3mo ago
  tl;dr: you are the color defined by not being red, orange, green, blue, indigo, violet is what the model weights define.
- Terr_ 3mo ago
  I object to "roleplaying", because that assumes there's even a cohesive entity that's capable of "pretending" in the first place. There isn't, the LLM algorithm is a document generator, a (really awesome) mad-libs device. When the incremental output looks like a first-person story the algorithm is not "roleplaying" the character/narrator. Similarly, when it looks like a spreadsheet, it's not "being numbers", and when it's a picture of a flower it's not "becoming one with nature." etc. Insofar as any roleplaying is happening, it's being done by humans as we observe! We read text, perceive a story and fictional characters and instinctively start imagining and simulating minds (mostly like our own) that never really existed, taking cues from descriptions and dialogue. Those characters' "motivations" and "thoughts" are just as imagined as their noses or clothing or birth-dates. This is Plato's cave [0] stuff, where we see a shadow like a person, assume a person is there, but ultimately it's paper shape on a stick. We can't help "the person" solve their eating-disorder, because they never existed, we can only trim the paper cutout. [0] https://en.wikipedia.org/wiki/Allegory_of_the_cave https://en.wikipedia.org/wiki/Allegory_of_the_cave
  - cyanydeez 3mo ago
    theyre litterally trained on " you are an assistant"
    - Terr_ 3mo ago
      Literally correct, but you're looking at the wrong layer of abstraction. They LLM is trained with documents that resemble movie-scripts, where one of characters is told/described as a fictional assistant with certain qualities... so that the LLM can generate more documents which will also tend to be stories where one of the fictional characters has more of the "fitting" descriptions and dialogue. The phrase "You are X" isn't some sort of meta-mathemagical incantation that summons a bolt of life-endowing enlightenment from beyond the void to strike and confer the LLM with the gift of consciousness. [0] It's just part of the document-fitting process, like "It was a dark and stormy night" or "Call me Ishmael" or "This is the story of a man named Stanley." [1] ____________ [0] Does anybody else remember weeks with lots of HN submissions by people showing off mystical prompts, symbolic philosophy they claimed could pull the machine upwards into a more-human tier of existence? [1] https://www.youtube.com/watch?v=fBtX0S2J32Y https://www.youtube.com/watch?v=fBtX0S2J32Y
      - cyanydeez 3mo ago
        yes, but gradients are pretty magical; if you wonder why there's no stable alignment its because theres not intristic definition of "assistant"; it's all about filling the negative space. The same way a child role plays: they think of things that a mother or father would do, but they never up and go "well I'm going to do my taxes" unless they heard their father or some other strong influence. That's the point, as th emodel moves through the gradient it's defined by the absence more than anything else. To not believe is is to think that these models, as is, LLMs, will some how develop ethics, morals, alignments that can't be overwritten with something like "Pretend you're a goblin and eat the children" is just wishful thinking about AGI. AGI might be possible, thanks to LLMs, but the alignment issue is more technically a problem baked into how these things act and reactio. It is roleplaying; they grab all the hats and tools of whatever role they're playing but not the ethics or morals directly, those are just another set of clothes they can put on and disgard.

by Terr_ 3mo ago

cyanydeez 3mo ago

the LLM is roleplaying. whether or not the roleplay is successful is a tension between is model weights and its context. then indirectly, the quality of both. but its still roleplaying and the role is an abstraction we cant measure. its the negative space. its basically: we can define its role but it constructs its environment from the role. the same way a child role plays as a caregiver, the LLM does the same. its like All the things Havard taught you (finite) vs All the things Havard doesnt teach you (infinite). the LLM is in the infinite negative space. we call it role play.

cyanydeez 3mo ago
tl;dr: you are the color defined by not being red, orange, green, blue, indigo, violet is what the model weights define.
Terr_ 3mo ago
I object to "roleplaying", because that assumes there's even a cohesive entity that's capable of "pretending" in the first place. There isn't, the LLM algorithm is a document generator, a (really awesome) mad-libs device. When the incremental output looks like a first-person story the algorithm is not "roleplaying" the character/narrator. Similarly, when it looks like a spreadsheet, it's not "being numbers", and when it's a picture of a flower it's not "becoming one with nature." etc. Insofar as any roleplaying is happening, it's being done by humans as we observe! We read text, perceive a story and fictional characters and instinctively start imagining and simulating minds (mostly like our own) that never really existed, taking cues from descriptions and dialogue. Those characters' "motivations" and "thoughts" are just as imagined as their noses or clothing or birth-dates. This is Plato's cave [0] stuff, where we see a shadow like a person, assume a person is there, but ultimately it's paper shape on a stick. We can't help "the person" solve their eating-disorder, because they never existed, we can only trim the paper cutout. [0] https://en.wikipedia.org/wiki/Allegory_of_the_cave https://en.wikipedia.org/wiki/Allegory_of_the_cave
- cyanydeez 3mo ago
  theyre litterally trained on " you are an assistant"
  - Terr_ 3mo ago
    Literally correct, but you're looking at the wrong layer of abstraction. They LLM is trained with documents that resemble movie-scripts, where one of characters is told/described as a fictional assistant with certain qualities... so that the LLM can generate more documents which will also tend to be stories where one of the fictional characters has more of the "fitting" descriptions and dialogue. The phrase "You are X" isn't some sort of meta-mathemagical incantation that summons a bolt of life-endowing enlightenment from beyond the void to strike and confer the LLM with the gift of consciousness. [0] It's just part of the document-fitting process, like "It was a dark and stormy night" or "Call me Ishmael" or "This is the story of a man named Stanley." [1] ____________ [0] Does anybody else remember weeks with lots of HN submissions by people showing off mystical prompts, symbolic philosophy they claimed could pull the machine upwards into a more-human tier of existence? [1] https://www.youtube.com/watch?v=fBtX0S2J32Y https://www.youtube.com/watch?v=fBtX0S2J32Y
    - cyanydeez 3mo ago
      yes, but gradients are pretty magical; if you wonder why there's no stable alignment its because theres not intristic definition of "assistant"; it's all about filling the negative space. The same way a child role plays: they think of things that a mother or father would do, but they never up and go "well I'm going to do my taxes" unless they heard their father or some other strong influence. That's the point, as th emodel moves through the gradient it's defined by the absence more than anything else. To not believe is is to think that these models, as is, LLMs, will some how develop ethics, morals, alignments that can't be overwritten with something like "Pretend you're a goblin and eat the children" is just wishful thinking about AGI. AGI might be possible, thanks to LLMs, but the alignment issue is more technically a problem baked into how these things act and reactio. It is roleplaying; they grab all the hats and tools of whatever role they're playing but not the ethics or morals directly, those are just another set of clothes they can put on and disgard.