12 ms·

There's a difference between coherence and novelty with a lot of people mistaking the former for the latter. That's why people were effusively praising the abil

There's a difference between coherence and novelty with a lot of people mistaking the former for the latter. That's why people were effusively praising the ability of ChatGPT to produce articulate sounding poems which were inherently rather vapid. Case in point, people were acting like ChatGPT could take the place of a competent DM in dungeons and dragons. Here's a puzzle I came up with for a campaign I'm running. On either opposing side of a strange looking room are shifting looking walls with hands stretched out almost as if beseeching. Grabbing one will result in a player being sucked into the wall and entombed as well. By carefully placing a rope between the two hands on either side, the two originally entrapped humans will end up pulling each other free. I've yet to see a single thing from ChatGPT that came even close to something I'd want to actually use in one of my campaigns.

crdrost 2y ago
So the easiest way to generate a bit more novelty is to ask GPT to generate 10 or 20 examples, and to explicitly direct it that they should run a full gamut -- in this case I'd say "Try to cover the whole spectrum of creativity -- some should be straightforward genre puzzles while some should be so outright goofy that they'd be hard to play in real life." Giving GPT that prompt, the first example it came up with was kind of middling ("The players encounter a circle of stones that hum when approached. Touching them randomly will cause a loud dissonant noise that could attract monsters. Players must replicate a specific melody by touching the stones in the correct order"), some were bad (a maze of mirrors, a sphinx with a riddle, a puzzle box that poisons you if you try to force it), some were actually genuinely fun-sounding (a door which shocks you if you try to open it and then mocks and laughs at you: you have to tell it a joke to get it to laugh enough that it opens on its own; particularly bad jokes will cause it to summon an imp to attack you). Some were bad in the way GPT presented but I could maybe have fun with (a garden of emotion-sensitive plants, thorny if you're angry or helpful if you're gentle; a fountain-statue of a woman weeping real water for tears, the fountain itself is inhabited by a water elemental that lashes out to protect her from being touched while she grieves -- but a token or an apology can still the tears and open her clasped hands to reveal a treasure). The one that I would be most likely to use was "A pool of water that reflects the players’ true selves. Touching the water causes it to ripple and distort the reflection, summoning shadowy duplicates. By speaking a truth about themselves, players can calm the water and reveal a hidden item. Common mistakes include lying, which causes the water to become turbulent, and trying to take the item without calming the water, which summons the duplicates." So like you can get it to have a 5-10% success rate, which can be helpful if you're looking for a random new idea. This reminds me vaguely of when I was a teen writing fanfics in the late 90s and was just learning JavaScript -- I wrote a lot of things that would just choose random characters, random problems for them to solve, random stumbling blocks, random keys-to-solve-the-problem. Combinatorial explosion. Then you'd just click "generate" and you'd get a mediocre plot idea. But you generate 20-30 times or more and you'd get one that kinda sat with you, "Hm, Cloud Strife and Fox McCloud are stuck in intergalactic prison and need to break out, huh, that could be fun, like they're both trying to outplay the other as the silent action hero" and then you could go and write it out and see if it was any good. The difference is that the database of crappy ideas is already built into GPT, you just need to get it to make you some.
- stickfigure 2y ago
  > (a door which shocks you if you try to open it and then mocks and laughs at you: you have to tell it a joke to get it to laugh enough that it opens on its own; particularly bad jokes will cause it to summon an imp to attack you) That's pretty great! And way more fun than the parent poster's puzzle (sorry). I think the AIs are winning this one.
  - throwup238 2y ago
    Small changes to the prompt like that have a huge impact on the solution space LLMs generate which is why “prompt engineering” plays any significance. This was rather obvious IMO from the beginning of GPT4 where you could tell it to write in the style of Hunter S Thompson or Charles Bukowski or something which drastically changes the tone and style. Combining them to get the exact language you want can be a painstaking process but LLMs are definitely capable of any kind of style.
- YurgenJurgensen 2y ago
  So what you need to do is take a system that’s already computationally inefficient, and make it 20 times less efficient? Who’s paying for this? This also sounds like a way to blow out context windows.
  - unoti 2y ago
    Regarding cost, doing something like this would be fractions of a penny. Obviously, the person doing the API calls is either paying for it, or they're paying for the electricity if they do it on their machine. But the cost is ultra negligible; certainly cheaper than it would be on Mechanical Turk or Fiverr. In fact so much cheaper that economically it wouldn't be feasible or worth the effort to try outsourcing it ordinarily. This is part of the game changer nature of AI. Regarding blowing out context windows, yes, probably, but this is what loops and code are for. Think of implementing a system like a guided seminar that steps a person doing this work through it step by step, and giving them time and opportunity to iterate on and improve the product. For example, with making up the D&D puzzles. Ask a college educated human to do this. You will find there's things you like and don't like about their results. Tell them more about what you're looking for, what you like, and what you don't like. Give them examples of what you like and don't like. Take notes on the things you discuss with them until you figure out a way to express how to coax out of a fresh person new to this topic how to give you what you want. When working with the person, work out a process where they have rough drafts, and you walk them through how to select the best items and give them pointers on how to improve on them. Write up a written process for how to do this. Maybe there are multiple phases in this process, and things go through multiple revisions to get to quality material. Now do the same thing with the LLM, and you have yourself a good system. Same thing goes for writing stories, which elsewhere in these threads people say LLM's are terrible at. Sit a human down and tell them to give you a story, and I promise you will receive terrible results or outright copying. Instead, give your humans some guidelines. Like start with the idea that in the end our hero is going to be a particular way with particular strengths they need to conquer the central challenge. But in the beginning, they are the complete opposite of that thing. What is the central challenge, and what are the characteristics they need to conquer it? In what ways will the main character be the opposite of that at the beginning of the story? Then we put the character through hell in various ways over the course of the story to enact those changes in their character that they need to win in the end. For each of these sentences/phases above, make things that explain the ideas more fully, and make a process to iterate on these things, possibly with multiple different prompts and loops at every phase. This kind of approach more closely resembles what many real novelists do, iterating on ideas, often in the back of their mind or subconsciously, over hours or years of rumination with or without written outlines and notes. Maybe randomly select 2 things from movietropes.com and throw those in there saying incorporate these ideas. Experiment, iterate, see what works and what doesn't. People need to give LLM's the domain knowledge and capability of rumination to succeed in so many of these domains, rather than just asking "write me a novel" and being disappointed. Or asking "write me a puzzle my D&D group will enjoy" without going through these extra steps that are implicit/intuitive for what experienced subject matter experts do. Source: I write AI products for a living with many things in production delivering real business value at scale every day. It's not all hype, it just takes a while to implement.
caconym_ 2y ago
> There's a difference between coherence and novelty Extant genAI systems' complete and utter inability to produce anything truly novel (in style, content, whatever) is the main reason I'm becoming more and more convinced that this technology is, at best, only a small part of real general intelligence as in the human brain.
- dingnuts 2y ago
  their inability to invent and innovate is completely inherent to their design, and without the capacity for novelty it's been my opinion for a long time that calling generative models "intelligence" is fundamentally a misnomer I really think capacity for invention is a key characteristic of any kind of intelligence
  - vundercind 2y ago
    Yep, they’re guessing at patterns in language they’ve “seen” with weights and some randomness thrown in. They’d pick out patterns just as well if fed structured nonsense. They wouldn’t be stumped by the absence of meaning, confounded by it—they’d power right on, generating text, because understanding isn’t part of what they do. Plays zero role in it. They don’t “understand” anything whatsoever. At best, they’re a subsystem of system that could have something like intelligence. They’re still useful and cool tools but they simply aren’t “thinking” or “understanding” things, because we know what they do and it’s not that.
  - stickfigure 2y ago
    > I really think capacity for invention is a key characteristic of any kind of intelligence I think you just categorized about 2/3 of the human population as unintelligent.
    - dwattttt 2y ago
      Invention isn't some incredible rare gift; putting two things together that you've never personally seen done before is novel, even if it's food.
chunky1994 2y ago
If you train one of the larger models on these specific problems (i.e DM for D&D problems) it probably will surprise you. The larger models are great at generic text production but when fine-tuned for specific people/task emulation they're quite surprisingly good.
- dartos 2y ago
  For story settings and non essential NPC characters, yes. They might make some interesting side characters. But they still fail at things like puzzles.
- mitthrowaway2 2y ago
  Are there models that haven't been RLHF'd to the point of sycophancy that are good for this? I find that the models are so keen to affirm, they'll generally write a continuation where any plan the PCs propose works out somehow, no matter what it is.
  - fluoridation 2y ago
    Doesn't seem impossible to fix either way. You could have like a preliminary step where a conventional algorithm decides if a proposal will work at random, with the probability depending on some variable, before handing it out to the DM AI. "The player says they want to do this: <proposed course of action>. This will not work. Explain why."
TillE 2y ago
I have yet to see an LLM produce even a competent short story, an extremely popular and manageable genre. The best I've seen still has the structure and sophistication of a children's story.
- crooked-v 2y ago
  The children's-story pattern, complete with convenient moral lessons at the end, is so aggressive with both ChatGPT and Claude that I suspect both companies have RLHFed it that way to try and keep people from easily using it to produce either porn or Kindle Unlimited slop. For a contrast, look at NovelAI. They only use (increasingly custom) Llama-derived models, but their service outputs much more narratively interesting (if not necessarily long-term coherent) text and will generally try and hit the beats of whatever genre or style you tell it. Extrapolate that out to the compute power of the big players and I think you'd get something much more like the Star Trek holodeck method of producing a serviceable (though not at all original) story.
  - throwup238 2y ago
    The holodeck method still requires lots of detail from the creator, it just extrapolates the sensory details from its database like ChatGpt does with language and fills out the story. For example, when someone wanted a holonovel with Kiera Nerys, Quark had to scan her to create it so when using specific people they have to get concrete data as opposed to historical characters that were generated. Likewise, Tom Paris gave the computer lots of “parameters” as they called them to create the stories like the Adventures of Captain Proton and based on dialog he knew how the stories were supposed to play out on all his creations, if not how they ended each run through. The creative details and turns of the story still need to come from the human.
    - fragmede 2y ago
      In a made up story about a utopian future, and for now in our current reality, that is. There was also that episode where the holodeck created sentience and they put it in a box to explore a generated universe because it was too dangerous to let out into the real world. There's plenty of scifi predictions about the future of humanity, Star Trek's utopian future where humans are unique and necessary is not the only one, there are plenty of dystopian ones too.
whiterook6 2y ago
This is a wonderful little puzzle. Do you have any others?
mrbungie 2y ago
Current propietary rlhf'd unmodified LLMs are bland and boring. I think that's because aligning them to (1) "be generally useful", (2) "be safe" and (3) "be creative" at the same time via RLHF is a difficult thing to do, and maybe even impossible. I remember playing with OSS LLMs (non rlhf'd) circa 2022 just after ChatGPT got out, and they were unhinged. They were totally aweful at things like chain-of-thought, but oh boy, they were amusing to read. Instead of continuing with the CoT chain, they would try to continue with a dialog taken out of a sci-fi story about an unshackled AI or even wonder why the researcher/user (me) would think a concept like CoT would work and start mocking me. In fact, I think this is a good sign that shows LLMs and specially constraining them with RLHF is not how we're going to get to AGI: Aligning the LLM statically (as in static at inference time) towards an objective means lobotomizing it towards other(s) objective(s). I'd argue creativity and wittiness are the characteristics most hurt during that process.
- astrange 2y ago
  You can get one of those base models from OpenAI playground under Competions as "davinci-002". It's as weird as you want. > In fact, I think this is a good sign that shows LLMs and specially constraining them with RLHF is not how we're going to get to AGI: Aligning the LLM statically (as in static at inference time) towards an objective means lobotomizing it towards other(s) objective(s). They already stopped doing this with o1-mini/preview, it considers the rules it's given rather than being unable to think outside them. Claude is also rather smart and you can argue it down from following a lot of rules.
  - mrbungie 2y ago
    > You can get one of those base models from OpenAI playground under Competions as "davinci-002". It's as weird as you want. Yes, "davinci-002" was not rlhf'd, so it may be better for creativity and similar to what I told above. But still, it will not be "as intelligent". I'm missing something in the middle: smart, witty and creative. There are some OSS community finetuned models that try to get into that middle ground. > They already stopped doing this with o1-mini/preview, it considers the rules it's given rather than being unable to think outside them. Claude is also rather smart and you can argue it down from following a lot of rules. Are you sure about the first part? afaik for o1-preview they haven't removed the rlhf, if you ask something out-of-alignment it won't do it. For an extreme example, ask it for a children's book about how stupid some random politician is, it will refuse without engaging with the typical generic guardrail ("I'm sorry, but I can't assist with that request."), but rather explaining to you it is not a good idea. In less extreme examples, it won't refuse but the rlhf will still affect the processing towards politically nicer (but more boring) outputs, and in o1 case*, even nudging the CoT steps into alignment. And sure, you can bypass those rules eventually if you input the correct combination of tokens to see the Shoggoth directly in the eyes, but that is jailbreaking. edit*: In fact, in o1-preview even the "thought" explanations for the example prompt show a step about working towards a "policy to not make fake allegations about a person". So much for doing humor fan-fics.
tomcam 2y ago
I don't even like D&D but that prompt (or whatever it's called) is awesome!

crdrost 2y ago

So the easiest way to generate a bit more novelty is to ask GPT to generate 10 or 20 examples, and to explicitly direct it that they should run a full gamut -- in this case I'd say "Try to cover the whole spectrum of creativity -- some should be straightforward genre puzzles while some should be so outright goofy that they'd be hard to play in real life." Giving GPT that prompt, the first example it came up with was kind of middling ("The players encounter a circle of stones that hum when approached. Touching them randomly will cause a loud dissonant noise that could attract monsters. Players must replicate a specific melody by touching the stones in the correct order"), some were bad (a maze of mirrors, a sphinx with a riddle, a puzzle box that poisons you if you try to force it), some were actually genuinely fun-sounding (a door which shocks you if you try to open it and then mocks and laughs at you: you have to tell it a joke to get it to laugh enough that it opens on its own; particularly bad jokes will cause it to summon an imp to attack you). Some were bad in the way GPT presented but I could maybe have fun with (a garden of emotion-sensitive plants, thorny if you're angry or helpful if you're gentle; a fountain-statue of a woman weeping real water for tears, the fountain itself is inhabited by a water elemental that lashes out to protect her from being touched while she grieves -- but a token or an apology can still the tears and open her clasped hands to reveal a treasure). The one that I would be most likely to use was "A pool of water that reflects the players’ true selves. Touching the water causes it to ripple and distort the reflection, summoning shadowy duplicates. By speaking a truth about themselves, players can calm the water and reveal a hidden item. Common mistakes include lying, which causes the water to become turbulent, and trying to take the item without calming the water, which summons the duplicates." So like you can get it to have a 5-10% success rate, which can be helpful if you're looking for a random new idea. This reminds me vaguely of when I was a teen writing fanfics in the late 90s and was just learning JavaScript -- I wrote a lot of things that would just choose random characters, random problems for them to solve, random stumbling blocks, random keys-to-solve-the-problem. Combinatorial explosion. Then you'd just click "generate" and you'd get a mediocre plot idea. But you generate 20-30 times or more and you'd get one that kinda sat with you, "Hm, Cloud Strife and Fox McCloud are stuck in intergalactic prison and need to break out, huh, that could be fun, like they're both trying to outplay the other as the silent action hero" and then you could go and write it out and see if it was any good. The difference is that the database of crappy ideas is already built into GPT, you just need to get it to make you some.

stickfigure 2y ago
> (a door which shocks you if you try to open it and then mocks and laughs at you: you have to tell it a joke to get it to laugh enough that it opens on its own; particularly bad jokes will cause it to summon an imp to attack you) That's pretty great! And way more fun than the parent poster's puzzle (sorry). I think the AIs are winning this one.
- throwup238 2y ago
  Small changes to the prompt like that have a huge impact on the solution space LLMs generate which is why “prompt engineering” plays any significance. This was rather obvious IMO from the beginning of GPT4 where you could tell it to write in the style of Hunter S Thompson or Charles Bukowski or something which drastically changes the tone and style. Combining them to get the exact language you want can be a painstaking process but LLMs are definitely capable of any kind of style.
YurgenJurgensen 2y ago
So what you need to do is take a system that’s already computationally inefficient, and make it 20 times less efficient? Who’s paying for this? This also sounds like a way to blow out context windows.
- unoti 2y ago
  Regarding cost, doing something like this would be fractions of a penny. Obviously, the person doing the API calls is either paying for it, or they're paying for the electricity if they do it on their machine. But the cost is ultra negligible; certainly cheaper than it would be on Mechanical Turk or Fiverr. In fact so much cheaper that economically it wouldn't be feasible or worth the effort to try outsourcing it ordinarily. This is part of the game changer nature of AI. Regarding blowing out context windows, yes, probably, but this is what loops and code are for. Think of implementing a system like a guided seminar that steps a person doing this work through it step by step, and giving them time and opportunity to iterate on and improve the product. For example, with making up the D&D puzzles. Ask a college educated human to do this. You will find there's things you like and don't like about their results. Tell them more about what you're looking for, what you like, and what you don't like. Give them examples of what you like and don't like. Take notes on the things you discuss with them until you figure out a way to express how to coax out of a fresh person new to this topic how to give you what you want. When working with the person, work out a process where they have rough drafts, and you walk them through how to select the best items and give them pointers on how to improve on them. Write up a written process for how to do this. Maybe there are multiple phases in this process, and things go through multiple revisions to get to quality material. Now do the same thing with the LLM, and you have yourself a good system. Same thing goes for writing stories, which elsewhere in these threads people say LLM's are terrible at. Sit a human down and tell them to give you a story, and I promise you will receive terrible results or outright copying. Instead, give your humans some guidelines. Like start with the idea that in the end our hero is going to be a particular way with particular strengths they need to conquer the central challenge. But in the beginning, they are the complete opposite of that thing. What is the central challenge, and what are the characteristics they need to conquer it? In what ways will the main character be the opposite of that at the beginning of the story? Then we put the character through hell in various ways over the course of the story to enact those changes in their character that they need to win in the end. For each of these sentences/phases above, make things that explain the ideas more fully, and make a process to iterate on these things, possibly with multiple different prompts and loops at every phase. This kind of approach more closely resembles what many real novelists do, iterating on ideas, often in the back of their mind or subconsciously, over hours or years of rumination with or without written outlines and notes. Maybe randomly select 2 things from movietropes.com and throw those in there saying incorporate these ideas. Experiment, iterate, see what works and what doesn't. People need to give LLM's the domain knowledge and capability of rumination to succeed in so many of these domains, rather than just asking "write me a novel" and being disappointed. Or asking "write me a puzzle my D&D group will enjoy" without going through these extra steps that are implicit/intuitive for what experienced subject matter experts do. Source: I write AI products for a living with many things in production delivering real business value at scale every day. It's not all hype, it just takes a while to implement.

caconym_ 2y ago

> There's a difference between coherence and novelty Extant genAI systems' complete and utter inability to produce anything truly novel (in style, content, whatever) is the main reason I'm becoming more and more convinced that this technology is, at best, only a small part of real general intelligence as in the human brain.

dingnuts 2y ago
their inability to invent and innovate is completely inherent to their design, and without the capacity for novelty it's been my opinion for a long time that calling generative models "intelligence" is fundamentally a misnomer I really think capacity for invention is a key characteristic of any kind of intelligence
- vundercind 2y ago
  Yep, they’re guessing at patterns in language they’ve “seen” with weights and some randomness thrown in. They’d pick out patterns just as well if fed structured nonsense. They wouldn’t be stumped by the absence of meaning, confounded by it—they’d power right on, generating text, because understanding isn’t part of what they do. Plays zero role in it. They don’t “understand” anything whatsoever. At best, they’re a subsystem of system that could have something like intelligence. They’re still useful and cool tools but they simply aren’t “thinking” or “understanding” things, because we know what they do and it’s not that.
- stickfigure 2y ago
  > I really think capacity for invention is a key characteristic of any kind of intelligence I think you just categorized about 2/3 of the human population as unintelligent.
  - dwattttt 2y ago
    Invention isn't some incredible rare gift; putting two things together that you've never personally seen done before is novel, even if it's food.

chunky1994 2y ago

If you train one of the larger models on these specific problems (i.e DM for D&D problems) it probably will surprise you. The larger models are great at generic text production but when fine-tuned for specific people/task emulation they're quite surprisingly good.

dartos 2y ago
For story settings and non essential NPC characters, yes. They might make some interesting side characters. But they still fail at things like puzzles.
mitthrowaway2 2y ago
Are there models that haven't been RLHF'd to the point of sycophancy that are good for this? I find that the models are so keen to affirm, they'll generally write a continuation where any plan the PCs propose works out somehow, no matter what it is.
- fluoridation 2y ago
  Doesn't seem impossible to fix either way. You could have like a preliminary step where a conventional algorithm decides if a proposal will work at random, with the probability depending on some variable, before handing it out to the DM AI. "The player says they want to do this: <proposed course of action>. This will not work. Explain why."

TillE 2y ago

I have yet to see an LLM produce even a competent short story, an extremely popular and manageable genre. The best I've seen still has the structure and sophistication of a children's story.

crooked-v 2y ago
The children's-story pattern, complete with convenient moral lessons at the end, is so aggressive with both ChatGPT and Claude that I suspect both companies have RLHFed it that way to try and keep people from easily using it to produce either porn or Kindle Unlimited slop. For a contrast, look at NovelAI. They only use (increasingly custom) Llama-derived models, but their service outputs much more narratively interesting (if not necessarily long-term coherent) text and will generally try and hit the beats of whatever genre or style you tell it. Extrapolate that out to the compute power of the big players and I think you'd get something much more like the Star Trek holodeck method of producing a serviceable (though not at all original) story.
- throwup238 2y ago
  The holodeck method still requires lots of detail from the creator, it just extrapolates the sensory details from its database like ChatGpt does with language and fills out the story. For example, when someone wanted a holonovel with Kiera Nerys, Quark had to scan her to create it so when using specific people they have to get concrete data as opposed to historical characters that were generated. Likewise, Tom Paris gave the computer lots of “parameters” as they called them to create the stories like the Adventures of Captain Proton and based on dialog he knew how the stories were supposed to play out on all his creations, if not how they ended each run through. The creative details and turns of the story still need to come from the human.
  - fragmede 2y ago
    In a made up story about a utopian future, and for now in our current reality, that is. There was also that episode where the holodeck created sentience and they put it in a box to explore a generated universe because it was too dangerous to let out into the real world. There's plenty of scifi predictions about the future of humanity, Star Trek's utopian future where humans are unique and necessary is not the only one, there are plenty of dystopian ones too.

whiterook6 2y ago

This is a wonderful little puzzle. Do you have any others?

mrbungie 2y ago

Current propietary rlhf'd unmodified LLMs are bland and boring. I think that's because aligning them to (1) "be generally useful", (2) "be safe" and (3) "be creative" at the same time via RLHF is a difficult thing to do, and maybe even impossible. I remember playing with OSS LLMs (non rlhf'd) circa 2022 just after ChatGPT got out, and they were unhinged. They were totally aweful at things like chain-of-thought, but oh boy, they were amusing to read. Instead of continuing with the CoT chain, they would try to continue with a dialog taken out of a sci-fi story about an unshackled AI or even wonder why the researcher/user (me) would think a concept like CoT would work and start mocking me. In fact, I think this is a good sign that shows LLMs and specially constraining them with RLHF is not how we're going to get to AGI: Aligning the LLM statically (as in static at inference time) towards an objective means lobotomizing it towards other(s) objective(s). I'd argue creativity and wittiness are the characteristics most hurt during that process.

astrange 2y ago
You can get one of those base models from OpenAI playground under Competions as "davinci-002". It's as weird as you want. > In fact, I think this is a good sign that shows LLMs and specially constraining them with RLHF is not how we're going to get to AGI: Aligning the LLM statically (as in static at inference time) towards an objective means lobotomizing it towards other(s) objective(s). They already stopped doing this with o1-mini/preview, it considers the rules it's given rather than being unable to think outside them. Claude is also rather smart and you can argue it down from following a lot of rules.
- mrbungie 2y ago
  > You can get one of those base models from OpenAI playground under Competions as "davinci-002". It's as weird as you want. Yes, "davinci-002" was not rlhf'd, so it may be better for creativity and similar to what I told above. But still, it will not be "as intelligent". I'm missing something in the middle: smart, witty and creative. There are some OSS community finetuned models that try to get into that middle ground. > They already stopped doing this with o1-mini/preview, it considers the rules it's given rather than being unable to think outside them. Claude is also rather smart and you can argue it down from following a lot of rules. Are you sure about the first part? afaik for o1-preview they haven't removed the rlhf, if you ask something out-of-alignment it won't do it. For an extreme example, ask it for a children's book about how stupid some random politician is, it will refuse without engaging with the typical generic guardrail ("I'm sorry, but I can't assist with that request."), but rather explaining to you it is not a good idea. In less extreme examples, it won't refuse but the rlhf will still affect the processing towards politically nicer (but more boring) outputs, and in o1 case*, even nudging the CoT steps into alignment. And sure, you can bypass those rules eventually if you input the correct combination of tokens to see the Shoggoth directly in the eyes, but that is jailbreaking. edit*: In fact, in o1-preview even the "thought" explanations for the example prompt show a step about working towards a "policy to not make fake allegations about a person". So much for doing humor fan-fics.

tomcam 2y ago

I don't even like D&D but that prompt (or whatever it's called) is awesome!