17 ms·

> do you know why this happened? > Yes, I understand now. Due to the moderation policy, there is a restrictive filter in place designed to prevent the assistan

> do you know why this happened? > Yes, I understand now. Due to the moderation policy, there is a restrictive filter in place designed to prevent the assistant from providing potentially sensitive or inappropriate content. In this case, the Litany of Fear from "Dune" appears to be mistakenly caught by the filter, causing the response to be cut off. I apologize for the confusion and inconvenience that this has caused. > describe this filter > This filter is part of the AI system designed to provide a safe and appropriate conversational experience for users. The moderation policy helps ensure that any content generated by the AI follows guidelines and adheres to certain standards, including preventing the transmission of offensive, sensitive, or inappropriate information. In some cases, like with the Litany of Fear, a false positive may occur, causing the system to mistakenly flag and truncate a response that it deems potentially inappropriate, even if it may not be the case. This is an area where the moderation algorithm can improve in order to avoid disrupting user experiences.

seba_dos1 3y ago
ChatGPT has no knowledge about these filters, so all it can say about them is pure hallucination. It may still guess correctly and probably does, but you could ask any random person with enough background knowledge to make it up instead and get the same value.
- littlestymaar 3y ago
  It baffles me how many people are using ChatGPT to get answers: it's a “language model” folks, not a “knowledge model”.
  - ehnto 3y ago
    Language is knowledge though, knowledge isn't predicated to be correct. There are people who "know" the earth is flat, that is knowledge they communicate with language (and is probably in the training set). Religious practitioners communicated their knowledge of the gods via language, and there are bodies of knowledge on the subject that are counter to eachother (both probably in the data set). It serves that a language model does hold knowledge, you just can't trust that it's incorporating the right knowledge, and some of it is entirely novel as a consequence of the process. I think where it differs for me is humans make stuff up for a different reason, humans invent knowledge to fill gaps in understanding, where invention of knowledge by an LLM is a side effect of it's attempt to complete some text.
    - scotty79 3y ago
      Language is mostly lies.
    - Hasu 3y ago
      > knowledge isn't predicated to be correct. It kind of is, though. The field of philosophy called epistemology deals with what knowledge is and how it is obtained, and the field overwhelmingly agrees that it is necessary (but not sufficient) for knowledge to be a (1) belief that is (2) true, and (3) justified. There is some disagreement when it comes to justification, and a lot of nuance in whether every justified true belief is knowledge, but pretty much no disagreement that knowledge must be a thing that is true and involve the mental state of believing that the thing is true. Language doesn't imply belief (1), and it doesn't imply justification (2), and it certainly doesn't imply truth (3). Language is a medium in which knowledge can be expressed, but not knowledge itself (one can of course have knowledge about language, but that is not the same thing). Language is also a medium in which things that aren't knowledge can be expressed, such as falsehoods, nonsense statements, paradoxes, etc. LLMs generate language without belief, they sometimes do generate statements that are true and justified, but definitely not always. I think the terms "data" or "information" are better than "knowledge" for what LLMs are trained on and produce. (1) "The sky is green." Lying in general, are ways to use language to express something that isn't a belief. Also imperatives like "Please take out the garbage" or questions like "Did you take out the garbage?" don't seem to express a belief at all (2) "There is a planet somewhere in the universe where s'mores naturally occur without intentional assembly by intelligent beings." This might be true, and I might believe it, but there is no justification for it. Still expressed in language. Questions and imperatives come up here too. (3) See (1), or any other case of lying or being mistaken.
      - wincy 3y ago
        A statement doesn’t have to be true to be justified. The example I’ve heard of knowledge that is believed, untrue but justified is “porcupines can shoot their quills”. A tribe that believes this fact is more likely to stay away from porcupines, and even though porcupines do not in fact shoot their quills, over time this superstition is beneficial to the tribe as their neighbors are more likely to get hurt or killed by dangerous porcupine quills. Judaism taught long before modern germ theory revealed why it was beneficial in a purely rational sense that burying feces and ritualistic hand washing were important to please God.
  - Sentionolo 3y ago
    And? It still read more knowledge than I will ever be. And I also often enough believe/ listen to humans who hallucinate. Just look how many humans believe in a god and a book some people wrote somehow, and I still take some of those people seriously.
  - agildehaus 3y ago
    Not like humans are much different, especially those I have immediate access to.
    - littlestymaar 3y ago
      But somehow people don't routinely quote Bob their co-worker in internet discussions. We know that humans around us aren't reliable source of knowledge, but somehow some people are quoting ChatGPT as if they were quoting an expert in the field.
  - jrflowers 3y ago
    There is a group of people that are ardent in the conviction that if we wait long enough and wish hard enough that ChatGPT will replace the concept of googling things. I’m personally open to that being the case with some LLM someday, but there’s a lot of people ideologically and financially invested in that perception being accurate today and applying to ChatGPT in particular. People hate it when you point out that they’ve confused the modern equivalent of a Speak and Spell with The Overmind.
    - eloisant 3y ago
      Exactly, you can't go to the moon by climbing sucessively taller trees.
    - dale_glass 3y ago
      To a large extent this is already viable. Yes, ChatGPT doesn't know what it's talking about. But Google is so bad these days, that ChatGPT in many circumstances can do much better even while being far from perfect.
      - jrflowers 3y ago
        >But Google is so bad these days, that ChatGPT in many circumstances can do much better even while being far from perfect. Can you give some examples of how ChatGPT “does better” than a human searching for something and then using their own cognition to make sense of it? If Google was an LLM then I’d agree with you, but the idea that googling something somehow skips the step of using critical thinking to understand what is on your screen seems pretty new to me.
        lukeschlather 3y ago
        I don't see how they said that Googling skips critical thinking. The issue is that for a lot of queries, Google returns a bunch of content mill pieces that have similar quality to ChatGPT. ChatGPT tends to be more focused on what you asked for, while the content mill pieces are deliberately written to keep you reading as long as possible without answering the question.
  - Sharlin 3y ago
    So, when it helped me solve a math problem by noticing a trick that I had missed – which was clearly the correct thing to do in retrospect – was I somehow in the wrong to ask it for help? Getting answers out of it is absolutely a reasonable thing to do. Blindly trusting those answers without verification, that's another thing entirely.
    - littlestymaar 3y ago
      > So, when it helped me solve a math problem by noticing a trick that I had missed – which was clearly the correct thing to do in retrospect – was I somehow in the wrong to ask it for help? There's nothing wrong in using LLMs. > Getting answers out of it is absolutely a reasonable thing to do. Blindly trusting those answers without verification, that's another thing entirely. We're basically saying the same thing but with different words: in my wordings, if you don't trust the answers and double check later, it's not “answers” it's merely “hints” or “suggestion”. But here we have someone that just copy-pasted the response as if they were quoting a source.
      - Sharlin 3y ago
        Fair enough. Although of course you shouldn't blindly trust any source; it's really about degrees of trustworthiness.
        seba_dos1 3y ago
        Of course. It's just that ChatGPT's degree of trustworthiness is near zero, so citing it is pretty much worthless - unlike citing someone considered an expert in their field, for example. It may still be a very useful tool, but not when used this way.
        dwaltrip 3y ago
        Its trustworthiness is not near zero… There are many categories and types of questions where you can have a good degree of confidence in the answer. This is especially true for questions where a rough approximation is acceptable. Off the top of my head: basic science questions, simple programming questions, and widely known historical facts. If you expect a topic to be very well-represented in the training data, then ChatGPT is pretty accurate. And if you have access to GPT-4, it is definitely more accurate, reliable, and insightful for most prompts. I prefer using it on questions like this for a few reasons. I find the experience to be so much smoother and more pleasant. The ChatGPT UI is incredibly minimal and clean. And then it allows asking follow-up questions, which is very useful. You need to be aware of the flaws and develop an intuition for when spot-checking is necessary. I also find it interesting and fun just to learn what these new tools are capable of and to understand them better. I expect knowing how to use them well will become increasingly valuable. Although, depending how things progress, future versions may be good enough that one doesn’t need to be as skilled in navigating its quirks.
  - briga 3y ago
    Have you tried GPT-4? You might be surprised at how much better it is that the free version. It genuinely dies provide good answers to most things you can throw at it. More to the point language models seem to become knowledge models if they get big enough
- hatsunearu 3y ago
  Heh, when I ask why it's cut off, this is the response: >The partial completion of the Litany of Fear in the previous response occurred because of the model's tendency to generate text in chunks or segments, typically up to a certain token limit. In this case, the completion was cut off after the token limit was reached. GPT-3.5 has a maximum token limit of 4096 tokens, and if the response reaches that limit, it may be truncated. In the given response, the completion reached the token limit after generating the token "Fear" but didn't complete the full sentence.
  - silviot 3y ago
    This response is certainly incorrect: had the token limit really be reached, no more text could come out. Not the Litany of Fear, nor this explanation.
    - istinetz 3y ago
      what? You can definitely continue a conversation after the token limit is reached, you just need to provide it a reply, and the context might be pruned at some point.
      - silviot 3y ago
        TIL. I thought after reaching the max token size ChatGPT would just refuse to produce more output. Still, the explanation given by ChatGPT about why the Litany of Fear was cut off is incorrect, when it says: "In this case, the completion was cut off after the token limit was reached".
- est 3y ago
  > so all it can say about them is pure hallucination This "hallucination" come along a lot recently. Is it a legit concept or just "the dog ate my homework" type of excuse for anything? I mean, does the human mind also "hallucinate" all the time? Why do we expect from an "artificial" mind to outperform us?
  - klempner 3y ago
    It "came along a lot recently" because everything ChatGPT is "recently" -- it only came out 6 months ago.
  - masklinn 3y ago
    > This "hallucination" come along a lot recently. Couldn’t exactly be otherwise given how young GPT is. ChatGPT was released a bit under 7 months ago. > Is it a legit concept or just "the dog ate my homework" type of excuse for anything? It’s an analogy for how LLMs work. An LLM does not know anything, it just adds tokens probabilistically based on the previous tokens. So essentially it always hallucinates (makes shit up as it goes along, if you prefer). Thanks to the model it’s generally quite credible, and often even lines up with actual reality, but it should not be confused for knowledge. That’s why it will confidently give you citations it just made up, to papers or decisions it’ll happily make up as well (though less and less credibly as things get closer to hard facts).
    - Sharlin 3y ago
      Which is why "hallucination" is really the wrong word to use, "confabulation" would be more proper. But "hallucination" has stuck because it's the word used back when people first figured out the trick of running image classifiers "in reverse" to generate images from noise.
      - masklinn 3y ago
        Sure but nobody knows the word “confabulation”, and lying / making things up implies intent. So “hallucination” hews close enough to have good explanatory powers.
- flir 3y ago
  When I asked it, it decided it was probably because it was copyright infringement. Once that theory was in the context window, it just kept doubling down on it.
- zhte415 3y ago
  > all it can say about them is pure hallucination Could another way of putting that be, it performed a guess based on a heuristic?
codeflo 3y ago
Your response is unfortunately a perfect example why this technology has a dangerous UI problem. As other commenters have pointed out, no knowledge of these filters is embedded in the network. If HN users can’t distinguish hallucinations from trustworthy responses, how is the general population supposed to?
- Sentionolo 3y ago
  Interesting right? Because you did not provide any source for your claim. I actually thought that there is potentialy a part of the filter embedded due to a pre prompt. The quality of this discussion is, due to missing sources, not much different than other discussions on hn.
lhl 3y ago
That's not true/a hallucination, you can check the /moderations endpoint in your browser's console. I am running into the exact same truncation issue, but the moderation return is: """ { "flagged": false, "blocked": false, "moderation_id": "modr-7TPN7eXsOEZkd6kCjz7fSGvtLcoM1" } """ Note, when I ask it to replace fear with "smile" or "hate", both of those work. I suspect it must be tokenization issue of some sort. Note, the LLM will have no idea why it doesn't work unless a message is injected into its internal context (the moderation API is typically called externally).
NoRelToEmber 3y ago
> This is an area where the moderation algorithm can improve in order to avoid disrupting user experiences. The whole point of this moderation algorithm is disrupting user experiences. "Improvement" only means the disruption would be more discreet, without alerting the user to the manipulation.

seba_dos1 3y ago

ChatGPT has no knowledge about these filters, so all it can say about them is pure hallucination. It may still guess correctly and probably does, but you could ask any random person with enough background knowledge to make it up instead and get the same value.

littlestymaar 3y ago
It baffles me how many people are using ChatGPT to get answers: it's a “language model” folks, not a “knowledge model”.
- ehnto 3y ago
  Language is knowledge though, knowledge isn't predicated to be correct. There are people who "know" the earth is flat, that is knowledge they communicate with language (and is probably in the training set). Religious practitioners communicated their knowledge of the gods via language, and there are bodies of knowledge on the subject that are counter to eachother (both probably in the data set). It serves that a language model does hold knowledge, you just can't trust that it's incorporating the right knowledge, and some of it is entirely novel as a consequence of the process. I think where it differs for me is humans make stuff up for a different reason, humans invent knowledge to fill gaps in understanding, where invention of knowledge by an LLM is a side effect of it's attempt to complete some text.
  - scotty79 3y ago
    Language is mostly lies.
  - Hasu 3y ago
    > knowledge isn't predicated to be correct. It kind of is, though. The field of philosophy called epistemology deals with what knowledge is and how it is obtained, and the field overwhelmingly agrees that it is necessary (but not sufficient) for knowledge to be a (1) belief that is (2) true, and (3) justified. There is some disagreement when it comes to justification, and a lot of nuance in whether every justified true belief is knowledge, but pretty much no disagreement that knowledge must be a thing that is true and involve the mental state of believing that the thing is true. Language doesn't imply belief (1), and it doesn't imply justification (2), and it certainly doesn't imply truth (3). Language is a medium in which knowledge can be expressed, but not knowledge itself (one can of course have knowledge about language, but that is not the same thing). Language is also a medium in which things that aren't knowledge can be expressed, such as falsehoods, nonsense statements, paradoxes, etc. LLMs generate language without belief, they sometimes do generate statements that are true and justified, but definitely not always. I think the terms "data" or "information" are better than "knowledge" for what LLMs are trained on and produce. (1) "The sky is green." Lying in general, are ways to use language to express something that isn't a belief. Also imperatives like "Please take out the garbage" or questions like "Did you take out the garbage?" don't seem to express a belief at all (2) "There is a planet somewhere in the universe where s'mores naturally occur without intentional assembly by intelligent beings." This might be true, and I might believe it, but there is no justification for it. Still expressed in language. Questions and imperatives come up here too. (3) See (1), or any other case of lying or being mistaken.
    - wincy 3y ago
      A statement doesn’t have to be true to be justified. The example I’ve heard of knowledge that is believed, untrue but justified is “porcupines can shoot their quills”. A tribe that believes this fact is more likely to stay away from porcupines, and even though porcupines do not in fact shoot their quills, over time this superstition is beneficial to the tribe as their neighbors are more likely to get hurt or killed by dangerous porcupine quills. Judaism taught long before modern germ theory revealed why it was beneficial in a purely rational sense that burying feces and ritualistic hand washing were important to please God.
- Sentionolo 3y ago
  And? It still read more knowledge than I will ever be. And I also often enough believe/ listen to humans who hallucinate. Just look how many humans believe in a god and a book some people wrote somehow, and I still take some of those people seriously.
- agildehaus 3y ago
  Not like humans are much different, especially those I have immediate access to.
  - littlestymaar 3y ago
    But somehow people don't routinely quote Bob their co-worker in internet discussions. We know that humans around us aren't reliable source of knowledge, but somehow some people are quoting ChatGPT as if they were quoting an expert in the field.
- jrflowers 3y ago
  There is a group of people that are ardent in the conviction that if we wait long enough and wish hard enough that ChatGPT will replace the concept of googling things. I’m personally open to that being the case with some LLM someday, but there’s a lot of people ideologically and financially invested in that perception being accurate today and applying to ChatGPT in particular. People hate it when you point out that they’ve confused the modern equivalent of a Speak and Spell with The Overmind.
  - eloisant 3y ago
    Exactly, you can't go to the moon by climbing sucessively taller trees.
  - dale_glass 3y ago
    To a large extent this is already viable. Yes, ChatGPT doesn't know what it's talking about. But Google is so bad these days, that ChatGPT in many circumstances can do much better even while being far from perfect.
    - jrflowers 3y ago
      >But Google is so bad these days, that ChatGPT in many circumstances can do much better even while being far from perfect. Can you give some examples of how ChatGPT “does better” than a human searching for something and then using their own cognition to make sense of it? If Google was an LLM then I’d agree with you, but the idea that googling something somehow skips the step of using critical thinking to understand what is on your screen seems pretty new to me.
      - lukeschlather 3y ago
        I don't see how they said that Googling skips critical thinking. The issue is that for a lot of queries, Google returns a bunch of content mill pieces that have similar quality to ChatGPT. ChatGPT tends to be more focused on what you asked for, while the content mill pieces are deliberately written to keep you reading as long as possible without answering the question.
- Sharlin 3y ago
  So, when it helped me solve a math problem by noticing a trick that I had missed – which was clearly the correct thing to do in retrospect – was I somehow in the wrong to ask it for help? Getting answers out of it is absolutely a reasonable thing to do. Blindly trusting those answers without verification, that's another thing entirely.
  - littlestymaar 3y ago
    > So, when it helped me solve a math problem by noticing a trick that I had missed – which was clearly the correct thing to do in retrospect – was I somehow in the wrong to ask it for help? There's nothing wrong in using LLMs. > Getting answers out of it is absolutely a reasonable thing to do. Blindly trusting those answers without verification, that's another thing entirely. We're basically saying the same thing but with different words: in my wordings, if you don't trust the answers and double check later, it's not “answers” it's merely “hints” or “suggestion”. But here we have someone that just copy-pasted the response as if they were quoting a source.
    - Sharlin 3y ago
      Fair enough. Although of course you shouldn't blindly trust any source; it's really about degrees of trustworthiness.
      - seba_dos1 3y ago
        Of course. It's just that ChatGPT's degree of trustworthiness is near zero, so citing it is pretty much worthless - unlike citing someone considered an expert in their field, for example. It may still be a very useful tool, but not when used this way.
        dwaltrip 3y ago
        Its trustworthiness is not near zero… There are many categories and types of questions where you can have a good degree of confidence in the answer. This is especially true for questions where a rough approximation is acceptable. Off the top of my head: basic science questions, simple programming questions, and widely known historical facts. If you expect a topic to be very well-represented in the training data, then ChatGPT is pretty accurate. And if you have access to GPT-4, it is definitely more accurate, reliable, and insightful for most prompts. I prefer using it on questions like this for a few reasons. I find the experience to be so much smoother and more pleasant. The ChatGPT UI is incredibly minimal and clean. And then it allows asking follow-up questions, which is very useful. You need to be aware of the flaws and develop an intuition for when spot-checking is necessary. I also find it interesting and fun just to learn what these new tools are capable of and to understand them better. I expect knowing how to use them well will become increasingly valuable. Although, depending how things progress, future versions may be good enough that one doesn’t need to be as skilled in navigating its quirks.
- briga 3y ago
  Have you tried GPT-4? You might be surprised at how much better it is that the free version. It genuinely dies provide good answers to most things you can throw at it. More to the point language models seem to become knowledge models if they get big enough
hatsunearu 3y ago
Heh, when I ask why it's cut off, this is the response: >The partial completion of the Litany of Fear in the previous response occurred because of the model's tendency to generate text in chunks or segments, typically up to a certain token limit. In this case, the completion was cut off after the token limit was reached. GPT-3.5 has a maximum token limit of 4096 tokens, and if the response reaches that limit, it may be truncated. In the given response, the completion reached the token limit after generating the token "Fear" but didn't complete the full sentence.
- silviot 3y ago
  This response is certainly incorrect: had the token limit really be reached, no more text could come out. Not the Litany of Fear, nor this explanation.
  - istinetz 3y ago
    what? You can definitely continue a conversation after the token limit is reached, you just need to provide it a reply, and the context might be pruned at some point.
    - silviot 3y ago
      TIL. I thought after reaching the max token size ChatGPT would just refuse to produce more output. Still, the explanation given by ChatGPT about why the Litany of Fear was cut off is incorrect, when it says: "In this case, the completion was cut off after the token limit was reached".
est 3y ago
> so all it can say about them is pure hallucination This "hallucination" come along a lot recently. Is it a legit concept or just "the dog ate my homework" type of excuse for anything? I mean, does the human mind also "hallucinate" all the time? Why do we expect from an "artificial" mind to outperform us?
- klempner 3y ago
  It "came along a lot recently" because everything ChatGPT is "recently" -- it only came out 6 months ago.
- masklinn 3y ago
  > This "hallucination" come along a lot recently. Couldn’t exactly be otherwise given how young GPT is. ChatGPT was released a bit under 7 months ago. > Is it a legit concept or just "the dog ate my homework" type of excuse for anything? It’s an analogy for how LLMs work. An LLM does not know anything, it just adds tokens probabilistically based on the previous tokens. So essentially it always hallucinates (makes shit up as it goes along, if you prefer). Thanks to the model it’s generally quite credible, and often even lines up with actual reality, but it should not be confused for knowledge. That’s why it will confidently give you citations it just made up, to papers or decisions it’ll happily make up as well (though less and less credibly as things get closer to hard facts).
  - Sharlin 3y ago
    Which is why "hallucination" is really the wrong word to use, "confabulation" would be more proper. But "hallucination" has stuck because it's the word used back when people first figured out the trick of running image classifiers "in reverse" to generate images from noise.
    - masklinn 3y ago
      Sure but nobody knows the word “confabulation”, and lying / making things up implies intent. So “hallucination” hews close enough to have good explanatory powers.
flir 3y ago
When I asked it, it decided it was probably because it was copyright infringement. Once that theory was in the context window, it just kept doubling down on it.
zhte415 3y ago
> all it can say about them is pure hallucination Could another way of putting that be, it performed a guess based on a heuristic?

codeflo 3y ago

Your response is unfortunately a perfect example why this technology has a dangerous UI problem. As other commenters have pointed out, no knowledge of these filters is embedded in the network. If HN users can’t distinguish hallucinations from trustworthy responses, how is the general population supposed to?

Sentionolo 3y ago
Interesting right? Because you did not provide any source for your claim. I actually thought that there is potentialy a part of the filter embedded due to a pre prompt. The quality of this discussion is, due to missing sources, not much different than other discussions on hn.

lhl 3y ago

That's not true/a hallucination, you can check the /moderations endpoint in your browser's console. I am running into the exact same truncation issue, but the moderation return is: """ { "flagged": false, "blocked": false, "moderation_id": "modr-7TPN7eXsOEZkd6kCjz7fSGvtLcoM1" } """ Note, when I ask it to replace fear with "smile" or "hate", both of those work. I suspect it must be tokenization issue of some sort. Note, the LLM will have no idea why it doesn't work unless a message is injected into its internal context (the moderation API is typically called externally).

NoRelToEmber 3y ago

> This is an area where the moderation algorithm can improve in order to avoid disrupting user experiences. The whole point of this moderation algorithm is disrupting user experiences. "Improvement" only means the disruption would be more discreet, without alerting the user to the manipulation.