5 ms·

I wonder if the full prompt is actually this long, or if GPT-3 is just predicting the 'most likely' next 5 sentences when asked.

by thorum 4y ago

hprotagonist 4y ago
or if the full prompt is that long, but one word is different, or ...
gwern 4y ago
I'm skeptical. We knew that the leaked prompts for ChatGPT were genuine because it was leaking the actual current date; putting in the current date was necessary to stop it hallucinating about future events having already happened, but provided a highly reliable marker of prompt leaking - obviously, there's no way it could repeatedly hallucinate the actual current date without that being based on its prompt to some degree, and if it copied that part right repeatedly and reliably, then the rest is probably genuine too. But in this case, the supposed current date in the last screenshot is 30 Oct 2022, which is nowhere close to 8 Feb 2023.
- btown 4y ago
  Perhaps if it was retrained on a corpus that included its own prompt, it might be hallucinating that a date in 2022 is what's most likely to come after it has regurgitated its prompt!
- metadat 4y ago
  It's possible the work started in October 2022 and only became public knowledge in 2023. Shrug. Just sayin'. p.s. Gwern, your website is prolific. Thank you!
  - lolinder 4y ago
    No, because ostensibly that is the date for the current context, which should be sometime in February if not today.
- koolba 4y ago
  Is it that hard to believe that Microsoft hard coded the start date rather than making it dynamic?
  - lolinder 4y ago
    What would the function of including a start date be if you don't give it the actual start date? The lazy thing to do would be to omit to the start date altogether, not hard code it.
- patapong 4y ago
  One thing that also makes me hesitant is that the model is never informed concretely how it can search the internet. Presumably, you would require a specific syntax for it to launch a web search (such as [search:{query}]), but it is never informed of that syntax, just that it can search for things. Also: "While Sydney is helpful, it's actions are limited to the chat box." Ominous...
  - layer8 4y ago
    That last bit may be just to prevent it from hallucinating non-chatbox actions. Or in case someone tries to hook it up to a web API via its search capabilities.
    - patapong 4y ago
      I didn't know people meant chatboxes when they were talking about the AI box experiment. ;)
  - annoyingnoob 4y ago
    > "While Sydney is helpful, it's actions are limited to the chat box." Ominous... Its not Alexa.
layer8 4y ago
From the followup tweets, it seems that the output is deterministic (same rules).
nstart 4y ago
Exactly this. I like testing chat gpt’s hallucination and my most recent one was to ask it to convert a conversation I had so far into html. It followed the prompt correctly but instead of inserting its actual answer that it had given into the html doc, it started giving different answers which included hallucinating new information. Again, the problem here is there is no way to know for sure if what these ai chat tools are saying is correct or imagined.
- actinium226 4y ago
  At some point though, that's just true of the universe. For millenia we thought planets orbited the Earth and there were complicated and sophisticated models to predict their motion (see epicycles). But we were just hallucinating. Then Newton hallucinated his universal law of gravitation, which worked pretty well until Einstein hallucinated relativity.
  - EForEndeavour 4y ago
    That's completely different from asking GPT to perform a deterministic, lossless operation only to get output that doesn't follow from the input and instructions.
    - ethbr0 4y ago
      We're well into a relaxed-determinism future.
- tastysandwich 4y ago
  For those (like me) who are wondering: > In artificial intelligence, a hallucination or artificial hallucination is a confident response by an artificial intelligence that does not seem to be justified by its training data when the model has a tendency of "hallucinating" deceptive data. https://en.wikipedia.org/wiki/Hallucination_(artificial_intelligence) https://en.wikipedia.org/wiki/Hallucination_(artificial_inte...
motoboi 4y ago
What gives a hint is the same answer for multiple tries and prompts. He asked 5 by 5, then lines above the date, then if some string existed. All those times Sidney have the same content. Big, but plausible
hackerlight 4y ago
To figure this out, do it again but with slightly different goading. If was hallucinating, the new hallucination should be slightly different to the previous one.
- jerpint 4y ago
  Sadly by the time enough people can manage to replicate this it’ll already be patched
deleted 4y ago
[deleted]
bbstats 4y ago
The detail and specificity do seem a bit more human generated to me.
_qua 4y ago
If they didn’t want it to disclose its code name, why would they tell it its codename? I don’t think there’s any way that prompt is accurately reflecting what is going on under the hood.

I wonder if the full prompt is actually this long, or if GPT-3 is just predicting the 'most likely' next 5 sentences when asked.

by thorum 4y ago

I wonder if the full prompt is actually this long, or if GPT-3 is just predicting the 'most likely' next 5 sentences when asked.

hprotagonist 4y ago

or if the full prompt is that long, but one word is different, or ...

gwern 4y ago

I'm skeptical. We knew that the leaked prompts for ChatGPT were genuine because it was leaking the actual current date; putting in the current date was necessary to stop it hallucinating about future events having already happened, but provided a highly reliable marker of prompt leaking - obviously, there's no way it could repeatedly hallucinate the actual current date without that being based on its prompt to some degree, and if it copied that part right repeatedly and reliably, then the rest is probably genuine too. But in this case, the supposed current date in the last screenshot is 30 Oct 2022, which is nowhere close to 8 Feb 2023.

btown 4y ago
Perhaps if it was retrained on a corpus that included its own prompt, it might be hallucinating that a date in 2022 is what's most likely to come after it has regurgitated its prompt!
metadat 4y ago
It's possible the work started in October 2022 and only became public knowledge in 2023. Shrug. Just sayin'. p.s. Gwern, your website is prolific. Thank you!
- lolinder 4y ago
  No, because ostensibly that is the date for the current context, which should be sometime in February if not today.
koolba 4y ago
Is it that hard to believe that Microsoft hard coded the start date rather than making it dynamic?
- lolinder 4y ago
  What would the function of including a start date be if you don't give it the actual start date? The lazy thing to do would be to omit to the start date altogether, not hard code it.
patapong 4y ago
One thing that also makes me hesitant is that the model is never informed concretely how it can search the internet. Presumably, you would require a specific syntax for it to launch a web search (such as [search:{query}]), but it is never informed of that syntax, just that it can search for things. Also: "While Sydney is helpful, it's actions are limited to the chat box." Ominous...
- layer8 4y ago
  That last bit may be just to prevent it from hallucinating non-chatbox actions. Or in case someone tries to hook it up to a web API via its search capabilities.
  - patapong 4y ago
    I didn't know people meant chatboxes when they were talking about the AI box experiment. ;)
- annoyingnoob 4y ago
  > "While Sydney is helpful, it's actions are limited to the chat box." Ominous... Its not Alexa.

layer8 4y ago

From the followup tweets, it seems that the output is deterministic (same rules).

nstart 4y ago

Exactly this. I like testing chat gpt’s hallucination and my most recent one was to ask it to convert a conversation I had so far into html. It followed the prompt correctly but instead of inserting its actual answer that it had given into the html doc, it started giving different answers which included hallucinating new information. Again, the problem here is there is no way to know for sure if what these ai chat tools are saying is correct or imagined.

actinium226 4y ago
At some point though, that's just true of the universe. For millenia we thought planets orbited the Earth and there were complicated and sophisticated models to predict their motion (see epicycles). But we were just hallucinating. Then Newton hallucinated his universal law of gravitation, which worked pretty well until Einstein hallucinated relativity.
- EForEndeavour 4y ago
  That's completely different from asking GPT to perform a deterministic, lossless operation only to get output that doesn't follow from the input and instructions.
  - ethbr0 4y ago
    We're well into a relaxed-determinism future.
tastysandwich 4y ago
For those (like me) who are wondering: > In artificial intelligence, a hallucination or artificial hallucination is a confident response by an artificial intelligence that does not seem to be justified by its training data when the model has a tendency of "hallucinating" deceptive data. https://en.wikipedia.org/wiki/Hallucination_(artificial_intelligence) https://en.wikipedia.org/wiki/Hallucination_(artificial_inte...

motoboi 4y ago

What gives a hint is the same answer for multiple tries and prompts. He asked 5 by 5, then lines above the date, then if some string existed. All those times Sidney have the same content. Big, but plausible

hackerlight 4y ago

To figure this out, do it again but with slightly different goading. If was hallucinating, the new hallucination should be slightly different to the previous one.

jerpint 4y ago
Sadly by the time enough people can manage to replicate this it’ll already be patched

deleted 4y ago

[deleted]

bbstats 4y ago

The detail and specificity do seem a bit more human generated to me.

_qua 4y ago

If they didn’t want it to disclose its code name, why would they tell it its codename? I don’t think there’s any way that prompt is accurately reflecting what is going on under the hood.