8 ms·

Fyi, asking the model about it's cutoff date is not a reliable way of finding out what it has been trained on. The model doesn't know that, unless it was provi

by maest 3y ago

Fyi, asking the model about it's cutoff date is not a reliable way of finding out what it has been trained on. The model doesn't know that, unless it was provided that info during fine-tuning or something -- but even then it doesn't guarantee anything. The more useful thing to do is ask it about events which happened during a particular time range. That way, you'll know it has at least seen the news.

retrofuturism 3y ago
Information about cut-off date is very much part of its fine-tuning.
- kesor 3y ago
  Apparently it is also part of its system prompt, since otherwise it wouldn't know what the cutoff date is just by feeding it fresher information - it has to be told the date explicitely somewhere.
  - haltist 3y ago
    It's possible the date is hallucinated. There is no reason that a combination of system prompt and regular prompt when combined can not generate a hallucinated cut off date that does not match the actual date. LLMs are statistical models and simply generate probable sequences of tokens based on a context (very much like sampling from Markov chains) so there is no a priori reason to believe that the cut off date is accurate. More generally, all output from the model that seems to be model metadata should be assumed to be a hallucination.
    - flangola7 3y ago
      When it can be repeated dozens of times consistently that is strong reason to believe it is part of the system prompt. Baseless hallucinations will be different everytime.
      - vineyardmike 3y ago
        If the model didn’t change, why would the hallucinations change?
        kesor 3y ago
        Temperature.
        flangola7 3y ago
        I believe the model did change.
hnben 3y ago
back when prompt-hacking was a thing, you could ask chatgpt to print out the system prompt (i.e. the part of the conversation before the user entered the chat). Iirc the system prompt hat this exact info in it. Iirc it was surprisingly light. Only the name and role of the AI, the cutoff-date and the current date. edit: found an example https://old.reddit.com/r/ChatGPT/comments/zuhkvq// https://old.reddit.com/r/ChatGPT/comments/zuhkvq//
- p-e-w 3y ago
  That is a common misunderstanding. Even if no safeguards are in place, asking an LLM what its "system prompt" is does not guarantee it will accurately reproduce the same. LLMs are not databases. They don't have perfect recall. What they print when asked such a question may or may not be the actual system prompt, and there is no way to tell for sure.
  - benoliver999 3y ago
    Especially now if it's hoovered up endless blog posts about prompt hacking
  - circuit10 3y ago
    If you get the same result over and over again it's more likely to be true
    - whoisthemachine 3y ago
      If you get the same result over and over again, it means the model is more overfit to a certain result. It does not mean the result is correct.
      - Jerrrry 3y ago
        Thank you, this is one of the most understood 'facts', especially regarding "prompt hacking/jailbreaking"
      - 3y ago
xkcd1963 3y ago
The model doesn't have the capability of "knowing something"
- kesor 3y ago
  The model doesn't "know" anything. But if you provide it with certain assertions, it will repeat them back to you. For example, if you make a prompt saying "XX represents up, and YY represents down. Tell me how to write upside-down using these." ; Then you can argue that it "knows" which pairs of characters represent which direction. Not real knowledge, just regurgitation of whatever it was prompted with. https://chat.openai.com/share/f6c1d260-2246-4388-8c98-290a1d87ecf2 https://chat.openai.com/share/f6c1d260-2246-4388-8c98-290a1d... In exactly the same way your prompt is preceded with OpenAI's prefix that sets up some "facts" about itself, telling the model what its name is, what today's date is, and what it's knowledge cutoff date is. Even though they might have trained the model on much older (or much newer) data and texts, it will still tell you the cutoff date that is part of the prompt it was provided with.
  - xkcd1963 3y ago
    I'm not arguing against the useful capabilities of openai but rather the language we are using to describe LLMs. The terms "Thinking" and "Knowing" are not be used on computer systems. We should say "processing" and "in memory" instead, or similar. Why is it important not to use the false terms? Because its not the same and some people might start confusing machines for humans.

by maest 3y ago

retrofuturism 3y ago

Information about cut-off date is very much part of its fine-tuning.

kesor 3y ago
Apparently it is also part of its system prompt, since otherwise it wouldn't know what the cutoff date is just by feeding it fresher information - it has to be told the date explicitely somewhere.
- haltist 3y ago
  It's possible the date is hallucinated. There is no reason that a combination of system prompt and regular prompt when combined can not generate a hallucinated cut off date that does not match the actual date. LLMs are statistical models and simply generate probable sequences of tokens based on a context (very much like sampling from Markov chains) so there is no a priori reason to believe that the cut off date is accurate. More generally, all output from the model that seems to be model metadata should be assumed to be a hallucination.
  - flangola7 3y ago
    When it can be repeated dozens of times consistently that is strong reason to believe it is part of the system prompt. Baseless hallucinations will be different everytime.
    - vineyardmike 3y ago
      If the model didn’t change, why would the hallucinations change?
      - kesor 3y ago
        Temperature.
      - flangola7 3y ago
        I believe the model did change.

hnben 3y ago

back when prompt-hacking was a thing, you could ask chatgpt to print out the system prompt (i.e. the part of the conversation before the user entered the chat). Iirc the system prompt hat this exact info in it. Iirc it was surprisingly light. Only the name and role of the AI, the cutoff-date and the current date. edit: found an example https://old.reddit.com/r/ChatGPT/comments/zuhkvq// https://old.reddit.com/r/ChatGPT/comments/zuhkvq//

p-e-w 3y ago
That is a common misunderstanding. Even if no safeguards are in place, asking an LLM what its "system prompt" is does not guarantee it will accurately reproduce the same. LLMs are not databases. They don't have perfect recall. What they print when asked such a question may or may not be the actual system prompt, and there is no way to tell for sure.
- benoliver999 3y ago
  Especially now if it's hoovered up endless blog posts about prompt hacking
- circuit10 3y ago
  If you get the same result over and over again it's more likely to be true
  - whoisthemachine 3y ago
    If you get the same result over and over again, it means the model is more overfit to a certain result. It does not mean the result is correct.
    - Jerrrry 3y ago
      Thank you, this is one of the most understood 'facts', especially regarding "prompt hacking/jailbreaking"
    - 3y ago

xkcd1963 3y ago

The model doesn't have the capability of "knowing something"

kesor 3y ago
The model doesn't "know" anything. But if you provide it with certain assertions, it will repeat them back to you. For example, if you make a prompt saying "XX represents up, and YY represents down. Tell me how to write upside-down using these." ; Then you can argue that it "knows" which pairs of characters represent which direction. Not real knowledge, just regurgitation of whatever it was prompted with. https://chat.openai.com/share/f6c1d260-2246-4388-8c98-290a1d87ecf2 https://chat.openai.com/share/f6c1d260-2246-4388-8c98-290a1d... In exactly the same way your prompt is preceded with OpenAI's prefix that sets up some "facts" about itself, telling the model what its name is, what today's date is, and what it's knowledge cutoff date is. Even though they might have trained the model on much older (or much newer) data and texts, it will still tell you the cutoff date that is part of the prompt it was provided with.
- xkcd1963 3y ago
  I'm not arguing against the useful capabilities of openai but rather the language we are using to describe LLMs. The terms "Thinking" and "Knowing" are not be used on computer systems. We should say "processing" and "in memory" instead, or similar. Why is it important not to use the false terms? Because its not the same and some people might start confusing machines for humans.