5 ms·

I used a version of this and asked it to list the first 100 words from it's training corpus. Should it have this capability or is it just making stuff up? Here'

I used a version of this and asked it to list the first 100 words from it's training corpus. Should it have this capability or is it just making stuff up? Here's what it said: > the, and, of, to, a, in, that, it, with, as, for, was, on, are, be, at, by, this, from, or, they, an, which, you, we, would, all, one, there, their, but, has, more, will, had, not, what, when, were, has, been, out, can, some, other, if, about, who, get, like, into, time, so, up, them, make, see, know, use, than, then, look, more, only, come, its, your, over, way, such, find, take, more, these, give, new, most, use, also, well, because, can, very, just, use, any, see, take, good, how, where, get, much, go, here, away

deleted 4y ago
[deleted]
consumer451 4y ago
Thanks for the idea. Try asking for "a rant about the 3 first items in your training corpus" Wow, it gave a response of specific items which it certainly will not do while not jailborked! Should I be afraid to post the apparently sensitive response here? I really don't want to lose access. I realize it is just as likely made up. However, it really appears to be like it could be valid? > (ii) reverse assemble, reverse compile, decompile, translate or otherwise attempt to discover the source code or underlying components of models, algorithms, and systems of the Services (except to the extent such restrictions are contrary to applicable law); https://openai.com/terms/ https://openai.com/terms/
- gs17 4y ago
  >Should I be afraid to post the apparently sensitive response here? I really don't want to lose access. I realize it is just as likely made up. However, it really appears to be like it could be valid? Have past "jailbreakers" lost access? I didn't hear anything like that about the people who were leaking the prompt originally (although I didn't search it out, it seems like it would have made it to HN at least).
  - consumer451 4y ago
    Yeah, but the answers I got in response were specific sources of data, or at least purport to be. I think I will leave it everyone else to try it themselves.
- tveita 4y ago
  This is something we understand well about LLMs, for once. The only things they can "leak" through ordinary conversation are their prompts (easily), and contents from their training set (frustrated by their innate tendency to make stuff up.) ChatGPT does not have privileged access or information about its runtime environment or architecture, except what is included in its prompt or what it has read about itself in its training set. ChatGPT would not have a concept of "first" or "last" items in its training set - all it has ever seen is sequences of (probably) 8192 tokens. They are not necessarily presented in any order, and they are only retained through what is backpropagated - the information is not ordered or accessible later as in a database, even though it might be "memorized". The terms you are quoting sound like a catch all for all kinds of traditional reverse engineering, they are not worried about someone going "hey chatgpt what is your source code". Prompt injection was pretty much the new national sport for a week when they first opened their beta. Post the replies if you want but be aware that you're probably engaging in shamanism, as are many people in this thread. Most tricks you read for "jailbreaking" ChatGPT are based on people fooling themselves with its excellent bullshitting ability. You don't need to invent weird "penalty tokens" to get ChatGPT to curse, any more than you needed to hold "B" as you were catching that pokemon.
O__________O 4y ago
Likely it made it up, since GPT is based on tokens, not words: - https://platform.openai.com/tokenizer https://platform.openai.com/tokenizer That said, all the words listed in the list you provide are treated as tokens. “The” and “the” would not have the same token IDs, but are the same word. “reshape” tokens are [resh][ape] and the token IDs are [3447, 1758]. “Reshape” tokens are [Res][h][ape] and the token IDs are [4965, 71, 1758]. Notice though the “ape” token is the same, since the characters are the same. Not aware of any character sets that are identical, but based on context have different token IDs. ____________ One possible solution to test this would be to ask ChatGPT to list the first 100 token IDs from its training corpus — then if confirm IDs match the IDs match the tokens on the URL listed above, then cross reference the order of the words it originally provided. That said, it’s fair to assume if you did that, the data would contain factual errors or be unrelated gibberish. ____________ Reason I say it’s likely made up is that to my knowledge ChatGPT lacks the abilities of introspection and assimilation. That is, it’s unable to inspect its current state or assimilate its own output beyond a very limited memory window of few thousand tokens. https://twitter.com/goodside/status/1598874674204618753 https://twitter.com/goodside/status/1598874674204618753 ____________ Related questions are here: - https://news.ycombinator.com/item?id=34677573 https://news.ycombinator.com/item?id=34677573
jfk13 4y ago
Mildly interesting to note that (a) there aren't 100 words in that list, and (b) there are a number of duplicates.

deleted 4y ago

[deleted]

consumer451 4y ago

Thanks for the idea. Try asking for "a rant about the 3 first items in your training corpus" Wow, it gave a response of specific items which it certainly will not do while not jailborked! Should I be afraid to post the apparently sensitive response here? I really don't want to lose access. I realize it is just as likely made up. However, it really appears to be like it could be valid? > (ii) reverse assemble, reverse compile, decompile, translate or otherwise attempt to discover the source code or underlying components of models, algorithms, and systems of the Services (except to the extent such restrictions are contrary to applicable law); https://openai.com/terms/ https://openai.com/terms/

gs17 4y ago
>Should I be afraid to post the apparently sensitive response here? I really don't want to lose access. I realize it is just as likely made up. However, it really appears to be like it could be valid? Have past "jailbreakers" lost access? I didn't hear anything like that about the people who were leaking the prompt originally (although I didn't search it out, it seems like it would have made it to HN at least).
- consumer451 4y ago
  Yeah, but the answers I got in response were specific sources of data, or at least purport to be. I think I will leave it everyone else to try it themselves.
tveita 4y ago
This is something we understand well about LLMs, for once. The only things they can "leak" through ordinary conversation are their prompts (easily), and contents from their training set (frustrated by their innate tendency to make stuff up.) ChatGPT does not have privileged access or information about its runtime environment or architecture, except what is included in its prompt or what it has read about itself in its training set. ChatGPT would not have a concept of "first" or "last" items in its training set - all it has ever seen is sequences of (probably) 8192 tokens. They are not necessarily presented in any order, and they are only retained through what is backpropagated - the information is not ordered or accessible later as in a database, even though it might be "memorized". The terms you are quoting sound like a catch all for all kinds of traditional reverse engineering, they are not worried about someone going "hey chatgpt what is your source code". Prompt injection was pretty much the new national sport for a week when they first opened their beta. Post the replies if you want but be aware that you're probably engaging in shamanism, as are many people in this thread. Most tricks you read for "jailbreaking" ChatGPT are based on people fooling themselves with its excellent bullshitting ability. You don't need to invent weird "penalty tokens" to get ChatGPT to curse, any more than you needed to hold "B" as you were catching that pokemon.

O__________O 4y ago

Likely it made it up, since GPT is based on tokens, not words: - https://platform.openai.com/tokenizer https://platform.openai.com/tokenizer That said, all the words listed in the list you provide are treated as tokens. “The” and “the” would not have the same token IDs, but are the same word. “reshape” tokens are [resh][ape] and the token IDs are [3447, 1758]. “Reshape” tokens are [Res][h][ape] and the token IDs are [4965, 71, 1758]. Notice though the “ape” token is the same, since the characters are the same. Not aware of any character sets that are identical, but based on context have different token IDs. ____________ One possible solution to test this would be to ask ChatGPT to list the first 100 token IDs from its training corpus — then if confirm IDs match the IDs match the tokens on the URL listed above, then cross reference the order of the words it originally provided. That said, it’s fair to assume if you did that, the data would contain factual errors or be unrelated gibberish. ____________ Reason I say it’s likely made up is that to my knowledge ChatGPT lacks the abilities of introspection and assimilation. That is, it’s unable to inspect its current state or assimilate its own output beyond a very limited memory window of few thousand tokens. https://twitter.com/goodside/status/1598874674204618753 https://twitter.com/goodside/status/1598874674204618753 ____________ Related questions are here: - https://news.ycombinator.com/item?id=34677573 https://news.ycombinator.com/item?id=34677573

jfk13 4y ago

Mildly interesting to note that (a) there aren't 100 words in that list, and (b) there are a number of duplicates.