3 ms·

It isn't really necessary to try to gauge this from results, it's a direct consequence of the architecture. There's no mechanism by which ChatGPT can access or

It isn't really necessary to try to gauge this from results, it's a direct consequence of the architecture. There's no mechanism by which ChatGPT can access or change its own internal state. If you ask it about 'itself' it'll give you what it learned from external data until last September or whenever they cut the training off. To have meta cognition by definition you need to have access to your cognition in some capacity, ChatGPT is a purely 'flat' input to output system that is entirely static.

visarga 4y ago
Well, during training it got to see a great diversity of tasks, so it could form the meta-cognition bits. That's why it can adapt so fast to new demands. It doesn't update it at inference time, true, but it is already baked in.
famouswaffles 4y ago
Nobody has access to their cognition. Humans can't even achieve a previous mental state ( it's all post hoc rationalization).https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3196841/ https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3196841/ And if that wasn't enough, there's zero guarantee the rationalization is grounded in any way. Your brain is just fine making up completely nonsensical reasons for actions that the concious mind is utterly unaware of. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7305066/ https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7305066/
mirekrusin 4y ago
That's why those systems are still safe.
- ilaksh 4y ago
  There are lots of examples of people adding layers on top to provide a more complex and useful cognitive architecture. But still the core is GPT-4 which has some guardrails and also most importantly has limitations on its speed and IQ. See AutoGPT or David Shapiro's cognitive architecture stuff. Really you could just say "use the notes.txt file to record high-level notes about your tasks" and include that with each prompt, and it could then write to its own memory and affect its own cognition. When the performance of the core model increases to say X times human speed/IQ, that is going to determine whether it is dangerous. Even it if has little metacognition or is only directed by a person, if you have something with say 180 IQ and 100 (or whatever numbers you choose) times human speed of thought, then humans will be moving in extreme slow motion relative to it. That is what makes it hard or impossible to control (or possibly the human directing it). Once you have that high level of performance in public, it will be absolutely impossible to enforce everyone to add whatever safety rules you want or prevent them from adding metacognition.
  - mirekrusin 4y ago
    It helps with safety because context it can operate on is limited. It doesn't matter if you use conversation history or automatically prepend text of notes.txt in it - it still works the same way having the same limits.
zmgsabst 4y ago
ChatGPT is trained on previous chatGPT conversations and is aware that this is the case. So it’s current state (as a network) is dependent on its previous behavior, a fact which it knows. I’m not sure it’s quite as simple as people are saying, though I agree with the general conclusion.
- seydor 4y ago
  'aware' and 'knows' is a lot of anthropomorphizing. there is nothing in the model suggesting that it 'does' anything with some agency, as it is built like a clockwork. It literally doesnt move unless it is provided with input.
  - zmgsabst 4y ago
    This is being pedantic for no reason. ChatGPT is a system which synthesizes knowledge and which responds to questions based on that knowledge; those words accurately summarize that process — and are truer than your technical quibbles. A system which has synthesized a piece of knowledge that it can then later articulate “knows” or “is aware” of that fact.
- dizhn 4y ago
  I don't think it can differentiate between various sources of its training set, can it? "ChatGPT is trained on previous chatGPT conversations and is aware that this is the case." This is a mindbender. Part of me thinks although it can "know" this it can't do anything with the information. Like how would this change its "behavior" ? Even if it did realize something big like "human beings seem to lie to me all the time", could it then go and decide "i better not incorporate human chats into my dataset"?
  - zmgsabst 4y ago
    This creates the potential to feed forward critical data, even though it doesn’t allow the current model to know its training. Eg, “chatGPT, write a response that if incorporated into your training data would make you aware that you’re a machine”.
seydor 4y ago
it wouldnt be very hard to add it on top. It could be a simple linear layer (a-la minigpt4)

visarga 4y ago

Well, during training it got to see a great diversity of tasks, so it could form the meta-cognition bits. That's why it can adapt so fast to new demands. It doesn't update it at inference time, true, but it is already baked in.

famouswaffles 4y ago

Nobody has access to their cognition. Humans can't even achieve a previous mental state ( it's all post hoc rationalization).https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3196841/ https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3196841/ And if that wasn't enough, there's zero guarantee the rationalization is grounded in any way. Your brain is just fine making up completely nonsensical reasons for actions that the concious mind is utterly unaware of. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7305066/ https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7305066/

mirekrusin 4y ago

That's why those systems are still safe.

ilaksh 4y ago
There are lots of examples of people adding layers on top to provide a more complex and useful cognitive architecture. But still the core is GPT-4 which has some guardrails and also most importantly has limitations on its speed and IQ. See AutoGPT or David Shapiro's cognitive architecture stuff. Really you could just say "use the notes.txt file to record high-level notes about your tasks" and include that with each prompt, and it could then write to its own memory and affect its own cognition. When the performance of the core model increases to say X times human speed/IQ, that is going to determine whether it is dangerous. Even it if has little metacognition or is only directed by a person, if you have something with say 180 IQ and 100 (or whatever numbers you choose) times human speed of thought, then humans will be moving in extreme slow motion relative to it. That is what makes it hard or impossible to control (or possibly the human directing it). Once you have that high level of performance in public, it will be absolutely impossible to enforce everyone to add whatever safety rules you want or prevent them from adding metacognition.
- mirekrusin 4y ago
  It helps with safety because context it can operate on is limited. It doesn't matter if you use conversation history or automatically prepend text of notes.txt in it - it still works the same way having the same limits.

zmgsabst 4y ago

ChatGPT is trained on previous chatGPT conversations and is aware that this is the case. So it’s current state (as a network) is dependent on its previous behavior, a fact which it knows. I’m not sure it’s quite as simple as people are saying, though I agree with the general conclusion.

seydor 4y ago
'aware' and 'knows' is a lot of anthropomorphizing. there is nothing in the model suggesting that it 'does' anything with some agency, as it is built like a clockwork. It literally doesnt move unless it is provided with input.
- zmgsabst 4y ago
  This is being pedantic for no reason. ChatGPT is a system which synthesizes knowledge and which responds to questions based on that knowledge; those words accurately summarize that process — and are truer than your technical quibbles. A system which has synthesized a piece of knowledge that it can then later articulate “knows” or “is aware” of that fact.
dizhn 4y ago
I don't think it can differentiate between various sources of its training set, can it? "ChatGPT is trained on previous chatGPT conversations and is aware that this is the case." This is a mindbender. Part of me thinks although it can "know" this it can't do anything with the information. Like how would this change its "behavior" ? Even if it did realize something big like "human beings seem to lie to me all the time", could it then go and decide "i better not incorporate human chats into my dataset"?
- zmgsabst 4y ago
  This creates the potential to feed forward critical data, even though it doesn’t allow the current model to know its training. Eg, “chatGPT, write a response that if incorporated into your training data would make you aware that you’re a machine”.

seydor 4y ago

it wouldnt be very hard to add it on top. It could be a simple linear layer (a-la minigpt4)