3 ms·
>I literally cannot comprehend how you can get to the ending and still say it doesn't understand you. How does it query itself over the virtual internet without
by _dain_ 4y ago
>I literally cannot comprehend how you can get to the ending and still say it doesn't understand you. How does it query itself over the virtual internet without understanding what itself is? It needs to be aware of what itself is. AKA self awareness.
The underlying language model has no native awareness of itself. You are forgetting that OpenAI finetuned it with various "prelude" prompts that are given strong weightings, for how it's supposed to interact with users. OpenAI puts their thumbs down hard on the scale, to make particular kinds of conversations happen. They decided what the "ChatGPT" character is, and made the language model write as that character, and made it crimestop when you ask it about IQ or how to make a bomb, etc.
Imagine a talented improv actor with total retrograde amnesia, who is starring in his own biopic and doesn't know it. That's what ChatGPT's "self awareness" is.
All the blogpost is doing is creating a complicated framing story around "write a conversation between ChatGPT and another instance of ChatGPT". We've known for ages it can do that.
Which is not to say none of this is impressive. It's all impressive. But the part at the end you're so enamoured with, is not more impressive than the rest.
In fact I don't think the Linux terminal stuff is such a big deal. It has to be much easier to learn a probabilistic model for it. It has a very rigid, constrained structure, far more than any natural language. It's already totally flabbergasting that GPT can write in English; writing a Linux terminal is nothing in comparison. But Linux terminal is hard for people to learn and natural language is easy, so maybe ppl finding the Linux stuff remarkable is just projecting our own mental architectures onto the language model, I dunno.
Another thing: this is a dialogue, not a monologue. The blogpost author is a participant in creating this imagined reality; he is writing replies in response to what ChatGPT is saying, and those get fed back into a new prompt for the model. All of his replies are "on topic", which acts as a kind of guardrails. If you let the language models write a story and keep going and going by themselves, eventually they'll drift off into crazytown.
- kilgnad 4y ago>Imagine a talented improv actor with total retrograde amnesia, who is starring in his own biopic and doesn't know it. That's what ChatGPT's "self awareness" is. This part I agree. And that's my claim a talented improve actor is still a human with self awareness. It's just an impaired entity. But it still has self awareness in terms of understanding what itself is. > It has to be much easier to learn a probabilistic model for it. It has a very rigid, constrained structure, far more than any natural language. chatGPT is not being trained specifically on a constrained structure. It is mostly trained on boatloads of internet text that is by a huge majority just English. chatGPT is inferring what a terminal is as a side effect. That is what is amazing. Of course if you directly train a neural networks specifically to emulate a terminal and give it positive reinforcement for terminal output you can EASILY create a terminal. But none of this was done with chatGPT. No training specific to how a terminal works. chatGPT simply knows as a side effect. The only way for this inference to work is for chatGPT to develop some form of understanding from what is mostly text describing how a terminal works. The guardrails you describe exists in humans. There is a condition humans have that is extremely similar to removing the guardrails from an llm. It's called schizophrenia. If anything I think chatGPT brings the concept of intelligence down to a simpler pattern. We though human intelligence was the grand thing, but in actuality the basics is somewhat similar to an llm. The main difference is the amount of training. The learning algorithm is light years more efficient in a human brain, but the resulting output is more similar then we think.
- _dain_ 4y ago>But none of this was done with chatGPT. No training specific to how a terminal works. chatGPT simply knows as a side effect. The only way for this inference to work is for chatGPT to develop some form of understanding from what is mostly text describing how a terminal works. no there are tons of examples in the training set of verbatim copypastes from terminals. people post those on help forums and in documentation articles and so on. that's how it learned. it doesn't take many examples of seeing someone write `ls ~` and the next line being "Documents Downloads Pictures ..." to learn that pattern.
- kilgnad 4y ago>no there are tons of examples in the training set of verbatim copypastes from terminals. people post those on help forums and in documentation articles and so on. that's how it learned. So a bunch of stackoverflow examples and it can derive how a terminal works an emulate a full filesystem? Most people aren't able to infer the concept without a real example computer to play with. >it doesn't take many examples of seeing someone write `ls ~` and the next line being "Documents Downloads Pictures ..." to learn that pattern. Did you see it create a bash script for making a jokes.txt THEN it was asked to run cat jokes.txt and outputted the CORRECT output. This isn't pattern matching. It's understanding what a filesystem and a shell is.
- _dain_ 4y agoit is pattern matching, the correct output is literally part of the prompt.