4 ms·

I'm not much interested in the prompt itself, but I am continually amazed that ChatGPT is able to make sense of prompts like this. That is pretty far from "lan

I'm not much interested in the prompt itself, but I am continually amazed that ChatGPT is able to make sense of prompts like this. That is pretty far from "language", and I can't see how any of that has been seen in its "training data". I mean ... you can add something like "//! = loop mode = do loop: (ask USR for instruct (string, num); iterate string num times); Nesting ok." and it can not only parse that (or "tokenize" it), then somehow find relationships in its internal high-dimensional vector space that are sufficient to ... pseudo-"execute" it? I don't know. Obviously, not my area of expertise, although I can say I've spent a lot of time trying to understand even the basics. But then I'll see an example like this, and be reminded of how little I understand any of it.

jadbox 3y ago
I wonder how much "gpt" within openai understands and how much syntax is conditionally parsed and run though state machines prior to the prompt sent to transformers.
huijzer 3y ago
I did my MSc thesis on ML in 2019 and so I‘m definitely no expert but one mental model worked well for me. Machine learning models learn thousands of small functions which map input to output. A higher level "function" will choose which function is used to map the input to the output. So that’s what amazed me in about ChatGPT: think about all the little things it must understand to do what it does. For example, the model learned how to interpret if clauses and other language nuances, how variable definitions should be used, what words are in other languages.
AndrewKemendo 3y ago
If you think about it relative to the level of specific interaction you’re doing with a computer, then it makes sense So, for example, if I’m talking to a compiler, I have to talk to it in effectively perfect sequential structure If I’m talking to a web server, then it’s with a scripting language in order to communicate something from another programming language that’s interacting it lower level At the other extreme if I’m talking to a kernel, then I really only have a couple of things I can say because its language is memory blocks In this case, because I’m talking to what’s effectively an interpreter, then the default language looks like a scripting language rather than a programming language or natural language Which in effect is what’s happening? These are basically scripts people are running against the structure of the GPT sequence interpretation - so it’s a weird mix
com2kid 3y ago
> I'm not much interested in the prompt itself, but I am continually amazed that ChatGPT is able to make sense of prompts like this. GPT4 (and even 3.5) can do all sorts of magic that makes even the latest Bard look like a joke. You can use GPT4 to compress inputs down to 1/2 the original number of tokens while retaining 100% of the original meaning. (When the GPT4 generated output is pasted back into GPT4 it's, seemingly, the same as posting the original text). Defining DSLs in GPT3.5 and 4 is easy, and IMHO better than relying on JSON output (especially before the function calling stuff was introduced). You can give GPT3.5 and 4 a DSL and it'll suggest improvements to the grammar. You can tell GPT4 it is an experienced LLM prompt engineer and then ask it to write prompts for GPT4 which you feed into GPT4. You can tell GPT4 it is an experienced python engineer and that it needs to write a program to parse the output of another prompt which you then paste in. One time after several hours of fruitless debugging, I pasted a source file into GPT4, told it "there is a concurrency bug here, can you find it for me" and out popped the exact line I had screwed things up on. (Hilariously I had explained how to avoid exactly that same bug a couple years prior to some junior engineers... oops) I'd pay $100 a month for access to a more powerful programmer centric version of GPT4. For that matter if OpenAI made an announcement that next week they were gating code generation behind a $100 a month fee, I'd probably just be forced to pay up. GPT4 is absurdly powerful and I am well aware I'm not even making the best use of it compared to what I see others doing.
- jackphilson 3y ago
  Can we compress this prompt so it doesn't clog up the context window as much?
  - airesearcher 3y ago
    I challenge you to do it and still get it to actually work. The context window is pretty large.. I don't think this will clog it up that much...
  - airesearcher 3y ago
    The most essential part of the prompt is the first section(s) that does message numbering, and then the parts that enable you to do operations using message numbers or ranges of them. After using message numbering in ChatGPT I can never go back to working without that. For example suppose you have a long chat with a bunch of iterations on some idea or piece of content. You can simply say “Continue from //16” to continue the chat from that message (basically making a branch) without having to copy and paste it again. Or you can say “revise //12 but make it more professional and also merge with //distill 5-7” to revise message 12 and merge with the distilled points from messages 5 through 7. There are many more powerful things you can do with the /t tagging and /s sets, and the //! Loop command and //f function command to automate repetitive tasks such as expanding content, pulling sets of messages for a purpose, or iteratively writing a longer article etc.

jadbox 3y ago

I wonder how much "gpt" within openai understands and how much syntax is conditionally parsed and run though state machines prior to the prompt sent to transformers.

huijzer 3y ago

I did my MSc thesis on ML in 2019 and so I‘m definitely no expert but one mental model worked well for me. Machine learning models learn thousands of small functions which map input to output. A higher level "function" will choose which function is used to map the input to the output. So that’s what amazed me in about ChatGPT: think about all the little things it must understand to do what it does. For example, the model learned how to interpret if clauses and other language nuances, how variable definitions should be used, what words are in other languages.

AndrewKemendo 3y ago

If you think about it relative to the level of specific interaction you’re doing with a computer, then it makes sense So, for example, if I’m talking to a compiler, I have to talk to it in effectively perfect sequential structure If I’m talking to a web server, then it’s with a scripting language in order to communicate something from another programming language that’s interacting it lower level At the other extreme if I’m talking to a kernel, then I really only have a couple of things I can say because its language is memory blocks In this case, because I’m talking to what’s effectively an interpreter, then the default language looks like a scripting language rather than a programming language or natural language Which in effect is what’s happening? These are basically scripts people are running against the structure of the GPT sequence interpretation - so it’s a weird mix

com2kid 3y ago

> I'm not much interested in the prompt itself, but I am continually amazed that ChatGPT is able to make sense of prompts like this. GPT4 (and even 3.5) can do all sorts of magic that makes even the latest Bard look like a joke. You can use GPT4 to compress inputs down to 1/2 the original number of tokens while retaining 100% of the original meaning. (When the GPT4 generated output is pasted back into GPT4 it's, seemingly, the same as posting the original text). Defining DSLs in GPT3.5 and 4 is easy, and IMHO better than relying on JSON output (especially before the function calling stuff was introduced). You can give GPT3.5 and 4 a DSL and it'll suggest improvements to the grammar. You can tell GPT4 it is an experienced LLM prompt engineer and then ask it to write prompts for GPT4 which you feed into GPT4. You can tell GPT4 it is an experienced python engineer and that it needs to write a program to parse the output of another prompt which you then paste in. One time after several hours of fruitless debugging, I pasted a source file into GPT4, told it "there is a concurrency bug here, can you find it for me" and out popped the exact line I had screwed things up on. (Hilariously I had explained how to avoid exactly that same bug a couple years prior to some junior engineers... oops) I'd pay $100 a month for access to a more powerful programmer centric version of GPT4. For that matter if OpenAI made an announcement that next week they were gating code generation behind a $100 a month fee, I'd probably just be forced to pay up. GPT4 is absurdly powerful and I am well aware I'm not even making the best use of it compared to what I see others doing.

jackphilson 3y ago
Can we compress this prompt so it doesn't clog up the context window as much?
- airesearcher 3y ago
  I challenge you to do it and still get it to actually work. The context window is pretty large.. I don't think this will clog it up that much...
- airesearcher 3y ago
  The most essential part of the prompt is the first section(s) that does message numbering, and then the parts that enable you to do operations using message numbers or ranges of them. After using message numbering in ChatGPT I can never go back to working without that. For example suppose you have a long chat with a bunch of iterations on some idea or piece of content. You can simply say “Continue from //16” to continue the chat from that message (basically making a branch) without having to copy and paste it again. Or you can say “revise //12 but make it more professional and also merge with //distill 5-7” to revise message 12 and merge with the distilled points from messages 5 through 7. There are many more powerful things you can do with the /t tagging and /s sets, and the //! Loop command and //f function command to automate repetitive tasks such as expanding content, pulling sets of messages for a purpose, or iteratively writing a longer article etc.