3 ms·

Interesting. It's almost the exact same structure (although better organized) that I have built in my "AI Programmer" project (also in Node). Which by the way I

by ilaksh 4y ago

Interesting. It's almost the exact same structure (although better organized) that I have built in my "AI Programmer" project (also in Node). Which by the way I hope to have a new release within a week or so. I am not mentioning the domain name again until I have the new release because it much, much better than the version I have up now. The core idea is that you need a certain structure in order to deal with the limitations of GPT-3.5 or whatever. To get the best output, you give it a brief overview of what you want, a few examples, and the history of interaction. OpenAI's API does not remember the interaction history for you. And the limit is 4000 or 8000 tokens so you will need to truncate the beginning of the conversation at some point. So this library provides a structure since it's something you will need to repeat for just about every significant text completion application or module.

knome 4y ago
>so you will need to truncate the beginning of the conversation at some point I wrote a little python script to keep track of a running conversation when I started playing with openai's completions API. I keep track of how many tokens the prompt is taking up, and when it gets too close to some configurable threshold, I then have a different prompt to tell the AI to summarize the conversation and any previous summary, with various demands to keep track of certain specific bits of state ( like names and things ). Works pretty well for marching conversations past the token limit.
- davedx 4y ago
  Fascinating, and somewhat comparable to how the human brain "compresses" visual scenes when encoding them into long term memory. This is such an interesting space to follow :)
- pixl97 4y ago
  Has anyone played around with GPT itself to see if you can get it to track the number of tokens it's tracking (best if implemented with the least number of tokens possible)
  - lasermatts 4y ago
    I haven't played too much with doing this programmatically, but OpenAI's Tokenizer tool is a fun way to get an intuition around tokens and text length https://platform.openai.com/tokenizer https://platform.openai.com/tokenizer
- singularity2001 4y ago
  simple but brilliant, should be the default behavior for all token limited agents!
- davedx 4y ago
  I honestly keep thinking about this approach and talking about it to other people I know. If you don't mind sharing the code I'd be super interested to see how it works
politician 4y ago
So, actually, I just had a good long talk with ChatGPT about how it works internally. Turns out that while it doesn't remember the concatenated sequence of prompts and response, GPT-3 does maintain a sort of impression or representation of the entire conversation - the conversation context. Also, turns out that there's about a dozen predefined special tokens that can be used to control how the model considers or ignores the conversation context when responding to a prompt. There's also some internal use only special tokens that it can accidentally emit, and if you aren't careful in specifying how it should print the tokens there's an internal token that'll cause it to basically hard reset the conversation context. Anyway, the point is that at least the Default (paid) ChatGPT model and probably the GPT-3 model does have a representation of the conversation which it can and does reference. You can ask it to explain how and when it considers or ignores the context and how to control that. (Yes, I understand that it's not describing its own architecture, but regurgitating the average of all of the papers on GPT and weighting the ones that refer to ChatGPT higher due to the fine-tuning effect.)

by ilaksh 4y ago

knome 4y ago

>so you will need to truncate the beginning of the conversation at some point I wrote a little python script to keep track of a running conversation when I started playing with openai's completions API. I keep track of how many tokens the prompt is taking up, and when it gets too close to some configurable threshold, I then have a different prompt to tell the AI to summarize the conversation and any previous summary, with various demands to keep track of certain specific bits of state ( like names and things ). Works pretty well for marching conversations past the token limit.

davedx 4y ago
Fascinating, and somewhat comparable to how the human brain "compresses" visual scenes when encoding them into long term memory. This is such an interesting space to follow :)
pixl97 4y ago
Has anyone played around with GPT itself to see if you can get it to track the number of tokens it's tracking (best if implemented with the least number of tokens possible)
- lasermatts 4y ago
  I haven't played too much with doing this programmatically, but OpenAI's Tokenizer tool is a fun way to get an intuition around tokens and text length https://platform.openai.com/tokenizer https://platform.openai.com/tokenizer
singularity2001 4y ago
simple but brilliant, should be the default behavior for all token limited agents!
davedx 4y ago
I honestly keep thinking about this approach and talking about it to other people I know. If you don't mind sharing the code I'd be super interested to see how it works

politician 4y ago

So, actually, I just had a good long talk with ChatGPT about how it works internally. Turns out that while it doesn't remember the concatenated sequence of prompts and response, GPT-3 does maintain a sort of impression or representation of the entire conversation - the conversation context. Also, turns out that there's about a dozen predefined special tokens that can be used to control how the model considers or ignores the conversation context when responding to a prompt. There's also some internal use only special tokens that it can accidentally emit, and if you aren't careful in specifying how it should print the tokens there's an internal token that'll cause it to basically hard reset the conversation context. Anyway, the point is that at least the Default (paid) ChatGPT model and probably the GPT-3 model does have a representation of the conversation which it can and does reference. You can ask it to explain how and when it considers or ignores the context and how to control that. (Yes, I understand that it's not describing its own architecture, but regurgitating the average of all of the papers on GPT and weighting the ones that refer to ChatGPT higher due to the fine-tuning effect.)