3 ms·

Yeah, I don't know exactly what at an AGI model will look like, but I think it would have more than 200k context window.

kurthr 1y ago
I'd think it would be able to at least suggest which model to use rather than just having 6 for you to choose from.
doug_durham 1y ago
Do you have a 200k context window? I don't. Most humans can only keep 6 or 7 things in short term memory. Beyond those 6 or 7 you are pulling data from your latent space, or replacing of the short term slots with new content.
- sixQuarks 1y ago
  But context windows for LLMs include all the “long term memory” things you’re excluding from humans
  - kadushka 1y ago
    Long term memory in an LLM is its weights.
    - echoangle 1y ago
      Not really, because humans can form long term memories from conversations, but LLM users aren’t finetuning models after every chat so the model remembers.
      - esafak 1y ago
        He's right, but most people don't have the resources, nor indeed the weights themselves, to keep training the models. But the weights are very much long term memory.
      - kadushka 1y ago
        users aren’t finetuning models after every chat Users can do that if they want, but it’s more effective and more efficient to do that after every billion chats, and I’m sure OpenAI does it.
        echoangle 1y ago
        If you want the entire model to remember everything it talked about with every user, sure. But ideally, I would want the model to remember what I told it a few million tokens ago, but not what you told it (because to me, the model should look like my private copy that only talks to me).
        kadushka 1y ago
        ideally, I would want the model to remember what I told it a few million tokens ago Yes, you can keep finetuning your model on every chat you have with it. You can definitely make it remember everything you have ever said. LLMs are excellent at remembering their training data.
boznz 1y ago
I'm not quite AGI, but I work quite adequately with a much, much smaller memory. Maybe AGI just needs to know how to use other computers and work with storage a bit better.

kurthr 1y ago

I'd think it would be able to at least suggest which model to use rather than just having 6 for you to choose from.

doug_durham 1y ago

Do you have a 200k context window? I don't. Most humans can only keep 6 or 7 things in short term memory. Beyond those 6 or 7 you are pulling data from your latent space, or replacing of the short term slots with new content.

sixQuarks 1y ago
But context windows for LLMs include all the “long term memory” things you’re excluding from humans
- kadushka 1y ago
  Long term memory in an LLM is its weights.
  - echoangle 1y ago
    Not really, because humans can form long term memories from conversations, but LLM users aren’t finetuning models after every chat so the model remembers.
    - esafak 1y ago
      He's right, but most people don't have the resources, nor indeed the weights themselves, to keep training the models. But the weights are very much long term memory.
    - kadushka 1y ago
      users aren’t finetuning models after every chat Users can do that if they want, but it’s more effective and more efficient to do that after every billion chats, and I’m sure OpenAI does it.
      - echoangle 1y ago
        If you want the entire model to remember everything it talked about with every user, sure. But ideally, I would want the model to remember what I told it a few million tokens ago, but not what you told it (because to me, the model should look like my private copy that only talks to me).
        kadushka 1y ago
        ideally, I would want the model to remember what I told it a few million tokens ago Yes, you can keep finetuning your model on every chat you have with it. You can definitely make it remember everything you have ever said. LLMs are excellent at remembering their training data.

boznz 1y ago

I'm not quite AGI, but I work quite adequately with a much, much smaller memory. Maybe AGI just needs to know how to use other computers and work with storage a bit better.