4 ms·

Llms only work in one direction, they produce the next token only. It can't go back and edit. They would need to be able to back track and edit in place somehow

by tayo42 1y ago

DANmode 1y ago
"Somehow", like caching multiple layers of context, like all the free tools are now doing?
- tayo42 1y ago
  That's different then seeing if it's current output made a mistake or not. It's not editing in place. Your just rolling the dice again with a different prompt
  - thunky 1y ago
    No, the session history is all in the prompt, including the LLM's previous responses.
    - tayo42 1y ago
      Appending more context to the existing prompt means it's a different prompt still... The text isn't the same
      - thunky 1y ago
        I'm not sure what your point is? Think of it like an append only journal. To correct an entry you add a new one with the correction. The LLM sees the mistake and the correction. That's no worse then mutating the history.
        tayo42 1y ago
        thats not how it works. You put in its context window some more information, then roll the dice again. And it produces text again token by token. Its still not planning ahead and its not looking back at what was just generated. There's no guarantee everything stays the same except the mistake. This is different then editing in place. you are rolling the dice again.
        thunky 1y ago
        > its not looking back at what was just generated It is, though. The LLM gets the full history in every prompt until you start a new session. That's why it gets slower as the conversation/context gets big. The developer could choose to rewrite or edit the history before sending it back to the LLM but the user typically can't. > There's no guarantee everything stays the same except the mistake Sure, but there's no guarantee about anything it will generate. But that's a separate issue.
thunky 1y ago
Loops. Plus, the entire session/task history goes into every LLM prompt, not just the last message. So for every turn of the loop the LLM has the entire context with everything that previously happened in it, along with added "memories" and instructions.

Llms only work in one direction, they produce the next token only. It can't go back and edit. They would need to be able to back track and edit in place somehow

by tayo42 1y ago

Llms only work in one direction, they produce the next token only. It can't go back and edit. They would need to be able to back track and edit in place somehow

DANmode 1y ago

"Somehow", like caching multiple layers of context, like all the free tools are now doing?

tayo42 1y ago
That's different then seeing if it's current output made a mistake or not. It's not editing in place. Your just rolling the dice again with a different prompt
- thunky 1y ago
  No, the session history is all in the prompt, including the LLM's previous responses.
  - tayo42 1y ago
    Appending more context to the existing prompt means it's a different prompt still... The text isn't the same
    - thunky 1y ago
      I'm not sure what your point is? Think of it like an append only journal. To correct an entry you add a new one with the correction. The LLM sees the mistake and the correction. That's no worse then mutating the history.
      - tayo42 1y ago
        thats not how it works. You put in its context window some more information, then roll the dice again. And it produces text again token by token. Its still not planning ahead and its not looking back at what was just generated. There's no guarantee everything stays the same except the mistake. This is different then editing in place. you are rolling the dice again.
        thunky 1y ago
        > its not looking back at what was just generated It is, though. The LLM gets the full history in every prompt until you start a new session. That's why it gets slower as the conversation/context gets big. The developer could choose to rewrite or edit the history before sending it back to the LLM but the user typically can't. > There's no guarantee everything stays the same except the mistake Sure, but there's no guarantee about anything it will generate. But that's a separate issue.

thunky 1y ago

Loops. Plus, the entire session/task history goes into every LLM prompt, not just the last message. So for every turn of the loop the LLM has the entire context with everything that previously happened in it, along with added "memories" and instructions.