3 ms·

You compared it to diffusion models, which work entirely differently. Could you elaborate what you mean? I understand how transformers work and their implementa

You compared it to diffusion models, which work entirely differently. Could you elaborate what you mean? I understand how transformers work and their implementation, but I'm unsure how with the current transformer architecture you can do what you described by "playing with vectors directly". Which vectors are you referring to? attention vectors?

jncfhnb 2y ago
The simplest example is contextual prompts. Let’s say you have a 3000 token prompt that you put in front of every query. You could A) have the LLM ingest and reprocess the same prompt every single time at needless cost Or B) process it one time and reuse the vector embedding of the prompt OpenAI forces you to do A.
- lossolo 2y ago
  I'm confused now, how does it solve the need to write "I REPEAT"? As a strengthening the signal to follow instructions. Besides that, in transformer models, the computation of each token’s output vector in a sequence depends on all other tokens in the sequence due to the self-attention mechanism. This interdependency means that you cannot simply "reuse"/add on to an existing processed sequence without re-processing the entire sequence, as the presence of new tokens (user input) alters the calculations of the attention mechanisms throughout the sequence. So what you basically proposing is not possible in current architecture.
  - jncfhnb 2y ago
    Tokens are generally consumed sequentially. The model does not care what token 3001 is when it’s creating the embedding representation of tokens 1-3000. You can use the cached state of the 3000th token rather than reprocessing it each time.
    - lossolo 2y ago
      I think you are confused on how transformers work. You are confusing between static embeddings and the dynamic, contextual outputs calculated by transformers. While initial embeddings (representations BEFORE processing through the transformer layers) are relatively static, the output embeddings from each layer of the transformer are highly dynamic and context-dependent. They incorporate information from the entire input sequence via the attention mechanisms. So when transformers process an input sequence, they don't just look at each token in isolation but consider the ENTIRE sequence's context through complex inter-token relationships. This makes the technique of caching outputs non-trivial and context-specific. The sequential processing does not imply independence of tokens but underscores the integral role of context and sequence in generating accurate and coherent outputs. This should help you understand better how transformers work: https://bbycroft.net/llm https://bbycroft.net/llm
      - jncfhnb 2y ago
        I’m not saying they look at each token in isolation. I’m saying they look at the current token and previous tokens. it is not accurate that these things consider the entire sequence. They (generally) consider the entire sequence up until that token. Which means if you have a large prefix prompt that comes before everything, it’s always the same embedding of that prefix prompt. You want to reuse the embedding representation of the prefix prompt before the new user tokens are confiscated, otherwise you are recalculating the embedding over and over and over. God forbid it’s not a small prefix but a huge document.
        lossolo 2y ago
        No, you are mistaken. You are still confusing initial embeddings, which are basically computationally free compared to the rest. I explained this in detail in my previous post. The ENTIRE sequence is processed by transformers to generate the first predicted token. Please check the link I provided and click on 'continue' to see a visual representation of how vectors are created from embeddings and how calculations are made. This should clarify what I mean.
        2y ago

jncfhnb 2y ago

The simplest example is contextual prompts. Let’s say you have a 3000 token prompt that you put in front of every query. You could A) have the LLM ingest and reprocess the same prompt every single time at needless cost Or B) process it one time and reuse the vector embedding of the prompt OpenAI forces you to do A.

lossolo 2y ago
I'm confused now, how does it solve the need to write "I REPEAT"? As a strengthening the signal to follow instructions. Besides that, in transformer models, the computation of each token’s output vector in a sequence depends on all other tokens in the sequence due to the self-attention mechanism. This interdependency means that you cannot simply "reuse"/add on to an existing processed sequence without re-processing the entire sequence, as the presence of new tokens (user input) alters the calculations of the attention mechanisms throughout the sequence. So what you basically proposing is not possible in current architecture.
- jncfhnb 2y ago
  Tokens are generally consumed sequentially. The model does not care what token 3001 is when it’s creating the embedding representation of tokens 1-3000. You can use the cached state of the 3000th token rather than reprocessing it each time.
  - lossolo 2y ago
    I think you are confused on how transformers work. You are confusing between static embeddings and the dynamic, contextual outputs calculated by transformers. While initial embeddings (representations BEFORE processing through the transformer layers) are relatively static, the output embeddings from each layer of the transformer are highly dynamic and context-dependent. They incorporate information from the entire input sequence via the attention mechanisms. So when transformers process an input sequence, they don't just look at each token in isolation but consider the ENTIRE sequence's context through complex inter-token relationships. This makes the technique of caching outputs non-trivial and context-specific. The sequential processing does not imply independence of tokens but underscores the integral role of context and sequence in generating accurate and coherent outputs. This should help you understand better how transformers work: https://bbycroft.net/llm https://bbycroft.net/llm
    - jncfhnb 2y ago
      I’m not saying they look at each token in isolation. I’m saying they look at the current token and previous tokens. it is not accurate that these things consider the entire sequence. They (generally) consider the entire sequence up until that token. Which means if you have a large prefix prompt that comes before everything, it’s always the same embedding of that prefix prompt. You want to reuse the embedding representation of the prefix prompt before the new user tokens are confiscated, otherwise you are recalculating the embedding over and over and over. God forbid it’s not a small prefix but a huge document.
      - lossolo 2y ago
        No, you are mistaken. You are still confusing initial embeddings, which are basically computationally free compared to the rest. I explained this in detail in my previous post. The ENTIRE sequence is processed by transformers to generate the first predicted token. Please check the link I provided and click on 'continue' to see a visual representation of how vectors are created from embeddings and how calculations are made. This should clarify what I mean.
        2y ago