3 ms·

There's one thing I haven't quite grasped about transformers. Are the query, key, and value vectors trained per token? Does each token have specific QVK vectors

There's one thing I haven't quite grasped about transformers. Are the query, key, and value vectors trained per token? Does each token have specific QVK vectors in each head, or does each attention head have one set that are trained using a lot of tokens?

danielmarkbruce 3y ago
The q,k,v projections are trained, usually per head. But each token goes through the same projection (per head). In some architectures (see the Falcon model) the k, v projections are shared across the heads. During the forward pass a "query" is "created" for each token using the query projection (again, one projection per head, all the tokens run through the same projection). The keys are created using the key projection, the values are created using the dot product of query with the keys for each "token" then projected using the value projection. But again, different models do different things. Some models bring in the positional encoding into the attention calculation rather than adding it in earlier. Practically every combination of things has been tried since that paper was published.
- two_in_one 3y ago
  Thanks, good explanation. More questions. Models have the initial N embeddings passed through, processed by the attention+linear blocks which add something to the input. Sort of memory bus. After the last block we still have array of the same dimensions like N initial embeddings, but they mean something else now. How do we select the next token, based on the last element in that array? Another question, that bus technically doesn't have to have the same width, right? It should be possible, for the same model's size, to trade bus width for the number of heads. Even have sort of U-Net. And the last, on each loop resulting embedding (1) is converted into token, which is being added to the input. Converted to embedding first, it should be possible to just reuse embedding (1), and use token only for user output, right? PS: not sure about every combination tried, we just started. Some of the problems still don't have satisfying solutions. Like hallucinations, online training. Or responsibility.
  - danielmarkbruce 3y ago
    in go X tokens. Out comes a probability distribution across every possible next token. In your analogy, the bus could be any width. In practice people tend to trade "bus width" for heads, but it need not be that way. I'm not sure understand the last part. It's quite easy to try stuff with LLMs/transformers. The fact that a paper hasn't been written on every combination doesn't mean the haven't been tried in some way. It's not as though the architecture is the only thing.

danielmarkbruce 3y ago

The q,k,v projections are trained, usually per head. But each token goes through the same projection (per head). In some architectures (see the Falcon model) the k, v projections are shared across the heads. During the forward pass a "query" is "created" for each token using the query projection (again, one projection per head, all the tokens run through the same projection). The keys are created using the key projection, the values are created using the dot product of query with the keys for each "token" then projected using the value projection. But again, different models do different things. Some models bring in the positional encoding into the attention calculation rather than adding it in earlier. Practically every combination of things has been tried since that paper was published.

two_in_one 3y ago
Thanks, good explanation. More questions. Models have the initial N embeddings passed through, processed by the attention+linear blocks which add something to the input. Sort of memory bus. After the last block we still have array of the same dimensions like N initial embeddings, but they mean something else now. How do we select the next token, based on the last element in that array? Another question, that bus technically doesn't have to have the same width, right? It should be possible, for the same model's size, to trade bus width for the number of heads. Even have sort of U-Net. And the last, on each loop resulting embedding (1) is converted into token, which is being added to the input. Converted to embedding first, it should be possible to just reuse embedding (1), and use token only for user output, right? PS: not sure about every combination tried, we just started. Some of the problems still don't have satisfying solutions. Like hallucinations, online training. Or responsibility.
- danielmarkbruce 3y ago
  in go X tokens. Out comes a probability distribution across every possible next token. In your analogy, the bus could be any width. In practice people tend to trade "bus width" for heads, but it need not be that way. I'm not sure understand the last part. It's quite easy to try stuff with LLMs/transformers. The fact that a paper hasn't been written on every combination doesn't mean the haven't been tried in some way. It's not as though the architecture is the only thing.