2 ms·

Isn't that weird? I mean weren't transformers/attention explicitly designed to avoid this problem faces by RNNs?

stoniejohnson 3y ago
I don't know much, but this isn't surprising based on the little I know. Transformers predict the next token. If your question is at the end of the prompt, the start of an answer is a more likely next token than if the question is at the beginning of the prompt followed by a ton of other relevant, but non-question-forming tokens. Still, if you had to put the question at the beginning of your prompt, a transformer is more likely to give an answer than an RNN.
- jumpCastle 3y ago
  It is fine tuned to maximize reward though, not likelihood. And it provides an answer in both cases, just not as well.
  - stoniejohnson 3y ago
    So since a model is fine tuned via RLHF my point doesn't stand? Genuine question; it would be interesting if some other mechanism was at play here.
    - jumpCastle 3y ago
      For an answer I would expect it to get the same reward for both question orderings. So naively I would expect it to not be affected by the ordering.
furyofantares 3y ago
If you've got 20 tokens of query at the start and then 200 tokens of text data that it's querying, it seems really impressive that it's able to work out (via instruct tuning) to answer the query rather than continue the text data. A continuation of the text data is the actual most likely next token. I don't know about the super large contexts but you can also just make the text data clearly delimited instead of putting the query at the end, so that "predict the next token" isn't fighting the instruction-following training
TeMPOraL 3y ago
IDK. Ignoring the "transformers predict the next token" statement, which feels at best technically correct but missing the point, I imagine this comes down to the network learning "low-frequency" patterns in the training data. That is, in both training and instruct fine-tuning, the model is likely to encounter text structured like: DATA DATA DATA DATA ... -- boundary -- QUERY or the arguably equivalent: QUOTED TEXT -- boundary -- REPLY / COMMENTARY The inverse shape is also common: INSTRUCTIONS -- boundary -- DATA / TEXT ON WHICH TO WORK For example, most exercise lists and test books are written like that. The somewhat less frequent patterns are more random mix of: WHAT -- boundary -- ON WHAT -- boundary -- WHAT ELSE -- boundary -- ON WHAT ELSE -- boundary -- (...) CLOSING REMARKS Most of my HN comments are structured like that, for example. Including this one. Boundary here can take many forms. Extra newlines, --, ``` blocks ```, > - prefixed text, and lists (both OL and UL) are all common methods used to structure text, and are seen both in training data and in inference. We know LLM picks up on those structure markets at high-frequency level (e.g. using extra newlines or -- lines to separate distinct blocks seems effective). But I imagine it also picks up on the low-frequency patterns, which is why payload followed by, or bracketed with, instructions is something it "knows" how to process, whereas if you use less common structuring patterns, you're more likely to confuse the LLM.

I don't know much, but this isn't surprising based on the little I know. Transformers predict the next token. If your question is at the end of the prompt, the start of an answer is a more likely next token than if the question is at the beginning of the prompt followed by a ton of other relevant, but non-question-forming tokens. Still, if you had to put the question at the beginning of your prompt, a transformer is more likely to give an answer than an RNN.

jumpCastle 3y ago
It is fine tuned to maximize reward though, not likelihood. And it provides an answer in both cases, just not as well.
- stoniejohnson 3y ago
  So since a model is fine tuned via RLHF my point doesn't stand? Genuine question; it would be interesting if some other mechanism was at play here.
  - jumpCastle 3y ago
    For an answer I would expect it to get the same reward for both question orderings. So naively I would expect it to not be affected by the ordering.

furyofantares 3y ago

If you've got 20 tokens of query at the start and then 200 tokens of text data that it's querying, it seems really impressive that it's able to work out (via instruct tuning) to answer the query rather than continue the text data. A continuation of the text data is the actual most likely next token. I don't know about the super large contexts but you can also just make the text data clearly delimited instead of putting the query at the end, so that "predict the next token" isn't fighting the instruction-following training

TeMPOraL 3y ago

IDK. Ignoring the "transformers predict the next token" statement, which feels at best technically correct but missing the point, I imagine this comes down to the network learning "low-frequency" patterns in the training data. That is, in both training and instruct fine-tuning, the model is likely to encounter text structured like: DATA DATA DATA DATA ... -- boundary -- QUERY or the arguably equivalent: QUOTED TEXT -- boundary -- REPLY / COMMENTARY The inverse shape is also common: INSTRUCTIONS -- boundary -- DATA / TEXT ON WHICH TO WORK For example, most exercise lists and test books are written like that. The somewhat less frequent patterns are more random mix of: WHAT -- boundary -- ON WHAT -- boundary -- WHAT ELSE -- boundary -- ON WHAT ELSE -- boundary -- (...) CLOSING REMARKS Most of my HN comments are structured like that, for example. Including this one. Boundary here can take many forms. Extra newlines, --, ``` blocks ```, > - prefixed text, and lists (both OL and UL) are all common methods used to structure text, and are seen both in training data and in inference. We know LLM picks up on those structure markets at high-frequency level (e.g. using extra newlines or -- lines to separate distinct blocks seems effective). But I imagine it also picks up on the low-frequency patterns, which is why payload followed by, or bracketed with, instructions is something it "knows" how to process, whereas if you use less common structuring patterns, you're more likely to confuse the LLM.