3 ms·
One strategy I’ve been playing around with is to take an instruction I want the model to follow and squish the positional encodings for the keys down to positio
by valine 2y ago
One strategy I’ve been playing around with is to take an instruction I want the model to follow and squish the positional encodings for the keys down to position zero, and the new queries out slightly further in the window. The model will still follow the instruction but the behaviors are more global. It’s behaves more like a fine-tune and less like the instruction is part of the conversation.
- bhickey 2y ago> squish the positional encodings for the keys down to position zero I might be misunderstanding, but wouldn't this turn your instructions into a bag of words?
- valine 2y agoNo, and that’s because we are talking about relative positions. Every query can have its own set of keys. From the perspective of token 100 token 3 would be squished down, but from the perspective of token 3 it is still at position 3 and can see tokens 0,1,2 without them being squished.