3 ms·
> Also the attention mechanism is baked in during pretraining IIUC, this is no longer necessarily true with positional encodings like ALiBi: https://github.com
by rewq4321 3y ago
> Also the attention mechanism is baked in during pretraining
IIUC, this is no longer necessarily true with positional encodings like ALiBi: https://github.com/ofirpress/attention_with_linear_biases https://github.com/ofirpress/attention_with_linear_biases