4 ms·
>Alibi Do you have a source on Mistral/Mixtral using that?
by logicchains 3y ago
>Alibi
Do you have a source on Mistral/Mixtral using that?
- benreesman 3y agoNo, they could be using any of the variants of pointwise scalar trig-style embedding, one imagines it's at least a little custom to their particular training setup. It was just an example of a modern positional encoding. I regret that I implied inside knowledge about that level of detail. They're doing something clever on scalar pointwise positional encoding but as for what who knows.