3 ms·
I think changing the positional encoding to ALiBi would help in this case but I guess it wouldn't be Llama 2 anymore.
by GaggiX 3y ago
I think changing the positional encoding to ALiBi would help in this case but I guess it wouldn't be Llama 2 anymore.
- karpathy 3y agoYes :(