4 ms·

I think you've done a great explanation expansion except I believe it's ALiBi ("Attention with Linear Biases Enables Input Length Extrapolation"), a method of p

I think you've done a great explanation expansion except I believe it's ALiBi ("Attention with Linear Biases Enables Input Length Extrapolation"), a method of positional encoding (i.e. telling the Transformer model how much to weight a distant token when computing the current output token). This has been used on various other LLMs[2]. [1]: https://arxiv.org/abs/2108.12409 https://arxiv.org/abs/2108.12409 [2]: n.b. Ofir Press is co-creator of ALiBi https://twitter.com/OfirPress/status/1654538361447522305 https://twitter.com/OfirPress/status/1654538361447522305

benreesman 3y ago
This is indeed what I was referring to and along with RoPE and related techniques is a sort of "meta-attention" in which a cost-effective scalar pointwise calculation can hint the heavyweight attention mechanism with super-linear returns in practical use cases. In more intuitive terms, your bog-standard transformer overdoes it in terms of considering all context equally in the final prediction, and we historically used rather blunt-force instruments like causally masking everything to zero. These techniques are still heuristic and I imagine every serious shop has tweaks and tricks that go with their particular training setup, but the Rope shit in general is kind of a happy medium and exploits locality at a much cheaper place in the overall computation.
- Kerbonut 3y ago
  imo mistral-medium is worse than mixtral. Do you have API access?
- lhl 3y ago
  My understanding is that Mistral uses a regular 4K RoPE that is "extends" the window size with SWA. This is based on looking at the results of Nous Research's Yarn-Mistral extension: https://huggingface.co/NousResearch/Yarn-Mistral-7b-128k https://huggingface.co/NousResearch/Yarn-Mistral-7b-128k and Self-Extend, both of which only apply to RoPE models. There are quite a few recent attention extension techniques recently published: * Activation Beacons - up to 100X context length extension in as little as 72 A800 hours https://huggingface.co/papers/2401.03462 https://huggingface.co/papers/2401.03462 * Self-Extend - a no-training RoPE modification that can give "free" context extension with 100% passkey retrieval (works w/ SWA as well) https://huggingface.co/papers/2401.01325 https://huggingface.co/papers/2401.01325 * DistAttention/DistKV-LLM - KV cache segmentation for 2-19X context length at runtime https://huggingface.co/papers/2401.02669 https://huggingface.co/papers/2401.02669 * YaRN - aforementioned efficient RoPE extension https://huggingface.co/papers/2309.00071 https://huggingface.co/papers/2309.00071 You could imagine combining a few of these together to basically "solve" the context issue while largely training for shorter context length. There are of course some exciting new alternative architectures, notably Mamba https://huggingface.co/papers/2312.00752 https://huggingface.co/papers/2312.00752 and Megabyte https://huggingface.co/papers/2305.07185 https://huggingface.co/papers/2305.07185 that can efficiently process up to 1M tokens...
EmilStenstrom 3y ago
Thank you!

benreesman 3y ago

This is indeed what I was referring to and along with RoPE and related techniques is a sort of "meta-attention" in which a cost-effective scalar pointwise calculation can hint the heavyweight attention mechanism with super-linear returns in practical use cases. In more intuitive terms, your bog-standard transformer overdoes it in terms of considering all context equally in the final prediction, and we historically used rather blunt-force instruments like causally masking everything to zero. These techniques are still heuristic and I imagine every serious shop has tweaks and tricks that go with their particular training setup, but the Rope shit in general is kind of a happy medium and exploits locality at a much cheaper place in the overall computation.

Kerbonut 3y ago
imo mistral-medium is worse than mixtral. Do you have API access?
lhl 3y ago
My understanding is that Mistral uses a regular 4K RoPE that is "extends" the window size with SWA. This is based on looking at the results of Nous Research's Yarn-Mistral extension: https://huggingface.co/NousResearch/Yarn-Mistral-7b-128k https://huggingface.co/NousResearch/Yarn-Mistral-7b-128k and Self-Extend, both of which only apply to RoPE models. There are quite a few recent attention extension techniques recently published: * Activation Beacons - up to 100X context length extension in as little as 72 A800 hours https://huggingface.co/papers/2401.03462 https://huggingface.co/papers/2401.03462 * Self-Extend - a no-training RoPE modification that can give "free" context extension with 100% passkey retrieval (works w/ SWA as well) https://huggingface.co/papers/2401.01325 https://huggingface.co/papers/2401.01325 * DistAttention/DistKV-LLM - KV cache segmentation for 2-19X context length at runtime https://huggingface.co/papers/2401.02669 https://huggingface.co/papers/2401.02669 * YaRN - aforementioned efficient RoPE extension https://huggingface.co/papers/2309.00071 https://huggingface.co/papers/2309.00071 You could imagine combining a few of these together to basically "solve" the context issue while largely training for shorter context length. There are of course some exciting new alternative architectures, notably Mamba https://huggingface.co/papers/2312.00752 https://huggingface.co/papers/2312.00752 and Megabyte https://huggingface.co/papers/2305.07185 https://huggingface.co/papers/2305.07185 that can efficiently process up to 1M tokens...

EmilStenstrom 3y ago

Thank you!