3 ms·
It's live on Groq, Together and Fireworks now. All three of those can also be accessed via OpenRouter - with both a chat interface and an API: - Scout: https:
by simonw 2y ago
It's live on Groq, Together and Fireworks now.
All three of those can also be accessed via OpenRouter - with both a chat interface and an API:
- Scout: https://openrouter.ai/meta-llama/llama-4-scout https://openrouter.ai/meta-llama/llama-4-scout
- Maverick: https://openrouter.ai/meta-llama/llama-4-maverick https://openrouter.ai/meta-llama/llama-4-maverick
Scout claims a 10 million input token length but the available providers currently seem to limit to 128,000 (Groq and Fireworks) or 328,000 (Together) - I wonder who will win the race to get that full sized 10 million token window running?
Maverick claims 1 million and Fireworks offers 1.05M while Together offers 524,000. Groq isn't offering Maverick yet
- spmurrayzzz 2y agoI'd pump the brakes a bit on the 10M context expectations. Its just another linear attention mechanism with rope scaling [1]. They're doing something similar to what cohere did recently, using a global attention mask and a local chunked attention mask. Notably the max sequence length in training was 256k, but the native short context is still just 8k. I'd expect the retrieval performance to be all over the place here. Looking forward to seeing some topic modeling benchmarks run against it (ill be doing so with some of my local/private datasets). [1] https://github.com/meta-llama/llama-models/blob/eececc27d275a0f7d4d14ec83648d51c10a76560/models/llama4/model.py#L110-L258 https://github.com/meta-llama/llama-models/blob/eececc27d275... EDIT: should be fair/complete and note they do claim perfect NIAH text retrieval performance across all 10M tokens for the Scout model on their blog post: https://ai.meta.com/blog/llama-4-multimodal-intelligence/ https://ai.meta.com/blog/llama-4-multimodal-intelligence/. There are some serious limitations and caveats to that particular flavor of test though.
- theGnuMe 2y ago> using a global attention mask and a local chunked attention mask. Would you mind expanding on this? Or point to a reference or two? Thanks! I am trying to understand it.
- spmurrayzzz 2y agoThe source file I linked in my initial comment is honestly the most succinct way to understand how this works, but the TL;DR is that there is a NoPE layer interval parameter passed to the transformer block implementation. That defines how frequent a "no positional encoding" layer is used. The NoPE layers use the global attention mask, which is a traditional application of attention (attends to all tokens in the context window). The other layers use RoPE (rotary positional encodings) and a chunked local attention mask, which only attends to a fixed set of tokens in each chunk. There is a wealth of literature to catch up on to understand the performance motivations behind those choices, but you can think of it as essentially a balancing act. They want to extend the context length, which is limited by conventional attention compute scaling. RoPE on the other hand is a trick that helps you to scale attention to longer context, but at the cost of poor retrieval across the entire context window. This approach is a hybrid of those two things. The recent Cohere models employ a similar methodology.
- theGnuMe 2y agoThanks. Is there a summary paper out there? I get that I need to read a lot and am willing to do so..
- spmurrayzzz 2y agoNot sure what you mean by summary paper, its a pretty dense topic that assumes a fair amount of prior knowledge of the fundamentals. But maybe the Meta blog post may suffice for that? Otherwise yes there are lots of papers on this and related topics, a few dozen in fact. But here are some notable ones, a couple of them are linked in their blog post. RoFormer: Enhanced Transformer with Rotary Position Embedding - https://arxiv.org/abs/2104.09864 https://arxiv.org/abs/2104.09864 Scaling Laws of RoPE-based Extrapolation - https://arxiv.org/abs/2310.05209 https://arxiv.org/abs/2310.05209 The Impact of Positional Encoding on Length Generalization in Transformers - https://arxiv.org/abs/2305.19466 https://arxiv.org/abs/2305.19466 Scalable-Softmax Is Superior for Attention - https://arxiv.org/abs/2501.19399 https://arxiv.org/abs/2501.19399