5 ms·
Llama 4 Now Live on Groq
- greeneggs 2y agoFYI, the last sentence, "Start building today on GroqCloud – sign up for free access here…" links to https://conosle.groq.com/ https://conosle.groq.com/ (instead of "console")
- snikch 2y agoFixed. Thanks for the report.
- vessenes 2y agoJust tried this thank you. Couple qs - looked like just scout access for now, do you have plans for larger model access? Also, seems like context length is always fairly short with you guys, is that architectural or cost-based decisions?
- ozenhati 2y agoamazing! and yes, we'll have maverick available today. the reason we limit ctx window is because demand > capacity. we're pretty busy with building out more capacity so we can get to a state where we give everyone access to larger context windows without melting our currently available lpus, haha.
- vessenes 2y agocool. I would so happily pay you guys for long context API that aider could point at -- the speed is just game changing. I know your arch is different, so I understand it's an engineering lift. But, I bet you'd find some pareto optimal point in the curve where you could charge a lott more for the speed you guys can do if it's long enough for coding.
- Game_Ender 2y agoTo help those who got a bit confused (like me) this Groq the company making accelerators designed specifically for LLM's that they call LPUs (Language Process Units) [0]. So they want to sell you their custom machines that, while expensive, will be much more efficient at running LLMs for you. While there is also Grok [0] which is xAI's series of LLMs and competes with ChatGPT and other models like Claude and DeepSeek. EDIT - Seems that Groq has stopped selling their chips and now will only partner to fund large build outs of their cloud [2]. 0 - https://groq.com/the-groq-lpu-explained/ https://groq.com/the-groq-lpu-explained/ 1 - https://grok.com/ https://grok.com/ 2 - https://www.eetimes.com/groq-ceo-we-no-longer-sell-hardware https://www.eetimes.com/groq-ceo-we-no-longer-sell-hardware
- simonw 2y agoIt's live on Groq, Together and Fireworks now. All three of those can also be accessed via OpenRouter - with both a chat interface and an API: - Scout: https://openrouter.ai/meta-llama/llama-4-scout https://openrouter.ai/meta-llama/llama-4-scout - Maverick: https://openrouter.ai/meta-llama/llama-4-maverick https://openrouter.ai/meta-llama/llama-4-maverick Scout claims a 10 million input token length but the available providers currently seem to limit to 128,000 (Groq and Fireworks) or 328,000 (Together) - I wonder who will win the race to get that full sized 10 million token window running? Maverick claims 1 million and Fireworks offers 1.05M while Together offers 524,000. Groq isn't offering Maverick yet
- spmurrayzzz 2y agoI'd pump the brakes a bit on the 10M context expectations. Its just another linear attention mechanism with rope scaling [1]. They're doing something similar to what cohere did recently, using a global attention mask and a local chunked attention mask. Notably the max sequence length in training was 256k, but the native short context is still just 8k. I'd expect the retrieval performance to be all over the place here. Looking forward to seeing some topic modeling benchmarks run against it (ill be doing so with some of my local/private datasets). [1] https://github.com/meta-llama/llama-models/blob/eececc27d275a0f7d4d14ec83648d51c10a76560/models/llama4/model.py#L110-L258 https://github.com/meta-llama/llama-models/blob/eececc27d275... EDIT: should be fair/complete and note they do claim perfect NIAH text retrieval performance across all 10M tokens for the Scout model on their blog post: https://ai.meta.com/blog/llama-4-multimodal-intelligence/ https://ai.meta.com/blog/llama-4-multimodal-intelligence/. There are some serious limitations and caveats to that particular flavor of test though.
- theGnuMe 2y ago> using a global attention mask and a local chunked attention mask. Would you mind expanding on this? Or point to a reference or two? Thanks! I am trying to understand it.
- spmurrayzzz 2y ago
- imcritic 2y agoAll I get is {"error":{"message":"Not Found"}}
- ozenhati 2y agocan you reach out to us via live chat on console.groq.com with your organization id?
- growdark 2y agoWould it be realistic to buy and self-host the hardware to run, for example, the latest Llama 4 models, assuming a budget of less than $500,000?
- hhh 2y agoYou can do it with regular gpus for less
- deleted 2y ago[deleted]
- mrajcok 2y agoYes - I'm able to run Llama 3.1 405B on 3x A6000 + 3x 4090. Will have Llama 4 Maverick running in 4bit quantization (typically results in only minor quality degradation) once llama.cpp support is merged. Total hardware cost well under $50,000. The 2T Behemoth model is tougher, but enough Blackwell 6000 Pro cards (16) should be able to run it for under $200k.
- latchkey 2y agoA box of AMD MI300x (1.5TB of memory) is much less than $500k and AMD made sure to have day zero support with vLLM. That said, I'm obviously biased but you're probably better off renting it.
- briandw 2y agoLlama scout is a 17B x 16 MOE. So that 17B active parameters. That makes it faster to run. But the memory requirements are still large. They claim it fits on an H100. So under 80GB. A mac studio at 96GB could run this. By run i mean inference, Ollama is easy to use for this. 4x3090 nvidia cards would also work but its not the easiest pc build. The tinybox https://tinygrad.org/#tinybox https://tinygrad.org/#tinybox is 15k and you can do Lora fine tuning. Could also do a regular pc with 128gb of ram, but its would be quite slow.
- jasonjmcghee 2y agoSeems to be about 500 tk/s. That's actually significantly less than I expected / hoped for, but fantastic compared to nearly anything else. (specdec when?) Out of curiosity, the console is letting me set max output tokens to 131k but errors above 8192. what's the max intended to be? (8192 max output tokens would be rough after getting spoiled with 128K output of Claude 3.7 Sonnet and 64K of gemini models.)
- ozenhati 2y agodo you happen to be trying this out on free tier right now? because our rate limits are at 6k tokens per minute on free tier for this model, which might be what you're running into.
- jasonjmcghee 2y agoWhen I tried llama4 scout and tried to set the max output tokens above 8192 it told me the max was 8192. Once I set it below, it worked. This was in the console
- parhamn 2y agoI might be biased by the products I'm building but it feels to me that function support is table stakes now? Are open source models are just missing the dataset to fine tune one? Very few of the models supported on Groq/Together/Fireworks support function calling. And rarely the interesting ones (DeepSeek V3, large llamas, etc)
- garfij 2y agoThank you for saying this out loud. I've been losing my mind wondering where the discussion on this was. LLMs without Tool Use/Function Calling is basically a non starter for anything I want to do.
- jstanley 2y agoWhen I was working with LLMs without function calling I made the scaffold put some information in the system prompt that tells it some JSON-ish syntax it can use to invoke function calls. It places more of a "mental burden" on the model to output tool calls in your custom format, but it worked enough to be useful.
- ozenhati 2y ago100%. we've found that llama-3.3-70b-versatile and qwen-qwq-32b perform exceptionally well with reliable function calling. we had recognized the need for this and our engineers partnered with glaive ai to create fine tunes of llama 3.0 specifically for better function calling performance until the llama 3.3 models came along and performed even better. i'd actually love to hear your experience with llama scout and maverick for function calling. i'm going to dig into it with our resident function calling expert rick lamers this week.
- minimaxir 2y agoAlthough Llama 4 is too big for mere mortals to run without many caveats, the economics of call a dedicated-hosting Llama 4 are more interesting than expected. $0.11 per 1M tokens, a 10 million content window (not yet implemented in Groq), and faster inference due to fewer activated parameters allows for some specific applications that were not cost-feasible to be done with GPT-4o/Claude 3.7 Sonnet. That's all dependent on whether the quality of Llama 4 is as advertised, of course, particularly around that 10M context window.
- latchkey 2y ago> Llama 4 is too big for mere mortals to run without many caveats AMD MI300x has day zero support to run it using vLLM. Easy enough to rent them for decent pricing.
- zozbot234 2y agoIt's possible that we'll see smaller Llama 4-based models in the future, though. Similar to Llama 3.2 1B, which was released later than other Llama 3.x models.
- sroussey 2y agoYeah, I too am looking forward to their small text only models at 3B and 1B.
- deleted 2y ago[deleted]
- geor9e 2y agoI'm glad I saw this because llama-3.3-70b-versatile just stopped working in my app. I switched it to meta-llama/llama-4-scout-17b-16e-instruct and it started working again. Maybe groq stopped supporting the old one?
- sinab 2y agoI got an error when passing a prompt with about 20k tokens to the Llama 4 Scout model on groq (despite Llama 4 supporting up to 10M token context). groq responds with a POST https://api.groq.com/openai/v1/chat/completions https://api.groq.com/openai/v1/chat/completions 413 (Payload Too Large) error. Is there some technical limitation on the context window size with LPUs or is this a temporary stop-gap measure to avoid overloading groq's resources? Or something else?