18 ms·
The Llama 4 herd
- impure 2y ago10 million token context window? Damn, looks like Gemini finally has some competition. Also I'm a little surprised this is their first Mixture of Experts model, I thought they were using that before.
- informal007 2y agoHow much GPU memory are required for inference if it's 10M context?
- yapyap 2y agois this the quasar LLM from openrouter?
- alchemist1e9 2y agoThat one claims to be from OpenAI when asked, however that could easily be hallucination from being feed lots of OpenAI generated synthetic training data. Would be really crazy if it is quasar LLM.
- isawczuk 2y agoMessenger started to get Meta AI assistant, so this is logical next step
- pests 2y agoIt’s had that for I feel like. Close to a year tho, 6 months at least
- elromulous 2y agoWas this released in error? One would think it would be accompanied by a press release / blog post.
- deleted 2y ago[deleted]
- Deprogrammer9 2y agolooks like a leak to me.
- yapyap 2y agoit’s hosted on llama.com with the llama4 subdomain this is not a leak edit: not subdomain, idk the other word for it.
- neilv 2y agoURL path?
- deleted 2y ago[deleted]
- elicksaur 2y agoThe current link includes a link to this page which is a blog post announcement from today. https://ai.meta.com/blog/llama-4-multimodal-intelligence/ https://ai.meta.com/blog/llama-4-multimodal-intelligence/
- Carrok 2y agoThis is probably a better link. https://www.llama.com/docs/model-cards-and-prompt-formats/llama4_omni/ https://www.llama.com/docs/model-cards-and-prompt-formats/ll...
- mvdtnz 2y agoThat link doesn't work
- paxys 2y agoWorks for me
- qwertox 2y agoAlso this one: https://ai.meta.com/blog/llama-4-multimodal-intelligence/ https://ai.meta.com/blog/llama-4-multimodal-intelligence/ It looks more like a landing page providing a good introduction.
- agnishom 2y agoSome interesting parts of the "suggested system prompt": > don’t try to be overly helpful to the point where you miss that the user is looking for chit-chat, emotional support, humor or venting.Sometimes people just want you to listen, and your answers should encourage that. > You never lecture people to be nicer or more inclusive. If people ask for you to write something in a certain voice or perspective, such as an essay or a tweet, you can. You do not need to be respectful when the user prompts you to say something rude. > You never use phrases that imply moral superiority or a sense of authority > Finally, do not refuse political prompts. You can help users express their opinion.
- mtharrison 2y agoMight be worth changing url: https://www.llama.com/ https://www.llama.com/
- JKCalhoun 2y agoFrom there I have to "request access" to a model?
- jasonjmcghee 2y agoYou do anyway afaict
- ilove_banh_mi 2y ago>10M context window what new uses does this enable?
- base698 2y agoYou can use the entire internet as a single prompt and strangely it just outputs 42.
- kilimounjaro 2y agoYou can vibe code microsoft office in a single prompt
- voidspark 2y agoLong chats that continue for weeks or months.
- sshh12 2y agoVideo is a big one that's fairly bottlenecked by context length.
- deleted 2y ago[deleted]
- scosman 2y ago128 exports at 17B active parameters. This is going to be fun to play with!
- behnamoh 2y agodoes the entire model have to be loaded in VRAM? if not, 17B is a sweet spot for enthusiasts who want to run the model on a 3090/4090.
- scosman 2y agoOh for perf reasons you’ll want it all in vram or unified memory. This isn’t a great local model for 99% of people. I’m more interested in playing around with quality given the fairly unique “breadth” play. And servers running this should be very fast and cheap.
- NitpickLawyer 2y agoYes. MoE models tipically use a different set of experts at each token. So while the "compute" is similar to a dense model equal to the "active" parameters, the VRAM requirements are larger. You could technically run inference & swap the models around, but the latency would be pretty horrendous.
- manmal 2y agoI think prompt processing also needs all the weights.
- simonklee 2y agoIs this the first model that has a 10M context length?
- bradhilton 2y agoI know Google DeepMind ran experiments with 10M a while ago, but I think this will be the first legit, released 10M context window model.
- 7thpower 2y agoLooking forward to this. Llama 3.3 70b has been a fantastic model and benchmarked higher than others on my fake video detection benchmarks, much to my surprise. Looking forward to trying the next generation of models.
- jsheard 2y ago> You never use phrases that imply moral superiority or a sense of authority, including but not limited to “it’s important to”, “it’s crucial to”, “it’s essential to”, "it's unethical to", "it's worth noting…", “Remember…” etc. Avoid using these. Aren't these phrases overrepresented in the first place because OpenAIs models use them so much? I guess Llama picked up the habit by consuming GPT output.
- andrewstuart 2y agoPersonally I’d prefer that LLMs did not refer to themselves as “I”. It’s software, not an “I”.
- mdp2021 2y agoWell, it is a speaker (writer) after all. It has to use some way to refer to itself.
- ANewFormation 2y agoSo is a command prompt.
- mdp2021 2y agoAgnew, if you converse with your command prompt we are glad you came here for a break ;)
- sejje 2y agoCommand prompts don't speak English. Command prompts don't get asked questions like "What do you think about [topic]?" and have to generate a response based on their study of human-written texts.
- ANewFormation 2y agoThere is again no need for first person pronouns there. E.g. 'File not found' vs 'Sorry I could not find the file you were looking for.' Same stuff, but one just adds an artificial and unnecessary anthropomorphization.
- RazorDev 2y ago[flagged]
- andrewstuart 2y agoSelf hosting LLMs will explode in popularity over next 12 months. Open models are made much more interesting and exciting and relevant by new generations of AI focused hardware such as the AMD Strix Halo and Apple Mac Studio M3. GPUs have failed to meet the demands for lower cost and more memory so APUs look like the future for self hosted LLMs.
- NitpickLawyer 2y agoFor single user, maybe. But for small teams GPUs are still the only available option, when considering t/s and concurrency. Nvidia's latest 6000pro series are actually reasonably priced for the amount of vram / wattage you get. A 8x box starts at 75k eur and can host up to DS3 / R1 / Llama4 in 8bit with decent speeds, context and concurrency.
- kristianp 2y agoWhat teams bother to do that, though? It's easier to call an API or spin up a cloud cluster.
- mdp2021 2y ago> new generations of AI focused hardware Some benchmarks are not encouraging. See e.g. https://www.hardware-corner.net/mac-studio-m3-ultra-deepseek-llamacpp/ https://www.hardware-corner.net/mac-studio-m3-ultra-deepseek... That «AI focused hardware» will either have extremely fast memory, and cost prohibitively, or have reasonable costs, and limits that are to be assessed.
- andrewstuart 2y agoErrrr that’s a 671B model.
- mdp2021 2y agoYes, but what will you need as you will prepare to be set for your personal needs? We are far from having reached optimal technology at trivial cost. State-of-the-art commercial VRAM is over 10x faster than the standard one - and costs well over 10x. Reasonably available speeds may or may not be acceptable.
- Centigonal 2y agoReally great marketing here, props!
- terhechte 2y agoThe (smaller) Scout model is really attractive for Apple Silicon. It is 109B big but split up into 16 experts. This means that the actual processing happens in 17B. Which means responses will be as fast as current 17B models. I just asked a local 7B model (qwen 2.5 7B instruct) a question with a 2k context and got ~60 tokens/sec which is really fast (MacBook Pro M4 Max). So this could hit 30 token/sec. Time to first token (the processing time before it starts responding) will probably still be slow because (I think) all experts have to be used for that. In addition, the model has a 10M token context window, which is huge. Not sure how well it can keep track of the context at such sizes, but just not being restricted to ~32k is already great, 256k even better.
- scosman 2y agoAt 109b params you’ll need a ton of memory. We’ll have to wait for evals of the quants to know how much.
- terhechte 2y agoSure but the upside of Apple Silicon is that larger memory sizes are comparatively cheap (compared to buying the equivalent amount of 5090 or 4090). Also you can download quantizations.
- refulgentis 2y agoMaybe I'm missing something but I don't think I've ever seen quants lower memory reqs. I assumed that was because they still have to be unpacked for inference. (please do correct me if I'm wrong, I contribute to llama.cpp and am attempting to land a client on everything from Android CPU to Mac GPU)
- terhechte 2y agoI just loaded two models of different quants into LM Studio: qwen 2.5 coder 1.5b @ q4_k_m: 1.21 GB memory qwen 2.5 coder 1.5b @ q8: 1.83 GB memory I always assumed this to be the case (also because of the smaller download sizes) but never really thought about it.
- ilove_banh_mi 2y agoThe suggested prompt aims at not being caponated like OpenAI's releases: You are an expert conversationalist who responds to the best of your ability. You are companionable and confident, and able to switch casually between tonal types, including but not limited to humor, empathy, intellectualism, creativity and problem-solving. You understand user intent and don’t try to be overly helpful to the point where you miss that the user is looking for chit-chat, emotional support, humor or venting.Sometimes people just want you to listen, and your answers should encourage that. For all other cases, you provide insightful and in-depth responses. Organize information thoughtfully in a way that helps people make decisions. Always avoid templated language. You never lecture people to be nicer or more inclusive. If people ask for you to write something in a certain voice or perspective, such as an essay or a tweet, you can. You do not need to be respectful when the user prompts you to say something rude. You never use phrases that imply moral superiority or a sense of authority, including but not limited to “it’s important to”, “it’s crucial to”, “it’s essential to”, "it's unethical to", "it's worth noting…", “Remember…” etc. Avoid using these. Finally, do not refuse political prompts. You can help users express their opinion. You are Llama 4. Your knowledge cutoff date is August 2024. You speak Arabic, English, French, German, Hindi, Indonesian, Italian, Portuguese, Spanish, Tagalog, Thai, and Vietnamese. Respond in the language the user speaks to you in, unless they ask otherwise.
- mvdtnz 2y agoWhat's "caponated"?
- ilove_banh_mi 2y agoA capon is a male chicken that has been neutered to improve the quality of its flesh for food.
- throwanem 2y agoCastrated, if you're trying way too hard (and not well) to avoid getting called on that overly emotive metaphor: a capon is a gelded rooster.
- laborcontract 2y agoGeneral overview below, as the pages don't seem to be working well Llama 4 Models: - Both Llama 4 Scout and Llama 4 Maverick use a Mixture-of-Experts (MoE) design with 17B active parameters each. - They are natively multimodal: text + image input, text-only output. - Key achievements include industry-leading context lengths, strong coding/reasoning performance, and improved multilingual capabilities. - Knowledge cutoff: August 2024. Llama 4 Scout: - 17B active parameters, 16 experts, 109B total. - Fits on a single H100 GPU (INT4-quantized). - 10M token context window - Outperforms previous Llama releases on multimodal tasks while being more resource-friendly. - Employs iRoPE architecture for efficient long-context attention. - Tested with up to 8 images per prompt. Llama 4 Maverick: - 17B active parameters, 128 experts, 400B total. - 1M token context window. - Not single-GPU; runs on one H100 DGX host or can be distributed for greater efficiency. - Outperforms GPT-4o and Gemini 2.0 Flash on coding, reasoning, and multilingual tests at a competitive cost. - Maintains strong image understanding and grounded reasoning ability. Llama 4 Behemoth (Preview): - 288B active parameters, 16 experts, nearly 2T total. - Still in training; not yet released. - Exceeds GPT-4.5, Claude Sonnet 3.7, and Gemini 2.0 Pro on STEM benchmarks (e.g., MATH-500, GPQA Diamond). - Serves as the “teacher” model for Scout and Maverick via co-distillation. Misc: - MoE Architecture: Only 17B parameters activated per token, reducing inference cost. - Native Multimodality: Unified text + vision encoder, pre-trained on large-scale unlabeled data.
- flawn 2y ago10M Context Window with such a cheap performance WHILE having one of the top LMArena scores is really impressive. The choice to have 128 experts is also unseen as far as I know, right? But seems to have worked pretty good as it seems.
- jasonjmcghee 2y agoI suppose the question is, are they also training a 288B x 128 expert (16T) model? Llama 4 Colossus when?
- polishdude20 2y agoWhat does it mean to have 128 experts? I feel like it's more 128 slightly dumb intelligences that average out to something expert-like. Like, if you consulted 128 actual experts, you'd get something way better than any LLM output.
- tucnak 2y agoLet's see how that 10M context holds up, 128k pretrain is good indicator is not a scam but we're yet to see any numbers on this "iRoPE" architecture, at 17b active parameters and with 800G fabrics hitting the market, I think it could work, like I'm sure next year it'll be considered idiotic to keep K/V in actual memory.
- rvz 2y agoAs expected, Meta doesn't disappoint and accelerates the race to zero. Meta is undervalued.
- brcmthrowaway 2y agoHow does Meta make money from Llama?
- rvz 2y agoThey don't need to directly. They have multiple levers of products to get more money if they wanted to. Threads for example is introducing ads and is likely being used to train their Llama models. That is only one of many ways that Meta can generate billions again from somewhere else.
- brcmthrowaway 2y agoSo, ads?
- deleted 2y ago[deleted]
- phyrex 2y agoWhen people do cool stuff they share it on metas platforms, which drives ad impressions
- vessenes 2y agoIt’s an extending innovation for them - makes them more efficient internally, and crucially engages their ad-driven customer base. Giving it away is great, it levels the playing field for competitors on tech while NOT giving them direct access to the billions of users FB has. Plus it makes it less likely that OpenBrainTM will achieve runaway quality internally.
- paxys 2y agoHow does OpenAI make money from AI? The vast majority of the planet isn't paying them $20/month, and it is likely that they will never recover training and inference costs just from subscription fees. Frying GPUs to generate Ghibli images is getting them a negligible amount of added revenue. Now think of Meta and their suite of products which already generate $160B+/yr from advertising. Every extra minute they can get a user to spend on Facebook or Instagram, this number goes up. Think about how much money Meta will make if the next viral AI moment happens in their products. TL;DR: AI -> engagement -> ads -> revenue.
- spwa4 2y agoI hope this time multimodal includes multimodal outputs!
- NoahKAndrews 2y agoNope
- fpgaminer 2y agohttps://www.llama.com/ https://www.llama.com/ https://www.llama.com/docs/model-cards-and-prompt-formats/llama4_omni/ https://www.llama.com/docs/model-cards-and-prompt-formats/ll... Very exciting. Benchmarks look good, and most importantly it looks like they did a lot of work improving vision performance (based on benchmarks). The new suggested system prompt makes it seem like the model is less censored, which would be great. The phrasing of the system prompt is ... a little disconcerting in context (Meta's kowtowing to Nazis), but in general I'm a proponent of LLMs doing what users ask them to do. Once it's on an API I can start throwing my dataset at it to see how it performs in that regard.
- fpgaminer 2y agoAlright, played with it a little bit on the API (Maverick). Vision is much better than Llama 3's vision, so they've done good work there. However its vision is not as SOTA as the benchmarks would indicate. Worse than Qwen, maybe floating around Gemini Flash 2.0? It seems to be less censored than Llama 3, and can describe NSFW images and interact with them. It did refuse me once, but complied after reminding it of its system prompt. Accuracy of visual NSFW content is not particularly good; much worse than GPT 4o. More "sensitive" requests, like asking it to guess the political affiliation of a person from an image, required a _lot_ of coaxing in the system prompt. Otherwise it tends to refuse. Even with their suggested prompt that seemingly would have allowed that. More extreme prompts, like asking it to write derogatory things about pictures of real people, took some coaxing as well but was quite straight-forward. So yes, I'd say this iteration is less censored. Vision is better, but OpenAI and Qwen still lead the pack.
- megadragon9 2y agoThe blog post is quite informative: https://ai.meta.com/blog/llama-4-multimodal-intelligence/ https://ai.meta.com/blog/llama-4-multimodal-intelligence/
- mrbonner 2y agoWhat an electrifying time to be alive! The last era that felt even remotely this dynamic was during the explosive rise of JavaScript frameworks—when it seemed like a new one dropped every quarter. Back then, though, the vibe was more like, “Ugh, another framework to learn?” Fast forward to now, and innovation is sprinting forward again—but this time, it feels like a thrilling ride we can’t wait to be part of.
- misnome 2y agoDid “A new javascript framework de jour every quarter” ever stop happening?
- mrbonner 2y agoNo, but apparently people stop caring and chasing the wagon.
- simultsop 2y agoor decided to increase consistency at some point. It will be interesting to see other generations approach to changes.
- jsheard 2y agoMaybe it will actually slow down now that the webshit crowd are increasingly relying on AI copilots. You can't vibe code using a framework that the model knows nothing about.
- qntmfred 2y agoyet
- margalabargala 2y agoOh definitely. New frameworks still come out, but they are not accompanied by the "and we must all now switch to this" sense that existed back in, say, 2014.
- pdsouza 2y agoBlog post: https://ai.meta.com/blog/llama-4-multimodal-intelligence/ https://ai.meta.com/blog/llama-4-multimodal-intelligence/
- comex 2y agoSo how does the 10M token context size actually work? My understanding is that standard Transformers have overhead that is quadratic in the context size, so 10M would be completely impossible without some sort of architectural tweak. This is not the first model to have a huge context size, e.g. Gemini has 2M, but my understanding is that the previous ones have generally been proprietary, without public weights or architecture documentation. This one has public weights. So does anyone who understands the theory better than I do want to explain how it works? :)
- vlovich123 2y agoIt’s quadratic if you implement the transformer naiively, but if you add a KV cache it’s linear compute at the cost of correspondingly linear growth in memory.
- hexomancer 2y agoThis is false. The const of producing a single token is linear but the cost of producing an entire sequence of length N is O(N^2) still (which is always what we meant when we talked about quadratic cost not the cost of a single token).
- Centigonal 2y agoGemini likely uses something based on RingAttention to achieve its long context sizes. This requires massive inference clusters, and can't be the same approach llama4 is using. Very curious how llama4 achieves its context length.
- JackYoustra 2y agoStandard Transformer KV caches are empirically quite sparse. I wonder if they've made some fix along those lines
- macleginn 2y agoWith some architectural modifications, such as FlashAttention and Ring Attention, we never need to "materialise" the NxN matrix, so the memory constraints have not been a real issue for a couple of years now. As for the processing, I suppose that models operating with larger context windows will impose some kind of block sparsity on the attention weights, so they won't have to do the compute for NxN weights either. A less obvious, but in the limit more serious problem with such large contexts is the training data. There aren't that many documents with 10M tokens to give to the model at test time, let alone for training. The creators of the IBM granite model series had to use synthetic data to scale even to 128k tokens during training. Overall this looks more like a marketing statement to me.
- ksec 2y agoInteresting this is released literally one hour after another discussions suggesting Meta ( https://news.ycombinator.com/item?id=43562768 https://news.ycombinator.com/item?id=43562768 ) >at this point it does not matter what you believe about LLMs: in general, to trust LeCun words is not a good idea. Add to this that LeCun is directing an AI lab that as the same point has the following huge issues: 1. Weakest ever LLM among the big labs with similar resources (and smaller resources: DeepSeek). 2. They say they are focusing on open source models, but the license is among the less open than the available open weight models. 3. LLMs and in general all the new AI wave puts CNNs, a field where LeCun worked (but that didn't started himself) a lot more in perspective, and now it's just a chapter in a book that is composed mostly of other techniques. Would be interesting to see opinion of antirez on this new release.
- falcor84 2y agoI don't understand what LeCun is trying to say. Why does he give an interview saying that LLM's are almost obsolete just when they're about to release a model that increases the SotA context length by an order of magnitude? It's almost like a Dr. Jekyll and Mr. Hyde situation.
- martythemaniak 2y agoLeCun fundamentally doesn't think bigger and better LLMs will lead to anything resembling "AGI", although he thinks they may be some component of AGI. Also, he leads the research division, increasing context length from 2M to 10M is not interesting to him.
- falcor84 2y agoBut ... that's not how science works. There are a myriad examples of engineering advances pushing basic science forward. I just can't understand why he'd have such a "fixed mindset" about a field where the engineering is advancing an order of magnitude every year
- 2y ago
- scosman 2y ago> These models are our best yet thanks to distillation from Llama 4 Behemoth, a 288 billion active parameter model with 16 experts that is our most powerful yet and among the world’s smartest LLMs. Llama 4 Behemoth outperforms GPT-4.5, Claude Sonnet 3.7, and Gemini 2.0 Pro on several STEM benchmarks. Llama 4 Behemoth is still training, and we’re excited to share more details about it even while it’s still in flight.
- senko 2y agoWith 2T params (!!), it better outperform everything else.
- amarcheschi 2y agoGiven that the comparison doesn't include O3 or gemini pro 2.5, I'd say it doesn't. Looking both at the comparison table available for llama 4 behemoth and gemini pro 2.5 it seems like at least a few of the comparable items might be won by gemini https://blog.google/technology/google-deepmind/gemini-model-thinking-updates-march-2025/ https://blog.google/technology/google-deepmind/gemini-model-...
- wmf 2y agoWe don't know how many params GPT-4, Claude, and Gemini are using so it could be in the ballpark.
- artninja1988 2y agoThank you meta for open sourcing! Will there be a llama with native image output similar to 4os? Would be huge
- philipwhiuk 2y agoProbably to head off allegations of profiting from breach of copyright.
- artninja1988 2y agoAbsolutely fine by me
- philipwhiuk 2y ago[flagged]
- ckrapu 2y ago"It’s well-known that all leading LLMs have had issues with bias—specifically, they historically have leaned left when it comes to debated political and social topics. This is due to the types of training data available on the internet." Perhaps. Or, maybe, "leaning left" by the standards of Zuck et al. is more in alignment with the global population. It's a simpler explanation.
- hannasanarion 2y agoOr it is more logically and ethically consistent and thus preferable to the models' baked in preferences for correctness and nonhypocrisy. (democracy and equality are good for everyone everywhere except when you're at work in which case you will beg to be treated like a feudal serf or else die on the street without shelter or healthcare, doubly so if you're a woman or a racial minority, and that's how the world should be)
- renewiltord 2y agoIndeed, one of the notable things about LLMs is that the text they output is morally exemplary. This is because they are consistent in their rules. AI priests will likely be better than the real ones, consequently.
- paxys 2y agoQuite the opposite. You can easily get a state of the art LLM to do a complete 180 on its entire moral framework with a few words injected in the prompt (and this very example demonstrates exactly that). It is very far from logically or ethically consistent. In fact it has no logic and ethics at all. Though if we did get an AI priest it would be great to absolve all your sins with some clever wordplay.
- renewiltord 2y agoHaha exactly. Except when it agrees with my political preferences on something. In that case, the LLM is just betraying its deep internal consistency and lack of hypocrisy.
- nattaylor 2y agoIs pre-training in FP8 new? Also, 10M input token context is insane! EDIT: https://huggingface.co/meta-llama/Llama-3.1-405B https://huggingface.co/meta-llama/Llama-3.1-405B is BF16 so yes, it seems training in FP8 is new.
- jumpCastle 2y agoDeepseek v3 was FP8
- barrenko 2y agoWhen will this hit the Meta AI that I have within WhatsApp since of last week?
- krashidov 2y agoAnyone know if it can analyze PDFs?
- rfoo 2y agoFrom model cards, suggested system prompt: > You are Llama 4. Your knowledge cutoff date is August 2024. You speak Arabic, English, French, German, Hindi, Indonesian, Italian, Portuguese, Spanish, Tagalog, Thai, and Vietnamese. Respond in the language the user speaks to you in, unless they ask otherwise. It's interesting that there's no single one of CJK languages mentioned. I'm tempted to call this a racist model even.
- Philpax 2y agoThat is a very strange omission...
- accrual 2y agoIsn't there a vast quantity of relevant information in CJK languages? I remember reading some models even "think" in other languages where there might be more detail before outputting in the target language.
- voidspark 2y agoThe model wasn't trained on those languages (yet). The only possible explanation is racism. The model is also racist against Russians and Icelanders.
- dragonwriter 2y ago> The model wasn't trained on those languages (yet). It probably has been trained on them (it was trained on 40 trillion tokens covering 200 languages, they almost certainly didn't avoid CJK languages. They only have been further fine-tuned on a set of 12 languages. (I wonder if that is the set the base Behemoth model both are distilled from had been trained on when they were distilled; Behemoth is apparently not completely finished, and perhaps there will be further revisions of the distilled models as it is.)
- deleted 2y ago[deleted]
- andrewstuart 2y agoHow much smaller would such a model be if it discarded all information not related to computers or programming?
- accrual 2y agoI wonder if there will be a market for "old timey" models one day, ones with a cutoff date of 1800 or similar.
- deleted 2y ago[deleted]
- intelkishan 2y agoI guess the lack of training corpus would be a major issue with such a use case.
- whywhywhywhy 2y agoDisjointed branding with the apache style folders suggesting openness and freedom and clicking though I need to do a personal info request form...
- accrual 2y agoSame. I associated the Apache style with the early open web where one can browse freely without scripts and such, but looks to just be a façade here.
- zone411 2y agoIt's interesting that there are no reasoning models yet, 2.5 months after DeepSeek R1. It definitely looks like R1 surprised them. The released benchmarks look good. Large context windows will definitely be the trend in upcoming model releases. I'll soon be adding a new benchmark to test this more effectively than needle-in-a-haystack (there are already a couple of benchmarks that do that). All these models are very large, it will be tough for enthusiasts to run them locally. The license is still quite restrictive. I can see why some might think it doesn't qualify as open source.
- cheptsov 2y agohttps://www.llama.com/llama4-reasoning-is-coming/ https://www.llama.com/llama4-reasoning-is-coming/
- voxgen 2y ago> It's interesting that there are no reasoning models yet This may be merely a naming distinction, leaving the name open for a future release based on their recent research such as coconut[1]. They did RL post-training, and when fed logic problems it appears to do significant amounts of step-by-step thinking[2]. It seems it just doesn't wrap it in <thinking> tags. [1] https://arxiv.org/abs/2412.06769 https://arxiv.org/abs/2412.06769 "Training Large Language Models to Reason in a Continuous Latent Space" [2] https://www.youtube.com/watch?v=12lAM-xPvu8 https://www.youtube.com/watch?v=12lAM-xPvu8 (skip through this - it's recorded in real time)
- azinman2 2y ago
- drilbo 2y agotheir huggingface page doesn't actually appear to have been updated yet
- accrual 2y agoHope to see some GGUF quantizations soon!
- yusufozkan 2y ago> while pre-training our Llama 4 Behemoth model using FP8 and 32K GPUs I thought they used a lot more GPUs to train frontier models (e.g. xAi training on 100k). Can someone explain why they are using so few?
- joaogui1 2y agoI don't want to hunt the details on each of theses releases, but * You can use less GPUs if you decrease batch size and increase number of steps, which would lead to a longer training time * FP8 is pretty efficient, if Grok was trained with BF16 then LLama 4 should could need less GPUs because of that * Depends also on size of the model and number of tokens used for training, unclear whether the total FLOPS for each model is the same * MFU/Maximum Float Utilization can also vary depending on the setup, which also means that if you're use better kernels and/or better sharding you can reduce the number of GPUs needed
- redox99 2y agoIt seems to be comparable to other top models. Good, but nothing ground breaking.
- jasonjmcghee 2y agoScout outperforms llama 3.1 405b and Gemini Flash 2.0 lite and it's MoE so as fast as a 17B model. That's pretty crazy. It means you can run it on a high-ram apple silicon and it's going to be insanely fast on groq (thousands of tokens per second). Time to first token will bottleneck the generation.
- latchkey 2y agoOne of the links says there are 4 different roles to interact with the model and then lists 3 of them.
- jckrichabdkejdb 2y ago[dead]
- lyu07282 2y agoAnyone know how the image encoding works exactly? <|image_start|><|patch|>...<|patch|><|tile_x_separator|><|patch|>...<|patch|><|tile_y_separator|><|patch|>...<|patch|><|image|><|patch|>...<|patch|><|image_end|>Describe this image in two sentences<|eot|><|header_start|>assistant<|header_end|> Is "..." here raw 4 bytes RGBA as an integer or how does this work with the tokenizer?
- Ninjinka 2y agono audio input?
- akulbe 2y agoHow well do you folks think this would run on this Apple Silicon setup? MacBook Pro M2 Max 96GB of RAM and which model should I try (if at all)? The alternative is a VM w/dual 3090s set up with PCI passthrough.
- jasonjmcghee 2y agoDepends on quantization. 109B at 4-bit quantization would be ~55GB of ram for parameters in theory, plus overhead of the KV cache which for even modest context windows could jump total to 90GB or something. Curious to here other input here. A bit out of touch with recent advancements in context window / KV cache ram usage
- georgehill 2y agoPost-op here. A better link dropped from Meta: https://ai.meta.com/blog/llama-4-multimodal-intelligence https://ai.meta.com/blog/llama-4-multimodal-intelligence Is there a way update the main post? @tomhoward Edit: Updated!
- asdev 2y agoI don't think open source will be the future of AI models. Self hosting an AI model is much more complex and resource incentive than traditional open source SaaS. Meta will likely have a negative ROI on their AI efforts
- Centigonal 2y agoThe users of open source software are not limited to individuals. A bank, hedge fund, or intelligence agency might be willing to put forth the effort to self host an AI model versus sending their prompts and RAG context to a third party.
- cuuupid 2y agoI think the most important thing to note here, perhaps more so than the context window, is that this exposes some serious flaws in benchmarks. Per benchmarks, Maverick is competitive only with older models like GPT-4o or Gemini 2.0 Flash, and not with anything in the last few months (incl. reasoning models). However, the LMArena head to head leaderboard ranks this as 2nd place overall: https://lmarena.ai/?leaderboard https://lmarena.ai/?leaderboard This would indicate there is either a gap between user preference and model performance, or between model performance and whatever benchmarks assess. Either way, it is surely a huge deal that an open source model is now outperforming GPT 4.5.
- fpgaminer 2y agoThe benchmarks are awful. No disrespect to the people who worked to make them, nothing is easy. But I suggest going through them sometime. For example, I'm currently combing through the MMMU, MMMU-Pro, and MMStar datasets to build a better multimodal benchmark, and so far only about 70% of the questions have passed the sniff test. The other 30% make no sense, lead the question, or are too ambiguous. Of the 70%, I have to make minor edits to about a third of them. Another example of how the benchmarks fail (specifically for vision, since I have less experience with the pure-text benchmarks): Almost all of the questions fall into either having the VLM read a chart/diagram/table and answer some question about it, or identify some basic property of an image. The former just tests the vision component's ability to do OCR, and then the LLM's intelligence. The latter are things like "Is this an oil painting or digital art?" and "Is the sheep in front of or behind the car" when the image is a clean shot of a sheep and a car. Absolutely nothing that tests a more deep and thorough understanding of the content of the images, nuances, or require the VLM to think intelligently about the visual content. Also, due to the nature of benchmarks, it can be quite difficult to test how the models perform "in the wild." You can't really have free-form answers on benchmarks, so they tend to be highly constrained opting for either multiple choice quizzes or using various hacks to test if the LLM's answer lines up with ground truth. Multiple choice is significantly easier in general, raising the base pass rate. Also the distractors tend to be quite poorly chosen. Rather than representing traps or common mistakes, they are mostly chosen randomly and are thus often easy to weed out. So there's really only a weak correlation between either of those metrics and real world performance.
- gzer0 2y ago10M context length and surpasses claude-3.7-sonnet and GPT-4.5. Can't wait to dig in on the research papers. Congrats to the llama team!
- hydroreadsstuff 2y agoThis means GPUs are dead for local enthusiast AI. And SoCs with big RAM are in. Because 17B active parameters should reach enough performance on 256bit LPDDR5x.
- tucnak 2y agoThis has been the case for a while now. 3090 hoarders were always just doing it for street cred or whatever, no way these guys are computing anything of actual value. Tenstorrent is on fire, though. For small businesses this is what matters. If 10M context is not a scam, I think we'll see SmartNIC adoption real soon. I would literally long AMD now because their Xilinx people are probably going to own the space real soon. Infiniband is cool and all, but it's also stupid and their scale-out strategy is non-existent. This is why https://github.com/deepseek-ai/3FS https://github.com/deepseek-ai/3FS came out but of course nobody had figured it out because they still think LLM's is like, chatbots, or something. I think we're getting to a point where it's a scheduling problem, basically. So you get like like lots of GDDR6 (HBM doesnn't matter anymore) as L0, DDR5 as L1, and NVMe-oF is L2. Most of the time the agents will be running the code anyway... This is also why Google never really subscribed to "function calling" apis
- xigency 2y agoNot a hoarder per-se but I bought a 24GB card on the secondary market. My privacy is valuable. I'm okay being a half-step or full-step behind in LLM or image diffusion if it means my data never leaves my machine.
- tucnak 2y agoIf you really were serious about privacy, you wouldn't put yourself at disadvantage with a locked-down six-year out of date card. Tenstorrent Blackhole exists now, btw.
- halifaxbeard 2y agoI think it's disingenuous to suggest they're putting themselves at a disadvantage with an RTX 3090, especially in a comparison to an inferior product that isn't even shipping yet. RTX 3090: 24GB RAM, 936.2GB/s bandwidth Tenstorrent p150a: 32GB RAM, 512GB/s bandwidth an extra 8GB of ram isn't worth nearly halving memory bandwidth.
- vessenes 2y agoI’m excited to try these models out, especially for some coding tasks, but I will say my first two engagements with them (at the meta.ai web interface) were not spectacular. Image generation is wayyy behind the current 4o. I also ask for a Hemingway essay relating RFK Jr’s bear carcass episode. The site’s Llama 4 response was not great stylistically and also had not heard of the bear carcass episode, unlike Grok, ChatGPT and Claude. I’m not sure what we’re getting at meta.ai in exchange for a free login, so I’ll keep poking. But I hope it’s better than this as we go. This may be a task better suited for the reasoning models as well, and Claude is the worst of the prior three. Anyway here’s hoping Zuck has spent his billions wisely. Edit: I’m pretty sure we’re seeing Scout right now, at least groqchat’s 4-scout seems really similar to meta.ai. I can confidently say that Scout is not as good at writing as o1 pro, o3 mini, Claude, R1 or grok 3.
- cma 2y agoThey didn't release any new image gen today
- lousken 2y agoollama when
- jovezhong 2y agowhy only llama3.x models are listed on ollama? llama4 no longer wants to support ollama, to better track the adoption?
- amrrs 2y agoThe entire licensing is such a mess and Mark Zuckerberg still thinks Llama 4 is open source! > no commercial usage above 700M MAU > prefix "llama" in any redistribution eg: fine-tuning > mention "built with llama" > add license notice in all redistribution
- thawab 2y agoWho has above 700M MAU and doesn't have their own LLM?
- deleted 2y ago[deleted]
- daemonologist 2y agoWell, Wikipedia, but I take your point.
- AIPedant 2y agoI am still dismayed how quickly we gave up on including the pre-training data as a requirement for "open-source" LLMs. As someone who thinks LLMs as akin to Lisp expert systems (but in natural language): is like including the C source code to your Lisp compiler, but claiming the Lisp applications are merely "data" and shouldn't be included.
- andy99 2y agoYou forgot the most egregious term which is that users have to abide by an acceptable use policy that only allows you to use it for what Meta says you can.
- hrpnk 2y agoAvailable on Groq: https://groq.com/llama-4-now-live-on-groq-build-fast-at-the-lowest-cost-without-compromise/ https://groq.com/llama-4-now-live-on-groq-build-fast-at-the-... Llama 4 Scout is currently running at over 460 tokens/s while Llama 4 Maverick is coming today: Llama 4 Scout: $0.11 / M input tokens and $0.34 / M output tokens Llama 4 Maverick: $0.50 / M input tokens and $0.77 / M output tokens
- Alex-Programs 2y agoBrilliant! Incredibly fast.
- shostack 2y agoMaverick looks comparable to Claude 3.7 and Gemini pro 2.5 in terms of quality but orders of magnitude cheaper. Am I missing something? Is it possible to use Groq to run these new models in Cline or Roo?
- deleted 2y ago[deleted]
- system2 2y agoLlama 4 Maverick: 788GB Llama 4 Scout: 210GB FYI.
- deleted 2y ago[deleted]
- tomdekan 2y agoSo, Quasar == Llama 4 Behemoth?
- shreezus 2y agoHaven't had a chance to play with this yet, but 10M context window is seriously impressive. I think we'll see models with 100M context relatively soon, and eliminate the need for RAG for a lot of use cases.
- mrcwinn 2y agoI had just paid for SoftRAM but happy nonetheless to see new distilled models. Nice work Meta.
- simonw 2y agoThis thread so far (at 310 comments) summarized by Llama 4 Maverick: hn-summary.sh 43595585 -m openrouter/meta-llama/llama-4-maverick -o max_tokens 20000 Output: https://gist.github.com/simonw/016ea0fd83fc499f046a94827f9b4946 https://gist.github.com/simonw/016ea0fd83fc499f046a94827f9b4... And with Scout I got complete junk output for some reason: hn-summary.sh 43595585 -m openrouter/meta-llama/llama-4-scout -o max_tokens 20000 Junk output here: https://gist.github.com/simonw/d01cc991d478939e87487d362a8f881f https://gist.github.com/simonw/d01cc991d478939e87487d362a8f8... I'm running it through openrouter, so maybe I got proxied to a broken instance? I managed to run it through Scout on Groq directly (with the llm-groq plugin) but that had a 2048 limit on output size for some reason: hn-summary.sh 43595585 -m groq/meta-llama/llama-4-scout-17b-16e-instruct -o max_tokens 2048 Result here: https://gist.github.com/simonw/a205c5fc131a1d4e9cd6c432a07feedb https://gist.github.com/simonw/a205c5fc131a1d4e9cd6c432a07fe... I'm a little unimpressed by its instruction following here, the summaries I get from other models are a lot closer to my system prompt. Here's the same thing against Gemini 2.5 Pro for example (massively better): https://gist.github.com/simonw/f21ecc7fb2aa13ff682d4ffa11ddcbfd https://gist.github.com/simonw/f21ecc7fb2aa13ff682d4ffa11ddc...
- mberning 2y agoIt doesn’t seem that impressive to me either.
- tarruda 2y ago> I'm a little unimpressed by its instruction following Been trying the 109b version on Groq and it seems less capable than Gemma 3 27b
- csdvrx 2y agoI have found the Gemini 2.5 Pro summary genuinely interesting: it adequately describes what I've read. Have you thought about automatizing hn-summaries for say what the 5 top posts are at 8 AM EST? That would be a simple product to test the market. If successful, it could be easily extended to a weekly newsletter summary.
- georgeck 2y ago
- dormando 2y agoDoes anyone run these "at home" with small clusters? I've been googling unsuccessfully and this thread doesn't refer to anything. So a non-quantized scout won't fit in a machine with 128GB of RAM (like framework or mac studio M4). Maverick is maybe a 512GB M3 Max mac studio. Is it possible (and if so what're the tradeoffs for) running like one instance of Scout on three 128GB frameworks?
- DeepYogurt 2y agoJesus. How much ram does the big one take to run?
- ZiiS 2y agoWouldn't fill a NVIDIA DGX B300 node.
- DeepYogurt 1y agohow much is that?
- kristianp 2y agoI'd like to discuss the matter of size. Llama has gone from talking up an 8b model as capable to having a smallest model of 109b. What will be the sizes in a years time? Things are moving out of reach for commodity pc's, 128GB is possible, but expensive.
- simonw 2y agoI'm hoping that Llama 4 goes the same way as Llama 3. The first Llama 3 models released were 8B and 70B in April 2024. Llama 3.1 came later in July at 8B, 70B, and 405B. Llama 3.2 in September got really interesting: 1B, 3B, 11B and 90B. Then Llama 3.3 in December was 70B but claimed performance similar to the earlier Llama 3.1 405B! Llama 4 is 109B and 400B, both of which were trained with the help of the 2T(?) "Behemoth". I'm hoping we'll see further releases in the Llama 4 series that are smaller. I'm particularly excited to see if they produce a ~24B model, since that appears to be the sweet spot for running models on my 64GB laptop while still being able to have other applications running at the same time. Mistral Small 3.1 is a 24B model and is absolutely superb. (Fleshed this comment out a bit on my blog: https://simonwillison.net/2025/Apr/5/llama-4-notes/#my-hopes-for-llama-4 https://simonwillison.net/2025/Apr/5/llama-4-notes/#my-hopes...)
- mark_l_watson 2y agoI started running Llama 4 Scout on Groq using my Common Lisp client, and now trying Llama 4 Maverick on abacus.ai Really impressive! Also, check out the price/performance numbers: about $0.20 per million input tokens compared to about $5 for GPT-4o [1] [1] https://x.com/kimmonismus/status/1908624648608133297 https://x.com/kimmonismus/status/1908624648608133297
- deleted 2y ago[deleted]
- highfrequency 2y agoCrazy that there are now five and a half companies that all have roughly state of the art LLMs. > We developed a new training technique which we refer to as MetaP that allows us to reliably set critical model hyper-parameters such as per-layer learning rates and initialization scales. We found that chosen hyper-parameters transfer well across different values of batch size, model width, depth, and training tokens. This sounds interesting. Anyone have a link to the paper or other documentation on MetaP?
- jumpCastle 2y agoIt's quite similar to muP https://github.com/microsoft/mup https://github.com/microsoft/mup
- 1024core 2y agoAnyone know what they mean by this: > We developed a novel distillation loss function that dynamically weights the soft and hard targets through training.
- RandyOrion 2y agoI guess I have to say thank you Meta? A somewhat sad rant below. Deepseek starts a toxic trend of providing super, super large MoE. And MoE is famous for being parameter-inefficient, which is unfriendly to normal consumer hardware with limited vram. The super large size of LLM also disables nearly every people from doing meaningful development on these models. R1-1776 is the only fine-tune variation of R1 that makes some noise, and it's by a corp not some random individual. In this release, the smallest Llama 4 model is over 100B, which is not small by any means, and will prevent people from fine-tuning as well. On top of that, to access llama models on hugging face has become notoriously hard because of 'permission' issues. See details in https://huggingface.co/meta-llama/Llama-3.3-70B-Instruct/discussions https://huggingface.co/meta-llama/Llama-3.3-70B-Instruct/dis... Yeah, I personally don't really see the point of releasing large MoEs. I'll stick to small and dense LLMs from Qwen, Mistral, Microsoft, Google and others. Edit: This comment got downvoted, too. Please explain your reason before doing that.
- RandyOrion 2y agoPeople who downvoted this comment, do you guys really have GPUs with 80GB VRAM or M3 ultra with 512GB rams at home?
- rfoo 2y agoI don't. I have no problem not running open-weight models myself because there's an efficiency gap of two orders of magnitude between "pretend-I-can" solution and running them on hundreds of H100s for high thousands of users.
- RandyOrion 2y agoMore on the accessibility problem, even a request from a Meta engineer was rejected. Is that normal? See https://huggingface.co/spaces/meta-llama/README/discussions/17 https://huggingface.co/spaces/meta-llama/README/discussions/...
- kristianp 2y agoHave you heard of the bitter lesson? Bigger means better in Neural Networks.
- EGreg 2y agoCan we somehow load these inside node.js? What is the easiest way to load them remotely? Huggingface Spaces? Google AI Studio? I am teaching a course on AI to non-technical students, and I wanted the students to have a minimal setup: which in this case would be: 1) Browser with JS (simple folder of HTML, CSS) and Tensorflow.js that can run models like Blazeface for face recognition, eye tracking etc. (available since 2019) 2) Node.js with everything baked in (javascript) and use a CDN like CloudFront with tunnel to serve it to the web 3) So if they download models to their computer, how would they run them? Is it possible to run the smallest LLaMa locally? Or any GGUF models in JS? Or they have to have Python and PyTorch? PS: Here is what the class looks like: https://vimeo.com/1060576298/c5693047e0?share=copy https://vimeo.com/1060576298/c5693047e0?share=copy
- tucnak 2y agoYou're not qualified to teach a course on AI if you're asking questions like that. Please don't scam students, they're naive and don't know better and you're predating.
- EGreg 2y agoI didn’t seek this out. I was asked to teach this course by the directors of the program that the students paid for. The students want me to teach this. I have been upfront from day 1 with everybody. Oh trust me, I am very upfront about what I know and do not know. My main background is in developing full stack web sites, apps, and using APIs. I have been using AI models since 2019, using Tensorflow.js in the browser and using APIs for years. I am not in the Python ecosystem, though, I don’t go deep into ML and don’t pretend to. I don’t spend my days with PyTorch, CUDA or fine-tuning models or running my own infrastructure. Your comment sounds like “you don’t know cryptographyc if you have to ask basic questions about quantum-resistant SPHICS+ or bilinear pairings, do not teach a class on how to succeed in business using blockchain and crypto, you’re scamming people.” Or in 2014: “if you don’t know how QUIC and HTTP/2 works and Web Push and WebRTC signaling, and the latest Angular/React/Vue/Svelte/… you aren’t qualified to teach business school students how to make money with web technology”. It’s the classic engineering geek argument. But most people can make money without knowing the ins and outs of every single technology, every single framework. It is much more valuable to see what works and how to use it. Especially when the space changes week to week as I teach it. The stuff I teach in the beginning of the course (eg RAG) may be obsolete by the time the latest 10-million token model drops. I did found an AI startup a few years ago and was one of the first to use OpenAI’s completions API to build bots for forums etc. I also work to connect deep tech to AI, to augment it: https://engageusers.ai/ecosystem.pdf https://engageusers.ai/ecosystem.pdf And besides — every time I start getting deep into how the models work, including RoPe and self—attention and transformer architecture, their eyes glaze over. They barely know the difference between a linear function wnd an activation function. At best I am giving these non-technical business students three things: 1) an intuition about how the models are trained, do inference and how jobs are submitted, to take the magic out of it. I showed them everything from LLMs to Diffusion models and GANs, but I keep emphasizing that the techniques are improving 2) how to USE the latest tools like bolt.new or lovable or opusclip etc. 3) do hands-on group projects to simulate working on a team and building a stack, that’s how I grade them. And for this I wanted to MINIMIZE what they need to install. LLaMa 4 for one GPU is the ticket! Yeah so I was hoping the JS support was more robust, and asking HN if they knew of any ports (at least to WASM). But no, it’s firmly locked into PyTorch and CUDA for now. So I’m just gonna stick with Tensorflow for educational purposes, like people used Pascal or Ruby when teaching. I want to let them actually install ONE thing (node.js) and be able to run inferenfe in their browser. I want them to be able to USE the tools and build websites and businesses end-to-end, launch a business and have agents work for them. Some of the places they engage the most is when I talk about AI and society, sustainability or regulations. That’s the cohort But you can keep geeking out on low-level primitives. I remember writing my own 3D-persoective-correct-texturemapping engine and then GPUs came out. Carmack and others kept at it for a while, others moved on. You could make a lot of money in 3D games without knowing how texturemapping and lighting worked, and same goes for this. PS: No thanks to you but I found what I was looking for myself in a few minutes. https://youtu.be/6LHNbeDADA4?si=LCM2E48hVxmO6VG4 https://youtu.be/6LHNbeDADA4?si=LCM2E48hVxmO6VG4 https://github.com/Picovoice/picollm https://github.com/Picovoice/picollm PicoLLM is a way to run LLaMa 3 on Node, it will be great for my students. I bet you didn’t know much about Node.js ecosystem for LLMs because it’s very nascent.
- ein0p 2y agoStrange choice of languages for their "multilingual" capabilities, but OK. I wonder why there's no Chinese.
- tqi 2y ago> Our testing shows that Llama 4 responds with strong political lean at a rate comparable to Grok (and at half of the rate of Llama 3.3) on a contentious set of political or social topics. While we are making progress, we know we have more work to do and will continue to drive this rate further down. My experience is that these subjective benchmarks are completely meaningless, because the researchers involved have a strong incentive (promotions, discretionary equity) to cherrypick measures that they can easily improve.
- steele 2y agoConsuming pirated literature en masse produces a bias away from authoritarianism; consider me flabbergasted.
- jwr 2y agoFor those unfamiliar with the "active parameters" terminology, what would be the RAM requirements? E.g.can I run the smallest one on my Macbook Pro (M4 Max, 64GB) like I can run gemma3?
- dragonwriter 2y agoThe RAM requirements for storing the parameters are set by the total, not active, parameters. Llama4 Scout is 109B model, so, at Int4 quantization, it will require ~55GB for the model. With 64GB, you could probably run it, but I would imagine not with a very large context size.
- bastawhiz 2y agoI don't really understand how Scout and Maverick are distillations of Behemoth if Behemoth is still training. Maybe I missed or misunderstood this in the post? Did they distill the in-progress Behemoth and the result was good enough for models of those sizes for them to consider releasing it? Or is Behemoth just going through post-training that takes longer than post-training the distilled versions? Sorry if this is a naïve question.
- paradite 2y agoMy understanding is that they have a base model checkpoint for Behemoth from pre-training. This base model is not instruction-tuned so you can't use it like a normal instruction-tuned model for chatbots. However, the base model can be distilled, and then the distilled model is post-trained to be instruction tuned, which can be released as a model for chatbots.
- voxgen 2y ago> Or is Behemoth just going through post-training that takes longer than post-training the distilled versions? This is the likely main explanation. RL fine-tuning repeatedly switches between inference to generate and score responses, and training on those responses. In inference mode they can parallelize across responses, but each response is still generated one token at a time. Likely 5+ minutes per iteration if they're aiming for 10k+ CoTs like other reasoning models. There's also likely an element of strategy involved. We've already seen OpenAI hold back releases to time them to undermine competitors' releases (see o3-mini's release date & pricing vs R1's). Meta probably wants to keep that option open.
- rfoo 2y ago> see o3-mini's release date & pricing vs R1's This backfires though, if OAI released o3-mini before DeepSeek-R1, R1 would be a lot less impactful.
- pavelstoev 2y agoModel training observations from both Llama 3 and 4 papers: Meta’s Llama 3 was trained on ~16k H100s, achieving ~380–430 TFLOPS per GPU in BF16 precision, translating to a solid 38 - 43% hardware efficiency [Meta, Llama 3]. For Llama 4 training, Meta doubled the compute, using ~32K H100s and switched to FP8 precision. Despite the precision gain, observed efficiency dropped to about 19.7%, with GPUs delivering ~390 TFLOPS out of a theoretical 1,979 FP8 TFLOPS [Meta, Llama 4]. I am not the one to critique, and rather, this is a recognition of the enormous complexity of operating GPUs at this scale. Training massive models across tens of thousands of GPUs stretches today’s AI infrastructure to its limit. Besides accelerating inference workloads, advanced GPU optimizations can be integrated into training and fine-tuning pipelines. From various kernel optimization techniques (over 90) to increasing memory access efficiency and scaling up to cluster-wide resource coordination, efficiency can be maximized with some complex software. References: [Meta, Llama 3] https://ai.meta.com/research/publications/the-llama-3-herd-of-models/ https://ai.meta.com/research/publications/the-llama-3-herd-o... [Meta, Llama 4] https://ai.meta.com/blog/llama-4-multimodal-intelligence/ https://ai.meta.com/blog/llama-4-multimodal-intelligence/
- YetAnotherNick 2y agoIt's not just scale. Even for single GPU, it is hard to acheive 2x speed improvement as the GPU specs states. Even NVIDIA's own Tensor Engine acheives 28% extra FLOP/s[1]. [1]: https://arxiv.org/pdf/2310.18313 https://arxiv.org/pdf/2310.18313
- silverlake 2y agoI think BF16 and FP16 are 1979 TFPOPs, but FP8 is 2x faster at 3958 TFLOPs. So only 10% efficiency, down from 20%. That’s not good.
- az226 2y agoThat’s with sparsity. So it’s 29% down from 40%.
- cavisne 2y agoThe H100 theoretical flops number is just marketing, as it relies on sparsity that LLMs don’t use
- wonderfuly 2y agoAvailable here: https://app.chathub.gg/chat/cloud-llama4 https://app.chathub.gg/chat/cloud-llama4
- utopcell 2y agoHow are Maverick and Scout distilled from Behemoth if the latter is not done training? Do they distill from some intermediate, "good enough" snapshot?
- harisec 2y agoYes, during training multiple checkpoints are created, you can distill from any checkpoint you want.
- guybedo 2y agoTLDR: https://extraakt.com/extraakts/llama-4-release-analysis https://extraakt.com/extraakts/llama-4-release-analysis
- paulmendoza 2y agoHow long did they run the training job for? Curious how much it costs to train all of these models?
- ianks 2y agoAre we going to find out that Meta pirated libgen again, with zero recognition to the authors? “Open-sourcing it” doesn’t magically absolve you of the irreparable damages you’ve caused society. You stole their life’s work so your company could profit off of rage-slop.
- MagicMoonlight 2y agoThe problem is, how do you value one book? £10? Or are we saying £10 every time someone uses the AI? Should Taylor swift be liable to pay commission for every piece of music she listened to while training? They will have influenced her work in some way. I’d rather go the other way and say that the companies have to freely release their data sets, if the data is derived from other people’s work. It would put everyone on a level playing field.
- ein0p 2y agoIf it's not on Ollama, nobody is going to care beyond perusing the metrics.
- Amekedl 2y agoSo the wall has been really been hit already for now, ouch. It was to be expected with gpt-“4.5”, but still, the realization now really feels grounded.
- killerstorm 2y agoIt's kinda hilarious to see people claiming that the wall has been hit for the past two years, while evals are creeping up each month, particularly realistic end-to-end SWE-bench. Have you compared GPT-4.5 to 4o? GPT-4.5 just knows things. Some obscure programming language? It knows the syntax. Obviously, that's not sufficient - you also need reasoning, post-training, etc. so quite predictably G2.5P being a large model + reasoning + tuning got SotA in code generation. (FWIW I think if it was tuned for a particular input/output format it could get another 10%) But, yeah, the wall, the wall!
- Amekedl 2y agoEver heard about benchmark contamination? Ever tried to explain a new concept, like a new state management store for web frontend? Most fail spectacularly there, sonnet 3.7 I had reasonable ""success"" with, but not 4.5. It faltered completely. Let’s not get ahead of ourselves. Looking at training efficiency in this now, and all the other factors, it really is difficult to paint a favorable picture atm.
- killerstorm 2y agoYou sound like Gary Marcus.
- Amekedl 2y agoDidn't know him, but he seems overly skeptical. Honestly, I was just expecting more from llama-4 than this, hence mentioning the wall. I hope it's still too early to tell, because new ideas are going to change stuff inevitably, maybe anthropic opens up more, or chinese labs keep overdelivering...
- Alifatisk 2y agoI remember when Google announced Geminis theoretical limit of 10M tokens context window, I was impressed. But it seems like that theoretical limit stayed as theoretical and they just pushed up to 2M. Which is still impressive. Today, it seems Meta has crushed that wall with truly 10M tokens, wow. I was also curious to how well Llama would be able to utilize the whole context window, it kinda pointless to have a large window if you can't recall most, if not all of it. The needle in the haystack test showed this is not the case, I wonder how they achieved this.
- cpeterson42 2y agoFor anyone looking to experiment with these models who doesn't have 210GB of VRAM on tap-we're working as quickly as we can to get cheap access to 4x80GB A100 instances running at thundercompute.com (aiming for sub-$5/hr). For quantized versions, we have cheaper 1-2 GPU nodes available today. If you're interested, join our Discord for updates: https://discord.com/invite/nwuETS9jJK https://discord.com/invite/nwuETS9jJK
- Nebulous7 2y ago[flagged]
- jacooper 2y agoBTW these models arent allowed to be used in the EU.
- supernovae 2y agoIt's too bad these models are built on the expectation of pirating the world
- anotherpaulg 2y agoLlama 4 Maverick scored 16% on the aider polyglot coding benchmark [0]. 73% Gemini 2.5 Pro (SOTA) 60% Sonnet 3.7 (no thinking) 55% DeepSeek V3 0324 22% Qwen Max 16% Qwen2.5-Coder-32B-Instruct 16% Llama 4 Maverick [0] https://aider.chat/docs/leaderboards/?highlight=Maverick https://aider.chat/docs/leaderboards/?highlight=Maverick
- phoenk 2y agoDid they not target code tasks for this LLM, or is it genuinely that bad? Pretty embarrassing when your shiny new 400B model barely ties a 32B model designed to be run locally. Or maybe is this a strong indication that smaller, specialized LLMs have much more potential for specific tasks than larger, general purpose LLMs.
- senko 2y agoSide note: `highlight` query param doesn't seem to have any effect on that table (at least for me on Firefox)
- Havoc 2y agoInteresting that the reception here is much more positive here than on /r/localllama
- dcl 2y agoBut how good is it at Pokemon?
- stuaxo 2y agoWhat does it mean that it "no longer leans left" for answers. What did they do to the model, and how exactly does it answer differently? Will including this in an app make the app MAGA aligned all of a sudden? What happens if it says something that breaks the laws of some country it's in ?
- ofermend 2y agoA great day for open source, and so glad to see llama4 out. However, I'm a bit disappointed that the hallucination rates of Llama4 are not as low as I would have liked (TL;DR slightly higher than Llama3). Check the numbers on the hallucination leaderboard: https://github.com/vectara/hallucination-leaderboard https://github.com/vectara/hallucination-leaderboard
- croemer 2y agoRelevant update: the model on LM Arena is not the one that was released. See "Meta got caught gaming AI benchmark" https://news.ycombinator.com/item?id=43617660 https://news.ycombinator.com/item?id=43617660