4 ms·

The Llama 4 launch looks like a real debacle for Meta. The model doesn't look great. All the coverage I've seen has been negative. This is about what I expecte

by Mond_ 2y ago

The Llama 4 launch looks like a real debacle for Meta. The model doesn't look great. All the coverage I've seen has been negative. This is about what I expected, but it makes you wonder what they're going to do next. At this point it looks like they are falling behind the other open models, and made an ambitious bet on MoEs, without this paying off. Did Zuck push for the release? I'm sure they knew it wasn't ready yet.

bko 2y ago
I remember reading that they were in panic mode when the DeepSeek model came out so they must have scrambled and had to re-work a lot of things since DeepSeek was so competitive and open source as well
- blueboo 2y ago
  Fear of R2 looms large as well. I suspect they succumbed to the nuance collapse along the lines of “Is double checking results worth it if DeepSeek eats our lunch?”
nightski 2y ago
Do you know that they made a bet on MoE? Meaning they abandonded dense models? I doubt that is the case. Just releasing MoE Llama 4 does not constitute a "bet" without further information. Also from what I can tell this performs better than models with parameter counts equal to one expert, and worse than fully dense models equal to total parameter count. Isn't that kind of what we'd expect? in what way is that a failure? Maybe I am missing some details. But it feels like you have an axe to grind.
- genewitch 2y ago
  A 4x8 MOE performs better than an 8B but worse than a 32B, is your statement? My response would be, "so why bother with MOE?" However deepseek r1 is MOE from my understanding, but the "E" are all =>32B parameters. There's > 20 experts. I could be misinformed; however, even so, I'd say a MOE with 32B or even 70B experts will outperform (define this!) Models with equal parameter counts, because deepseek outperforms (define?) ChatGPT et al.
  - nightski 2y ago
    Easy, vastly improved inference performance on machines with larger RAM but lower bandwidth/compute. These are becoming more popular such as Apple's M series chips, AMD's strix halo series, and the upcoming DGX Spark from Nvidia.
    - genewitch 2y ago
      yes i understand all that. I was saying the claim is incorrect. My understanding of deepseek is mechanically correct but apparently they use 3B models as experts, per your sibling comment. I don't buy it, regardless of what they put in the paper - 3B models are pretty dumb, and R1 isn't dumb. No amount of shuffling between "dumb" experts will make the output not dumb. it's more likely 32x32B experts, based on the quant sizes i've seen. A deepseek employee is welcome to correct me.
      - zamadatix 2y ago
        I'm not a DeepSeek employee but I think there is more clarification needed on what an "expert" is before the conversation can make any sense. Much like physics, one needs at least take a glance at how the math is going to be used to be able to sanity check a claim. A model includes many components. There will be bits that encode/decode tokens as vectors, transformer blocks which do the actual processing on the data, some post-transformer block filtering to normalize that output, and maybe some other stuff depending on the model architecture. The part we're interested in involves parts of the transformer blocks, which handle using encoded relational information about the vectors involved (part 1) to transform the input vector using a feed forward network of weights (part 2) and then all that gets post processed in various ways (part 3). A model will chain these transformers together and each part of the chain is called a layer. A vector will run from the first layer through to the last, being modified by each transformer along the way. In an MoE model the main part about the transformer block changed is part 2, which goes from "using a feed forward network of weights" to "using a subset of feed forward network weights chosen by a router for the given token and then recombined". In the MoE case each subset of weights per feed forward layer is what is called the "expert". Importantly, the expert is not a whole model - it's just a group of the weights available in a given layer. Each layer's router choses which group(s) of weights to use independently. As an example, if a 10 layer model had a total of 10 billion 8 bit parameters in the feed forward layers (so a >10 billion parameter model in overall parameters) and 10 experts that means each expert is ~100 MB (10 billion bytes / 10 layers / 10 experts per layer). These 10 billion parameters would be referred to as the sparse parameters (not always used each token) while the rest of the model would be referred to as the dense parameters (always used each token). Note: folks on the internet have a strong tendency to label this incorrectly as "10x1B" or "10x{ActiveParameters}" instead. The "Mixture" part of MoE extends a bit further than "the parameter groups sit next to each other in the transformer block" though. Similar in concept to how transformer attention is combined in part 1, more than 1 expert can be activated and the outputs combined. At minimum there is usually 1 expert which is always used in a layer (the "shared expert") and at least 1 expert which is selected by the router (the "routed experts"). The shared expert exists to make the utilization of the routed experts more even by ensuring base information which needs to be used all the time is dedicated to it which increases training performance since the other experts can be more evenly selected by the router as a result. With that understanding, the important takeaways are: - Experts are parts of individual parameter networks in each layer, not a sub-model carved out. - More than 1 expert can be used, and the way the data is combined is not like feeding the output of a low parameter LLM to another, it's more like how the first phase of the transformer has multiple attention heads which are combined to give the full attention information. - There are a lot of other weights beyond the sparse weights used by experts in a model. The same is true for the equivalent portions of a dense model as well though. This ultimately makes comparing active parameters of a sparse model to total parameters of a dense model a valid comparison. As to the original conversation: DeepSeek v3/R1 has 37 Billion active parameters so that should set the floor for a comparative dense model, not whatever the size of an individual expert in part of a single transformer layer is (which acts as a bit of a red herring in these kinds of conversations, doubly so since more than 1 experts worth of weights are used anyways). In reality a bit more than that, though definitely less than if all 671 Billion parameters were dense. While we don't have much concrete public information about modern version of ChatGPT, one thing we're relatively certain of is "ChatGPT 4.5 has a TON more parameters and has basically nothing to show for it". Meanwhile Gemma 3, a 27 B locally runnable model from Google, is a hair behind DeepSeek v3 in many popular benchmarks. Truth is, there are just a lot of other things beyond parameter count that go into making a well performing model and DeepSeek v3/r1 hit (or invented) a lot of them. If I had to place a bet on this whole comparison, I'd say OpenAI is very likely also using sparse/MoE-style architectures in their current products anyways. As a final note, don't just dismiss the architecture of DeepSeek with "regardless of what they put in the paper"! The model files are available and they include this sort of layer/parameter information so your device is able to run them (well, "run" requires a bit of RAM in this case... but you can still read the model file on disk and see it's laid out as the paper claims regardless).
throwaway25251 2y ago
I don't know about Llama 4. Competition is intense in this field so you can't expect everybody to be number 1. However, I think the performance culture at Meta is counterproductive. Incentives are misaligned, I hope leadership will try to improve it. Employees are encouraged to ship half-baked features and move to another project. Quality isn't rewarded at all. The recent layoffs have made things even worse. Skilled people were fired, slowing down teams. I assume the goal was to push remaining employees to work even more, but I doubt this is working. I haven't worked in enough companies of this size to be able to tell if alternatives are better, but it's very clear to me that Meta doesn't get the best from their employees.
- berkes 2y ago
  I've never liked it, but > Move fast and break things is really a bad concept in this space, where you get limited shots at releasing something that generates interest. > Employees are encouraged to ship half-baked features And this is why I never liked that motto and have always pushed back at startups where I was hired that embraced this line of thought. Quality matters. It's context-dependent, so sometimes it matters a lot, and sometimes hardly. But "moving fast and breaking things" should be a deliberate choice, made for every feature, module, sprint, story all over again, IMO. If at all.
  - Rebuff5007 2y ago
    I'd argue its a bad concept in any spaces that involve teams of people working together and deliverables that enter the real world.
  - diggan 2y ago
    > is really a bad concept in this space, where you get limited shots at releasing something that generates interest. Sure, but long term effects are more depending on the actual performance of the model, than anything. Say they launch a model that is hyped to be the best, but when people try it, it's worse than other models. People will quickly forget about it, unless it's particularly good at something. Alternatively, say they launch a model that doesn't even get a press release, or any benchmark results published ahead of launch, but the model actually rocks at a bunch of use cases. People will start using it regardless of the initial release, and continue to do so as long as it's a best model.
qoez 2y ago
> it makes you wonder what they're going to do next They're just gonna keep throwing money at it. This is a hobby and talent magnet for them, instagram is the money printer. They've been working on VR for like a decade with barely much results in terms of users (compared to costs). This will be no different.
- wongarsu 2y ago
  Both are also decent long-terms bets. Being VR market leader now means they will be VR market-leader with plenty of inhouse talent and IP when the technology matures and the market grows. Being in the AI race, even if they are not leading, means they have in-house talent and technology to be able to react to wherever the market is going with AI. They have one of the biggest messengers and one of the biggest image-posting sites, there is a decent chance AI will become important to them in some not-yet-obvious way. One of Meta's biggest strengths is Zuckerberg being able to play these kinds of bets. Those bets being great for PR and talent acquisition is the cherry on top
  - TheOtherHobbes 2y ago
    This assumes no upstart will create a game changing innovation which upends everything. Companies become complacent and confused when they get too big. Employees become trapped in a maze of performative careerism, and customer focus and a realistic understanding of threats from potential competitors both disappear. It's been a consistent pattern since the earliest days of computing. Nothing coming out of Big Tech at the moment is encouraging me to revise that assumption.
ntlm1686 2y ago
"made an ambitious bet on MoEs"? No, DeepSeek is MoE, and they succeeded. Meta is not betting on MoE, it just does what other people have done.
- antirez 2y ago
  Llama4 seems in many ways a cut and paste of DeepSeek. Including the shared expert and the high sparsity. It's a DeepSeek that does not work well.
thefourthchime 2y ago
I mean, there's a reason they released it on a Saturday.
root_axis 2y ago
It's not a big deal. Llama 4 feels like a flop because the expectations are really high based on their previous releases and the sense of momentum in the ecosystem because of DeepSeek. At the end of the day, LLama 4 didn't meet the elevated expectations, but they're fine. They'll continue to improve and iterate and maybe the next one will be more hype worthy, or maybe expectations will be readjusted as the specter of diminishing returns continues to creep in.
- lerchmo 2y ago
  The switching costs are so low (zero) that anyone using these models just jumps to the best performer. I also agree that this is not a brand or narrative sensitive project.
- int_19h 2y ago
  It feels like a flop because it is objectively worse than models many times smaller that shipped sometimes ago. In fact, it is worse than earlier LLaMA releases on may points. It's so bad that people who initially ran into it assumed that the downloaded weights must be corrupted somehow.
dkkergoog 2y ago
[dead]
agilob 2y ago
They knew they can't beat DeepSeek 2 months ago https://old.reddit.com/r/LocalLLaMA/comments/1i88g4y/meta_panicked_by_deepseek/ https://old.reddit.com/r/LocalLLaMA/comments/1i88g4y/meta_pa...
overfeed 2y ago
> Did Zuck push for the release? All I know is it is the first Llama release since Zuck brought "masculine energy" back to Meta.

by Mond_ 2y ago

bko 2y ago

I remember reading that they were in panic mode when the DeepSeek model came out so they must have scrambled and had to re-work a lot of things since DeepSeek was so competitive and open source as well

blueboo 2y ago
Fear of R2 looms large as well. I suspect they succumbed to the nuance collapse along the lines of “Is double checking results worth it if DeepSeek eats our lunch?”

nightski 2y ago

Do you know that they made a bet on MoE? Meaning they abandonded dense models? I doubt that is the case. Just releasing MoE Llama 4 does not constitute a "bet" without further information. Also from what I can tell this performs better than models with parameter counts equal to one expert, and worse than fully dense models equal to total parameter count. Isn't that kind of what we'd expect? in what way is that a failure? Maybe I am missing some details. But it feels like you have an axe to grind.

genewitch 2y ago
A 4x8 MOE performs better than an 8B but worse than a 32B, is your statement? My response would be, "so why bother with MOE?" However deepseek r1 is MOE from my understanding, but the "E" are all =>32B parameters. There's > 20 experts. I could be misinformed; however, even so, I'd say a MOE with 32B or even 70B experts will outperform (define this!) Models with equal parameter counts, because deepseek outperforms (define?) ChatGPT et al.
- nightski 2y ago
  Easy, vastly improved inference performance on machines with larger RAM but lower bandwidth/compute. These are becoming more popular such as Apple's M series chips, AMD's strix halo series, and the upcoming DGX Spark from Nvidia.
  - genewitch 2y ago
    yes i understand all that. I was saying the claim is incorrect. My understanding of deepseek is mechanically correct but apparently they use 3B models as experts, per your sibling comment. I don't buy it, regardless of what they put in the paper - 3B models are pretty dumb, and R1 isn't dumb. No amount of shuffling between "dumb" experts will make the output not dumb. it's more likely 32x32B experts, based on the quant sizes i've seen. A deepseek employee is welcome to correct me.
    - zamadatix 2y ago
      I'm not a DeepSeek employee but I think there is more clarification needed on what an "expert" is before the conversation can make any sense. Much like physics, one needs at least take a glance at how the math is going to be used to be able to sanity check a claim. A model includes many components. There will be bits that encode/decode tokens as vectors, transformer blocks which do the actual processing on the data, some post-transformer block filtering to normalize that output, and maybe some other stuff depending on the model architecture. The part we're interested in involves parts of the transformer blocks, which handle using encoded relational information about the vectors involved (part 1) to transform the input vector using a feed forward network of weights (part 2) and then all that gets post processed in various ways (part 3). A model will chain these transformers together and each part of the chain is called a layer. A vector will run from the first layer through to the last, being modified by each transformer along the way. In an MoE model the main part about the transformer block changed is part 2, which goes from "using a feed forward network of weights" to "using a subset of feed forward network weights chosen by a router for the given token and then recombined". In the MoE case each subset of weights per feed forward layer is what is called the "expert". Importantly, the expert is not a whole model - it's just a group of the weights available in a given layer. Each layer's router choses which group(s) of weights to use independently. As an example, if a 10 layer model had a total of 10 billion 8 bit parameters in the feed forward layers (so a >10 billion parameter model in overall parameters) and 10 experts that means each expert is ~100 MB (10 billion bytes / 10 layers / 10 experts per layer). These 10 billion parameters would be referred to as the sparse parameters (not always used each token) while the rest of the model would be referred to as the dense parameters (always used each token). Note: folks on the internet have a strong tendency to label this incorrectly as "10x1B" or "10x{ActiveParameters}" instead. The "Mixture" part of MoE extends a bit further than "the parameter groups sit next to each other in the transformer block" though. Similar in concept to how transformer attention is combined in part 1, more than 1 expert can be activated and the outputs combined. At minimum there is usually 1 expert which is always used in a layer (the "shared expert") and at least 1 expert which is selected by the router (the "routed experts"). The shared expert exists to make the utilization of the routed experts more even by ensuring base information which needs to be used all the time is dedicated to it which increases training performance since the other experts can be more evenly selected by the router as a result. With that understanding, the important takeaways are: - Experts are parts of individual parameter networks in each layer, not a sub-model carved out. - More than 1 expert can be used, and the way the data is combined is not like feeding the output of a low parameter LLM to another, it's more like how the first phase of the transformer has multiple attention heads which are combined to give the full attention information. - There are a lot of other weights beyond the sparse weights used by experts in a model. The same is true for the equivalent portions of a dense model as well though. This ultimately makes comparing active parameters of a sparse model to total parameters of a dense model a valid comparison. As to the original conversation: DeepSeek v3/R1 has 37 Billion active parameters so that should set the floor for a comparative dense model, not whatever the size of an individual expert in part of a single transformer layer is (which acts as a bit of a red herring in these kinds of conversations, doubly so since more than 1 experts worth of weights are used anyways). In reality a bit more than that, though definitely less than if all 671 Billion parameters were dense. While we don't have much concrete public information about modern version of ChatGPT, one thing we're relatively certain of is "ChatGPT 4.5 has a TON more parameters and has basically nothing to show for it". Meanwhile Gemma 3, a 27 B locally runnable model from Google, is a hair behind DeepSeek v3 in many popular benchmarks. Truth is, there are just a lot of other things beyond parameter count that go into making a well performing model and DeepSeek v3/r1 hit (or invented) a lot of them. If I had to place a bet on this whole comparison, I'd say OpenAI is very likely also using sparse/MoE-style architectures in their current products anyways. As a final note, don't just dismiss the architecture of DeepSeek with "regardless of what they put in the paper"! The model files are available and they include this sort of layer/parameter information so your device is able to run them (well, "run" requires a bit of RAM in this case... but you can still read the model file on disk and see it's laid out as the paper claims regardless).

throwaway25251 2y ago

I don't know about Llama 4. Competition is intense in this field so you can't expect everybody to be number 1. However, I think the performance culture at Meta is counterproductive. Incentives are misaligned, I hope leadership will try to improve it. Employees are encouraged to ship half-baked features and move to another project. Quality isn't rewarded at all. The recent layoffs have made things even worse. Skilled people were fired, slowing down teams. I assume the goal was to push remaining employees to work even more, but I doubt this is working. I haven't worked in enough companies of this size to be able to tell if alternatives are better, but it's very clear to me that Meta doesn't get the best from their employees.

berkes 2y ago
I've never liked it, but > Move fast and break things is really a bad concept in this space, where you get limited shots at releasing something that generates interest. > Employees are encouraged to ship half-baked features And this is why I never liked that motto and have always pushed back at startups where I was hired that embraced this line of thought. Quality matters. It's context-dependent, so sometimes it matters a lot, and sometimes hardly. But "moving fast and breaking things" should be a deliberate choice, made for every feature, module, sprint, story all over again, IMO. If at all.
- Rebuff5007 2y ago
  I'd argue its a bad concept in any spaces that involve teams of people working together and deliverables that enter the real world.
- diggan 2y ago
  > is really a bad concept in this space, where you get limited shots at releasing something that generates interest. Sure, but long term effects are more depending on the actual performance of the model, than anything. Say they launch a model that is hyped to be the best, but when people try it, it's worse than other models. People will quickly forget about it, unless it's particularly good at something. Alternatively, say they launch a model that doesn't even get a press release, or any benchmark results published ahead of launch, but the model actually rocks at a bunch of use cases. People will start using it regardless of the initial release, and continue to do so as long as it's a best model.

qoez 2y ago

> it makes you wonder what they're going to do next They're just gonna keep throwing money at it. This is a hobby and talent magnet for them, instagram is the money printer. They've been working on VR for like a decade with barely much results in terms of users (compared to costs). This will be no different.

wongarsu 2y ago
Both are also decent long-terms bets. Being VR market leader now means they will be VR market-leader with plenty of inhouse talent and IP when the technology matures and the market grows. Being in the AI race, even if they are not leading, means they have in-house talent and technology to be able to react to wherever the market is going with AI. They have one of the biggest messengers and one of the biggest image-posting sites, there is a decent chance AI will become important to them in some not-yet-obvious way. One of Meta's biggest strengths is Zuckerberg being able to play these kinds of bets. Those bets being great for PR and talent acquisition is the cherry on top
- TheOtherHobbes 2y ago
  This assumes no upstart will create a game changing innovation which upends everything. Companies become complacent and confused when they get too big. Employees become trapped in a maze of performative careerism, and customer focus and a realistic understanding of threats from potential competitors both disappear. It's been a consistent pattern since the earliest days of computing. Nothing coming out of Big Tech at the moment is encouraging me to revise that assumption.

ntlm1686 2y ago

"made an ambitious bet on MoEs"? No, DeepSeek is MoE, and they succeeded. Meta is not betting on MoE, it just does what other people have done.

antirez 2y ago
Llama4 seems in many ways a cut and paste of DeepSeek. Including the shared expert and the high sparsity. It's a DeepSeek that does not work well.

thefourthchime 2y ago

I mean, there's a reason they released it on a Saturday.

root_axis 2y ago

It's not a big deal. Llama 4 feels like a flop because the expectations are really high based on their previous releases and the sense of momentum in the ecosystem because of DeepSeek. At the end of the day, LLama 4 didn't meet the elevated expectations, but they're fine. They'll continue to improve and iterate and maybe the next one will be more hype worthy, or maybe expectations will be readjusted as the specter of diminishing returns continues to creep in.

lerchmo 2y ago
The switching costs are so low (zero) that anyone using these models just jumps to the best performer. I also agree that this is not a brand or narrative sensitive project.
int_19h 2y ago
It feels like a flop because it is objectively worse than models many times smaller that shipped sometimes ago. In fact, it is worse than earlier LLaMA releases on may points. It's so bad that people who initially ran into it assumed that the downloaded weights must be corrupted somehow.

dkkergoog 2y ago

[dead]

agilob 2y ago

They knew they can't beat DeepSeek 2 months ago https://old.reddit.com/r/LocalLLaMA/comments/1i88g4y/meta_panicked_by_deepseek/ https://old.reddit.com/r/LocalLLaMA/comments/1i88g4y/meta_pa...

overfeed 2y ago

> Did Zuck push for the release? All I know is it is the first Llama release since Zuck brought "masculine energy" back to Meta.