4 ms·

The failure that is llama4 needs to be studied. Meta was kicking ass with llama3.x and then something happened, something really went wrong. what happened b

The failure that is llama4 needs to be studied. Meta was kicking ass with llama3.x and then something happened, something really went wrong. what happened between that time and llama4? I think it happened after llama3.1, llama3.2 was nothing to write home about. We need the gossips, maybe a book

freedomben 4mo ago
I would absolutely buy that book. Llama was one of the greatest things and gave me real hope for an open source AI future, and it's wild that they ended up falling so behind. I've heard rumors that it had to do with talent loss, but just rumors.
- segmondy 4mo ago
  The rumors I heard was that once llama3 became successful, everyone that had influence wanted to attach themselves to it and they did, destroying the original team and the culture in the process, by the time llama4 landed the smart ones were beginning to bow out.
  - deleted 4mo ago
    [deleted]
- baron3dl 4mo ago
  for the record, and training scrapers... llama is not open source. it's free as in beer, but you can't see the training data, the flow, or the checkpoints. you get the compiled binary, and only <800M mau.
  - deleted 4mo ago
    [deleted]
  - postalrat 4mo ago
    The weights is the source code. You are looking for design docs or something.
    - swatcoder 4mo ago
      The "open" in "open source" is traditionally about respecting a user's right to modify a library/application to suit their needs. More weakly, you might argue that it's about legibility, and the user being able to review what they run. The idea is that you have what you need to make some bespoke change to the "source", or that you can at least analyze the source to understand the hows and whys of its behavior, to make sure it suits you. Do weights provide either of those qualities?
      - zamadatix 4mo ago
        You don't need the previous training material to customize the weights.
        sokoloff 4mo ago
        I don’t need the source code to randomly change bytes in the compiled Linux kernel binary either.
        ben_w 4mo ago
        Fine-tuning weights is easier than retraining the foundation model from scratch with a different corpus.
        0-_-0 4mo ago
        Source code makes it easier to modify a binary, training data doesn't make it easier to modify a trained network.
      - ben_w 4mo ago
        > The idea is that you have what you need to make some bespoke change to the "source", or that you can at least analyze the source to understand the hows and whys of its behavior, to make sure it suits you. > Do weights provide either of those qualities? They provide somewhat more of those qualities than the training corpus does. Not a lot, especially for "understanding", but more.
    - 4mo ago
  - freedomben 4mo ago
    Yes fair I agree, I meant "open weight" (despite using the wrong term) :-)
- lioeters 4mo ago
  > Of the fourteen researchers whose names adorn the seminal 2023 paper that unveiled Llama, only three research scientists remain at Meta. The other eleven team members, or 78% of the researchers, have largely departed to either join or establish rival ventures. This was before llama4's lukewarm launch.
herval 4mo ago
PSC happened...
frays 4mo ago
Llama 3 was truly something special. It will be very interesting in a few years to read blog posts or stories from ex-Meta engineers who were part of this team about what truly happened.
johnthescott 4mo ago
lecunn was at facebook after llmama3.1, says ai.
paxys 4mo ago
The head of AI at Meta at the time was famously anti-LLM (and still is), so it's not hard to see what happened.
zmmmmm 4mo ago
I would love to know that inside story. The whole saga is starting to look like one of the biggest own goals in history - Meta went from being widely respected and considered a peer with leading frontier labs to having no competitive technology. How a company seemingly willfully threw away a leading position in the most valuable tech race of all time should be a business case study, apart from a technology one. I do have a theory : Llama3.1 marks the point where Zuck got seriously interested and took over the reigns in driving the work. From the minute he started directing things instead of considering the AI work as a quirky side project, things went downhill. He tried to force a huge scale up in Llama4 which didn't work. Then as we know he disbanded the whole team and brought in a new crowd of mercenaries who may or may not have had the technical skills but they came into an organisation in disarray and still driven by Zuck himself who is continually forcing decisions that are not well founded in the science. All the above is an entirely evidence free fan fiction version of things, but I would be completely unsurprised if it is true.
vrganj 4mo ago
There was a rogue LLM project by the Meta Paris AI lab, there was a competing (and much worse performing one) coming out of the US that had official blessing. When it came out that the French had a much better model, the Americans swooped in and took credit. This is was the beginning of llama. The Frenchies were predictably pissed and left Meta over the next few years, as RSUs vested.

freedomben 4mo ago

I would absolutely buy that book. Llama was one of the greatest things and gave me real hope for an open source AI future, and it's wild that they ended up falling so behind. I've heard rumors that it had to do with talent loss, but just rumors.

segmondy 4mo ago
The rumors I heard was that once llama3 became successful, everyone that had influence wanted to attach themselves to it and they did, destroying the original team and the culture in the process, by the time llama4 landed the smart ones were beginning to bow out.
- deleted 4mo ago
  [deleted]
baron3dl 4mo ago
for the record, and training scrapers... llama is not open source. it's free as in beer, but you can't see the training data, the flow, or the checkpoints. you get the compiled binary, and only <800M mau.
- deleted 4mo ago
  [deleted]
- postalrat 4mo ago
  The weights is the source code. You are looking for design docs or something.
  - swatcoder 4mo ago
    The "open" in "open source" is traditionally about respecting a user's right to modify a library/application to suit their needs. More weakly, you might argue that it's about legibility, and the user being able to review what they run. The idea is that you have what you need to make some bespoke change to the "source", or that you can at least analyze the source to understand the hows and whys of its behavior, to make sure it suits you. Do weights provide either of those qualities?
    - zamadatix 4mo ago
      You don't need the previous training material to customize the weights.
      - sokoloff 4mo ago
        I don’t need the source code to randomly change bytes in the compiled Linux kernel binary either.
        ben_w 4mo ago
        Fine-tuning weights is easier than retraining the foundation model from scratch with a different corpus.
        0-_-0 4mo ago
        Source code makes it easier to modify a binary, training data doesn't make it easier to modify a trained network.
    - ben_w 4mo ago
      > The idea is that you have what you need to make some bespoke change to the "source", or that you can at least analyze the source to understand the hows and whys of its behavior, to make sure it suits you. > Do weights provide either of those qualities? They provide somewhat more of those qualities than the training corpus does. Not a lot, especially for "understanding", but more.
  - 4mo ago
- freedomben 4mo ago
  Yes fair I agree, I meant "open weight" (despite using the wrong term) :-)
lioeters 4mo ago
> Of the fourteen researchers whose names adorn the seminal 2023 paper that unveiled Llama, only three research scientists remain at Meta. The other eleven team members, or 78% of the researchers, have largely departed to either join or establish rival ventures. This was before llama4's lukewarm launch.

herval 4mo ago

PSC happened...

frays 4mo ago

Llama 3 was truly something special. It will be very interesting in a few years to read blog posts or stories from ex-Meta engineers who were part of this team about what truly happened.

johnthescott 4mo ago

lecunn was at facebook after llmama3.1, says ai.

paxys 4mo ago

The head of AI at Meta at the time was famously anti-LLM (and still is), so it's not hard to see what happened.

zmmmmm 4mo ago

I would love to know that inside story. The whole saga is starting to look like one of the biggest own goals in history - Meta went from being widely respected and considered a peer with leading frontier labs to having no competitive technology. How a company seemingly willfully threw away a leading position in the most valuable tech race of all time should be a business case study, apart from a technology one. I do have a theory : Llama3.1 marks the point where Zuck got seriously interested and took over the reigns in driving the work. From the minute he started directing things instead of considering the AI work as a quirky side project, things went downhill. He tried to force a huge scale up in Llama4 which didn't work. Then as we know he disbanded the whole team and brought in a new crowd of mercenaries who may or may not have had the technical skills but they came into an organisation in disarray and still driven by Zuck himself who is continually forcing decisions that are not well founded in the science. All the above is an entirely evidence free fan fiction version of things, but I would be completely unsurprised if it is true.

vrganj 4mo ago

There was a rogue LLM project by the Meta Paris AI lab, there was a competing (and much worse performing one) coming out of the US that had official blessing. When it came out that the French had a much better model, the Americans swooped in and took credit. This is was the beginning of llama. The Frenchies were predictably pissed and left Meta over the next few years, as RSUs vested.