3 ms·

I wrote about this when llama-3 came out, and this launch confirms it: Meta's goal from the start was to target OpenAI and the other proprietary model players

I wrote about this when llama-3 came out, and this launch confirms it: Meta's goal from the start was to target OpenAI and the other proprietary model players with a "scorched earth" approach by releasing powerful open models to disrupt the competitive landscape. Meta can likely outspend any other AI lab on compute and talent: - OpenAI makes an estimated revenue of $2B and is likely unprofitable. Meta generated a revenue of $134B and profits of $39B in 2023. - Meta's compute resources likely outrank OpenAI by now. - Open source likely attracts better talent and researchers. - One possible outcome could be the acquisition of OpenAI by Microsoft to catch up with Meta. The big winners of this: devs and AI product startups

foolswisdom 2y ago
It could definitely be seen as part of that strategy, but do you mind elaborating why you think "this launch confirms it"?
adam_arthur 2y ago
It's pretty clear the base model is a race to the bottom on pricing. There is no defensible moat unless a player truly develops some secret sauce on training. As of now seems that the most meaningful techniques are already widely known and understood. The money will be made on compute and on applications of the base model (that are sufficiently novel/differentiated). Investors will lose big on OpenAI and competitors (outside of greater fool approach)
- lolinder 2y ago
  > There is no defensible moat unless a player truly develops some secret sauce on training. This is why Altman has gone all out pushing for regulation and playing up safety concerns while simultaneously pushing out the people in his company that actually deeply worry about safety. Altman doesn't care about safety, he just wants governments to build him a moat that doesn't naturally exist.
changoplatanero 2y ago
> Open source likely attracts better talent and researchers I work at OpenAI and used to work at meta. Almost every person from meta that I know has asked me for a referral to OpenAI. I don’t know anyone who left OpenAI to go to meta.
- beeboobaa3 2y ago
  So they just pay better?
- tintor 2y ago
  What % of them were from FAIR vs non-FAIR?
  - kkielhofner 2y ago
    Sample size of one but I know someone who went from FAIR to OpenAI.
- lossolo 2y ago
  When was that?
- deleted 2y ago
  [deleted]
moffkalast 2y ago
https://gwern.net/complement https://gwern.net/complement Classic strategy.
jeffchao 2y ago
This is very impressive, though an adjacent question — does anyone know roughly how much time and compute cost it takes to train something like the 405B? I would imagine with all the compute Meta has that the moat is incredibly large in terms of being able to train multiple 405B-level morels and compete.
- sva_ 2y ago
  30.84M H100 compute-hours, according to the model card https://github.com/meta-llama/llama-models/blob/main/models/llama3_1/MODEL_CARD.md https://github.com/meta-llama/llama-models/blob/main/models/...
  - Escapado 2y ago
    Interestingly that’s less energy than the mass energy equivalent of one gram of matter or roughly 5 seconds worth of the worlds average energy consumption (according to wolfram alpha). Still an absolute insane amount of energy, as in about 5 million dollars at household electricity rates. Absolutely wild how much compute goes into this.
deleted 2y ago
[deleted]

foolswisdom 2y ago

It could definitely be seen as part of that strategy, but do you mind elaborating why you think "this launch confirms it"?

adam_arthur 2y ago

It's pretty clear the base model is a race to the bottom on pricing. There is no defensible moat unless a player truly develops some secret sauce on training. As of now seems that the most meaningful techniques are already widely known and understood. The money will be made on compute and on applications of the base model (that are sufficiently novel/differentiated). Investors will lose big on OpenAI and competitors (outside of greater fool approach)

lolinder 2y ago
> There is no defensible moat unless a player truly develops some secret sauce on training. This is why Altman has gone all out pushing for regulation and playing up safety concerns while simultaneously pushing out the people in his company that actually deeply worry about safety. Altman doesn't care about safety, he just wants governments to build him a moat that doesn't naturally exist.

changoplatanero 2y ago

> Open source likely attracts better talent and researchers I work at OpenAI and used to work at meta. Almost every person from meta that I know has asked me for a referral to OpenAI. I don’t know anyone who left OpenAI to go to meta.

beeboobaa3 2y ago
So they just pay better?
tintor 2y ago
What % of them were from FAIR vs non-FAIR?
- kkielhofner 2y ago
  Sample size of one but I know someone who went from FAIR to OpenAI.
lossolo 2y ago
When was that?
deleted 2y ago
[deleted]

moffkalast 2y ago

https://gwern.net/complement https://gwern.net/complement Classic strategy.

jeffchao 2y ago

This is very impressive, though an adjacent question — does anyone know roughly how much time and compute cost it takes to train something like the 405B? I would imagine with all the compute Meta has that the moat is incredibly large in terms of being able to train multiple 405B-level morels and compete.

sva_ 2y ago
30.84M H100 compute-hours, according to the model card https://github.com/meta-llama/llama-models/blob/main/models/llama3_1/MODEL_CARD.md https://github.com/meta-llama/llama-models/blob/main/models/...
- Escapado 2y ago
  Interestingly that’s less energy than the mass energy equivalent of one gram of matter or roughly 5 seconds worth of the worlds average energy consumption (according to wolfram alpha). Still an absolute insane amount of energy, as in about 5 million dollars at household electricity rates. Absolutely wild how much compute goes into this.

deleted 2y ago

[deleted]