5 ms·

Yes, I'd also like to know why Meta has chosen this path, whereas many of the other big players haven't. Usually they all settle upon the same viewpoint.

hlfshell 2y ago
It started by accident, with the original llama weights being leaked by two separate employees. They've since embraced opening the weights, which I'm all for. As for why? I have a theory: Meta is not in a position to capitalize upon the model itself. Yes, they can use it internally, and maybe their competitors can copy it to - but there are no real competitors to Facebook or Instagram that can benefit from it enough to make it a differentiating facet. Thus, releasing stuff for open source does two things: 1) Make them more attractive to research talent (Apple famously recently started to publish research because their traditional secrecy was causing issues with hiring top talent) and... 2) Continues to undermine the ability to make $$$ off of model alone, driving it towards being a commodity rather than the long term profit engine for other companies.
- sdenton4 2y ago
  In short: "Commoditize the complement."
  - deleted 2y ago
    [deleted]
- onurcel 2y ago
  > It started by accident, with the original llama weights being leaked by two separate employees. This is not true. Meta Fair has been built on openness from day 1. We published many papers and open-source d many repositories to reproduce the work
- ipsum2 2y ago
  Wrong. FAIR has been open sourcing ML models and source code for the last 10+ years. It did not start with llama. Also, llama was not leaked by employees, but people in the broader community, who the weights were shared with. For example: Faster R-CNN - state of the art image segmentation, released in 2017. FastText - text embedding models, 2016. FAISS - vector DB, 2018. https://github.com/orgs/facebookresearch/repositories https://github.com/orgs/facebookresearch/repositories has over 1,000 repos.
  - hlfshell 2y ago
    I was speaking about LLM weights specifically and llama, not all models and work at FAIR. Per your links, it's clear that FAIR does have a good history of open source work.
  - ninjin 2y ago
    This is very true. But we should also mention that Facebook sadly have been and are on a negative trajectory of openness. As someone working closely with them, there was a culture of nearly complete openness in the early years of their existence. Research was promptly shared in its entirety and licensing was compliant with open science. However, as the "AI boom" has grown, there is an increasing internal culture of holding parts of research back (my understanding is that this pressure comes from the C-suite). Licensing that previously was open source compliant, has had non-commercial clauses added more and more frequently and even non-standard complex agreements as we have seen for LLaMA 2 and 3. This is sad and the culture of openness is ultimately at risk as they become more and more like OpenAI, Google DeepMind, etc. As I frequently point out, Facebook are free to decide on their own culture as they see fit and I am not entitled to their work. But it saddens me that they believe that compromising on their initial ideals is the way forward, rather than sticking to them through thick and thin. This, ultimately, makes it more and more difficult for me as an academic that believe in these ideals to work with them.
- freehorse 2y ago
  > weights being leaked You can hardly call that "leak" when they basically were sending the weights to thousands of people who applied for access. It is not that they kept them secret.
righthand 2y ago
Same reason they open sourced reactjs or encourage jestjs usage. If they give it away they can benefit by being embedded in the stack. Then all the scale issues they can beat implementers on because they have the money to run it. To keep ahead of people implementing their tech they have their own data trove to train on. You only get a small piece of it. It’s all to position themselves as a sensible solution.
downWidOutaFite 2y ago
My guess is that when Llama leaked on 4chan and it blew up in the community it somewhat forced their hand to go with an open strategy. But they also have a history with pytorch of reaping the benefits of an open strategy. The benefits are well explained in Google's leaked "We Have No Moat" document.
bg24 2y ago
1/ Build a community (think Linux for OS, Android for mobile, React for frontend) and figure out monetization later. What is clear to most people is that something fundamental is changing in how we build and consume applications. 2/ Prevent OpenAI from cornering the future $$$ market. Unfortunately, Google search is hit as well, but it is more due to the generational shift. 3/ Attract the best AI researchers. A product is a good as its core set of people (often just a few).
timy2shoes 2y ago
They’ve chosen the path of commoditizing their complement. To ensure that ML capabilities are not a differentiating factor in the market, make ML capabilities a commodity available to everyone at the marginal cost.
nicce 2y ago
There isn't conflict in business model. And if people can help to improve their models, they can extract better value by themselves. Also, maybe they need to improve their brand. Hoarding data for over a decade, maybe bringing something back now.
gillesjacobs 2y ago
Facebook/Meta has been doing open research in ML and NLP for far longer than the current LLM era: Convolutional Neural Networks for Sentence Classification (2014), FastText (2016), PyTorch (2016), fairseq (2017), LASER (2018), RoBERTa (2019), XLM-R (2019), and BART (2019), They were always present at ACL with decent open research as far as I have been studying/working in NLP (2014). It's part of a strategy to attract top talent in the field. If you want top researchers you have to let them publish, which in turn hones a reputation of solid research, attracting more talent.
- behnamoh 2y ago
  > It's part of a strategy to attract top talent in the field. If you want top researchers you have to let them publish, which in turn hones a reputation of solid research, attracting more talent. This. Although, it turns out, if you pay them well enough (like OpenAI), they'll forego publishing. If you can make enough to retire by 40, why work at places that pay less? ofc, not all researchers think that way, and there are those who are in it for the science, not just money.
lllaaaammmaaaaa 2y ago
All their users are writing "content" for free and communicate with each other. AI generated "content" does not really fit into this. They do not want their users to go to ClosedAI or similar and communicate with an Artificial Stupidity instead talking to each other on Facebook. So it is in their interest to undermine the market for Artificial Stupidities by releasing the models for free.
- notarealllama 2y ago
  Apt user name and salient points. Just wait until they have a fully intelligent automated pipeline for lifestyle ingestion (e.g. pervasive analysis of all communication and AV) into AI management (Timeline / Memories) backposted into web 2.0 feeds like Facebook.
edmundsauto 2y ago
Zuckerberg clearly lays out his position in an interview with Dwarkish, from about 2-3 months ago. Worth a watch if you’re curious.
whiplash451 2y ago
It is called "tipping the market" and it is a well-known strategy in the business of platforms. [1] Google did the same thing when they released android for free. [1] https://www.harperacademic.com/book/9780062896322/the-business-of-platforms/ https://www.harperacademic.com/book/9780062896322/the-busine...

hlfshell 2y ago

It started by accident, with the original llama weights being leaked by two separate employees. They've since embraced opening the weights, which I'm all for. As for why? I have a theory: Meta is not in a position to capitalize upon the model itself. Yes, they can use it internally, and maybe their competitors can copy it to - but there are no real competitors to Facebook or Instagram that can benefit from it enough to make it a differentiating facet. Thus, releasing stuff for open source does two things: 1) Make them more attractive to research talent (Apple famously recently started to publish research because their traditional secrecy was causing issues with hiring top talent) and... 2) Continues to undermine the ability to make $$$ off of model alone, driving it towards being a commodity rather than the long term profit engine for other companies.

sdenton4 2y ago
In short: "Commoditize the complement."
- deleted 2y ago
  [deleted]
onurcel 2y ago
> It started by accident, with the original llama weights being leaked by two separate employees. This is not true. Meta Fair has been built on openness from day 1. We published many papers and open-source d many repositories to reproduce the work
ipsum2 2y ago
Wrong. FAIR has been open sourcing ML models and source code for the last 10+ years. It did not start with llama. Also, llama was not leaked by employees, but people in the broader community, who the weights were shared with. For example: Faster R-CNN - state of the art image segmentation, released in 2017. FastText - text embedding models, 2016. FAISS - vector DB, 2018. https://github.com/orgs/facebookresearch/repositories https://github.com/orgs/facebookresearch/repositories has over 1,000 repos.
- hlfshell 2y ago
  I was speaking about LLM weights specifically and llama, not all models and work at FAIR. Per your links, it's clear that FAIR does have a good history of open source work.
- ninjin 2y ago
  This is very true. But we should also mention that Facebook sadly have been and are on a negative trajectory of openness. As someone working closely with them, there was a culture of nearly complete openness in the early years of their existence. Research was promptly shared in its entirety and licensing was compliant with open science. However, as the "AI boom" has grown, there is an increasing internal culture of holding parts of research back (my understanding is that this pressure comes from the C-suite). Licensing that previously was open source compliant, has had non-commercial clauses added more and more frequently and even non-standard complex agreements as we have seen for LLaMA 2 and 3. This is sad and the culture of openness is ultimately at risk as they become more and more like OpenAI, Google DeepMind, etc. As I frequently point out, Facebook are free to decide on their own culture as they see fit and I am not entitled to their work. But it saddens me that they believe that compromising on their initial ideals is the way forward, rather than sticking to them through thick and thin. This, ultimately, makes it more and more difficult for me as an academic that believe in these ideals to work with them.
freehorse 2y ago
> weights being leaked You can hardly call that "leak" when they basically were sending the weights to thousands of people who applied for access. It is not that they kept them secret.

righthand 2y ago

Same reason they open sourced reactjs or encourage jestjs usage. If they give it away they can benefit by being embedded in the stack. Then all the scale issues they can beat implementers on because they have the money to run it. To keep ahead of people implementing their tech they have their own data trove to train on. You only get a small piece of it. It’s all to position themselves as a sensible solution.

downWidOutaFite 2y ago

My guess is that when Llama leaked on 4chan and it blew up in the community it somewhat forced their hand to go with an open strategy. But they also have a history with pytorch of reaping the benefits of an open strategy. The benefits are well explained in Google's leaked "We Have No Moat" document.

bg24 2y ago

1/ Build a community (think Linux for OS, Android for mobile, React for frontend) and figure out monetization later. What is clear to most people is that something fundamental is changing in how we build and consume applications. 2/ Prevent OpenAI from cornering the future $$$ market. Unfortunately, Google search is hit as well, but it is more due to the generational shift. 3/ Attract the best AI researchers. A product is a good as its core set of people (often just a few).

timy2shoes 2y ago

They’ve chosen the path of commoditizing their complement. To ensure that ML capabilities are not a differentiating factor in the market, make ML capabilities a commodity available to everyone at the marginal cost.

nicce 2y ago

There isn't conflict in business model. And if people can help to improve their models, they can extract better value by themselves. Also, maybe they need to improve their brand. Hoarding data for over a decade, maybe bringing something back now.

gillesjacobs 2y ago

Facebook/Meta has been doing open research in ML and NLP for far longer than the current LLM era: Convolutional Neural Networks for Sentence Classification (2014), FastText (2016), PyTorch (2016), fairseq (2017), LASER (2018), RoBERTa (2019), XLM-R (2019), and BART (2019), They were always present at ACL with decent open research as far as I have been studying/working in NLP (2014). It's part of a strategy to attract top talent in the field. If you want top researchers you have to let them publish, which in turn hones a reputation of solid research, attracting more talent.

behnamoh 2y ago
> It's part of a strategy to attract top talent in the field. If you want top researchers you have to let them publish, which in turn hones a reputation of solid research, attracting more talent. This. Although, it turns out, if you pay them well enough (like OpenAI), they'll forego publishing. If you can make enough to retire by 40, why work at places that pay less? ofc, not all researchers think that way, and there are those who are in it for the science, not just money.

lllaaaammmaaaaa 2y ago

All their users are writing "content" for free and communicate with each other. AI generated "content" does not really fit into this. They do not want their users to go to ClosedAI or similar and communicate with an Artificial Stupidity instead talking to each other on Facebook. So it is in their interest to undermine the market for Artificial Stupidities by releasing the models for free.

notarealllama 2y ago
Apt user name and salient points. Just wait until they have a fully intelligent automated pipeline for lifestyle ingestion (e.g. pervasive analysis of all communication and AV) into AI management (Timeline / Memories) backposted into web 2.0 feeds like Facebook.

edmundsauto 2y ago

Zuckerberg clearly lays out his position in an interview with Dwarkish, from about 2-3 months ago. Worth a watch if you’re curious.

whiplash451 2y ago

It is called "tipping the market" and it is a well-known strategy in the business of platforms. [1] Google did the same thing when they released android for free. [1] https://www.harperacademic.com/book/9780062896322/the-business-of-platforms/ https://www.harperacademic.com/book/9780062896322/the-busine...