6 ms·

Some snippets for folks who come just for the comments: > While our models still hold a slight edge in terms of quality, the gap is closing astonishingly quic

Some snippets for folks who come just for the comments: > While our models still hold a slight edge in terms of quality, the gap is closing astonishingly quickly. Open-source models are faster, more customizable, more private, and pound-for-pound more capable. They are doing things with $100 and 13B params that we struggle with at $10M and 540B. And they are doing so in weeks, not months. > A tremendous outpouring of innovation followed, with just days between major developments (see The Timeline for the full breakdown). Here we are, barely a month later, and there are variants with instruction tuning, quantization, quality improvements, human evals, multimodality, RLHF, etc. etc. many of which build on each other. > This recent progress has direct, immediate implications for our business strategy. Who would pay for a Google product with usage restrictions if there is a free, high quality alternative without them? > Paradoxically, the one clear winner in all of this is Meta. Because the leaked model was theirs, they have effectively garnered an entire planet’s worth of free labor. Since most open source innovation is happening on top of their architecture, there is nothing stopping them from directly incorporating it into their products. > And in the end, OpenAI doesn’t matter. They are making the same mistakes we are in their posture relative to open source, and their ability to maintain an edge is necessarily in question. Open source alternatives can and will eventually eclipse them unless they change their stance. In this respect, at least, we can make the first move.

sterlind 3y ago
I wonder if OpenAI knew they didn't have a moat, and that's why they've been moving so fast and opening ChatGPT publicly - making the most of their lead in the short time they have left. I find it incredibly cathartic to see these massive tech companies and their gatekeepers get their lunch eaten by OSS.
- techwiz137 3y ago
  Sorry for the dumb question. But in the context of the AI space, what is moat?
  - leereeves 3y ago
    Moat is a business term coined by Warren Buffett. It's a competitive advantage that isn't easily overcome and allows a company to earn high margins. I don't think there are any examples in the context of AI. As the post says, no one in the AI space has a moat right now.
    - sterlind 3y ago
      Historically, datasets have been a moat. Google had a massive head start from having a massive search index and user data. Then access to compute became the moat - fully training a trillion-parameter language model has only been in reach for megacorps. But now, there's a ton of publicly-available datasets, and LLaMA showed that you don't need massive numbers of parameters.
  - bluGill 3y ago
    That is the million/billion+ dollar question. If find it and get there fast enough you can own the moat, and thus become rich. Note that I am not in any way implying that a moat even exists. There may be some reason AI becomes a winner takes all scheme and nobody else should bother playing, but it is also possible that there is no way to make your product better than anyone else. Only time will tell.
- adamsbriscoe 3y ago
  Taking the "no moat" argument at face value, I think it's important to remember that some of the largest players in AI are lobbying for regulation too.
  - HDThoreaun 3y ago
    Yep, regulating the ladder behind you is a classic monopolist move
  - marcod 3y ago
    If past performance is any indication, it's pretty safe to lobby for regulations in the US...
- tyre 3y ago
  OpenAI doesn't need a moat and it's fine that they don't have one. From their charter: > We will attempt to directly build safe and beneficial AGI, but will also consider our mission fulfilled if our work aids others to achieve this outcome. This was from 2018 and they've taken large strides away from their originally stated mission. Overall, though, they should be happy to have made progress in what they set out to do.
  - enjo 3y ago
    That's all well and good. I suspect their investors have a pretty different idea about their positioning tho.
    - arcticbull 3y ago
      Yes... > "we will attempt to directly build safe and beneficial AGI, but will also consider our mission fulfilled if our work aids others to achieve this outcome" ... has big "don't be evil" energy. I believe the next step was "we can do little a evil, as a treat."
    - tyre 3y ago
      Oh yeah, sure. I'm not sure I care much about that though. MSFT had the opportunity to think through all of this before they invested, OpenAI itself has incredible sums of money, and employees get to work on things they care about. If MSFT doesn't make anything on this investment—which it still might, given that a big chunk of its investment will likely go into Azure—then…okay.
  - int_19h 3y ago
    Note that this implies that if anyone tries to build AGI that is not "safe and beneficial" by OpenAI standards, it's fair game to suppress.
- gitfan86 3y ago
  That is the nature of the singularity. Progress moves faster than any one person or any one company can keep up with.
  - ddalex 3y ago
    What happens when the society as a whole cannot keep up with progress? That's a scary thought.
borski 3y ago
> Since most open source innovation is happening on top of their architecture, there is nothing stopping them from directly incorporating it into their products. There’s also nothing stopping anybody else from incorporating it into their products.
- ketzo 3y ago
  There definitely is. LLaMA is not licensed for commercial use. It's impractical to prosecute 1,000 people tinkering on their laptops, but if Meta discovered that Amazon was using LLaMA for commercial purposes, it would be nuclear war.
  - borski 3y ago
    Touché.
  - intalentive 3y ago
    Open-LLaMA is already out. It's not the end-all-be-all either. Better, smaller, open source models will continue to be released.
  - ada1981 3y ago
    Let’s play… Global Thermo Nuclear War.
  - robinsord 3y ago
    [dead]
  - spullara 3y ago
    We don't know what legal protection a bunch of weights have. They may not be copyrightable.
    - ketzo 3y ago
      They may not be. But do you wanna be the person/people to argue that against one of the richest companies in the world? I sure don't, and I definitely wouldn't stake my company/product on it.
whatshisface 3y ago
Meta's leaked model isn't open-source. I can found a business using Linux, that's open-source. The LLM piracy community are unpaid FB employees; it is not legal for anyone but Meta to use the results of their labor. I know this might be hard news but it needs to be said... if you want to put your time into working on open source LLMs, you need to get behind something you have a real (and yes, open source) license for.
- AnthonyMouse 3y ago
  Most of the code isn't specific to a model. It happens that LLaMA is approximately the best LLM currently available to the public to run on their own hardware, so that's what people are doing. But as soon as anyone publishes a better one, people will use that, using largely the same code, and there is no reason it couldn't be open source. I'm also curious what the copyright status of these models even is, given the "algorithmic output isn't copyrightable" thing and that the models themselves are essentially the algorithmic output of a machine learning algorithm on third party data. What right does Meta have to impose restrictions on the use of that data against people who downloaded it from The Pirate Bay? Wouldn't it be the same model if someone just ran the same algorithm on the same public data? (Not that that isn't an impediment to people who don't want to risk the legal expenses of setting a precedent, which models explicitly in the public domain would resolve.)
  - leereeves 3y ago
    > I'm also curious what the copyright status of these models even is That's my question as well. The models are clearly derivative works based on other people's copyrighted texts. Only a twisted court system would allow Google/OpenAI/Facebook to build models on other people's work and then forbid other people to build new models based on GOF's models.
    - AnthonyMouse 3y ago
      > That's my question as well. The models are clearly derivative works based on other people's copyrighted texts. That's not that clear either. (Sometimes it's more clear. If you ask the model to write fan fiction, and it does, and you want to claim that isn't a derivative work, good luck with that.) But the model itself is essentially a collection of data. "In Harry Potter and the Philosopher's Stone, Harry Potter is a wizard" is a fact about a work of fiction, not a work of fiction in itself. Facts generally aren't copyrightable. If you collect enough facts about something you could in principle reconstruct it, but that's not really something we've seen before and it's not obvious how to deal with it. That's going to create a practical problem if the models get good enough to e.g. emit the full text of the book on request, but the alternative is that it's illegal to make a model that knows everything there is to know about popular culture. Interesting times.
avereveard 3y ago
Openai moat is the upcoming first party integration with ms office.
- kccqzy 3y ago
  By that logic Google's moat is the integration with Gmail and Google Docs. And frankly it's not. People will decide to copy some text from Office or Docs to some other non-integrated tool, get LLMs to work, and then paste back to Office or Docs.
  - sudosysgen 3y ago
    Also, one can make Office plugins.
  - HDThoreaun 3y ago
    Some people will. Many others will just use the autocomplete functionality that is coming to every office suite product.
  - goatlover 3y ago
    That sounds rudimentary compared an integrated LLM could do for all your documents, emails, appointments, etc.
Alifatisk 3y ago
> the one clear winner in all of this is Meta. Because the leaked model was theirs, they have effectively garnered an entire planet’s worth of free labor. Since most open source innovation is happening on top of their architecture, there is nothing stopping them from directly incorporating it into their products. This
- wendyshu 3y ago
  This... what?
  - yellowstuff 3y ago
    It's internet speak for "I agree with this."
    - deleted 3y ago
      [deleted]
- diordiderot 3y ago
  I think this type of comment is generally frowned upon on HN. Upvote serves the same purpose.
  - Alifatisk 3y ago
    My bad
lhl 3y ago
> Paradoxically, the one clear winner in all of this is Meta. Because the leaked model was theirs, they have effectively garnered an entire planet’s worth of free labor. Since most open source innovation is happening on top of their architecture, there is nothing stopping them from directly incorporating it into their products. One interesting related point to this is Zuck's comments on Meta's AI strategy during their earnings call: https://www.reddit.com/r/MachineLearning/comments/1373nhq/discussion_mark_zuckerberg_on_metas_strategy_on/ https://www.reddit.com/r/MachineLearning/comments/1373nhq/di... Summary: """ Some noteworthy quotes that signal the thought process at Meta FAIR and more broadly We’re just playing a different game on the infrastructure than companies like Google or Microsoft or Amazon We would aspire to and hope to make even more open than that. So, we’ll need to figure out a way to do that. ...lead us to do more work in terms of open sourcing, some of the lower level models and tools Open sourcing low level tools make the way we run all this infrastructure more efficient over time. On PyTorch: It’s generally been very valuable for us to provide that because now all of the best developers across the industry are using tools that we’re also using internally. I would expect us to be pushing and helping to build out an open ecosystem. """
samstave 3y ago
>>”T ey have effectively garnered an entire planet’s worth of free labor.” - THIS IS WHY WE NEED DATA FUCKING OWNERSHIP. Users should be able to have a recourse to the use of their data in both of terms utility (for the parent company) and in terms of financial value to the parent company to the financial extraction of that value. Let me use cannabis as an example… When multiple cannabis cultivators (growers) combine their product for extraction into a singular product we have to figure out how to divide and pay the taxes.. Same thing (I’ll edit this later because I’m at the dentist

sterlind 3y ago

I wonder if OpenAI knew they didn't have a moat, and that's why they've been moving so fast and opening ChatGPT publicly - making the most of their lead in the short time they have left. I find it incredibly cathartic to see these massive tech companies and their gatekeepers get their lunch eaten by OSS.

techwiz137 3y ago
Sorry for the dumb question. But in the context of the AI space, what is moat?
- leereeves 3y ago
  Moat is a business term coined by Warren Buffett. It's a competitive advantage that isn't easily overcome and allows a company to earn high margins. I don't think there are any examples in the context of AI. As the post says, no one in the AI space has a moat right now.
  - sterlind 3y ago
    Historically, datasets have been a moat. Google had a massive head start from having a massive search index and user data. Then access to compute became the moat - fully training a trillion-parameter language model has only been in reach for megacorps. But now, there's a ton of publicly-available datasets, and LLaMA showed that you don't need massive numbers of parameters.
- bluGill 3y ago
  That is the million/billion+ dollar question. If find it and get there fast enough you can own the moat, and thus become rich. Note that I am not in any way implying that a moat even exists. There may be some reason AI becomes a winner takes all scheme and nobody else should bother playing, but it is also possible that there is no way to make your product better than anyone else. Only time will tell.
adamsbriscoe 3y ago
Taking the "no moat" argument at face value, I think it's important to remember that some of the largest players in AI are lobbying for regulation too.
- HDThoreaun 3y ago
  Yep, regulating the ladder behind you is a classic monopolist move
- marcod 3y ago
  If past performance is any indication, it's pretty safe to lobby for regulations in the US...
tyre 3y ago
OpenAI doesn't need a moat and it's fine that they don't have one. From their charter: > We will attempt to directly build safe and beneficial AGI, but will also consider our mission fulfilled if our work aids others to achieve this outcome. This was from 2018 and they've taken large strides away from their originally stated mission. Overall, though, they should be happy to have made progress in what they set out to do.
- enjo 3y ago
  That's all well and good. I suspect their investors have a pretty different idea about their positioning tho.
  - arcticbull 3y ago
    Yes... > "we will attempt to directly build safe and beneficial AGI, but will also consider our mission fulfilled if our work aids others to achieve this outcome" ... has big "don't be evil" energy. I believe the next step was "we can do little a evil, as a treat."
  - tyre 3y ago
    Oh yeah, sure. I'm not sure I care much about that though. MSFT had the opportunity to think through all of this before they invested, OpenAI itself has incredible sums of money, and employees get to work on things they care about. If MSFT doesn't make anything on this investment—which it still might, given that a big chunk of its investment will likely go into Azure—then…okay.
- int_19h 3y ago
  Note that this implies that if anyone tries to build AGI that is not "safe and beneficial" by OpenAI standards, it's fair game to suppress.
gitfan86 3y ago
That is the nature of the singularity. Progress moves faster than any one person or any one company can keep up with.
- ddalex 3y ago
  What happens when the society as a whole cannot keep up with progress? That's a scary thought.

borski 3y ago

> Since most open source innovation is happening on top of their architecture, there is nothing stopping them from directly incorporating it into their products. There’s also nothing stopping anybody else from incorporating it into their products.

ketzo 3y ago
There definitely is. LLaMA is not licensed for commercial use. It's impractical to prosecute 1,000 people tinkering on their laptops, but if Meta discovered that Amazon was using LLaMA for commercial purposes, it would be nuclear war.
- borski 3y ago
  Touché.
- intalentive 3y ago
  Open-LLaMA is already out. It's not the end-all-be-all either. Better, smaller, open source models will continue to be released.
- ada1981 3y ago
  Let’s play… Global Thermo Nuclear War.
- robinsord 3y ago
  [dead]
- spullara 3y ago
  We don't know what legal protection a bunch of weights have. They may not be copyrightable.
  - ketzo 3y ago
    They may not be. But do you wanna be the person/people to argue that against one of the richest companies in the world? I sure don't, and I definitely wouldn't stake my company/product on it.

whatshisface 3y ago

Meta's leaked model isn't open-source. I can found a business using Linux, that's open-source. The LLM piracy community are unpaid FB employees; it is not legal for anyone but Meta to use the results of their labor. I know this might be hard news but it needs to be said... if you want to put your time into working on open source LLMs, you need to get behind something you have a real (and yes, open source) license for.

AnthonyMouse 3y ago
Most of the code isn't specific to a model. It happens that LLaMA is approximately the best LLM currently available to the public to run on their own hardware, so that's what people are doing. But as soon as anyone publishes a better one, people will use that, using largely the same code, and there is no reason it couldn't be open source. I'm also curious what the copyright status of these models even is, given the "algorithmic output isn't copyrightable" thing and that the models themselves are essentially the algorithmic output of a machine learning algorithm on third party data. What right does Meta have to impose restrictions on the use of that data against people who downloaded it from The Pirate Bay? Wouldn't it be the same model if someone just ran the same algorithm on the same public data? (Not that that isn't an impediment to people who don't want to risk the legal expenses of setting a precedent, which models explicitly in the public domain would resolve.)
- leereeves 3y ago
  > I'm also curious what the copyright status of these models even is That's my question as well. The models are clearly derivative works based on other people's copyrighted texts. Only a twisted court system would allow Google/OpenAI/Facebook to build models on other people's work and then forbid other people to build new models based on GOF's models.
  - AnthonyMouse 3y ago
    > That's my question as well. The models are clearly derivative works based on other people's copyrighted texts. That's not that clear either. (Sometimes it's more clear. If you ask the model to write fan fiction, and it does, and you want to claim that isn't a derivative work, good luck with that.) But the model itself is essentially a collection of data. "In Harry Potter and the Philosopher's Stone, Harry Potter is a wizard" is a fact about a work of fiction, not a work of fiction in itself. Facts generally aren't copyrightable. If you collect enough facts about something you could in principle reconstruct it, but that's not really something we've seen before and it's not obvious how to deal with it. That's going to create a practical problem if the models get good enough to e.g. emit the full text of the book on request, but the alternative is that it's illegal to make a model that knows everything there is to know about popular culture. Interesting times.

avereveard 3y ago

Openai moat is the upcoming first party integration with ms office.

kccqzy 3y ago
By that logic Google's moat is the integration with Gmail and Google Docs. And frankly it's not. People will decide to copy some text from Office or Docs to some other non-integrated tool, get LLMs to work, and then paste back to Office or Docs.
- sudosysgen 3y ago
  Also, one can make Office plugins.
- HDThoreaun 3y ago
  Some people will. Many others will just use the autocomplete functionality that is coming to every office suite product.
- goatlover 3y ago
  That sounds rudimentary compared an integrated LLM could do for all your documents, emails, appointments, etc.

Alifatisk 3y ago

> the one clear winner in all of this is Meta. Because the leaked model was theirs, they have effectively garnered an entire planet’s worth of free labor. Since most open source innovation is happening on top of their architecture, there is nothing stopping them from directly incorporating it into their products. This

wendyshu 3y ago
This... what?
- yellowstuff 3y ago
  It's internet speak for "I agree with this."
  - deleted 3y ago
    [deleted]
diordiderot 3y ago
I think this type of comment is generally frowned upon on HN. Upvote serves the same purpose.
- Alifatisk 3y ago
  My bad

lhl 3y ago

> Paradoxically, the one clear winner in all of this is Meta. Because the leaked model was theirs, they have effectively garnered an entire planet’s worth of free labor. Since most open source innovation is happening on top of their architecture, there is nothing stopping them from directly incorporating it into their products. One interesting related point to this is Zuck's comments on Meta's AI strategy during their earnings call: https://www.reddit.com/r/MachineLearning/comments/1373nhq/discussion_mark_zuckerberg_on_metas_strategy_on/ https://www.reddit.com/r/MachineLearning/comments/1373nhq/di... Summary: """ Some noteworthy quotes that signal the thought process at Meta FAIR and more broadly We’re just playing a different game on the infrastructure than companies like Google or Microsoft or Amazon We would aspire to and hope to make even more open than that. So, we’ll need to figure out a way to do that. ...lead us to do more work in terms of open sourcing, some of the lower level models and tools Open sourcing low level tools make the way we run all this infrastructure more efficient over time. On PyTorch: It’s generally been very valuable for us to provide that because now all of the best developers across the industry are using tools that we’re also using internally. I would expect us to be pushing and helping to build out an open ecosystem. """

samstave 3y ago

>>”T ey have effectively garnered an entire planet’s worth of free labor.” - THIS IS WHY WE NEED DATA FUCKING OWNERSHIP. Users should be able to have a recourse to the use of their data in both of terms utility (for the parent company) and in terms of financial value to the parent company to the financial extraction of that value. Let me use cannabis as an example… When multiple cannabis cultivators (growers) combine their product for extraction into a singular product we have to figure out how to divide and pay the taxes.. Same thing (I’ll edit this later because I’m at the dentist