5 ms·

Super impressive. Being able to beat a mega closed source model with an open source LLM and some fine tuning really calls into question the financial viability

by JCM9 3y ago

Super impressive. Being able to beat a mega closed source model with an open source LLM and some fine tuning really calls into question the financial viability of these big proprietary LLMs. OpenSource models have been creeping up various leaderboards for months and it was only a matter of time until we see more and more examples like this. Excellent work.

m3kw9 3y ago
It beat it on a specific subset of language
- brucethemoose2 3y ago
  Thats still interesting, as finetunes for different languages can be made using similar methodologies.
trickstra 3y ago
Let's not call it open source. Even Llama2 doesn't think that Llama2 is open source: https://imgur.com/AZFOzWk https://imgur.com/AZFOzWk
- dado3212 3y ago
  Is this really the line we want to draw in the sand? It’s not open source because it can’t be trivially used by AWS and Google? It feels like the popularism is the point, not the complete lack of restrictions.
  - trickstra 3y ago
    Well if we are talking about the point of open source, then really, the lack of restrictions is the point. Some commercial entities wish to redefine the term because it benefits their marketing, but is that the line we want to draw in the sand?
    - qup 3y ago
      You give them an inch...
      - tempaccount420 3y ago
        Slippery slope
    - jstummbillig 3y ago
      People have different ideas about what open source means. Defining that is what licenses are for. "Unrestricted in all cases" is simply not how that ever played out, historically, at least not for as long as I can recall.
  - jdthedisciple 3y ago
    Well, you see you are absolutely correct. But unfortunately none of the "True Open Source" crusaders I have seen so far can think beyond their tribalism-based schism.
  - dTal 3y ago
    It's not "open source" because it's not free software. We do not have access to the training data (source code). We cannot make new versions by editing the preferred representation. It's a big closed source blob released by a corporation that does a cool thing. At best, it's freeware. If it were really "open source", nobody would be complaining about its moralizing attitude. They'd just make a new one, without the attitude.
- deleted 3y ago
  [deleted]
cryptobruh 3y ago
I don't understand your reasoning. Those models are all trained by companies and open source primarily fine-tune them. It's impressive that we can do that but the base is still a lot of money and experts from the best companies/ai experts we have. There is a minimal chance that we will be able to somehow keep up if Google and co stop publishing or delaying publishing their papers and models. There are communities forming etc. And it's impressive but the financial 'viability' is key to the current ai progress
- brucethemoose2 3y ago
  Right now training is insanely expensive because Nvidia, but I don't think that's sustainable given the demand. Eventually, training hardware may be priced like commodity CPU instances, and won't require a bajillion infiniband-linked nodes for respectable throughput. But for now... I think you have a point. We would have seen more than Falcon, MPT, Llama, and the open Llama reproductions by now if open source foundational model training was viable.
  - adventured 3y ago
    The training will always be extremely expensive and beyond the grasp of commodity computing at a low'ish cost. The reason for that, is that the push will always be one direction in terms of size and improvement. That isn't going to stop. It will continue to push the edge of resources. It's not Nvidia that's holding back the premise. AMD and Intel also can't do anything for you beyond what Nvidia can. One might proclaim: yeah but in five years you'll be able to train CodeLlama 34B on a modest commodity desktop. Nobody will want to do that at that point, they'll want access to CodeLlama 204B. The money will be in hosting these as a service business, and building further ecosystems around that. Not much different than the way a lot of major open source oriented companies have made their money, despite their core product being largely free to use.
    - cbozeman 3y ago
      I think you're correct in the "right now", but I think things fall off dramatically in the next 5-10 years. NVIDIA's progress from 2018 to 2023 was insanely impressive. AMD's was too in fact, at least technologically. Intel is third, but at least still in the game. Where your premise here falls apart, is the age-old adage, "Don't let perfect be the enemy of the good." We probably don't need CodeLlama 204B to be highly effective in our jobs as developers. Just like physicians probably don't need DocLlama 566B to be more effective in their jobs. We only need good enough. Good enough to be useful and insightful; not the All-Seeing Oracle that has insight beyond mortal man. I think that revolution is probably coming sooner than we think. I'd argue that in 5-10 years, we'll have hardware powerful enough that a motivated and moderately well-heeled consumer can train their own LLM with domain-specific knowledge and have a product as useful as Llama8 1.3T or ChatGPT-14. There'll always be a market - I think anyway - for incredibly powerful general-purpose LLMs that are subscription-based (whatever that looks like in terms of cost / month), but locally trained, locally run LLMs designed to do one thing and do it well? I think that's the real money and the real future.
brucethemoose2 3y ago
I think it just highlights the utility of finetuning. This is something big tech can still offer with proprietary models.

by JCM9 3y ago

m3kw9 3y ago

It beat it on a specific subset of language

brucethemoose2 3y ago
Thats still interesting, as finetunes for different languages can be made using similar methodologies.

trickstra 3y ago

Let's not call it open source. Even Llama2 doesn't think that Llama2 is open source: https://imgur.com/AZFOzWk https://imgur.com/AZFOzWk

dado3212 3y ago
Is this really the line we want to draw in the sand? It’s not open source because it can’t be trivially used by AWS and Google? It feels like the popularism is the point, not the complete lack of restrictions.
- trickstra 3y ago
  Well if we are talking about the point of open source, then really, the lack of restrictions is the point. Some commercial entities wish to redefine the term because it benefits their marketing, but is that the line we want to draw in the sand?
  - qup 3y ago
    You give them an inch...
    - tempaccount420 3y ago
      Slippery slope
  - jstummbillig 3y ago
    People have different ideas about what open source means. Defining that is what licenses are for. "Unrestricted in all cases" is simply not how that ever played out, historically, at least not for as long as I can recall.
- jdthedisciple 3y ago
  Well, you see you are absolutely correct. But unfortunately none of the "True Open Source" crusaders I have seen so far can think beyond their tribalism-based schism.
- dTal 3y ago
  It's not "open source" because it's not free software. We do not have access to the training data (source code). We cannot make new versions by editing the preferred representation. It's a big closed source blob released by a corporation that does a cool thing. At best, it's freeware. If it were really "open source", nobody would be complaining about its moralizing attitude. They'd just make a new one, without the attitude.
deleted 3y ago
[deleted]

cryptobruh 3y ago

I don't understand your reasoning. Those models are all trained by companies and open source primarily fine-tune them. It's impressive that we can do that but the base is still a lot of money and experts from the best companies/ai experts we have. There is a minimal chance that we will be able to somehow keep up if Google and co stop publishing or delaying publishing their papers and models. There are communities forming etc. And it's impressive but the financial 'viability' is key to the current ai progress

brucethemoose2 3y ago
Right now training is insanely expensive because Nvidia, but I don't think that's sustainable given the demand. Eventually, training hardware may be priced like commodity CPU instances, and won't require a bajillion infiniband-linked nodes for respectable throughput. But for now... I think you have a point. We would have seen more than Falcon, MPT, Llama, and the open Llama reproductions by now if open source foundational model training was viable.
- adventured 3y ago
  The training will always be extremely expensive and beyond the grasp of commodity computing at a low'ish cost. The reason for that, is that the push will always be one direction in terms of size and improvement. That isn't going to stop. It will continue to push the edge of resources. It's not Nvidia that's holding back the premise. AMD and Intel also can't do anything for you beyond what Nvidia can. One might proclaim: yeah but in five years you'll be able to train CodeLlama 34B on a modest commodity desktop. Nobody will want to do that at that point, they'll want access to CodeLlama 204B. The money will be in hosting these as a service business, and building further ecosystems around that. Not much different than the way a lot of major open source oriented companies have made their money, despite their core product being largely free to use.
  - cbozeman 3y ago
    I think you're correct in the "right now", but I think things fall off dramatically in the next 5-10 years. NVIDIA's progress from 2018 to 2023 was insanely impressive. AMD's was too in fact, at least technologically. Intel is third, but at least still in the game. Where your premise here falls apart, is the age-old adage, "Don't let perfect be the enemy of the good." We probably don't need CodeLlama 204B to be highly effective in our jobs as developers. Just like physicians probably don't need DocLlama 566B to be more effective in their jobs. We only need good enough. Good enough to be useful and insightful; not the All-Seeing Oracle that has insight beyond mortal man. I think that revolution is probably coming sooner than we think. I'd argue that in 5-10 years, we'll have hardware powerful enough that a motivated and moderately well-heeled consumer can train their own LLM with domain-specific knowledge and have a product as useful as Llama8 1.3T or ChatGPT-14. There'll always be a market - I think anyway - for incredibly powerful general-purpose LLMs that are subscription-based (whatever that looks like in terms of cost / month), but locally trained, locally run LLMs designed to do one thing and do it well? I think that's the real money and the real future.

brucethemoose2 3y ago

I think it just highlights the utility of finetuning. This is something big tech can still offer with proprietary models.