5 ms·

Question to AI/ML folks : Is there no comparable open source model? Is the future going to be controlled by big corporations who own the models themselves? If m

Question to AI/ML folks : Is there no comparable open source model? Is the future going to be controlled by big corporations who own the models themselves? If models are so computationally intensive to produce does it mean than the more computational power a company has the better its models will be?

freediver 4y ago
Closest you can get is probably with Google T5-Flan [1]. It is not the size of the model or the text it was trained on that makes ChatGPT so performant. It is the additional human assisted training to make it respond well to instructions. Open source versions of that are just starting to see the light of day [2]. [1] https://huggingface.co/google/flan-t5-xxl https://huggingface.co/google/flan-t5-xxl [2] https://github.com/lucidrains/PaLM-rlhf-pytorch https://github.com/lucidrains/PaLM-rlhf-pytorch
- eternalban 4y ago
  From, [2]: ” This repository has gone viral without my permission. Next time, if you are promoting my unfinished repositories (notice the work in progress flag) for twitter engagement or eyeballs, at least (1) do your research or (2) be totally transparent with your readers about the capacity of the repository without resorting to clickbait. (1) I was not the first, CarperAI had been working on RLHF months before, link below. (2) There is no trained model. This is just the ship and overall map. We still need millions of dollars of compute + data to sail to the correct point in high dimensional parameter space. Even then, you need professional sailors (like Robin Rombach of Stable Diffusion fame) to actually guide the ship through turbulent times to that point.” [p.s. ^ was just fyi/heads-up + https://github.com/CarperAI https://github.com/CarperAI]
  - 2OEH8eoCRo0 4y ago
    I didn't know I needed permission when sharing things that I find publicly online. They should use a private repo and try to be more polite when making demands.
  - shepherdjerred 4y ago
    > This repository has gone viral without my permission That's a bizarre thing to write for a public repository
noobermin 4y ago
People who thought AI would ever be "open source" and not "stolen from the labor of millions or even billions in order to replace them" are fools.
- Der_Einzige 4y ago
  Why? Huggingface is most likely worth as much as OpenAI is today and everything they do is open sourced or basically open sourced. Huggingface is one of the most beloved companies in the world right now too. Lots of interest in them within open source developer communities.
  - politician 4y ago
    The content that these models ingest though have licenses that they ignore. That's the "labor" being "stolen from millions or billions."
localhost 4y ago
RE: Open Source models, there is the AnthropicAI model making the rounds on Twitter[1] and Stability.ai (makers of Stable Diffusion) are working on one [2]. If we use recent history as an example, OpenAI announced DALL-E in Jan 5, 2021 [3], announced v2 and a waitlist for public use in July 20, 2022, and Stable Diffusion shipped an open source model on August 22, 2022 [4] using ~$600K of compute (at retail prices on AWS) [5]. I don't see how it's likely that any company can acquire a durable technology moat here. There are scale barriers to entry, but even VC sized funding can overcome that. [1] https://twitter.com/goodside/status/1611556749726605312 https://twitter.com/goodside/status/1611556749726605312 [2] https://humanloop.com/blog/stability-ai-partnership https://humanloop.com/blog/stability-ai-partnership [3] https://en.wikipedia.org/wiki/DALL-E https://en.wikipedia.org/wiki/DALL-E [4] https://en.wikipedia.org/wiki/Stable_Diffusion https://en.wikipedia.org/wiki/Stable_Diffusion [5] https://twitter.com/emostaque/status/1563870674111832066?lang=en https://twitter.com/emostaque/status/1563870674111832066?lan...
- orra 4y ago
  Stable Diffusion's code and model weights can be downloaded, but it's proprietary, not Open Source.
  - localhost 4y ago
    This is the text of the license from this repo [1]. Seems pretty open to me? About this license The Responsible AI License allows users to take advantage of the model in a wide range of settings (including free use and redistribution) as long as they respect the specific use case restrictions outlined, which correspond to model applications the licensor deems ill-suited for the model or are likely to cause harm. [1] https://huggingface.co/CompVis/stable-diffusion-v-1-4-original https://huggingface.co/CompVis/stable-diffusion-v-1-4-origin...
    - schoen 4y ago
      Compare https://opensource.org/osd https://opensource.org/osd This isn't a very minor point, as this was an explicit discussion and is also OSI's translation of Debian's translation of Richard Stallman's "freedom 0". That is, it's an important, and explicit, tradition/consensus in FOSS that users aren't restricted in the purposes for which they may use the software.
spi 4y ago
The closest open source contender is BLOOM: https://huggingface.co/bigscience/bloom https://huggingface.co/bigscience/bloom. It has an almost identical architecture to GPT-3 (hence, to ChatGPT), and in particular the same number of parameters (175B). It was also trained on a similar amount of data, as far as we can know. Still, it's not like you can just "download it and run it", even just to _load_ the model into memory you need ~400GB of memory, to run it at any decent speed you need a lot of GPUs, so it's not really like consumer hardware. And the process to train it cost about 2 to 4 million $, so replicating it is definitely not for everybody. But also not just for "big corporations"...
- throwaway743 4y ago
  Saw this on here the other day. Uses BLOOM. Not as good as chatGPT, but it's something http://chat.petals.ml/ http://chat.petals.ml/
- espadrine 4y ago
  In terms of quality, I think BLOOMZ, or mT0, are the best open-source ones. The non-finetuned BLOOM does not appear favorably (in English) compared to GLM or OPT, which both have published weights: https://crfm.stanford.edu/helm/v0.1.0/?group=mmlu https://crfm.stanford.edu/helm/v0.1.0/?group=mmlu and Flan-T5 is above OPT-IML: https://arxiv.org/pdf/2212.12017.pdf https://arxiv.org/pdf/2212.12017.pdf > Is the future going to be controlled by big corporations who own the models themselves? On this subject, there is an effort stemming from BigScience to build an open, distributed inference network, so that people that don’t have enough GPUs at home can contribute theirs and get text generation at one word per second: https://github.com/bigscience-workshop/petals#how-does-it-work https://github.com/bigscience-workshop/petals#how-does-it-wo...
- metadat 4y ago
  Getting a server with > 400GB of RAM and a heap of GPUs can be done for less than $6,000 - $10,000 if you're scrappy. Not cheap, but also not out of reach for individuals.
  - spi 4y ago
ladino 4y ago
what about https://github.com/karpathy/nanoGPT https://github.com/karpathy/nanoGPT (currently trending on Github)
deleted 4y ago
[deleted]
amelius 4y ago
It is quite outrageous that despite the fact that nearly the entire tech world is built on FOSS, Open Source still doesn't get the recognition it deserves from governments.
- college_physics 4y ago
  Despite its recognition in techie circles open source is largely unknown in both corporate and government environments and there is probably a significant operational gap to overcome as these entities have very rigid rules of engagement. Among others, they generally dont think for themselves. The picture could change significantly if the various intermediaries, consultants etc who live off these ecosystems found ways to make open source profitable for them
kossTKR 4y ago
We really need this, because if it's only the absolute economic elite that will have access to this 99.9999% of people are going to be dominated, psyopped, outworked, censored and drowned in noise whether on the "left" or "right" for ultimate AI centrism aka. bootlicker discourse. A free for all will still result in a dizzying paradigm shift though, but there is no alternative. Like Guttenberg but exponentially faster - locality will become central, "corny pop culture hacker dungeons" / hackspaces could become important as no one will know whats real and only locally controlled compute and algo power is to be trusted.
Q6T46nT668w6i3m 4y ago
It depends what you mean by “open source.” You can find countless open source implementations. The issue is training. Meta has released their LLM weights. I believe Google has too.
matchagaucho 4y ago
The software seems fairly open. The computing bill around large language models is extraordinary (hence MSFT Azure being a strategic partner).
edgyquant 4y ago
It’s not the model that’s the edge here but the scale of compute. A PC user is not going to be able to run such a behemoth which has billions of input neurons. This is not an ideal ML model
Der_Einzige 4y ago
The existing open source models which compete at this point suck ass in comparison. I'm likely to do less NLP research going forward and more CV research because I can't locally run most LLMs but I sure as shit can run most of the diffusion models at home. It's a sad situation.

freediver 4y ago

Closest you can get is probably with Google T5-Flan [1]. It is not the size of the model or the text it was trained on that makes ChatGPT so performant. It is the additional human assisted training to make it respond well to instructions. Open source versions of that are just starting to see the light of day [2]. [1] https://huggingface.co/google/flan-t5-xxl https://huggingface.co/google/flan-t5-xxl [2] https://github.com/lucidrains/PaLM-rlhf-pytorch https://github.com/lucidrains/PaLM-rlhf-pytorch

eternalban 4y ago
From, [2]: ” This repository has gone viral without my permission. Next time, if you are promoting my unfinished repositories (notice the work in progress flag) for twitter engagement or eyeballs, at least (1) do your research or (2) be totally transparent with your readers about the capacity of the repository without resorting to clickbait. (1) I was not the first, CarperAI had been working on RLHF months before, link below. (2) There is no trained model. This is just the ship and overall map. We still need millions of dollars of compute + data to sail to the correct point in high dimensional parameter space. Even then, you need professional sailors (like Robin Rombach of Stable Diffusion fame) to actually guide the ship through turbulent times to that point.” [p.s. ^ was just fyi/heads-up + https://github.com/CarperAI https://github.com/CarperAI]
- 2OEH8eoCRo0 4y ago
  I didn't know I needed permission when sharing things that I find publicly online. They should use a private repo and try to be more polite when making demands.
- shepherdjerred 4y ago
  > This repository has gone viral without my permission That's a bizarre thing to write for a public repository

noobermin 4y ago

People who thought AI would ever be "open source" and not "stolen from the labor of millions or even billions in order to replace them" are fools.

Der_Einzige 4y ago
Why? Huggingface is most likely worth as much as OpenAI is today and everything they do is open sourced or basically open sourced. Huggingface is one of the most beloved companies in the world right now too. Lots of interest in them within open source developer communities.
- politician 4y ago
  The content that these models ingest though have licenses that they ignore. That's the "labor" being "stolen from millions or billions."

localhost 4y ago

RE: Open Source models, there is the AnthropicAI model making the rounds on Twitter[1] and Stability.ai (makers of Stable Diffusion) are working on one [2]. If we use recent history as an example, OpenAI announced DALL-E in Jan 5, 2021 [3], announced v2 and a waitlist for public use in July 20, 2022, and Stable Diffusion shipped an open source model on August 22, 2022 [4] using ~$600K of compute (at retail prices on AWS) [5]. I don't see how it's likely that any company can acquire a durable technology moat here. There are scale barriers to entry, but even VC sized funding can overcome that. [1] https://twitter.com/goodside/status/1611556749726605312 https://twitter.com/goodside/status/1611556749726605312 [2] https://humanloop.com/blog/stability-ai-partnership https://humanloop.com/blog/stability-ai-partnership [3] https://en.wikipedia.org/wiki/DALL-E https://en.wikipedia.org/wiki/DALL-E [4] https://en.wikipedia.org/wiki/Stable_Diffusion https://en.wikipedia.org/wiki/Stable_Diffusion [5] https://twitter.com/emostaque/status/1563870674111832066?lang=en https://twitter.com/emostaque/status/1563870674111832066?lan...

orra 4y ago
Stable Diffusion's code and model weights can be downloaded, but it's proprietary, not Open Source.
- localhost 4y ago
  This is the text of the license from this repo [1]. Seems pretty open to me? About this license The Responsible AI License allows users to take advantage of the model in a wide range of settings (including free use and redistribution) as long as they respect the specific use case restrictions outlined, which correspond to model applications the licensor deems ill-suited for the model or are likely to cause harm. [1] https://huggingface.co/CompVis/stable-diffusion-v-1-4-original https://huggingface.co/CompVis/stable-diffusion-v-1-4-origin...
  - schoen 4y ago
    Compare https://opensource.org/osd https://opensource.org/osd This isn't a very minor point, as this was an explicit discussion and is also OSI's translation of Debian's translation of Richard Stallman's "freedom 0". That is, it's an important, and explicit, tradition/consensus in FOSS that users aren't restricted in the purposes for which they may use the software.

spi 4y ago

The closest open source contender is BLOOM: https://huggingface.co/bigscience/bloom https://huggingface.co/bigscience/bloom. It has an almost identical architecture to GPT-3 (hence, to ChatGPT), and in particular the same number of parameters (175B). It was also trained on a similar amount of data, as far as we can know. Still, it's not like you can just "download it and run it", even just to _load_ the model into memory you need ~400GB of memory, to run it at any decent speed you need a lot of GPUs, so it's not really like consumer hardware. And the process to train it cost about 2 to 4 million $, so replicating it is definitely not for everybody. But also not just for "big corporations"...

throwaway743 4y ago
Saw this on here the other day. Uses BLOOM. Not as good as chatGPT, but it's something http://chat.petals.ml/ http://chat.petals.ml/
espadrine 4y ago
In terms of quality, I think BLOOMZ, or mT0, are the best open-source ones. The non-finetuned BLOOM does not appear favorably (in English) compared to GLM or OPT, which both have published weights: https://crfm.stanford.edu/helm/v0.1.0/?group=mmlu https://crfm.stanford.edu/helm/v0.1.0/?group=mmlu and Flan-T5 is above OPT-IML: https://arxiv.org/pdf/2212.12017.pdf https://arxiv.org/pdf/2212.12017.pdf > Is the future going to be controlled by big corporations who own the models themselves? On this subject, there is an effort stemming from BigScience to build an open, distributed inference network, so that people that don’t have enough GPUs at home can contribute theirs and get text generation at one word per second: https://github.com/bigscience-workshop/petals#how-does-it-work https://github.com/bigscience-workshop/petals#how-does-it-wo...
metadat 4y ago
Getting a server with > 400GB of RAM and a heap of GPUs can be done for less than $6,000 - $10,000 if you're scrappy. Not cheap, but also not out of reach for individuals.
- spi 4y ago

ladino 4y ago

what about https://github.com/karpathy/nanoGPT https://github.com/karpathy/nanoGPT (currently trending on Github)

deleted 4y ago

[deleted]

amelius 4y ago

It is quite outrageous that despite the fact that nearly the entire tech world is built on FOSS, Open Source still doesn't get the recognition it deserves from governments.

college_physics 4y ago
Despite its recognition in techie circles open source is largely unknown in both corporate and government environments and there is probably a significant operational gap to overcome as these entities have very rigid rules of engagement. Among others, they generally dont think for themselves. The picture could change significantly if the various intermediaries, consultants etc who live off these ecosystems found ways to make open source profitable for them

kossTKR 4y ago

We really need this, because if it's only the absolute economic elite that will have access to this 99.9999% of people are going to be dominated, psyopped, outworked, censored and drowned in noise whether on the "left" or "right" for ultimate AI centrism aka. bootlicker discourse. A free for all will still result in a dizzying paradigm shift though, but there is no alternative. Like Guttenberg but exponentially faster - locality will become central, "corny pop culture hacker dungeons" / hackspaces could become important as no one will know whats real and only locally controlled compute and algo power is to be trusted.

Q6T46nT668w6i3m 4y ago

It depends what you mean by “open source.” You can find countless open source implementations. The issue is training. Meta has released their LLM weights. I believe Google has too.

matchagaucho 4y ago

The software seems fairly open. The computing bill around large language models is extraordinary (hence MSFT Azure being a strategic partner).

edgyquant 4y ago

It’s not the model that’s the edge here but the scale of compute. A PC user is not going to be able to run such a behemoth which has billions of input neurons. This is not an ideal ML model

Der_Einzige 4y ago

The existing open source models which compete at this point suck ass in comparison. I'm likely to do less NLP research going forward and more CV research because I can't locally run most LLMs but I sure as shit can run most of the diffusion models at home. It's a sad situation.