5 ms·

Is there any reason to believe that the first open source chat GPT clone won't consume most mindshare, a la stable diffusion?

mouse_ 4y ago
My theory is that OpenAI is preying on venture capital, and they don't care who wins long term. As long as they're first to get the freshest ideas on the biggest computers, they can secure a large sum of money.
- eddsh1994 4y ago
  And do what with it?
- speed_spread 4y ago
  Sounds like SNL's "First Change Bank" skit: - You give us a dollar, we'll give you four quarters! - People ask us how we make money. The answer is simple: _volume_.
  - 13of40 4y ago
    I see what you're saying, but the truth is that for a moment there the bank had $2 on its books.
adamsmith143 4y ago
It all ends up on HuggingFace anyway so that's where the smart money is heading.
ShamelessC 4y ago
Deployment is the significant barrier with these models. They have far more parameters than stable diffusion/DALLE2.
antimatter15 4y ago
There are already open source LLMs with comparable parameter counts (Facebook's OPT-175B, BLOOM), but you'll need ~10x A100 GPUs to run them (which would cost ~$100K+). I suspect a big part of why stable diffusion managed to consume so much mindshare is that it can run on ordinary consumer hardware. On that point, I would be excited about an open-source RETRO (https://arxiv.org/pdf/2112.04426.pdf https://arxiv.org/pdf/2112.04426.pdf) model with comparable performance to GPT-3 that could run on consumer hardware with an NVMe SSD.
- rafaelero 4y ago
  They aren't as good as davinci-003, though. There is no open source model competitive with GPT-3.5 yet.
- lagrange77 4y ago
  GPT-J-6B is said to be also of comparable performance in certain areas to GPT-3 and can be run (inference) on a RTX3090. https://huggingface.co/EleutherAI/gpt-j-6B https://huggingface.co/EleutherAI/gpt-j-6B
- Roark66 4y ago
  One can in theory run even 175B bloom with just a modern multicore CPU, 32gb of RAM and 2TB of nvme storage. There is a library called accelerate witch with some slight modifications allows one to run in cpu/storage only mode models that don't fit in memory. Of course it takes a looong time to do inference, but one can at least have a taste. The biggest bloom I personally have run on cpu only in this fashion is 7B. It requires 4x7B of RAM plus some. On my hardware it tends to use all 32GB RAM and about ~4GB of storage during inference. At the moment I believe there is still a limitation of the smallest layer fitting in memory at once. This is why I haven't tried bigger bloom, but I believe there are ways to overcome it. Once this problem is resolved one should be able to use the same tech to use GPUs with less vram (like my 2070 with 8GB) for parts of larger models.
turmeric_root 4y ago
on top of the hardware requirements ($10s of thousands of GPUs are needed for something a language model as big as gpt-3) there's also a lot of work involved in RLHF models like chatgpt. you need to pay people to write and review thousands/tens of thousands of responses for training. see 'methods' here: https://openai.com/blog/chatgpt/ https://openai.com/blog/chatgpt/
- coolspot 4y ago
  LAION’s Open Assistant trying to address that with gamified crowdsourcing. https://github.com/LAION-AI/Open-Assistant https://github.com/LAION-AI/Open-Assistant

mouse_ 4y ago

My theory is that OpenAI is preying on venture capital, and they don't care who wins long term. As long as they're first to get the freshest ideas on the biggest computers, they can secure a large sum of money.

eddsh1994 4y ago
And do what with it?
speed_spread 4y ago
Sounds like SNL's "First Change Bank" skit: - You give us a dollar, we'll give you four quarters! - People ask us how we make money. The answer is simple: _volume_.
- 13of40 4y ago
  I see what you're saying, but the truth is that for a moment there the bank had $2 on its books.

adamsmith143 4y ago

It all ends up on HuggingFace anyway so that's where the smart money is heading.

ShamelessC 4y ago

Deployment is the significant barrier with these models. They have far more parameters than stable diffusion/DALLE2.

antimatter15 4y ago

There are already open source LLMs with comparable parameter counts (Facebook's OPT-175B, BLOOM), but you'll need ~10x A100 GPUs to run them (which would cost ~$100K+). I suspect a big part of why stable diffusion managed to consume so much mindshare is that it can run on ordinary consumer hardware. On that point, I would be excited about an open-source RETRO (https://arxiv.org/pdf/2112.04426.pdf https://arxiv.org/pdf/2112.04426.pdf) model with comparable performance to GPT-3 that could run on consumer hardware with an NVMe SSD.

rafaelero 4y ago
They aren't as good as davinci-003, though. There is no open source model competitive with GPT-3.5 yet.
lagrange77 4y ago
GPT-J-6B is said to be also of comparable performance in certain areas to GPT-3 and can be run (inference) on a RTX3090. https://huggingface.co/EleutherAI/gpt-j-6B https://huggingface.co/EleutherAI/gpt-j-6B
Roark66 4y ago
One can in theory run even 175B bloom with just a modern multicore CPU, 32gb of RAM and 2TB of nvme storage. There is a library called accelerate witch with some slight modifications allows one to run in cpu/storage only mode models that don't fit in memory. Of course it takes a looong time to do inference, but one can at least have a taste. The biggest bloom I personally have run on cpu only in this fashion is 7B. It requires 4x7B of RAM plus some. On my hardware it tends to use all 32GB RAM and about ~4GB of storage during inference. At the moment I believe there is still a limitation of the smallest layer fitting in memory at once. This is why I haven't tried bigger bloom, but I believe there are ways to overcome it. Once this problem is resolved one should be able to use the same tech to use GPUs with less vram (like my 2070 with 8GB) for parts of larger models.

turmeric_root 4y ago

on top of the hardware requirements ($10s of thousands of GPUs are needed for something a language model as big as gpt-3) there's also a lot of work involved in RLHF models like chatgpt. you need to pay people to write and review thousands/tens of thousands of responses for training. see 'methods' here: https://openai.com/blog/chatgpt/ https://openai.com/blog/chatgpt/

coolspot 4y ago
LAION’s Open Assistant trying to address that with gamified crowdsourcing. https://github.com/LAION-AI/Open-Assistant https://github.com/LAION-AI/Open-Assistant