6 ms·

Is there a Stable-Diffusion-esque open-source GPT yet? Given the incredible pace of advances in the image space this year, and my (perhaps naive) assumption tha

Is there a Stable-Diffusion-esque open-source GPT yet? Given the incredible pace of advances in the image space this year, and my (perhaps naive) assumption that text generation is less complex and less resource-intensive than image generation, I'm hoping we'll get something similar and surprised that we haven't yet.

espadrine 4y ago
BLOOM[0] is another 176B multilingual open-source model whose weights were published. However, running it locally requires 8 A6000 GPUs. The GPU cost alone is roughly $37K. Also, there is the concern of performance. The Chinchilla[1] paper indicates that most model sizes trained so far received too little token training (or were too big) to reach an optimal performance per energy expenditure. Furthermore, the InstructGPT[2] and more recently Flan[3] papers show that various finetuning techniques achieve significant gains, without which the result is a bit meh. The dark secret of GPT-3 is that the model available on the OpenAI website is heavily fine-tuned, even compared to the InstructGPT paper. [0]: https://huggingface.co/bigscience/bloom https://huggingface.co/bigscience/bloom [1]: https://arxiv.org/abs/2203.15556 https://arxiv.org/abs/2203.15556 [2]: https://arxiv.org/abs/2203.02155 https://arxiv.org/abs/2203.02155 [3]: https://arxiv.org/pdf/2210.11416.pdf https://arxiv.org/pdf/2210.11416.pdf
davnn 4y ago
There are multiple open-source GPTs, but GPT-3 is absolutely massive - larger than the image models actually! So, unfortunately, text generation is probably even more complex and resource intensive than image generation (especially to train). Additionally, in image generation, we appreciate the creativity of solutions, but in text generation the creative solutions seem like utter nonsense.
- joshuahedlund 4y ago
  I guess my intuition is based on the file size of text being so much smaller than images, but I guess that doesn't really map to the complexity of generating it. Fascinating!
  - davnn 4y ago
    I think large language models are still in their infancy. The models are extremely sparse, but we don‘t have the tooling yet to deal with these kinds of structures efficiently. Your intuition might be right in a future, maybe.
  - drusepth 4y ago
    If you think about the space both models are covering from a rate-of-failure perspective, it kind of makes sense that images end up being a bit easier than text: both text- and image-models can output results that look plausible at first glance, but when you analyze both outputs further there are a lot more gotchas in parsing meaning within language than there are in pixel placement within an image.
    - behnamoh 4y ago
      Actually, I'd argue that images generated by SD have much more flaws than texts produced by GPT-3. GPT-3 (at least, the full model) is quite capable of writing stuff that "make sense", but most eye candy results generated by SD are cherry picked, the others are simply crap.
      - macrolime 4y ago
        That's kinda the point here I think. GPT-3 is trained on much more data than SD and contains much more knowledge. SD is actually similar in size to GPT-2. Image models of the same size as GPT-3 should be much more impressive, the difference will probably be quite large like the difference between GPT-2 and GPT-3. Ask SD to write a step by step guide to do something and it will create an image that looks kinda like some instructions, but the contents will be nonsense. An image model of the size of GPT-3 could probably do this task quite well in many cases. Image models needs much better language understanding to get to the next level also though, so probably multimodal models may make more sense. Maybe feeding web pages rendered as images to an image model could give interesting results.
spijdar 4y ago
There is, called "GPT-NeoX", or its TPU-based predecessor GPT-Neo. However, even running inference on these models is much, much harder than Stable Diffusion -- the GPT-NeoX-20B weights for GPT-NeoX requires a minimum of two GPUs with 24 GB of VRAM each to simply run inference, never-mind training or fine-tuning. I believe there are some tricks for cutting down the VRAM requirements a bit by dropping precision at different points, but the gist is that these big text models are actually quite a bit more resource intensive than the image models.
- Tepix 4y ago
  Where can i find more information about running GPT-NeoX? Is it covered in the paper? Or is there a forum, HOWTO or Wiki somewhere?
  - juliensalinas 4y ago
    You could follow EleutherAI's official guide: https://github.com/EleutherAI/gpt-neox https://github.com/EleutherAI/gpt-neox You could also use a hosted service that proposes GPT-NeoX like https://nlpcloud.com https://nlpcloud.com or https://goose.ai https://goose.ai
- tinco 4y ago
  Interesting that they're saying they have a target of replicating GPT-3 with 175B parameters, but since the RETRO paper it should (in theory?) be possible to replicate GPT-3 with a tenth of the parameters. Are they not planning to use a retrieval system, is it difficult to adopt their system for retrieval, or is their readme out of date?
- deleted 4y ago
  [deleted]
jncraton 4y ago
There are smaller models that you can freely play around with that work in roughly the same way. If you're working on a fairly regular computer some reasonable options are GPT2 or GPT-Neo. These can both perform inference on your local CPU if you have 8GB or more of RAM. They are much less powerful that GPT-3, but they can still be fun for simple text generation or NLP tasks. You can play around with one of the smaller GPT-Neo models that should fit in RAM if run locally here: https://huggingface.co/EleutherAI/gpt-neo-1.3B https://huggingface.co/EleutherAI/gpt-neo-1.3B That page includes instructions to run this locally in Python. As others mentioned, there are larger models available, but they tend to be expensive to setup and use as an individual.
gamegoblin 4y ago
Emad Mostaque, CEO of Stability AI (behind Stable Diffusion) has said they are training language models that they plan on releasing. IIRC, though, even using all the most up-to-date techniques like 1-byte parameter quantization and Chinchilla scaling learnings, it will still probably be on the order of magnitude of 64GB or something, so not quite usable for most people locally yet, unless you happen to have a very beefy multi-GPU machine at home.
- ctoth 4y ago
  GLM-130b is available and uses 8-bit quantization, was trained on 400 billion tokens, and runs on 4x3090, which, of course, is why all the cheap Vast.AI instances are gone :)
- ShamelessC 4y ago
  Seems unlikely Emad would release
visarga 4y ago
Google recently released the Flan-T5 models that are very efficient, they rival GPT-3 and can run on a normal desktop. You can try them out. https://huggingface.co/google/flan-t5-xxl?text=Translate+to+German%3A++My+name+is+Arthur https://huggingface.co/google/flan-t5-xxl?text=Translate+to+...
dylan604 4y ago
Great. Soon, we'll have scripts written in GPT non-sense with actors and other imagery generated by stable diffusion. I'm guessing they'll still charge $15 prices at the theater for it.

espadrine 4y ago

BLOOM[0] is another 176B multilingual open-source model whose weights were published. However, running it locally requires 8 A6000 GPUs. The GPU cost alone is roughly $37K. Also, there is the concern of performance. The Chinchilla[1] paper indicates that most model sizes trained so far received too little token training (or were too big) to reach an optimal performance per energy expenditure. Furthermore, the InstructGPT[2] and more recently Flan[3] papers show that various finetuning techniques achieve significant gains, without which the result is a bit meh. The dark secret of GPT-3 is that the model available on the OpenAI website is heavily fine-tuned, even compared to the InstructGPT paper. [0]: https://huggingface.co/bigscience/bloom https://huggingface.co/bigscience/bloom [1]: https://arxiv.org/abs/2203.15556 https://arxiv.org/abs/2203.15556 [2]: https://arxiv.org/abs/2203.02155 https://arxiv.org/abs/2203.02155 [3]: https://arxiv.org/pdf/2210.11416.pdf https://arxiv.org/pdf/2210.11416.pdf

davnn 4y ago

There are multiple open-source GPTs, but GPT-3 is absolutely massive - larger than the image models actually! So, unfortunately, text generation is probably even more complex and resource intensive than image generation (especially to train). Additionally, in image generation, we appreciate the creativity of solutions, but in text generation the creative solutions seem like utter nonsense.

joshuahedlund 4y ago
I guess my intuition is based on the file size of text being so much smaller than images, but I guess that doesn't really map to the complexity of generating it. Fascinating!
- davnn 4y ago
  I think large language models are still in their infancy. The models are extremely sparse, but we don‘t have the tooling yet to deal with these kinds of structures efficiently. Your intuition might be right in a future, maybe.
- drusepth 4y ago
  If you think about the space both models are covering from a rate-of-failure perspective, it kind of makes sense that images end up being a bit easier than text: both text- and image-models can output results that look plausible at first glance, but when you analyze both outputs further there are a lot more gotchas in parsing meaning within language than there are in pixel placement within an image.
  - behnamoh 4y ago
    Actually, I'd argue that images generated by SD have much more flaws than texts produced by GPT-3. GPT-3 (at least, the full model) is quite capable of writing stuff that "make sense", but most eye candy results generated by SD are cherry picked, the others are simply crap.
    - macrolime 4y ago
      That's kinda the point here I think. GPT-3 is trained on much more data than SD and contains much more knowledge. SD is actually similar in size to GPT-2. Image models of the same size as GPT-3 should be much more impressive, the difference will probably be quite large like the difference between GPT-2 and GPT-3. Ask SD to write a step by step guide to do something and it will create an image that looks kinda like some instructions, but the contents will be nonsense. An image model of the size of GPT-3 could probably do this task quite well in many cases. Image models needs much better language understanding to get to the next level also though, so probably multimodal models may make more sense. Maybe feeding web pages rendered as images to an image model could give interesting results.

spijdar 4y ago

There is, called "GPT-NeoX", or its TPU-based predecessor GPT-Neo. However, even running inference on these models is much, much harder than Stable Diffusion -- the GPT-NeoX-20B weights for GPT-NeoX requires a minimum of two GPUs with 24 GB of VRAM each to simply run inference, never-mind training or fine-tuning. I believe there are some tricks for cutting down the VRAM requirements a bit by dropping precision at different points, but the gist is that these big text models are actually quite a bit more resource intensive than the image models.

Tepix 4y ago
Where can i find more information about running GPT-NeoX? Is it covered in the paper? Or is there a forum, HOWTO or Wiki somewhere?
- juliensalinas 4y ago
  You could follow EleutherAI's official guide: https://github.com/EleutherAI/gpt-neox https://github.com/EleutherAI/gpt-neox You could also use a hosted service that proposes GPT-NeoX like https://nlpcloud.com https://nlpcloud.com or https://goose.ai https://goose.ai
tinco 4y ago
Interesting that they're saying they have a target of replicating GPT-3 with 175B parameters, but since the RETRO paper it should (in theory?) be possible to replicate GPT-3 with a tenth of the parameters. Are they not planning to use a retrieval system, is it difficult to adopt their system for retrieval, or is their readme out of date?
deleted 4y ago
[deleted]

jncraton 4y ago

There are smaller models that you can freely play around with that work in roughly the same way. If you're working on a fairly regular computer some reasonable options are GPT2 or GPT-Neo. These can both perform inference on your local CPU if you have 8GB or more of RAM. They are much less powerful that GPT-3, but they can still be fun for simple text generation or NLP tasks. You can play around with one of the smaller GPT-Neo models that should fit in RAM if run locally here: https://huggingface.co/EleutherAI/gpt-neo-1.3B https://huggingface.co/EleutherAI/gpt-neo-1.3B That page includes instructions to run this locally in Python. As others mentioned, there are larger models available, but they tend to be expensive to setup and use as an individual.

gamegoblin 4y ago

Emad Mostaque, CEO of Stability AI (behind Stable Diffusion) has said they are training language models that they plan on releasing. IIRC, though, even using all the most up-to-date techniques like 1-byte parameter quantization and Chinchilla scaling learnings, it will still probably be on the order of magnitude of 64GB or something, so not quite usable for most people locally yet, unless you happen to have a very beefy multi-GPU machine at home.

ctoth 4y ago
GLM-130b is available and uses 8-bit quantization, was trained on 400 billion tokens, and runs on 4x3090, which, of course, is why all the cheap Vast.AI instances are gone :)
ShamelessC 4y ago
Seems unlikely Emad would release

visarga 4y ago

Google recently released the Flan-T5 models that are very efficient, they rival GPT-3 and can run on a normal desktop. You can try them out. https://huggingface.co/google/flan-t5-xxl?text=Translate+to+German%3A++My+name+is+Arthur https://huggingface.co/google/flan-t5-xxl?text=Translate+to+...

dylan604 4y ago

Great. Soon, we'll have scripts written in GPT non-sense with actors and other imagery generated by stable diffusion. I'm guessing they'll still charge $15 prices at the theater for it.