10 ms·

Emad (founder of Stability AI) has said they already have video model training underway, as well as text and audio. Exciting times.

ItsMonkk 4y ago
Is this going to end up into a single model, where its trained on text and images and audio and videos and 3d models, and it can do anything to anything depending on what you ask of it? Feels like the cross-training would help yield stronger results.
- minimaxir 4y ago
  These diffusion models are using a frozen text encoder (e.g. CLIP for Stable Diffusion, T5 for Imagen), which can be used in other applications. StabilityAI trained a new/better CLIP for the purpose of better Stable Diffusions.
- CuriouslyC 4y ago
  Probably not. We're actually headed towards many smaller models that call each other, because VRAM is the limiting factor in application, and if the domains aren't totally dependent on each other it's easier to have one model produce bad output, then detect that bad output and feed it into another model that cleans up the problem (like fixing faces in stable diffusion output). The human brain is modularized like this, so I don't think it'll be a limitation.
- deleted 4y ago
  [deleted]
rch 4y ago
And copilot-like code, possibly Q1 2023.
- RosanaAnaDana 4y ago
  "Generate the code base for an advanced diffusion model that can improve on the code base for an advanced diffusion model"
  - danuker 4y ago
    The road to Grey Goo is paved with artificial general intelligence.
  - TaylorAlexander 4y ago
    Oh no you forgot the important term “but do NOT start turning the universe in to paperclips”.
- moyix 4y ago
  Salesforce CodeGen (particularly the 16B-multi and 16B-mono models) is pretty good already and can be used with FauxPilot [1] to get an open Copilot-like experience with local compute :) I am also very excited about the upcoming BigCode project though, which is maybe what you're thinking of? Disclaimer: I am naturally biased since I made FauxPilot ;) [1] https://github.com/moyix/fauxpilot https://github.com/moyix/fauxpilot [2] https://www.bigcode-project.org/ https://www.bigcode-project.org/

ItsMonkk 4y ago

Is this going to end up into a single model, where its trained on text and images and audio and videos and 3d models, and it can do anything to anything depending on what you ask of it? Feels like the cross-training would help yield stronger results.

minimaxir 4y ago
These diffusion models are using a frozen text encoder (e.g. CLIP for Stable Diffusion, T5 for Imagen), which can be used in other applications. StabilityAI trained a new/better CLIP for the purpose of better Stable Diffusions.
CuriouslyC 4y ago
Probably not. We're actually headed towards many smaller models that call each other, because VRAM is the limiting factor in application, and if the domains aren't totally dependent on each other it's easier to have one model produce bad output, then detect that bad output and feed it into another model that cleans up the problem (like fixing faces in stable diffusion output). The human brain is modularized like this, so I don't think it'll be a limitation.
deleted 4y ago
[deleted]

rch 4y ago

And copilot-like code, possibly Q1 2023.

RosanaAnaDana 4y ago
"Generate the code base for an advanced diffusion model that can improve on the code base for an advanced diffusion model"
- danuker 4y ago
  The road to Grey Goo is paved with artificial general intelligence.
- TaylorAlexander 4y ago
  Oh no you forgot the important term “but do NOT start turning the universe in to paperclips”.
moyix 4y ago
Salesforce CodeGen (particularly the 16B-multi and 16B-mono models) is pretty good already and can be used with FauxPilot [1] to get an open Copilot-like experience with local compute :) I am also very excited about the upcoming BigCode project though, which is maybe what you're thinking of? Disclaimer: I am naturally biased since I made FauxPilot ;) [1] https://github.com/moyix/fauxpilot https://github.com/moyix/fauxpilot [2] https://www.bigcode-project.org/ https://www.bigcode-project.org/