7 ms·

Is there a reason why the pretrained models are not included as well? Or perhaps this is coming at some point in the future? There are many users who simply wa

Is there a reason why the pretrained models are not included as well? Or perhaps this is coming at some point in the future? There are many users who simply want to play around with the functionality, who lack the time and hardware required to replicate the training steps.

nl 4y ago
It literally says: > To train CLIP, you can either use x-clip package, or join the LAION discord, where a lot of replication efforts are already underway.
kvathupo 4y ago
As someone who has trained transformers with only ~5 layers on 1000 data point inputs for a decoder of length like 10 (so far fewer than the 12 billion parameters of DALL-E [1]), I learned two things: 1. Training is very finicky and time-consuming. 2. PyTorch model files are pretty large. For point 1, when I was training a different autoencoder for example, I would run it for a week on 4 GPUs, only to return and see that its results are subpar. For point 2, I would get 100 MB model files for a miniscule transformer (relative to DALL-E numbers). Combined with the strong dependence of the transformer on the training data, this can make it problematic when it comes to sharing pre-trained models for tasks that the user doesn't specify a priori. [1] - https://venturebeat.com/2021/01/05/openai-debuts-dall-e-for-generating-images-from-text/ https://venturebeat.com/2021/01/05/openai-debuts-dall-e-for-...
- Uehreka 4y ago
  I will gladly download a terabyte of model data if that’s what it would take, your numbers don’t scare me!
  - webmaven 4y ago
    Large models generally require beefier GPUs and more RAM. Wide (not just deep) models require more GPU cores, total model size requires more RAM. Some models require multiple GPUs to train and/or run. Some require non-consumer server GPUs. These are non-trivial hardware investments if purchased, and non-trivial cloud bills if rented, even at preemptible spot pricing.
    - brobinson 4y ago
      I'd pay money for pretrained models that can run on consumer gaming GPUs (8gb vram) for things like this. I wonder if anyone is selling these.
      - moffkalast 4y ago
        The main issue is that what neural nets are doing is just very fancy heuristic compression of vast amounts of data into a few gigs of vram (and a useful way to access it), so the smaller the model is the less it'll be able to do.
        brobinson 4y ago
        Most of the pretrained models I see for the stuff I'm working on (Coqui, ESRGAN, etc.) are really tiny in size. I'm sure the much better models are bigger but still fit under 8gb vram. It seems like there could be a market for models that fit into 8gb vram, but I'm not familiar enough with users of ML to say for sure.
sailingparrot 4y ago
> Is there a reason why the pretrained models are not included as well? Because every one else, too, lacks the time and hardware required to replicate the training steps. There is just not that many organizations that can pay for 100-200 thousands hours of training on A100 GPUs. So the reason why the pretained model is not included, is that it does not exist (yet).
- newswasboring 4y ago
  > There is just not that many organizations that can pay for 100-200 thousands hours of training on A100 GPUs. How much is this in money?
  - sailingparrot 4y ago
    With no discount, one A100 is about 3$/hour, so we are talking about 600K$ for a full training. Probably closer to 1M$ since in practice you never train successfully to the end on the first shot.
  - isoprophlex 4y ago
    IIRC a K80 goes for prices between 0.3 and 3 USD per GPU-hour Pretty expensive all in all
  - malka 4y ago
    between a a few hundred K and a few millions IMO
- lagrange77 4y ago
  There should be a system in place like Folding@Home or SETI@Home for situations like this.
  - HanClinto 4y ago
    Agreed. This feels like the next big step in open source machine learning. I would love to be able to dedicate CPU / GPU towards chunking out open-source GPT-3 / DALL-E 2 clones so long as the trained weights are shared with the world afterwards. That feels more long-term productive to me than crunching coins.

nl 4y ago

It literally says: > To train CLIP, you can either use x-clip package, or join the LAION discord, where a lot of replication efforts are already underway.

kvathupo 4y ago

As someone who has trained transformers with only ~5 layers on 1000 data point inputs for a decoder of length like 10 (so far fewer than the 12 billion parameters of DALL-E [1]), I learned two things: 1. Training is very finicky and time-consuming. 2. PyTorch model files are pretty large. For point 1, when I was training a different autoencoder for example, I would run it for a week on 4 GPUs, only to return and see that its results are subpar. For point 2, I would get 100 MB model files for a miniscule transformer (relative to DALL-E numbers). Combined with the strong dependence of the transformer on the training data, this can make it problematic when it comes to sharing pre-trained models for tasks that the user doesn't specify a priori. [1] - https://venturebeat.com/2021/01/05/openai-debuts-dall-e-for-generating-images-from-text/ https://venturebeat.com/2021/01/05/openai-debuts-dall-e-for-...

Uehreka 4y ago
I will gladly download a terabyte of model data if that’s what it would take, your numbers don’t scare me!
- webmaven 4y ago
  Large models generally require beefier GPUs and more RAM. Wide (not just deep) models require more GPU cores, total model size requires more RAM. Some models require multiple GPUs to train and/or run. Some require non-consumer server GPUs. These are non-trivial hardware investments if purchased, and non-trivial cloud bills if rented, even at preemptible spot pricing.
  - brobinson 4y ago
    I'd pay money for pretrained models that can run on consumer gaming GPUs (8gb vram) for things like this. I wonder if anyone is selling these.
    - moffkalast 4y ago
      The main issue is that what neural nets are doing is just very fancy heuristic compression of vast amounts of data into a few gigs of vram (and a useful way to access it), so the smaller the model is the less it'll be able to do.
      - brobinson 4y ago
        Most of the pretrained models I see for the stuff I'm working on (Coqui, ESRGAN, etc.) are really tiny in size. I'm sure the much better models are bigger but still fit under 8gb vram. It seems like there could be a market for models that fit into 8gb vram, but I'm not familiar enough with users of ML to say for sure.

sailingparrot 4y ago

> Is there a reason why the pretrained models are not included as well? Because every one else, too, lacks the time and hardware required to replicate the training steps. There is just not that many organizations that can pay for 100-200 thousands hours of training on A100 GPUs. So the reason why the pretained model is not included, is that it does not exist (yet).

newswasboring 4y ago
> There is just not that many organizations that can pay for 100-200 thousands hours of training on A100 GPUs. How much is this in money?
- sailingparrot 4y ago
  With no discount, one A100 is about 3$/hour, so we are talking about 600K$ for a full training. Probably closer to 1M$ since in practice you never train successfully to the end on the first shot.
- isoprophlex 4y ago
  IIRC a K80 goes for prices between 0.3 and 3 USD per GPU-hour Pretty expensive all in all
- malka 4y ago
  between a a few hundred K and a few millions IMO
lagrange77 4y ago
There should be a system in place like Folding@Home or SETI@Home for situations like this.
- HanClinto 4y ago
  Agreed. This feels like the next big step in open source machine learning. I would love to be able to dedicate CPU / GPU towards chunking out open-source GPT-3 / DALL-E 2 clones so long as the trained weights are shared with the world afterwards. That feels more long-term productive to me than crunching coins.