4 ms·
As someone who has trained transformers with only ~5 layers on 1000 data point inputs for a decoder of length like 10 (so far fewer than the 12 billion paramete
by kvathupo 4y ago
As someone who has trained transformers with only ~5 layers on 1000 data point inputs for a decoder of length like 10 (so far fewer than the 12 billion parameters of DALL-E [1]), I learned two things:
1. Training is very finicky and time-consuming.
2. PyTorch model files are pretty large.
For point 1, when I was training a different autoencoder for example, I would run it for a week on 4 GPUs, only to return and see that its results are subpar. For point 2, I would get 100 MB model files for a miniscule transformer (relative to DALL-E numbers). Combined with the strong dependence of the transformer on the training data, this can make it problematic when it comes to sharing pre-trained models for tasks that the user doesn't specify a priori.
[1] - https://venturebeat.com/2021/01/05/openai-debuts-dall-e-for-generating-images-from-text/ https://venturebeat.com/2021/01/05/openai-debuts-dall-e-for-...
- Uehreka 4y agoI will gladly download a terabyte of model data if that’s what it would take, your numbers don’t scare me!
- webmaven 4y agoLarge models generally require beefier GPUs and more RAM. Wide (not just deep) models require more GPU cores, total model size requires more RAM. Some models require multiple GPUs to train and/or run. Some require non-consumer server GPUs. These are non-trivial hardware investments if purchased, and non-trivial cloud bills if rented, even at preemptible spot pricing.
- brobinson 4y agoI'd pay money for pretrained models that can run on consumer gaming GPUs (8gb vram) for things like this. I wonder if anyone is selling these.
- moffkalast 4y agoThe main issue is that what neural nets are doing is just very fancy heuristic compression of vast amounts of data into a few gigs of vram (and a useful way to access it), so the smaller the model is the less it'll be able to do.
- brobinson 4y agoMost of the pretrained models I see for the stuff I'm working on (Coqui, ESRGAN, etc.) are really tiny in size. I'm sure the much better models are bigger but still fit under 8gb vram. It seems like there could be a market for models that fit into 8gb vram, but I'm not familiar enough with users of ML to say for sure.