4 ms·

What kind of computer configuration does it require to train a model like this? Speaking as noob, is it possible to create models like this on your average Ryze

What kind of computer configuration does it require to train a model like this? Speaking as noob, is it possible to create models like this on your average Ryzen 5 / GTX 1060 6GB type PC? If so, how much time would it take? I feel really excited to experiment on such projects in my spare time but reading how it costs researchers hundreds and thousands of dollar don't even try.

brujoand 6y ago
Not sure, but I suspect a google coral might be able to do this?
Aeolun 6y ago
It seems this was trained for two months! There is a blog post by the author somewhere else in this thread.
gwern 6y ago
If you don't have TFRC access, this is out of the question as a hobbyist. It's >32 GPU-months. Sorry. However, you can still transfer learn this particular model with <1 GPU-month, if you have some anime or artwork dataset you want (eg a specific character). The good news is that there's been huge progress in GAN efficiency over the past few years, and we are rapidly approaching the point where ImageNet-scale datasets can be reasonably trained with 1 GPU-month or less. Worth noting in that vein are VQ-GAN https://compvis.github.io/taming-transformers/ https://compvis.github.io/taming-transformers/ and not-so-BigGAN https://arxiv.org/abs/2009.04433 https://arxiv.org/abs/2009.04433 . There's also a resurgence of VAE work which is relatively efficient: VD-VAE https://arxiv.org/abs/2011.10650#openai https://arxiv.org/abs/2011.10650#openai and NVAE https://arxiv.org/abs/2007.03898#nvidia https://arxiv.org/abs/2007.03898#nvidia ; DALL-E https://openai.com/blog/dall-e/ https://openai.com/blog/dall-e/ also appears to be surprisingly lightweight on the VAE side. So, if you feel left out, don't worry. In the near future, between the RTX GPUs and optimization and new VAE/GANs, I expect things like Danbooru2020 https://www.gwern.net/Danbooru2020 https://www.gwern.net/Danbooru2020 to be totally doable on hobbyist hardware.
- ghoomketu 6y ago
  Amazing project and thank you so much for the detailed answer. In case you read it and I know it's too much to ask of you, but you seem to be an expert at ML, so what resources would you recommend me to read up on for a hobbyist who wants to create a project like this (I mean this x does not exist type or similar). Somebody recommend Andrew NG courses but I wasn't able to complete it and got bored. Then I tried watching sentdex and it was amazing and I'm able to fully comprehend what he does since he teaches more practical aspects like running gta v on autopilot etc. So just wondering what I could learn next.

brujoand 6y ago

Not sure, but I suspect a google coral might be able to do this?

Aeolun 6y ago

It seems this was trained for two months! There is a blog post by the author somewhere else in this thread.

gwern 6y ago

If you don't have TFRC access, this is out of the question as a hobbyist. It's >32 GPU-months. Sorry. However, you can still transfer learn this particular model with <1 GPU-month, if you have some anime or artwork dataset you want (eg a specific character). The good news is that there's been huge progress in GAN efficiency over the past few years, and we are rapidly approaching the point where ImageNet-scale datasets can be reasonably trained with 1 GPU-month or less. Worth noting in that vein are VQ-GAN https://compvis.github.io/taming-transformers/ https://compvis.github.io/taming-transformers/ and not-so-BigGAN https://arxiv.org/abs/2009.04433 https://arxiv.org/abs/2009.04433 . There's also a resurgence of VAE work which is relatively efficient: VD-VAE https://arxiv.org/abs/2011.10650#openai https://arxiv.org/abs/2011.10650#openai and NVAE https://arxiv.org/abs/2007.03898#nvidia https://arxiv.org/abs/2007.03898#nvidia ; DALL-E https://openai.com/blog/dall-e/ https://openai.com/blog/dall-e/ also appears to be surprisingly lightweight on the VAE side. So, if you feel left out, don't worry. In the near future, between the RTX GPUs and optimization and new VAE/GANs, I expect things like Danbooru2020 https://www.gwern.net/Danbooru2020 https://www.gwern.net/Danbooru2020 to be totally doable on hobbyist hardware.

ghoomketu 6y ago
Amazing project and thank you so much for the detailed answer. In case you read it and I know it's too much to ask of you, but you seem to be an expert at ML, so what resources would you recommend me to read up on for a hobbyist who wants to create a project like this (I mean this x does not exist type or similar). Somebody recommend Andrew NG courses but I wasn't able to complete it and got bored. Then I tried watching sentdex and it was amazing and I'm able to fully comprehend what he does since he teaches more practical aspects like running gta v on autopilot etc. So just wondering what I could learn next.