3 ms·
If you don't have TFRC access, this is out of the question as a hobbyist. It's >32 GPU-months. Sorry. However, you can still transfer learn this particular mode
by gwern 6y ago
If you don't have TFRC access, this is out of the question as a hobbyist. It's >32 GPU-months. Sorry. However, you can still transfer learn this particular model with <1 GPU-month, if you have some anime or artwork dataset you want (eg a specific character).
The good news is that there's been huge progress in GAN efficiency over the past few years, and we are rapidly approaching the point where ImageNet-scale datasets can be reasonably trained with 1 GPU-month or less.
Worth noting in that vein are VQ-GAN https://compvis.github.io/taming-transformers/ https://compvis.github.io/taming-transformers/ and not-so-BigGAN https://arxiv.org/abs/2009.04433 https://arxiv.org/abs/2009.04433 . There's also a resurgence of VAE work which is relatively efficient: VD-VAE https://arxiv.org/abs/2011.10650#openai https://arxiv.org/abs/2011.10650#openai and NVAE https://arxiv.org/abs/2007.03898#nvidia https://arxiv.org/abs/2007.03898#nvidia ; DALL-E https://openai.com/blog/dall-e/ https://openai.com/blog/dall-e/ also appears to be surprisingly lightweight on the VAE side.
So, if you feel left out, don't worry. In the near future, between the RTX GPUs and optimization and new VAE/GANs, I expect things like Danbooru2020 https://www.gwern.net/Danbooru2020 https://www.gwern.net/Danbooru2020 to be totally doable on hobbyist hardware.
- ghoomketu 6y agoAmazing project and thank you so much for the detailed answer. In case you read it and I know it's too much to ask of you, but you seem to be an expert at ML, so what resources would you recommend me to read up on for a hobbyist who wants to create a project like this (I mean this x does not exist type or similar). Somebody recommend Andrew NG courses but I wasn't able to complete it and got bored. Then I tried watching sentdex and it was amazing and I'm able to fully comprehend what he does since he teaches more practical aspects like running gta v on autopilot etc. So just wondering what I could learn next.