What sort of compute do you get access to? There's a lot of cool stuff you could do depending on whether or not you have decent GPU's and for how much time you'

What sort of compute do you get access to? There's a lot of cool stuff you could do depending on whether or not you have decent GPU's and for how much time you're allowed to experiment on them. Experimentation is fairly fundamental in practice. There are a lot of pretraining tasks in vision/multimodal that are cool. Largely techniques introduced or refined by OpenAI re-implemented as pytorch open source codebases with varying degrees of success: - Finetune your own CLIP https://github.com/mlfoundations/open_clip https://github.com/mlfoundations/open_clip - Train a (much smaller) DALLE https://github.com/lucidrains/DALLE-pytorch https://github.com/lucidrains/DALLE-pytorch - Train your own guided diffusion https://colab.research.google.com/drive/1javQRTkALBWLFWnx1K4VpRZkWLP3ozhr https://colab.research.google.com/drive/1javQRTkALBWLFWnx1K4... (pretty tough, may only be feasible on domain-specific data) - Train a variational autoencoder (VAE) - "VQGAN" from Heidelberg https://github.com/CompVis/taming-transformers https://github.com/CompVis/taming-transformers - "Discrete VAE", used as the backbone for OpenAI's DALL-E, reimplimented here (and other places) https://github.com/lucidrains/DALLE-pytorch https://github.com/lucidrains/DALLE-pytorch - "VQVAE2" https://github.com/tgisaturday/dalle-lightning https://github.com/tgisaturday/dalle-lightning