5 ms·
The model is open sourced on GitHub: https://github.com/openai/image-gpt https://github.com/openai/image-gpt Oddly, it still uses TensorFlow like the original
by minimaxir 6y ago
The model is open sourced on GitHub: https://github.com/openai/image-gpt https://github.com/openai/image-gpt
Oddly, it still uses TensorFlow like the original GPT-2 release despite OpenAI's declared switch to PyTorch, and it has dependency hell so it's not easy to create a wrapper tool for it.
Since it's still the GPT-2 architecture, it might be possible to port the weights to Huggingface Transformers (for the RGB generation), and then write a wrapper to extend it for the image rendering. (filed an issue here: https://github.com/huggingface/transformers/issues/5088 https://github.com/huggingface/transformers/issues/5088 )
- elAhmo 6y agoIt is quite amazing that this model is so concise and can be expressed with just a few hundred lines of code. (similar models as well, in general) Looking at the samples I am amazed by some of the results.
- gwern 6y ago> Oddly, it still uses TensorFlow like the original GPT-2 release despite OpenAI's declared switch to PyTorch Note the footnote in the paper where evaluations was interrupted by the move to MS Azure. This is relatively old work, and since it's literally GPT-2 but images, it's no surprise they didn't bother to rewrite it in PyTorch. If their next big from-scratch thing (presumably the multimodal GPT-like one-model-to-rule-them-all research I've been looking forward to ever since the TR article) is TensorFlow, then I'll be surprised.
- sillysaurusx 6y agoUnfortunately, there probably won’t be a next big-from-scratch thing unless it ties into their API somehow. It feels like we’ve seen the last of OpenAI open sourcing anything but toy models. Of course, I would be delighted to be mistaken.
- bigdict 6y agoThey say they used 1024 TPUs for training, maybe TF has better perf on that hardware at this point. EDIT: v2 of the paper says 2048 TPUs.
- AaronFriel 6y agoI'm still learning from deep learning papers and videos before dipping my toes in myself, is there a summary of why PyTorch vs TensorFlow? Does it matter for me?
- minimaxir 6y agoNowadays, there isn't a huge practical difference in terms of performance/tooling aside from edge cases and deployment options. It mostly depends on your syntax preference. (although there are flame wars from both sides) I noted the TensorFlow usage because the original GPT-2 release was TensorFlow 1.X, which led to issues when TensorFlow 2.0 was released soon after. For model training, I strongly recommend using the higher-level Keras APIs for TensorFlow/pytorch-lightning API for PyTorch than the respective base tools.
- m0zg 6y agoThere is a huge difference in developer experience still. When TF fails it typically offers little to no clue as to what you can do to rectify the situation. PyTorch is a lot more helpful most of the time. PyTorch is worth choosing on this basis alone, unless you intend to deploy to mobile hardware, where TFLite is the only real viable option.
- fxtentacle 6y agoTF can compile to optimized C++. When you use PyTorch with TPU, it'll internally use the TF XLA backend. If you intend to deploy on mobile, TF is easy and PyTorch is pain. But TF has obscure bugs and error messages.
- deleted 6y ago[deleted]
- currymj 6y agojust use PyTorch, it’s generally going to be the nicest. TF2 is much nicer than TF1 but most of the changes were just to make it more like PyTorch.