3 ms·
GPT-4 is actually multimodal. It can't be publicly prompted with images yet, but that's planned. I think that multimodal training will also solve the data sho
by benlivengood 4y ago
GPT-4 is actually multimodal. It can't be publicly prompted with images yet, but that's planned.
I think that multimodal training will also solve the data shortage problem. There are hundreds of times more bytes in video and audio than there are in text, so we'll likely be able to scale pre-training for quite a while before needing to go fully embodied on Transformers.
- famouswaffles 4y agoThey trained a text only model first and then made it multimodal. The experiments I was referring to (spatial understanding) were done with the text only model.
- est 4y ago> GPT-4 is actually multimodal The released version. Microsoft's paper was talking about an early internal text-only version.
- Hendrikto 4y ago> There are hundreds of times more bytes in video and audio than there are in text But the information density is similarly hundreds to thousand times less.
- benlivengood 4y agoI'm not actually sure that's true. There is a lot of detail in the world represented in audio and video, and presumably large transformers could learn from the textures and shadows and articulated movements and the physical modeling of how sounds are made, etc.