3 ms·
Am I correct to say this is a multi modal model using vision and audio? What model is it? And how is it understanding the image and the question? Can anyone sh
by stevofolife 3y ago
Am I correct to say this is a multi modal model using vision and audio?
What model is it? And how is it understanding the image and the question? Can anyone shed some light on this technical process?
- obscur 3y agoGPT4 is multimodal in the sense that it can take images as input. The person is using a speech to text system such as OpenAIs Whisper and serving screenshots and voice transcripts to GPT4 and GPT4 is returning a text response which is converted to speech using a text to speech system such as OpenAIs TTS API.
- stevofolife 3y agoAh got it! So basically the prompt to GPT4 is an image + text (converted from audio).