3 ms·

https://openai.com/blog/chatgpt-can-now-see-hear-and-speak https://openai.com/blog/chatgpt-can-now-see-hear-and-speak I think it's app only though

TaylorAlexander 3y ago
Ah that’s right. I guess my question is, is it a true multimodal model (able to produce arbitrary audio) or is it a speech to text system (OpenAI has a model called Whisper for this) feeding text to the model and then using text to speech to read it aloud. Though now that I am reading the Gemini technical report, it can only receive audio as input, it can’t produce audio as output. Still based on quickly glancing at their technical report it seems Gemini might have superior audio input capabilities. I am not sure of this though now that I think about it.
- sho_hn 3y ago
  One of the demo videos explicitly addresses this point: https://youtu.be/D64QD7Swr3s?si=_bBa9aPmqGbo-Iej https://youtu.be/D64QD7Swr3s?si=_bBa9aPmqGbo-Iej
  - TaylorAlexander 3y ago
    Oh that’s actually pretty good then. It also seems it does output audio despite the PDF from google I was reading saying otherwise. Hmm.
nojvek 3y ago
That’s different. It’s essentially using whisper model for audio to text and that inputs to ChatGPT. Multimodal would be watching YouTube without captions and asking “how did a certain character know it was raining outside?” Based on rain sound but no image of rain
- drusepth 3y ago
  I don't know if it's related to Gemini, but Bard seems to be able to do this by answering questions like "how many cups of sugar are called for in this video". Not sure if it relies on subtitles or not. From https://bard.google.com/updates https://bard.google.com/updates: > Expanding Bard’s understanding of YouTube videos > What: We're taking the first steps in Bard's ability to understand YouTube videos. For example, if you’re looking for videos on how to make olive oil cake, you can now also ask how many eggs the recipe in the first video requires. > Why: We’ve heard you want deeper engagement with YouTube videos. So we’re expanding the YouTube Extension to understand some video content so you can have a richer conversation with Bard about it.
  - nojvek 3y ago
    Interesting. Will take Bard for a spin.

TaylorAlexander 3y ago

Ah that’s right. I guess my question is, is it a true multimodal model (able to produce arbitrary audio) or is it a speech to text system (OpenAI has a model called Whisper for this) feeding text to the model and then using text to speech to read it aloud. Though now that I am reading the Gemini technical report, it can only receive audio as input, it can’t produce audio as output. Still based on quickly glancing at their technical report it seems Gemini might have superior audio input capabilities. I am not sure of this though now that I think about it.

sho_hn 3y ago
One of the demo videos explicitly addresses this point: https://youtu.be/D64QD7Swr3s?si=_bBa9aPmqGbo-Iej https://youtu.be/D64QD7Swr3s?si=_bBa9aPmqGbo-Iej
- TaylorAlexander 3y ago
  Oh that’s actually pretty good then. It also seems it does output audio despite the PDF from google I was reading saying otherwise. Hmm.

nojvek 3y ago

That’s different. It’s essentially using whisper model for audio to text and that inputs to ChatGPT. Multimodal would be watching YouTube without captions and asking “how did a certain character know it was raining outside?” Based on rain sound but no image of rain

drusepth 3y ago
I don't know if it's related to Gemini, but Bard seems to be able to do this by answering questions like "how many cups of sugar are called for in this video". Not sure if it relies on subtitles or not. From https://bard.google.com/updates https://bard.google.com/updates: > Expanding Bard’s understanding of YouTube videos > What: We're taking the first steps in Bard's ability to understand YouTube videos. For example, if you’re looking for videos on how to make olive oil cake, you can now also ask how many eggs the recipe in the first video requires. > Why: We’ve heard you want deeper engagement with YouTube videos. So we’re expanding the YouTube Extension to understand some video content so you can have a richer conversation with Bard about it.
- nojvek 3y ago
  Interesting. Will take Bard for a spin.