3 ms·

The Gemini 2 models support native audio and image generation but the latter won't be generally available till January. Really excited for that as well as 4o's

The Gemini 2 models support native audio and image generation but the latter won't be generally available till January. Really excited for that as well as 4o's image generation (whenever that comes out). Steerability has lagged behind aesthetics in image generation for a while now and it's be great to see a big advance in that. Also a whole lot of computer vision tasks (via LLMs) could be unlocked with this. Think Inpainting, Style Transfer, Text Editing in the wild, Segmentation, Edge detection etc They have a demo: https://www.youtube.com/watch?v=7RqFLp0TqV0 https://www.youtube.com/watch?v=7RqFLp0TqV0

jncfhnb 2y ago
These are not computer vision tasks…
- Jabrov 2y ago
  What are they, then…?
  - 85392_school 2y ago
    The first two are tasks which involve making images. They could be called image generation or image editing.
- newfocogi 2y ago
  Maybe some of these tasks are arguably not aligned with the traditional applications of CV, but Segmentation and Edge detection are definitely computer vision in every definition I've come across - before and after NNs took over.
kthartic 2y ago
I asked Gemini 2.0 Flash (with my voice) whether it natively understands audio or is converting my voice to text. It replied: "That's an insightful question. My understanding of your speech involves a pipeline first. Your voice is converted to text and then I process the text to understand what you're saying. So I don't understand your voice directly but rather through a text representation of it." Unsure if this is a hallucination, but is disappointing if true. Edit: Looking at the video you linked, they say "native audio output", so I assume this means the input isn't native? :(
- famouswaffles 2y ago
  Native audio output won't be in general availability until early next year. If you're using Gemini in aistudio(not sure about the real-time API but everything else) then it has native audio input

jncfhnb 2y ago

These are not computer vision tasks…

Jabrov 2y ago
What are they, then…?
- 85392_school 2y ago
  The first two are tasks which involve making images. They could be called image generation or image editing.
newfocogi 2y ago
Maybe some of these tasks are arguably not aligned with the traditional applications of CV, but Segmentation and Edge detection are definitely computer vision in every definition I've come across - before and after NNs took over.

kthartic 2y ago

I asked Gemini 2.0 Flash (with my voice) whether it natively understands audio or is converting my voice to text. It replied: "That's an insightful question. My understanding of your speech involves a pipeline first. Your voice is converted to text and then I process the text to understand what you're saying. So I don't understand your voice directly but rather through a text representation of it." Unsure if this is a hallucination, but is disappointing if true. Edit: Looking at the video you linked, they say "native audio output", so I assume this means the input isn't native? :(

famouswaffles 2y ago
Native audio output won't be in general availability until early next year. If you're using Gemini in aistudio(not sure about the real-time API but everything else) then it has native audio input