6 ms·

AudioPaLM: A large language model that can speak and listen

69 points by sheepscreek 3y ago

brucethemoose2 3y ago
Uh oh, here come the multimedia multimodels. I see a lot of talk about transformers LLMs being close to "topping out," which I am skeptical of for many reasons, but not the least of which is prompting/outputs other than pure text.
- mupuff1234 3y ago
  Is there really a difference between a multimodal vs a text LLM + stable diffusion?
  - valine 3y ago
    Multimodal can refer to a lot of different types of models, but feeding LLM text into stable diffusion definitely doesn’t count. LLaVA is the first one that comes to my mind, it takes images and text as input and outputs text. There’s an unreleased version of GPT4 that can do that same thing.
    - mupuff1234 3y ago
      Sure technically not the same, but won't there be the same affect? How do our brains work? Isn't there a separation between image and text processing?
      - valine 3y ago
        Surely there needs to be some amount of training with both models in the loop before it can be considered a multimodal system.
        lee101 3y ago
        [dead]
  - jamilton 3y ago
    What do you mean?
    - mupuff1234 3y ago
      That if you have 3 separate models, text -> text, image -> text and text -> image you can just glue them together to make it behave like a multimodal. (Just like gpt4 is rumored to be a few different sub models and not just one giant model)
valine 3y ago
Preserving the speakers voice after translation is super cool. It’s one of the things you don’t really think about, but voice inflection and identity is missing with our current translation tools. I like to watch political speeches from non english speaking politicians, and the speakers tone can easily be lost in translation. Emphasis is hard to discern when you don’t know which spoken word maps to which word in the subtitles. Dubbed speeches are even worse in that respect.
rhogar 3y ago
Additional discussion from post two days ago: https://news.ycombinator.com/item?id=36443676 https://news.ycombinator.com/item?id=36443676
ml_basics 3y ago
Demo website with speech-speech translation examples https://google-research.github.io/seanet/audiopalm/examples/ https://google-research.github.io/seanet/audiopalm/examples/
leobg 3y ago
Absolutely amazing. Anything like this available to play with?

69 points by sheepscreek 3y ago

Uh oh, here come the multimedia multimodels. I see a lot of talk about transformers LLMs being close to "topping out," which I am skeptical of for many reasons, but not the least of which is prompting/outputs other than pure text.

mupuff1234 3y ago
Is there really a difference between a multimodal vs a text LLM + stable diffusion?
- valine 3y ago
  Multimodal can refer to a lot of different types of models, but feeding LLM text into stable diffusion definitely doesn’t count. LLaVA is the first one that comes to my mind, it takes images and text as input and outputs text. There’s an unreleased version of GPT4 that can do that same thing.
  - mupuff1234 3y ago
    Sure technically not the same, but won't there be the same affect? How do our brains work? Isn't there a separation between image and text processing?
    - valine 3y ago
      Surely there needs to be some amount of training with both models in the loop before it can be considered a multimodal system.
      - lee101 3y ago
        [dead]
- jamilton 3y ago
  What do you mean?
  - mupuff1234 3y ago
    That if you have 3 separate models, text -> text, image -> text and text -> image you can just glue them together to make it behave like a multimodal. (Just like gpt4 is rumored to be a few different sub models and not just one giant model)

valine 3y ago

Preserving the speakers voice after translation is super cool. It’s one of the things you don’t really think about, but voice inflection and identity is missing with our current translation tools. I like to watch political speeches from non english speaking politicians, and the speakers tone can easily be lost in translation. Emphasis is hard to discern when you don’t know which spoken word maps to which word in the subtitles. Dubbed speeches are even worse in that respect.

rhogar 3y ago

Additional discussion from post two days ago: https://news.ycombinator.com/item?id=36443676 https://news.ycombinator.com/item?id=36443676

ml_basics 3y ago

Demo website with speech-speech translation examples https://google-research.github.io/seanet/audiopalm/examples/ https://google-research.github.io/seanet/audiopalm/examples/

leobg 3y ago

Absolutely amazing. Anything like this available to play with?