3 ms·
Does anyone know how they're doing the audio part where Mark breaths too hard? Does his breathing get turned into all-caps text (AA EE OO) and that GPT4-o inter
by christianqchung 2y ago
Does anyone know how they're doing the audio part where Mark breaths too hard? Does his breathing get turned into all-caps text (AA EE OO) and that GPT4-o interprets that as him breathing too hard, or is there something more going on?
- GalaxyNova 2y agoIt can natively interpret voice now.
- deleted 2y ago[deleted]
- deleted 2y ago[deleted]
- Jordan-117 2y agoThat's how it used to do it, but my understanding is that this new model processes audio directly. If it were a music generator, the original would have generated sheet music to send to a synthesizer (text to speech), while now it can create the raw waveform from scratch.
- modeless 2y agoThere is no text. The model understands ingests audio directly and also outputs audio directly.
- dclowd9901 2y agoIs it a stretch to think this thing could accurately "talk" with animals?
- jamilton 2y agoYes? Why would it be able to do that?
- ninininino 2y agoI think they are assuming a world where you took this existing model but it was trained on a dataset of animals making noises to each other, so that you could then feed the trained model the vocalization of one animal and the model would be able to produce a continuation of audio that has a better-than-zero chance of being a realistic sound coming from another animal - so in other words, if dogs have some type of bark that encodes a "I found something yummy" message and other dogs tend to have some bark that encodes "I'm on my way" and we're just oblivious to all of that sub-text, then maybe the model would be able to communicate back and forth with an animal in a way that makes "sense" to the animal. Probably substitute dogs for chimps though. But obviously that doesn't solve at all or human-understandability, unless maybe you have it all as audio+video and then ask the model to explain what visual often accompanies a specific type of audio? Maybe the model can learn what sounds accompany violence or accompany the discovery of a source of water or something?
- dclowd9901 2y agoYep, exactly what brought that to mind. Multimodal seems like the kind of thing needed for such a far-fetched idea.
- benlivengood 2y agoNot really a stretch in my mind. https://www.earthspecies.org/ https://www.earthspecies.org/ and others are working on it already.
- reisse 2y agoSo they retrained the whole model on audio datasets and the tokens are now sounds, not words/part of words?
- modeless 2y agoThey trained on text and audio and images. The model accepts tokens of all three types. And it can directly output audio as well as text.
- famouswaffles 2y agoIt can also directly output images. Some examples are up on the page. Though with how little coverage that's gotten, not sure if users will ever be able to play with that
- modeless 2y agoPeople are saying that GPT-4o still uses Dall-e for image generation. I think that it doesn't match the quality of dedicated image models yet. Which is understandable. I bet it can't generate music as well as Suno or Udio either. But the direction is clear and I'm sure someday it will generate great images, music, and video. You'll be able to do a video call with it where it generates its own avatar in real time. And they'll add more outputs for keyboard/mouse/touchscreen control, and eventually robot control. GPT-7o is going to be absolutely wild.