10 ms·
AudioPaLM: A Large Language Model That Can Speak and Listen
- ml_basics 3y ago> We introduce AudioPaLM, a large language model for speech understanding and generation. AudioPaLM fuses text-based and speech-based language models, PaLM-2 [Anil et al., 2023] and AudioLM [Borsos et al., 2022], into a unified multimodal architecture that can process and generate text and speech with applications including speech recognition and speech-to-speech translation. AudioPaLM inherits the capability to preserve paralinguistic information such as speaker identity and intonation from AudioLM and the linguistic knowledge present only in text large language models such as PaLM-2. Direct link to demo video showing speech-to-speech translation: https://google-research.github.io/seanet/audiopalm/examples/data/AudioPaLM_promo.mp4 https://google-research.github.io/seanet/audiopalm/examples/... (see website for more example)
- ChatGTP 3y agoI can't wait till everyone is using this and we have absolutely zero idea whether or not it's actually translating things correctly or using it's own interpretations of things, going to be...awesomeeeeeeeee!
- famouswaffles 3y agoSota Bi/Multilingual LLMs with good enough representation of the languages (takes much less data than you'd think) are human level translators. Hallucinations on tasks like Summaries, translations etc are near non-existent.
- deleted 3y ago[deleted]
- ChatGTP 3y agoThank you for reminding me that it's going to be, awesommeeee. Curious, do you speak more than one language? Edit: I just had a look at your comment history, do you realize you're like, incredibly pro LLM? Do you just scour HN looking for LLM articles and comment on them in a positive way? Not having a poke it's just interesting how keen you are.
- blovescoffee 3y agoI share the same sentiment as the original commenter and I speak more than one language. Why do you ask?
- ChatGTP 3y agoBecause virtually everyone tests these things with two languages they're familiar with, else you couldn't really verify if it was correct or not. For languages you're not familiar with, you don't have the "mental mode" to talk using a translator, that is there is more to this than just "talking", there is cultural norms, local dialects, slangs etc which are to be respected when learning and speaking languages with native speakers. When a person who speaks English and Italian tests these things. They know what they're in for an compensate a bit. Google translate screws up for me really, really hard sometimes when I'm speaking Korean but I'm already a pretty strong speaker, native so I know how to work with the screw ups...and laugh about the really bad ones. I'm not going to go into a meeting and blast off with an auto-translator without understanding what I'm saying or have someone to make sure I'm saying the right thing by talking with them first. I personally wouldn't feel comfortable using something like this for anything of real significance, a really good translator can ensure the message gets delivered.
- seanthemon 3y agoDo you use google translate for anything of significance?
- ChatGTP 3y agoOnly because I mostly understand the language I'm working with. This is why I know about these problems. I'd never just go to somewhere exotic and rely on it for anything significant based on my existing experience with these technologies.
- hfhdjdks 3y agoAre you american by any chance? Over here people speak multiple languages. I doubt we'll run out of people that speak multiple languages just because there's a language model that can do great translations.
- Kinrany 3y agoI wonder if it can translate from English into English Spoken By Five Year Old
- zb3 3y agoHey Google, what about finally giving me the access to MusicLM?
- famouswaffles 3y agoYou can use MusicLM on google's ai test kitchen
- zb3 3y agoI can't as I was not granted access to that (I filled the form).
- ot 3y agoImpressive that it translated "Morgenstund hat Gold im Mund" (morning hour has gold in the mouth) to the equivalent English expression "the early bird gets the worm", instead of going for a literal translation. I wonder though how much the text in the video was editorialized. For example, I doubt that the model would have correctly capitalized PaLM.
- famouswaffles 3y agoBilingual LLMs make less Literal Translations where appropriate. https://arxiv.org/abs/2305.16806 https://arxiv.org/abs/2305.16806
- ksaj 3y agoAnd this makes sense, as some sayings would be unrecognizable word salad in other languages and cultures. Early Arctic birds don't catch many worms. Gold in the mouth is something popularized by rappers (grills) back in the 90's, so that doesn't translate well at all for me.
- bamboozled 3y agoI actually really liked the literal translation, I thought it was cool even though I'd never heard it before, "oh well"...
- deleted 3y ago[deleted]
- ksaj 3y agoSince it is an LLM, you should be able to ask it for a literal translation if that's what you want.
- rhogar 3y agoThough inference for the 8B model is almost definitely not capable of near real time inference yet, we’re approaching babelfish territory. Main difference perhaps being this is powered by burning massive amounts of carbon as opposed to a fish brain.
- gwern 3y ago> Though inference for the 8B model is almost definitely not capable of near real time inference yet Google previously showed you could get the fullsized 540b-parameter PaLM-1 model down to "a low-batch-size latency of 29ms per token during generation (with int8 weight quantization)" https://arxiv.org/abs/2211.05102#google https://arxiv.org/abs/2211.05102#google . How many tokens per 1000ms do humans speak? I'm guessing fewer than 34. The real question is who wants to pay for it.
- criddell 3y agoFor some reason I’ve been getting 12-20 spam calls per day (all for the same Medicaid/Medicare scam). I’m on T-Mobile which was one of the first carriers to roll out STIR/SHAKEN and I have their Scam Buster app installed and they are getting by all of that. It’s frustrating. When I read about things like AudioPaLM, my first thought is of all the people in these call centers who seem to uniformly have pretty hard Indian accents and very American-sounding names (George Bush called me the other day!). Their days of working in a call center are numbered and their replacement is going to be a machine that is way cheaper to employ and better at the job.
- robterrell 3y agoEarlier this week I got spam call that was almost certainly an AI-generated human voice.
- zoklet-enjoyer 3y agoI get those all day everyday. Fun to mess with them sometimes. I usually tell them my name is Ben Chode and my birthday is April 20, 1969
- criddell 3y agoI just want there to be consequences for the abuse of the phone system and harassment that results. Nobody cares though. The phone company will change your number if you want. The FCC will let you report these - one call at a time. I actually thought about making an app to let me submit a report with a single click. If I started submitting 40-80 reports a week, would that get anybody’s attention? Would somebody at the FCC contact T-Mobile on my behalf and ask them to actually help me with this? Probably not.
- zoklet-enjoyer 3y agoAgreed. It basically makes my phone useless as a phone. I generally don't answer unrecognized calls unless I'm up for messing with a scammer, because that's who is usually calling. When I am expecting a phone call that's a problem because I sometimes don't recognize the number so don't answer. And my voicemail inbox is often just filled with this garbage.
- villgax 3y agoWhat a joke, 8Billion parameters to gain 1 percent compared to 1.5B of largest Whisper model