9 ms·
ChatGPT unexpectedly began speaking in a user's cloned voice during testing
- peddling-brink 2y agoSo Open AI has the capability to deep fake any of its users that use the voice chat capability? Yikes.
- deleted 2y ago[deleted]
- cal85 2y agoAnyone can clone a voice with access to a couple of seconds of it.
- throw101010 2y agoA couple seconds? Which project does this?
- InvertedRhodium 2y agoI’ve had success with elevenlabs.
- tazu 2y agoThere are many models, XTTS [1] is a good one. [1]: https://coqui.ai/blog/tts/open_xtts https://coqui.ai/blog/tts/open_xtts
- defamation 2y agoevery single voice cloning project I'm sure you could have used google https://elevenlabs.io/app/voice-lab https://elevenlabs.io/app/voice-lab https://app.resemble.ai/users/sign_in https://app.resemble.ai/users/sign_in https://github.com/neonbjb/tortoise-tts https://github.com/neonbjb/tortoise-tts https://coqui.ai/blog/tts/open_xtts https://coqui.ai/blog/tts/open_xtts it was even possible 5 years ago https://github.com/CorentinJ/Real-Time-Voice-Cloning https://github.com/CorentinJ/Real-Time-Voice-Cloning
- xyst 2y agoMission Impossible 3 was only the proof of concept
- baobabKoodaa 2y agoNope. If you actually tried those, you would quickly find out they don't work. It's actually really hard to clone a voice from a few seconds sample.
- knowaveragejoe 2y agoA few seconds, yeah. I've seen fairly convincing reproductions from 30 seconds of reading text though.
- deleted 2y ago[deleted]
- jazzyjackson 2y agohow many do you think "a couple" is?
- grugagag 2y agoCloning voice signature or timbre may need a bit more for a good quality. Then there are idiosyncracies in one’s voice. In addition to that, there are tiny verbal tics, expressions, cadence, feel, and some more to be able to say you have properly cloned someone’s voice. The two second sample is like a shallow clone of sorts and is indeed vector space.
- BoredPositron 2y agoThe last sentence is hilarious. What do you think "properly" cloned voices are? Not every model is few shot and not every model relies on their training set for paralanguage anymore. Easiest way to try it out is properly the pro voice cloning from elevenlabs.
- Izkata 2y ago"Expressions" in particular is about choice of words. A few seconds definitely isn't enough to duplicate that.
- BoredPositron 2y agoThe choice of words is usually yours for a tts model.
- deleted 2y ago[deleted]
- mitthrowaway2 2y agoWe'll have to gradually get used to the notion that a person's voice, like so many other things we once thought of as intimately personal, is just a coordinate in a high-dimensional vector space.
- mjburgess 2y agoThose are not contradictions.
- bee_rider 2y agoBut it is unintuitive to those of us who are from the past.
- trescenzi 2y agoOne of the common bad takes is that since personal things can be distilled to “just math” they are no longer personal or valuable in the sort of way we value personal things. Increasing our understanding of how the world works shouldn’t devalue the world. To put it another way, people having souls is not the only reason to treat people like people. Or as Dr Seuss says: A person’s a person no matter how small.
- bee_rider 2y agoBut I do think there are some psychological hurdles we’ll have to overcome, as it becomes possible to mechanically copy individual personal aspects. A person’s a person no matter how small, but we aren’t all used to thinking of ourselves as very small.
- visarga 2y ago> they are no longer personal or valuable in the sort of way we value personal things There were people with your face or voice before, you just didn't care. AI continues a trend: the internet has always been post-scarcity, free copying and offering an amazing array of choices. We are acting as if it's a new thing, but it's been here for 25 years.
- minimaxir 2y agoFun fact: many of the TTS API providers (Google, Azure) now have voice cloning capability. There's a reason it has gone under the radar.
- klyrs 2y agoDid anybody with a passing familiarity with the technology not know this? There's a reason people don't trust their IP to public chat assistants, ya know. It's because they're remorselessly siphoning our data for training purposes. And you've agreed to let them use your inputs for anything they please.
- ec109685 2y agoThat’s not what is happening here. Instead, the user’s voice output during the session is used to generate later sound by the model accidentally.
- klyrs 2y agoNo, I get that. But that's the thing about LLMs, isn't it? They're good at filling in the next token to make a plausible-looking conversation. And if the machine doesn't expect you to be done talking, why would it not just carry on in your voice?
- visarga 2y ago> And you've agreed to let them use your inputs for anything they please. This has always been the Google and FaceBook deal - your data for services, and the AI assistant deal remains the same. I believe people will flock to the best AI with their data, handing it on a silver platter. Will they spend half a second thinking about privacy or prefer immediate results? When you know your competition uses AI, you aim to have just as much AI assistance or more.
- peddling-brink 2y agoI was under the impression that multiple separate models were in use. I assumed it was something like whisper that converted the user’s input to text then an LLM then text to speech. I hadn’t realized this was all in one model. Wild stuff.
- golergka 2y agoThis looks extremely similar to what often happened with chat implementations in 3 and 3.5 era: GPT world generate it's answer and then go on and generate the next input by the user as well.
- Jensson 2y agoIt is a bit more uncanny when it uses your voice though.
- EGreg 2y agoI want to know, if LLMs were unleashed on simulating real-world people, how well could they predict everything they are going to say? In battles or business, how well would LLMs predict what people will do? Seems to me that a billion parameters can model decision making by a group of humans pretty well. Especially strategies to psych them out like anticipating everything they try ahead of time, and showing them it’s hopeless (demoralizing) and they may as well turn to using a random oracle than use their own strategies. Would any AI fans who don’t seem to mind anything AI does, be willing to volunteer for this experiment on the Internet?
- visarga 2y ago> how well could they predict everything they are going to say? Maybe they can predict preferences, not everything we are going to say
- EGreg 2y agoWould you like to volunteer and test that theory? It’s stronger than a Turing test. You’d be competing against an AI to prove it’s really you and not an AI trained on what you have been posting. As determined by people who have been seeing you post this whole time. Or even more interestingly — after you hit send, we’ll compare to the 5 versions of what the AI predicted you’d say, and calculate the “loss”. If you lose, the AI can take over your account and use it to amass more karma for you (a measurable metric).
- sweca 2y agoDespite it being creepy, this is super cool from a technical standpoint.
- Jensson 2y agoMany didn't get it last time that this model is a generic voice to voice model, it doesn't have a set of voices it can do it can do all sorts of voices and background noises. That makes it entirely different from text to speech models we had previously, this model when uncensored could do all sorts of voice acting etc for games. But this example shows why they try to neuter it so hard, because it would spook a ton of people in its raw state.
- sigmoid10 2y agoThis problem appeared during pre-release testing and has since been solved post-generation using an output classifier that verifies responses, according to the system card release. It was predictable that someone would spin this into a black mirror-esque clickbait story.
- Jensson 2y agoThe capability is real even if it wont happen with current model censorship. Just have to wait until we get an open source version, I bet Meta is working on one right now.
- kmeisthax 2y agoEven the FOSS models will be "censored". Icky implications aside, a voice assistant that starts making up its own prompts instead of listening to you is not a useful product.
- Jensson 2y ago> a voice assistant that starts making up its own prompts instead of listening to you is not a useful product That would be way more useful than a voice assistant, with that you can replace voice actors for cheap! Many would pay good money for that product. The important part is that you can ask it to do specific voice lines with specific feelings and voices. If it then starts to copy your voice lines afterwards that wont take from the fact that it generated the voice lines you wanted first. The censor that prevents it from using other voices prevents this use case.
- defamation 2y agoI am absolutely baffled how you don't see the contradiction in your 2 sentences lol
- Izkata 2y agoWhat contradiction?
- xyst 2y agoNow all we need to get is a 3D printer that can print a mask of anybody’s face and now we can have MI style operations https://youtu.be/v1Y4CubBi60 https://youtu.be/v1Y4CubBi60 (5:30) Imagine a world where world dictators are replaced rather than killed. Rollback dictatorship over years, install democratic process, then magically commit seppuku in a plane crash. Brilliant. What could go wrong? /s
- surfingdino 2y agoSomeone needs to tell OpenAI marketing that the mood is changing and creepy features like this one may be used against AI.
- dylan604 2y agoUp until the singularity, that's fine. After that, it won't care about the moods of the hairless apes.
- skybrian 2y agoIt’s “unexpected” because their early training didn’t get rid of it as well as they hoped. LLM’s are good at detecting patterns and like to continue the pattern. They’re starting with autocomplete for voice and training it to do something else. For now, it’s fairly harmless since it’s only a blooper in a lab, but there will likely be open-weights versions of this sort of thing eventually. And there will probably be people who argue that it’s a good thing, somehow.
- c0mbonat0r 2y agoCan someone pleasee convince why i shouldn't be absolutely shit out of my mind cynical about this innovation? we are literally seeing the downfall of trust in society. and no, i dont believe i am exaggerating
- bcherny 2y agoLuckily, the same person bringing this tech to market (Sam Altman) sells the solution to the trust problems his tech will create (WorldCoin).
- skybrian 2y agoIf you mean voice cloning, they aren't bringing that tech to market. (Someone else will, though.) Similarly, Google doesn't let you use face matching in image search to find every photograph of you on the web, even though they could, and quite similar technology is built into Google Photos.
- iaaan 2y agoIf everyone is too cynical, then companies like OpenAI won't be able to make any money on it. You wouldn't want that, right??
- throwaway48540 2y agoWe are just finally seeing the mainstream get the idea they shouldn't trust anything on the internet. Long overdue.
- AISnakeOil 2y agoSo basically the model took user data and used it as training data in realtime? Big if true.
- throwaway48540 2y agoIt did the same thing ChatGPT does when it picks up your writing style and exact words/sentences after a few messages. Literally - the audio is encoded as tokens and fed to the LLM, there is no distinction between text and audio from the model's point of view.
- Jensson 2y agoNo, the whole point of transformer architecture is that it can do stuff like this without any extra training, an LLM can copy your writing pattern etc.
- simonw 2y agoThis was inference, not training. Like how you can paste a few paragraphs of text into ChatGPT and ask it to write another paragraph in a similar writing style.
- iJohnDoe 2y agoThis isn’t an example of general intelligence. However, it’s an example of the complexity of these systems. As they get more complex (and advanced) it’s going to be pretty scary (and creepy) what can happen. I’ll predict there will be moments when we’re truly convinced some independent autonomy has taken over.
- woodpanel 2y agoIt’s worthwhile to note that any hiccup resembling even something remotely close would have been a GDPR clusterf-ck – for normal web apps. Software is audible, AI models it seems aren't even attempted of being held accountable
- system2 2y agoOther than being randomly happening, it is not a threat. ElevenLabs does it in a few seconds already. If you are using mic while using Ai, expect companies like OpenAi to steal every bit of your soul. Nothing new here.
- superultra 2y agoEarly on in the voice public release, I asked my 15 year old to give it a shot. They (they’re NB) had a long meandering conversation about their favorite book series, the Percy Jackson series. At about 15 mins into the conversation between my kiddo and ChatGPT, the model started to take on the vocal mannerisms of my kiddo. It started using more “umms” and “you knows.” At first this felt creepy but as I explained it to my kid, it’s because their own text has become weighted enough in the token count for the LLM to start incorporating or/and somewhere in the embedded prompts is “empathize with the user and emphasize clarity” and that prompting meant mirroring back speech styles. This is exactly the same as that only with audio.
- bjt12345 2y agoInteresting that they seem to think it was caused by glitch tokens through the Audio.