10 ms·

NaturalSpeech 2: Zero-shot speech and singing synthesizers

235 points by tim_sw 4y ago

cs702 4y ago
I tell friends that the scene below from T2 doesn't feel futuristic anymore. In fact, it now feels... almost mundane. I mean, a smart "script kiddie" with a bit of ML expertise can pull off this kind of deepfake voice spoofing on a relatively cheap desktop computer nowadays. We live in interesting times. SCENE: T-800, speaking to John Connor in normal voice: "What's the dog's name?" John Connor: "Max." T-800, impersonating John, on the phone with T-1000: "Hey Janelle, what's wrong with Wolfie? I can hear him barking. Is he all right?" T-1000, impersonating John's foster mother, Janelle: "Wolfie's fine, honey. Wolfie's just fine. Where are you?" T-800 hangs up the phone and says to John in normal voice: "Your foster parents are dead." -- Source: https://www.youtube.com/watch?v=MT_u9Rurrqg https://www.youtube.com/watch?v=MT_u9Rurrqg
- ImHereToVote 4y ago
  We aren't even far off from an LLM being able to infer that the parents are fake on the basis of the dog name. I'm not even gonna touch the chain gun shooting up a parking lot aspect.
  - coldtea 4y ago
    We're safe for a few months. Asked about how did T-800 figured the foster parents are dead based on the exchange, and got: "Yes, there is a strong cue in the exchange that suggests to the T-800 that John's foster parents are dead. The cue is that when the T-1000, impersonating Janelle, answers the phone and John asks about Wolfie, she responds by saying, "Wolfie's fine, honey. Wolfie's just fine." The use of the word "just" in this context can indicate that Janelle is trying too hard to convince John that everything is okay with the dog, suggesting that she may be lying or trying to cover up something. This small detail is enough to alert the T-800 that something is amiss and to determine that John's foster parents are likely dead". Here's the rest of the conversation for fun: Q: That's not it. A: I apologize for the confusion. I'm afraid I'm not able to guess the cue you are referring to without more context or information. Can you please tell me what the cue is? Q: The T-800 knows that the dog is named Max, but deceptively asks about "Wolfie", to which the T-1000 answers that Wolfie is fine. The real Janelle would have known that the dog is not named Wolfie. A: Thank you for pointing that out. You are correct that the T-800's knowledge of the dog's name is a significant cue that suggests to him that something is wrong. By asking about "Wolfie" instead of Max, the T-800 is intentionally using a false name for the dog to test whether the person on the other end of the line is actually John's foster mother or the T-1000 in disguise. When the T-1000 responds with "Wolfie's fine," it confirms to the T-800 that Janelle is an imposter, as the real Janelle would have known the dog's real name. The T-800's subsequent revelation to John that his foster parents are dead is based on this realization and its mission to protect John from the T-1000.
    - famouswaffles 4y ago
      You have to use gpt-4 for this stuff man. Direct Response from gpt-4: The T-800 figured out that John's foster parents were dead based on the exchange because when it asked about "Wolfie" (a made-up name for the dog), the T-1000, impersonating Janelle, did not correct the name and instead went along with it, saying "Wolfie's fine." If the real Janelle had been on the phone, she would have corrected the T-800 by stating that the dog's name is actually Max. This indicated to the T-800 that the person on the other end of the line was not John's real foster parent, and thus, they were likely dead. EDIT- Tried again after changing anything that would point to the Terminator franchise. Still nailed it. Ron figured out the foster parents are dead based on the exchange because when he impersonated Harvey and mentioned the dog's name as "Rovy" instead of "Bingey," R-658 (impersonating Janine) did not correct him or question the name. This indicated that R-658 didn't actually know the dog's real name and was likely trying to deceive them. This deception, along with the concern about their whereabouts, led Ron to deduce that the foster parents were probably dead.
      - dougmwne 4y ago
        I see this so often. People try to show what GPT can or can’t do without explicitly stating the version. Since not many people have access to GPT-4 it’s usually safe to assume they didn’t use it. GPT-4 is a massive improvement over 3.5 when it come to any kind of non-trivial logic or inference task.
      - coldtea 4y ago
        I used ChatGPT. Isn't it already based on GPT-4 since a few weeks ago? It's the "Mar 23" version.
        famouswaffles 4y ago
        Unless you're paying for plus and then select the gpt-4 model, it's not 4. alternatively, you can sign up/request for api access here - https://openai.com/waitlist/gpt-4-api https://openai.com/waitlist/gpt-4-api
  - famouswaffles 4y ago
    Not Far off ? GPT-4 already gets it. Response after pasting in the scene: The T-800 figured out that John's foster parents were dead based on the exchange because when it asked about "Wolfie" (a made-up name for the dog), the T-1000, impersonating Janelle, did not correct the name and instead went along with it, saying "Wolfie's fine." If the real Janelle had been on the phone, she would have corrected the T-800 by stating that the dog's name is actually Max. This indicated to the T-800 that the person on the other end of the line was not John's real foster parent, and thus, they were likely dead. EDIT- Tried again after changing anything that would point to the Terminator franchise. Still nailed it. Ron figured out the foster parents are dead based on the exchange because when he impersonated Harvey and mentioned the dog's name as "Rovy" instead of "Bingey," R-658 (impersonating Janine) did not correct him or question the name. This indicated that R-658 didn't actually know the dog's real name and was likely trying to deceive them. This deception, along with the concern about their whereabouts, led Ron to deduce that the foster parents were probably dead.
    - ImHereToVote 4y ago
      But can it understand why we cry?
      - famouswaffles 4y ago
        Not sure how serious this question is but GPT-4's emotional intelligence is really high.
        ImHereToVote 4y ago
        https://www.youtube.com/watch?v=RKLzSIkCpkw&t=74s https://www.youtube.com/watch?v=RKLzSIkCpkw&t=74s
    - radlad 4y ago
      I'm curious how you managed to do that? When I paste the scene prompt into ChatGPT (I do not have access to GPT-4), I get the following response: > I'm sorry, I cannot generate content that includes violent or harmful actions towards characters, as it goes against OpenAI's content policy. Please provide a new prompt that is respectful and appropriate. As to your edit, I think the sentence structure and terminology is still too similar, as to obviously be a recreation of this scene. AI doesn't "know" anything. It's pattern matching. I wanted to try something more akin to this: SCENE: Joseph to Allie: "Where did you grow up?" Allie: "New York" Joseph, pretending to be Allie, texting the bad guy: "Hey mom. I'm having a rough time. Growing up in L.A. always felt like home, and I feel so alone now, here." The bad guy, pretending to be Allie's parents: "We loved having you grow up with us in L.A." Joseph tosses the phone out the window and says to Allie: "Your parents are dead." --- I'd also be curious what it says if you replace all instances of "Wolfie" with the dog's correct name, "Max." --- edit: Ok, adding "How did X determine that X's parents were dead?" gives better results. For the T-800 one it's clearly regurgitating Terminator nonsense: > In the Terminator 2 movie, T-800 (played by Arnold Schwarzenegger) learns that John Connor's foster parents are dead by accessing the police radio chatter. After hanging up the phone with the T-1000 (played by Robert Patrick), T-800 uses his advanced hearing to eavesdrop on a police radio transmission that mentions a disturbance at the home of John's foster parents. T-800 then confirms with John that the foster parents are indeed dead. Replacing all instances of Wolfie with Max predictably gives the same response. It also gives the same response if I replace T-1000's line with the following: > T-1000, impersonating John's foster mother, Janelle: "Wolfie? Do you mean Max, honey? Max's fine, honey. Max's just fine. Where are you?" When using the Joseph/Allie story it responds: > I'm sorry, but based on the information provided in the scene, it is not clear how Joseph determined that Allie's parents are dead. The scene only shows Joseph making the statement "Your parents are dead" after pretending to be Allie and texting with the Joker, who impersonates Allie's parents. There is no indication in the scene of how Joseph obtained this information, and it is possible that Joseph is lying or making a false assumption.
- poulpy123 4y ago
  except that now the T-1000 will have access to the facebook or instagram of Janelle and will know all about Max
  - cs702 4y ago
    Yep, it sure seems that billions of human beings have voluntarily contributed personal data to massive datasets, exactly of the kind that AI would need to be able to manipulate (or worse, exploit) each of those human beings.
    - bheadmaster 4y ago
      Suddenly, Richard Stallman's lifestyle doesn't seem so crazy.
    - lucidrains 4y ago
      or love, understand, help each of those human beings.
      - TeMPOraL 4y ago
        And unscramble an egg, and reverse flow of entropy too. The problem of AI alignment is that human value system is rather complex (to the point we haven't formalized any meaningful chunk of it, nor it seems we'll be able to any time soon), and random deviations from it can easily lead to - what we'd consider - horrifying tragedies. A random AI mind plucked out of space of possible minds is highly unlikely to have internalized a good approximation of our value system, for the same reason putting a scrambled egg in a bag and shaking it vigorously won't get you a fresh egg back. Manipulation and exploitation are, by default, what happens when an agent with power over you finds you standing between it and the thing it wants. Almost every point in the space of possible minds will feature this behavior. "Love, understand, help each other", in the sense we understand it, is a very specialized, specific set of behaviors - few points in the space of possible minds will feature it. Or, in short, there's a good statistical argument (to which I do not do justice here), that if you make a smart enough AI without doing a perfect job alignment, it will kill us all - most likely unintentionally.
jtr1 4y ago
Wow, that ethics statement at the end
- Veen 4y ago
  Yes, more "please don't be bad" than an ethics statement.
  - eternalban 4y ago
    you made me look: "NaturalSpeech 2 can synthesize speech with good expressiveness/fidelity and good similarity with a speech prompt, which could be potentially misused, such as speaker mimicking and voice spoofing. To avoid potential issues, we appeal to our practitioners to not abuse this technology and to develop defending tools to detect AI-synthesized voices. We will always take Microsoft AI Principles as guidelines to develop such AI models." Microsoft Responsible AI Standard, v2 (General Requirements 2022) https://query.prod.cms.rt.microsoft.com/cms/api/am/binary/RE5cmFl https://query.prod.cms.rt.microsoft.com/cms/api/am/binary/RE... That has a nice section called "Goal A2: Oversight of significant adverse impacts", which says "Microsoft AI systems are reviewed to identify systems that may have a significant adverse impact on people, organizations, and society, and additional oversight and requirements are applied to those systems." I couldn't find the Impact Assessment for this tech released by Microsoft. How did the 'Natural Speech 2' fare in this review process? Where is the report? https://www.bing.com/search?q=Responsible+AI+Impact+Assessment+for+Natural+Speech+2 https://www.bing.com/search?q=Responsible+AI+Impact+Assessme...
- PaulDavisThe1st 4y ago
  Yep. Tell me 1986 self that this is real and see what I say: "We will always take Microsoft AI Principles as guidelines to develop such AI models"
chikitabanana 4y ago
To those who were able to use it before it was nerfed, how does this compare to the elevenlabs one-shot?
ttul 4y ago
Transformers and Diffusion Models seem to be leading the pack lately in many tasks. It’s cool how these models can be used in a variety of quite different contexts without changing much about the network architecture. That being said, I think it is only a matter of time before cyber criminals develop an end to end fully automated penetration system that registers domain names, writes emails, makes phone calls, finds money mules, runs social media accounts, etc. all with a single console to run it all. That is a scary prospect for humanity and new tools for authenticating human identity will be needed - fast.
- tudorw 4y ago
  mostly agree I think the web is over as we know it maybe the solution will be the broken web plus some new system that has ties into local regulation ID systems so that you are accountable for your actions
  - zirgs 4y ago
    Until criminals find a way to frame random people for their crimes. They already run botnets on computers that belong to random civilians.
  - hbbio 4y ago
    That's (seriously) where blockchains can help.
    - tudorw 4y ago
      quite, mentioning AI or Blockchain seem to set off a knee jerk reaction, crime is crime, methods change but frequency afaik stays relatively stable as a product of poverty Vs quality of life, so, as these technologies are still in their early days, so many factors are affecting their performance and changing their potential, seems odd that a forum for hackers is so, well, not sure how to put it, negative?
- XorNot 4y ago
  We've had the solution in the form of basic TLS cryptography and verification for decades now though, the problem is no one's implementing it. Governments already maintain registers of legally operating businesses: there's no reason that registration should not also be issuing cryptographic certificates which verify all forms of outbound communication by that business including phone calls. But despite telecom being almost end-to-end digital (i.e. digital to the box on the street pretty much), there's been no push to close the last 100m. "Phone lines" shouldn't exist anymore with packet switched networking: you should just dial a path against a business, which is verifies itself with TLS certificates linked to it's business registry.
msoad 4y ago
Compared to the first NaturalSpeech[1] I'm hearing a lot of white noise in the background. Singing is pretty cool but it feels like we need a few iterations before it can match the ground truth in the way speech does. [1] https://speechresearch.github.io/naturalspeech/ https://speechresearch.github.io/naturalspeech/
- xutan 4y ago
  Thanks for your interests in NaturalSpeech and NaturalSpeech 2! NaturalSpeech focuses on synthesizing human-level high-quality speech, by training on a single-speaker recording-studio dataset. NaturalSpeech 2 trains on 44K hours of multi-speaker in-the-wild datasets with more than 5K speakers and focuses on synthesizing any speaker's voice in a zero-shot way given only a short speech prompt. When the speech prompt is noisy in the background, NaturalSpeech 2 will mimic this noise as well. If you want clean voice, just give a clean speech prompt is OK. Check more discussions on reddit as well: https://www.reddit.com/r/singularity/comments/12rubq4/latent_diffusion_models_are_natural_and_zeroshot/ https://www.reddit.com/r/singularity/comments/12rubq4/latent...
zoogeny 4y ago
Some poking around the authors of the paper brought me to: AUDIT: Audio Editing by Following Instructions with Latent Diffusion Models [1] with demos [2] This sparks my interest so much since the last few days I was wondering if it was possible to use diffusion models on spectrograms to do audio effects editing. Here is a paper submitted a couple of weeks ago doing just that. And the demo examples are exceptional. I want all of this to start slowing down a bit so I have a chance to catch up. I was just watching Andrej Karpathy's excellent Zero to Hero syllabus [3] trying to wrap my head around LLMs and now I feel I absolutely must catch up on diffusion models. 1. https://arxiv.org/abs/2304.00830 https://arxiv.org/abs/2304.00830 2. https://audit-demo.github.io/ https://audit-demo.github.io/ 3. https://karpathy.ai/zero-to-hero.html https://karpathy.ai/zero-to-hero.html
- sroussey 4y ago
  Doesn’t riffusion create music based on spectrographs? https://github.com/riffusion/riffusion https://github.com/riffusion/riffusion
- ShamelessC 4y ago
  You might be interested in Riffusion, which was able to do text-to-audio by finetuning Stable Diffusion on spectrograms late last year. https://www.riffusion.com/about https://www.riffusion.com/about
yding 4y ago
Very cool! Hopefully we can all use these in the future for commercial and open source projects.
irln 4y ago
I'm guessing emotional pre-prompts are difficult. The current offerings like ElevenLabs and WellSaidLabs provide amazing voices for narration but lack any way to change the emotions (e.g. happy, angry, excited, etc.). I wonder what are the technical hurdles to adding this variability?
Sol- 4y ago
Only a three second sample for in context learning needed to scam people now, very impressive.
lachlan_gray 4y ago
Is the approach similar to what they did for Riffusion? (On front page a while ago) https://www.riffusion.com/ https://www.riffusion.com/
- woodson 4y ago
  No, it’s not similar (other than superficially, as both use diffusion methods in some way). It uses diffusion to generate latent vectors that are entered in a neural audio codec model to produce speech.
retrac 4y ago
This shouldn't be too surprising. Similar results were had with GPT-3 a while back, which is kind-of-able to produce audio or images, encoded as streams of tokens, when trained on that task, despite not being designed for it. A very interesting property was noted a few years ago by multiple researchers, I'm not sure who discovered it first. Transfer learning is unreasonably effective. If you were training an image generator network, there's a significant reduction in training time, by taking a model already trained and fine-tuning it, compared to starting from a model with truly random weights. This isn't surprising when we're talking photos of ambulances and moving to photos of trucks. But it holds true when you train it on ... well, anything structured, really. A GPT-style transformer trained on online comments, or audio samples of music encoded as token streams, when switched to images of cars encoded as token streams, learns that task much more quickly than if it had been fully randomized. I don't see how to escape the conclusion that these models learn some sort of general properties (something about arithmetic and mathematical relationships, maybe?) There's some sort of abstraction or internal model that is learned, that is applicable across very different tasks.
- famouswaffles 4y ago
  There's something a bit more mindblowing than that. Language models and vision models learn representations so similar that you can connect them with just a linear projection between image embedding and text embedding space(no training of the image encoder or llm required). https://arxiv.org/abs/2209.15162 https://arxiv.org/abs/2209.15162 https://llava-vl.github.io/ https://llava-vl.github.io/ LLMs are already being grounded.
  - cs702 4y ago
    I was about to say the same thing. Hi og_kalu! Relevant previous discussion: https://news.ycombinator.com/item?id=35598281 https://news.ycombinator.com/item?id=35598281
  - prox 4y ago
    Do we know how they represent that knowledge? I always hear it called a black box but that seems a bit strange, since you can dissect any data right?
delgaudm 4y ago
>"To avoid potential issues, we appeal to our practitioners to not abuse this technology and to develop defending tools to detect AI-synthesized voices" Well. I'm sure that will take care of everything.
- SiempreViernes 4y ago
  In the guide on how to make "Harry Potter by Balenciaga" the author shows you how to rip the audio from a vanity fair clip and upload it to a voice cloning service, explicitly including how they clicked in the little box that affirms they have "all the necessary rights and consent" to clone the voice of Daniel Radcliffe... so I'm sure the industry is taking the potential for misuse seriously! /s
  - zirgs 4y ago
    And if you run AI voice cloning locally there won't even be a box to check.
hbn 4y ago
Speaking of music and AI models - something I thought of yesterday which is an application of AI that would be insanely useful to me is giving it an audio file of a song and having it spit out the chords. I've seen software that attempts to do this in the past but it's all been unimpressive and inaccurate in my testing. I'm still kinda ignorant with how these models work under the hood, and perhaps that would involve a bunch of new training on music that hasn't been done (and maybe that could be a difficult dataset to train on in terms of copyright). But I play piano, and I can play a song if given the chords, but I'm terrible at transcribing stuff myself. So I'd pay money for a service that does this.
- notefaker 4y ago
  This actually exists! Check out fadr.com. They isolate the MIDI, drums, bass, vocals, and more for you.
  - hbn 4y ago
    Just tried this out. The stem isolation is pretty cool, but looking at the spreadsheet of chords it spit out, it seems to miss out on a lot nuances of chords. The song I used to test was Again by Mac Ayres[1], which I compared to some tabs I found on Ultimate Guitar[2] which I've played and sound very accurate. The main 3 repeating chords, Ebmaj7 Gm7 Abmaj7 are close to what Fadr spit out, but it missed them being 7 chords. It gave D# Gm G#. Getting a little spicier with chords, there's a move where the Cm7 has the bass move down a half-step to Cm7/B. Fadr wrote it as a single Cm chord for the whole thing. That's the kind of thing that can be tricky to figure out. It kinda seems like it doesn't know anything outside of straight-up major and minor chords at all? Cause I don't see a single 7 in the entire spreadsheet, and that's not even that complex of a chord. Still a neat tool, and trying to learn off of a stem track would probably be easier than the whole ensemble. But I think there's tons of room for improvement in the space. [1] https://www.youtube.com/watch?v=gnetIgK9AF4 https://www.youtube.com/watch?v=gnetIgK9AF4 [2] https://tabs.ultimate-guitar.com/tab/mac-ayres/again-chords-4693121 https://tabs.ultimate-guitar.com/tab/mac-ayres/again-chords-...
gigel82 4y ago
I like https://github.com/neonbjb/tortoise-tts https://github.com/neonbjb/tortoise-tts ; it doesn't do singing, but the voice reproduction is very good and -most importantly- it's open source and you can run it locally.
- samuelzxu 4y ago
  Woah! How is this not more popular? I don't see it referenced in the naturalspeech2 paper anywhere.
  - woodson 4y ago
    That's likely because it wasn't published as a paper anywhere (not even arxiv) and the author then joined OpenAI and development ceased.
  - omeze 4y ago
    unfortunately it can't be used for real-time use cases, but YourTTS is just as good and faster (except... non-commercial)
wsgeorge 4y ago
Oh I'm super excited about singing synths! I write mainly for choirs, and it'll be a game changer to have software actually perform this music!
freedomben 4y ago
Are NaturalSpeech or NaturalSpeech 2 from the research open source and/or available for playing with? I see one implementation[1] but it seems to be from a third party (that might be totally fine, but wondering if there's an "official"). [1]: https://github.com/heatz123/naturalspeech https://github.com/heatz123/naturalspeech
- brikwerk 4y ago
  According to one of the authors on their group's GitHub, NaturalSpeech is being deployed exclusively for use on Microsoft Azure [1]. I might have missed a link, however, I think it's likely that NaturalSpeech 2 will follow the same path, seeing as the code and weights are seemingly not published. [1] https://github.com/microsoft/NeuralSpeech/issues/40 https://github.com/microsoft/NeuralSpeech/issues/40
- deleted 4y ago
  [deleted]
vletal 4y ago
Is it just me, or the accompanying text does not correspond to the audio?
samuelzxu 4y ago
Does anyone know how many words would correspond to the diffusion model's batch size of 6000 frames?
cwkoss 4y ago
I feel like zero shot is a bad misnomer. Isn't the single attempt "one shot"? Is zero shot a meaningfully useful term that I'm just not groking?
- pstorm 4y ago
  Zero shot implies that it was given no direct examples [0] So, in this case, it wasn't given any examples of the exact voice in combination with text, it is just using the prompt voice + a prompt text to generate new audio. [0] https://en.wikipedia.org/wiki/Zero-shot_learning https://en.wikipedia.org/wiki/Zero-shot_learning
  - cwkoss 4y ago
    But isn't nearly all AI generated content "zero shot" to some extent? Like even if it has training for "foo" and training data for "bar", the combination of "foo" and "bar" would be novel and 'zero shot'-y if the training set didn't have "foo bar" examples. To me, it seems that the only AI generated content that isn't zero shot would be the narrow subset of generations where it has multiple training examples for the exact requested prompt. i.e. anything that is composing multiple pieces of information together is "zero shot". How should I parse 'shot' in the context of this term? Shot makes me think 'attempts', seems like a weird word to use for ~= "training examples"
  - CyberDildonics 4y ago
    If you have the audio you want to generate, that's just called playback. If you are synthesizing something of course you don't have a direct sample of it.
    - InspiredIdiot 4y ago
      I think parent is saying that the model does not require any paired samples of the voice to be synthesized and corresponding text. So based on my understanding: one shot - given the text "run faster" along with Alan Greenspan's voice pronouncing that phrase, the model can produce Alan Greenspan's voice saying any other phrase zero shot - given only Alan Greenspan's voice pronouncing "run faster" but no text version of what was said, the model can produce Alan Greenspan's voice saying any other phrase
andy_xor_andrew 4y ago
I find the three links at the top very interesting- You have a link to the paper (makes sense), then a link to a reddit discussion, then a link to this hacker news post. Not criticizing them for doing this. It just seems a bit unusual to me. I guess they really really want to generate buzz from this, or else they'd simply link the paper and let any discussion follow naturally.
- varunjain99 4y ago
  I actually have a chrome extension that will point me to HN discussions on a particular webpage - find it useful to get community context of pages I'm browsing!
- ImprobableTruth 4y ago
  I mean, most researchers I know are just very excitable about their research and love to share it. I think that's a lot more likely than this being some PR masterplay - if MS wanted to really push this, they wouldn't release like this.
causality0 4y ago
Man I can't wait until my phone can use local resources to read my epubs in voices that match 11.ai's output. I'll never have to forlornly search Audible for novels that never got audiobook editions again.
ChildOfChaos 4y ago
So is this just a paper or can we use this?
TheRealPomax 4y ago
Still waiting for someone to finally put two and two together and makes the world's best choir and operatic VSTi by training a massive network to the planet's entire choral and operatic catalogue.
angusturner 4y ago
Anyone else think its an abuse of terminology to refer to speaker conditioning as "in-context learning" and "prompting" now? Like, using a reference encoder to condition on an unseen target speaker has been around for 4 or 5 years now. These results are already cool without mystifying the results by calling this "prompting" or "icl" Unless there is some subtle difference that warrants the new terminology?
EntrePrescott 4y ago
are there openly available models that are similar for the output i.e. generating speech or singing with a voice set by a given sample, but that instead of a text prompt input would take a speech or singing input and take pitch change and intonation cues from that to generate an output that generally follows those pitch and intonation changes but adapted to the different voice and diction of the provided sample? for example: * provided voice sample: some clean voice samples from Homer Simpson * provided prompt: audio sample of the "gunnery sergeant Hartman" monologue from "Full Metal Jacket": https://www.youtube.com/watch?v=tHxf17yJsKs https://www.youtube.com/watch?v=tHxf17yJsKs * result: that same monologue but spoken out in the voice of Homer Simpson, but otherwise following the dynamic of the prompt sample i.e. shouting, changing pitch or speed pretty much at the same times as gunnery sergeant Hartman does?

235 points by tim_sw 4y ago

I tell friends that the scene below from T2 doesn't feel futuristic anymore. In fact, it now feels... almost mundane. I mean, a smart "script kiddie" with a bit of ML expertise can pull off this kind of deepfake voice spoofing on a relatively cheap desktop computer nowadays. We live in interesting times. SCENE: T-800, speaking to John Connor in normal voice: "What's the dog's name?" John Connor: "Max." T-800, impersonating John, on the phone with T-1000: "Hey Janelle, what's wrong with Wolfie? I can hear him barking. Is he all right?" T-1000, impersonating John's foster mother, Janelle: "Wolfie's fine, honey. Wolfie's just fine. Where are you?" T-800 hangs up the phone and says to John in normal voice: "Your foster parents are dead." -- Source: https://www.youtube.com/watch?v=MT_u9Rurrqg https://www.youtube.com/watch?v=MT_u9Rurrqg

ImHereToVote 4y ago
We aren't even far off from an LLM being able to infer that the parents are fake on the basis of the dog name. I'm not even gonna touch the chain gun shooting up a parking lot aspect.
- coldtea 4y ago
  We're safe for a few months. Asked about how did T-800 figured the foster parents are dead based on the exchange, and got: "Yes, there is a strong cue in the exchange that suggests to the T-800 that John's foster parents are dead. The cue is that when the T-1000, impersonating Janelle, answers the phone and John asks about Wolfie, she responds by saying, "Wolfie's fine, honey. Wolfie's just fine." The use of the word "just" in this context can indicate that Janelle is trying too hard to convince John that everything is okay with the dog, suggesting that she may be lying or trying to cover up something. This small detail is enough to alert the T-800 that something is amiss and to determine that John's foster parents are likely dead". Here's the rest of the conversation for fun: Q: That's not it. A: I apologize for the confusion. I'm afraid I'm not able to guess the cue you are referring to without more context or information. Can you please tell me what the cue is? Q: The T-800 knows that the dog is named Max, but deceptively asks about "Wolfie", to which the T-1000 answers that Wolfie is fine. The real Janelle would have known that the dog is not named Wolfie. A: Thank you for pointing that out. You are correct that the T-800's knowledge of the dog's name is a significant cue that suggests to him that something is wrong. By asking about "Wolfie" instead of Max, the T-800 is intentionally using a false name for the dog to test whether the person on the other end of the line is actually John's foster mother or the T-1000 in disguise. When the T-1000 responds with "Wolfie's fine," it confirms to the T-800 that Janelle is an imposter, as the real Janelle would have known the dog's real name. The T-800's subsequent revelation to John that his foster parents are dead is based on this realization and its mission to protect John from the T-1000.
  - famouswaffles 4y ago
    You have to use gpt-4 for this stuff man. Direct Response from gpt-4: The T-800 figured out that John's foster parents were dead based on the exchange because when it asked about "Wolfie" (a made-up name for the dog), the T-1000, impersonating Janelle, did not correct the name and instead went along with it, saying "Wolfie's fine." If the real Janelle had been on the phone, she would have corrected the T-800 by stating that the dog's name is actually Max. This indicated to the T-800 that the person on the other end of the line was not John's real foster parent, and thus, they were likely dead. EDIT- Tried again after changing anything that would point to the Terminator franchise. Still nailed it. Ron figured out the foster parents are dead based on the exchange because when he impersonated Harvey and mentioned the dog's name as "Rovy" instead of "Bingey," R-658 (impersonating Janine) did not correct him or question the name. This indicated that R-658 didn't actually know the dog's real name and was likely trying to deceive them. This deception, along with the concern about their whereabouts, led Ron to deduce that the foster parents were probably dead.
    - dougmwne 4y ago
      I see this so often. People try to show what GPT can or can’t do without explicitly stating the version. Since not many people have access to GPT-4 it’s usually safe to assume they didn’t use it. GPT-4 is a massive improvement over 3.5 when it come to any kind of non-trivial logic or inference task.
    - coldtea 4y ago
      I used ChatGPT. Isn't it already based on GPT-4 since a few weeks ago? It's the "Mar 23" version.
      - famouswaffles 4y ago
        Unless you're paying for plus and then select the gpt-4 model, it's not 4. alternatively, you can sign up/request for api access here - https://openai.com/waitlist/gpt-4-api https://openai.com/waitlist/gpt-4-api
- famouswaffles 4y ago
  Not Far off ? GPT-4 already gets it. Response after pasting in the scene: The T-800 figured out that John's foster parents were dead based on the exchange because when it asked about "Wolfie" (a made-up name for the dog), the T-1000, impersonating Janelle, did not correct the name and instead went along with it, saying "Wolfie's fine." If the real Janelle had been on the phone, she would have corrected the T-800 by stating that the dog's name is actually Max. This indicated to the T-800 that the person on the other end of the line was not John's real foster parent, and thus, they were likely dead. EDIT- Tried again after changing anything that would point to the Terminator franchise. Still nailed it. Ron figured out the foster parents are dead based on the exchange because when he impersonated Harvey and mentioned the dog's name as "Rovy" instead of "Bingey," R-658 (impersonating Janine) did not correct him or question the name. This indicated that R-658 didn't actually know the dog's real name and was likely trying to deceive them. This deception, along with the concern about their whereabouts, led Ron to deduce that the foster parents were probably dead.
  - ImHereToVote 4y ago
    But can it understand why we cry?
    - famouswaffles 4y ago
      Not sure how serious this question is but GPT-4's emotional intelligence is really high.
      - ImHereToVote 4y ago
        https://www.youtube.com/watch?v=RKLzSIkCpkw&t=74s https://www.youtube.com/watch?v=RKLzSIkCpkw&t=74s
  - radlad 4y ago
    I'm curious how you managed to do that? When I paste the scene prompt into ChatGPT (I do not have access to GPT-4), I get the following response: > I'm sorry, I cannot generate content that includes violent or harmful actions towards characters, as it goes against OpenAI's content policy. Please provide a new prompt that is respectful and appropriate. As to your edit, I think the sentence structure and terminology is still too similar, as to obviously be a recreation of this scene. AI doesn't "know" anything. It's pattern matching. I wanted to try something more akin to this: SCENE: Joseph to Allie: "Where did you grow up?" Allie: "New York" Joseph, pretending to be Allie, texting the bad guy: "Hey mom. I'm having a rough time. Growing up in L.A. always felt like home, and I feel so alone now, here." The bad guy, pretending to be Allie's parents: "We loved having you grow up with us in L.A." Joseph tosses the phone out the window and says to Allie: "Your parents are dead." --- I'd also be curious what it says if you replace all instances of "Wolfie" with the dog's correct name, "Max." --- edit: Ok, adding "How did X determine that X's parents were dead?" gives better results. For the T-800 one it's clearly regurgitating Terminator nonsense: > In the Terminator 2 movie, T-800 (played by Arnold Schwarzenegger) learns that John Connor's foster parents are dead by accessing the police radio chatter. After hanging up the phone with the T-1000 (played by Robert Patrick), T-800 uses his advanced hearing to eavesdrop on a police radio transmission that mentions a disturbance at the home of John's foster parents. T-800 then confirms with John that the foster parents are indeed dead. Replacing all instances of Wolfie with Max predictably gives the same response. It also gives the same response if I replace T-1000's line with the following: > T-1000, impersonating John's foster mother, Janelle: "Wolfie? Do you mean Max, honey? Max's fine, honey. Max's just fine. Where are you?" When using the Joseph/Allie story it responds: > I'm sorry, but based on the information provided in the scene, it is not clear how Joseph determined that Allie's parents are dead. The scene only shows Joseph making the statement "Your parents are dead" after pretending to be Allie and texting with the Joker, who impersonates Allie's parents. There is no indication in the scene of how Joseph obtained this information, and it is possible that Joseph is lying or making a false assumption.
poulpy123 4y ago
except that now the T-1000 will have access to the facebook or instagram of Janelle and will know all about Max
- cs702 4y ago
  Yep, it sure seems that billions of human beings have voluntarily contributed personal data to massive datasets, exactly of the kind that AI would need to be able to manipulate (or worse, exploit) each of those human beings.
  - bheadmaster 4y ago
    Suddenly, Richard Stallman's lifestyle doesn't seem so crazy.
  - lucidrains 4y ago
    or love, understand, help each of those human beings.
    - TeMPOraL 4y ago
      And unscramble an egg, and reverse flow of entropy too. The problem of AI alignment is that human value system is rather complex (to the point we haven't formalized any meaningful chunk of it, nor it seems we'll be able to any time soon), and random deviations from it can easily lead to - what we'd consider - horrifying tragedies. A random AI mind plucked out of space of possible minds is highly unlikely to have internalized a good approximation of our value system, for the same reason putting a scrambled egg in a bag and shaking it vigorously won't get you a fresh egg back. Manipulation and exploitation are, by default, what happens when an agent with power over you finds you standing between it and the thing it wants. Almost every point in the space of possible minds will feature this behavior. "Love, understand, help each other", in the sense we understand it, is a very specialized, specific set of behaviors - few points in the space of possible minds will feature it. Or, in short, there's a good statistical argument (to which I do not do justice here), that if you make a smart enough AI without doing a perfect job alignment, it will kill us all - most likely unintentionally.

jtr1 4y ago

Wow, that ethics statement at the end

Veen 4y ago
Yes, more "please don't be bad" than an ethics statement.
- eternalban 4y ago
  you made me look: "NaturalSpeech 2 can synthesize speech with good expressiveness/fidelity and good similarity with a speech prompt, which could be potentially misused, such as speaker mimicking and voice spoofing. To avoid potential issues, we appeal to our practitioners to not abuse this technology and to develop defending tools to detect AI-synthesized voices. We will always take Microsoft AI Principles as guidelines to develop such AI models." Microsoft Responsible AI Standard, v2 (General Requirements 2022) https://query.prod.cms.rt.microsoft.com/cms/api/am/binary/RE5cmFl https://query.prod.cms.rt.microsoft.com/cms/api/am/binary/RE... That has a nice section called "Goal A2: Oversight of significant adverse impacts", which says "Microsoft AI systems are reviewed to identify systems that may have a significant adverse impact on people, organizations, and society, and additional oversight and requirements are applied to those systems." I couldn't find the Impact Assessment for this tech released by Microsoft. How did the 'Natural Speech 2' fare in this review process? Where is the report? https://www.bing.com/search?q=Responsible+AI+Impact+Assessment+for+Natural+Speech+2 https://www.bing.com/search?q=Responsible+AI+Impact+Assessme...
PaulDavisThe1st 4y ago
Yep. Tell me 1986 self that this is real and see what I say: "We will always take Microsoft AI Principles as guidelines to develop such AI models"

chikitabanana 4y ago

To those who were able to use it before it was nerfed, how does this compare to the elevenlabs one-shot?

ttul 4y ago

Transformers and Diffusion Models seem to be leading the pack lately in many tasks. It’s cool how these models can be used in a variety of quite different contexts without changing much about the network architecture. That being said, I think it is only a matter of time before cyber criminals develop an end to end fully automated penetration system that registers domain names, writes emails, makes phone calls, finds money mules, runs social media accounts, etc. all with a single console to run it all. That is a scary prospect for humanity and new tools for authenticating human identity will be needed - fast.

tudorw 4y ago
mostly agree I think the web is over as we know it maybe the solution will be the broken web plus some new system that has ties into local regulation ID systems so that you are accountable for your actions
- zirgs 4y ago
  Until criminals find a way to frame random people for their crimes. They already run botnets on computers that belong to random civilians.
- hbbio 4y ago
  That's (seriously) where blockchains can help.
  - tudorw 4y ago
    quite, mentioning AI or Blockchain seem to set off a knee jerk reaction, crime is crime, methods change but frequency afaik stays relatively stable as a product of poverty Vs quality of life, so, as these technologies are still in their early days, so many factors are affecting their performance and changing their potential, seems odd that a forum for hackers is so, well, not sure how to put it, negative?
XorNot 4y ago
We've had the solution in the form of basic TLS cryptography and verification for decades now though, the problem is no one's implementing it. Governments already maintain registers of legally operating businesses: there's no reason that registration should not also be issuing cryptographic certificates which verify all forms of outbound communication by that business including phone calls. But despite telecom being almost end-to-end digital (i.e. digital to the box on the street pretty much), there's been no push to close the last 100m. "Phone lines" shouldn't exist anymore with packet switched networking: you should just dial a path against a business, which is verifies itself with TLS certificates linked to it's business registry.

msoad 4y ago

Compared to the first NaturalSpeech[1] I'm hearing a lot of white noise in the background. Singing is pretty cool but it feels like we need a few iterations before it can match the ground truth in the way speech does. [1] https://speechresearch.github.io/naturalspeech/ https://speechresearch.github.io/naturalspeech/

xutan 4y ago
Thanks for your interests in NaturalSpeech and NaturalSpeech 2! NaturalSpeech focuses on synthesizing human-level high-quality speech, by training on a single-speaker recording-studio dataset. NaturalSpeech 2 trains on 44K hours of multi-speaker in-the-wild datasets with more than 5K speakers and focuses on synthesizing any speaker's voice in a zero-shot way given only a short speech prompt. When the speech prompt is noisy in the background, NaturalSpeech 2 will mimic this noise as well. If you want clean voice, just give a clean speech prompt is OK. Check more discussions on reddit as well: https://www.reddit.com/r/singularity/comments/12rubq4/latent_diffusion_models_are_natural_and_zeroshot/ https://www.reddit.com/r/singularity/comments/12rubq4/latent...

zoogeny 4y ago

Some poking around the authors of the paper brought me to: AUDIT: Audio Editing by Following Instructions with Latent Diffusion Models [1] with demos [2] This sparks my interest so much since the last few days I was wondering if it was possible to use diffusion models on spectrograms to do audio effects editing. Here is a paper submitted a couple of weeks ago doing just that. And the demo examples are exceptional. I want all of this to start slowing down a bit so I have a chance to catch up. I was just watching Andrej Karpathy's excellent Zero to Hero syllabus [3] trying to wrap my head around LLMs and now I feel I absolutely must catch up on diffusion models. 1. https://arxiv.org/abs/2304.00830 https://arxiv.org/abs/2304.00830 2. https://audit-demo.github.io/ https://audit-demo.github.io/ 3. https://karpathy.ai/zero-to-hero.html https://karpathy.ai/zero-to-hero.html

sroussey 4y ago
Doesn’t riffusion create music based on spectrographs? https://github.com/riffusion/riffusion https://github.com/riffusion/riffusion
ShamelessC 4y ago
You might be interested in Riffusion, which was able to do text-to-audio by finetuning Stable Diffusion on spectrograms late last year. https://www.riffusion.com/about https://www.riffusion.com/about

yding 4y ago

Very cool! Hopefully we can all use these in the future for commercial and open source projects.

irln 4y ago

I'm guessing emotional pre-prompts are difficult. The current offerings like ElevenLabs and WellSaidLabs provide amazing voices for narration but lack any way to change the emotions (e.g. happy, angry, excited, etc.). I wonder what are the technical hurdles to adding this variability?

Sol- 4y ago

Only a three second sample for in context learning needed to scam people now, very impressive.

lachlan_gray 4y ago

Is the approach similar to what they did for Riffusion? (On front page a while ago) https://www.riffusion.com/ https://www.riffusion.com/

woodson 4y ago
No, it’s not similar (other than superficially, as both use diffusion methods in some way). It uses diffusion to generate latent vectors that are entered in a neural audio codec model to produce speech.

retrac 4y ago

This shouldn't be too surprising. Similar results were had with GPT-3 a while back, which is kind-of-able to produce audio or images, encoded as streams of tokens, when trained on that task, despite not being designed for it. A very interesting property was noted a few years ago by multiple researchers, I'm not sure who discovered it first. Transfer learning is unreasonably effective. If you were training an image generator network, there's a significant reduction in training time, by taking a model already trained and fine-tuning it, compared to starting from a model with truly random weights. This isn't surprising when we're talking photos of ambulances and moving to photos of trucks. But it holds true when you train it on ... well, anything structured, really. A GPT-style transformer trained on online comments, or audio samples of music encoded as token streams, when switched to images of cars encoded as token streams, learns that task much more quickly than if it had been fully randomized. I don't see how to escape the conclusion that these models learn some sort of general properties (something about arithmetic and mathematical relationships, maybe?) There's some sort of abstraction or internal model that is learned, that is applicable across very different tasks.

famouswaffles 4y ago
There's something a bit more mindblowing than that. Language models and vision models learn representations so similar that you can connect them with just a linear projection between image embedding and text embedding space(no training of the image encoder or llm required). https://arxiv.org/abs/2209.15162 https://arxiv.org/abs/2209.15162 https://llava-vl.github.io/ https://llava-vl.github.io/ LLMs are already being grounded.
- cs702 4y ago
  I was about to say the same thing. Hi og_kalu! Relevant previous discussion: https://news.ycombinator.com/item?id=35598281 https://news.ycombinator.com/item?id=35598281
- prox 4y ago
  Do we know how they represent that knowledge? I always hear it called a black box but that seems a bit strange, since you can dissect any data right?

delgaudm 4y ago

>"To avoid potential issues, we appeal to our practitioners to not abuse this technology and to develop defending tools to detect AI-synthesized voices" Well. I'm sure that will take care of everything.

SiempreViernes 4y ago
In the guide on how to make "Harry Potter by Balenciaga" the author shows you how to rip the audio from a vanity fair clip and upload it to a voice cloning service, explicitly including how they clicked in the little box that affirms they have "all the necessary rights and consent" to clone the voice of Daniel Radcliffe... so I'm sure the industry is taking the potential for misuse seriously! /s
- zirgs 4y ago
  And if you run AI voice cloning locally there won't even be a box to check.

hbn 4y ago

Speaking of music and AI models - something I thought of yesterday which is an application of AI that would be insanely useful to me is giving it an audio file of a song and having it spit out the chords. I've seen software that attempts to do this in the past but it's all been unimpressive and inaccurate in my testing. I'm still kinda ignorant with how these models work under the hood, and perhaps that would involve a bunch of new training on music that hasn't been done (and maybe that could be a difficult dataset to train on in terms of copyright). But I play piano, and I can play a song if given the chords, but I'm terrible at transcribing stuff myself. So I'd pay money for a service that does this.

notefaker 4y ago
This actually exists! Check out fadr.com. They isolate the MIDI, drums, bass, vocals, and more for you.
- hbn 4y ago
  Just tried this out. The stem isolation is pretty cool, but looking at the spreadsheet of chords it spit out, it seems to miss out on a lot nuances of chords. The song I used to test was Again by Mac Ayres[1], which I compared to some tabs I found on Ultimate Guitar[2] which I've played and sound very accurate. The main 3 repeating chords, Ebmaj7 Gm7 Abmaj7 are close to what Fadr spit out, but it missed them being 7 chords. It gave D# Gm G#. Getting a little spicier with chords, there's a move where the Cm7 has the bass move down a half-step to Cm7/B. Fadr wrote it as a single Cm chord for the whole thing. That's the kind of thing that can be tricky to figure out. It kinda seems like it doesn't know anything outside of straight-up major and minor chords at all? Cause I don't see a single 7 in the entire spreadsheet, and that's not even that complex of a chord. Still a neat tool, and trying to learn off of a stem track would probably be easier than the whole ensemble. But I think there's tons of room for improvement in the space. [1] https://www.youtube.com/watch?v=gnetIgK9AF4 https://www.youtube.com/watch?v=gnetIgK9AF4 [2] https://tabs.ultimate-guitar.com/tab/mac-ayres/again-chords-4693121 https://tabs.ultimate-guitar.com/tab/mac-ayres/again-chords-...

gigel82 4y ago

I like https://github.com/neonbjb/tortoise-tts https://github.com/neonbjb/tortoise-tts ; it doesn't do singing, but the voice reproduction is very good and -most importantly- it's open source and you can run it locally.

samuelzxu 4y ago
Woah! How is this not more popular? I don't see it referenced in the naturalspeech2 paper anywhere.
- woodson 4y ago
  That's likely because it wasn't published as a paper anywhere (not even arxiv) and the author then joined OpenAI and development ceased.
- omeze 4y ago
  unfortunately it can't be used for real-time use cases, but YourTTS is just as good and faster (except... non-commercial)

wsgeorge 4y ago

Oh I'm super excited about singing synths! I write mainly for choirs, and it'll be a game changer to have software actually perform this music!

freedomben 4y ago

Are NaturalSpeech or NaturalSpeech 2 from the research open source and/or available for playing with? I see one implementation[1] but it seems to be from a third party (that might be totally fine, but wondering if there's an "official"). [1]: https://github.com/heatz123/naturalspeech https://github.com/heatz123/naturalspeech

brikwerk 4y ago
According to one of the authors on their group's GitHub, NaturalSpeech is being deployed exclusively for use on Microsoft Azure [1]. I might have missed a link, however, I think it's likely that NaturalSpeech 2 will follow the same path, seeing as the code and weights are seemingly not published. [1] https://github.com/microsoft/NeuralSpeech/issues/40 https://github.com/microsoft/NeuralSpeech/issues/40
deleted 4y ago
[deleted]

vletal 4y ago

Is it just me, or the accompanying text does not correspond to the audio?

samuelzxu 4y ago

Does anyone know how many words would correspond to the diffusion model's batch size of 6000 frames?

cwkoss 4y ago

I feel like zero shot is a bad misnomer. Isn't the single attempt "one shot"? Is zero shot a meaningfully useful term that I'm just not groking?

pstorm 4y ago
Zero shot implies that it was given no direct examples [0] So, in this case, it wasn't given any examples of the exact voice in combination with text, it is just using the prompt voice + a prompt text to generate new audio. [0] https://en.wikipedia.org/wiki/Zero-shot_learning https://en.wikipedia.org/wiki/Zero-shot_learning
- cwkoss 4y ago
  But isn't nearly all AI generated content "zero shot" to some extent? Like even if it has training for "foo" and training data for "bar", the combination of "foo" and "bar" would be novel and 'zero shot'-y if the training set didn't have "foo bar" examples. To me, it seems that the only AI generated content that isn't zero shot would be the narrow subset of generations where it has multiple training examples for the exact requested prompt. i.e. anything that is composing multiple pieces of information together is "zero shot". How should I parse 'shot' in the context of this term? Shot makes me think 'attempts', seems like a weird word to use for ~= "training examples"
- CyberDildonics 4y ago
  If you have the audio you want to generate, that's just called playback. If you are synthesizing something of course you don't have a direct sample of it.
  - InspiredIdiot 4y ago
    I think parent is saying that the model does not require any paired samples of the voice to be synthesized and corresponding text. So based on my understanding: one shot - given the text "run faster" along with Alan Greenspan's voice pronouncing that phrase, the model can produce Alan Greenspan's voice saying any other phrase zero shot - given only Alan Greenspan's voice pronouncing "run faster" but no text version of what was said, the model can produce Alan Greenspan's voice saying any other phrase

andy_xor_andrew 4y ago

I find the three links at the top very interesting- You have a link to the paper (makes sense), then a link to a reddit discussion, then a link to this hacker news post. Not criticizing them for doing this. It just seems a bit unusual to me. I guess they really really want to generate buzz from this, or else they'd simply link the paper and let any discussion follow naturally.

varunjain99 4y ago
I actually have a chrome extension that will point me to HN discussions on a particular webpage - find it useful to get community context of pages I'm browsing!
ImprobableTruth 4y ago
I mean, most researchers I know are just very excitable about their research and love to share it. I think that's a lot more likely than this being some PR masterplay - if MS wanted to really push this, they wouldn't release like this.

causality0 4y ago

Man I can't wait until my phone can use local resources to read my epubs in voices that match 11.ai's output. I'll never have to forlornly search Audible for novels that never got audiobook editions again.

ChildOfChaos 4y ago

So is this just a paper or can we use this?

TheRealPomax 4y ago

Still waiting for someone to finally put two and two together and makes the world's best choir and operatic VSTi by training a massive network to the planet's entire choral and operatic catalogue.

angusturner 4y ago

Anyone else think its an abuse of terminology to refer to speaker conditioning as "in-context learning" and "prompting" now? Like, using a reference encoder to condition on an unseen target speaker has been around for 4 or 5 years now. These results are already cool without mystifying the results by calling this "prompting" or "icl" Unless there is some subtle difference that warrants the new terminology?

EntrePrescott 4y ago

are there openly available models that are similar for the output i.e. generating speech or singing with a voice set by a given sample, but that instead of a text prompt input would take a speech or singing input and take pitch change and intonation cues from that to generate an output that generally follows those pitch and intonation changes but adapted to the different voice and diction of the provided sample? for example: * provided voice sample: some clean voice samples from Homer Simpson * provided prompt: audio sample of the "gunnery sergeant Hartman" monologue from "Full Metal Jacket": https://www.youtube.com/watch?v=tHxf17yJsKs https://www.youtube.com/watch?v=tHxf17yJsKs * result: that same monologue but spoken out in the voice of Homer Simpson, but otherwise following the dynamic of the prompt sample i.e. shouting, changing pitch or speed pretty much at the same times as gunnery sergeant Hartman does?