4 ms·

What does "zero-shot" mean in this context?

nateb2022 10mo ago
> Zero-shot learning (ZSL) is a problem setup in deep learning where, at test time, a learner observes samples from classes which were not observed during training, and needs to predict the class that they belong to. The name is a play on words based on the earlier concept of one-shot learning, in which classification can be learned from only one, or a few, examples. https://en.wikipedia.org/wiki/Zero-shot_learning https://en.wikipedia.org/wiki/Zero-shot_learning edit: since there seems to be some degree of confusion regarding this definition, I'll break it down more simply: We are modeling the conditional probability P(Audio|Voice). If the model samples from this distribution for a Voice class not observed during training, it is by definition zero-shot. "Prediction" here is not a simple classification, but the estimation of this conditional probability distribution for a Voice class not observed during training. Providing reference audio to a model at inference-time is no different than including an AGENTS.md when interacting with an LLM. You're providing context, not updating the model weights.
- woodson 10mo ago
  This generic answer from Wikipedia is not very helpful in this context. Zero-shot voice cloning in TTS usually means that data of the target speaker you want the generated speech to sound like does not need to be included in the training data used to train the TTS models. In other words, you can provide an audio sample of the target speaker together with the text to be spoken to generate the audio that sounds like it was spoken by that speaker.
  - coder543 10mo ago
    Why wouldn’t that be one-shot voice cloning? The concept of calling it zero shot doesn’t really make sense to me.
    - geocar 10mo ago
      So if you get your target to record (say) 1 hour of audio, that's a one-shot. If you didn't do that (because you have 100 hours of other people talking), that's zero-shots, no?
      - nateb2022 10mo ago
        > So if you get your target to record (say) 1 hour of audio, that's a one-shot. No, that would still be zero shot. Providing inference-time context (in this case, audio) is no different than giving a prompt to an LLM. Think of it as analogous to an AGENTS.md included in a prompt. You're not retraining the model, you're simply putting the rest of the prompt into context. If you actually stopped and fine-tuned the model weights on that single clip, that would be one-shot learning.
        ImPostingOnHN 10mo ago
        > Providing inference-time context (in this case, audio) is no different than giving a prompt to an LLM. Right... And you have 0-shot prompts ("give me a list of animals"), 1-shot prompts ("give me a list of animals, for example: a cat"), 2-shot prompts ("give me a list of animals, for example: a cat; a dog"), etc. The "shot" refers to how many examples are provided to the LLM in the prompt, and have nothing to do with training or tuning, in every context I've ever seen.
        nateb2022 10mo ago
        > Right... And you have 0-shot prompts ("give me a list of animals"), 1-shot prompts ("give me a list of animals, for example: a cat"), 2-shot prompts ("give me a list of animals, for example: a cat; a dog"), etc. > The "shot" refers to how many examples are provided to the LLM in the prompt, and have nothing to do with training or tuning, in every context I've ever seen. In formal ML, "shot" refers to the number of samples available for a specific class during the training phase. You're describing a colloquial usage of the term found only in prompt engineering. You can't apply an LLMism to a voice cloning model where standard ML definitions apply.
  - nateb2022 10mo ago
    > This generic answer from Wikipedia is not very helpful in this context. Actually, the general definition fits this context perfectly. In machine learning terms, a specific 'speaker' is simply a 'class.' Therefore, a model generating audio for a speaker it never saw during training is the exact definition of the Zero-Shot Learning problem setup: "a learner observes samples from classes which were not observed during training," as I quoted. Your explanation just rephrases the very definition you dismissed.
    - woodson 10mo ago
      From your definition: > a learner observes samples from classes which were not observed during training, and needs to predict the class that they belong to. That's not what happens in zero-shot voice cloning, which is why I dismissed your definition copied from Wikipedia.
      - nateb2022 10mo ago
        > That's not what happens in zero-shot voice cloning It is exactly what happens. You are confusing the task (classification vs. generation) with the learning paradigm (zero-shot). In the voice cloning context, the class is the speaker's voice (not observed during training), samples of which are generated by the machine learning model. The definition applies 1:1. During inference, it is predicting the conditional probability distribution of audio samples that belong to that unseen class. It is "predict[ing] the class that they belong to," which very same class was "not observed during training." You're getting hung up on the semantics.
        woodson 10mo ago
        Jeez, OP asked what it means in this context (zero-shot voice cloning), where you quoted a generic definition copied from Wikipedia. I defined it concretely for this context. Don't take it as a slight, there is no need to get all argumentative.
- numpad0 10mo ago
  I think the point is it's not zero shot if a sample is needed. A system that require one sample is usually considered one-shot, or few-shot if it needs few, etc etc.
throwaway132448 10mo ago
The *-shot jargon is just in-crowd nonsense that has been meaningless since day one (or zero). Like Big O notation but even more arbitrary (as evidenced by all the answers to your comment).

nateb2022 10mo ago

> Zero-shot learning (ZSL) is a problem setup in deep learning where, at test time, a learner observes samples from classes which were not observed during training, and needs to predict the class that they belong to. The name is a play on words based on the earlier concept of one-shot learning, in which classification can be learned from only one, or a few, examples. https://en.wikipedia.org/wiki/Zero-shot_learning https://en.wikipedia.org/wiki/Zero-shot_learning edit: since there seems to be some degree of confusion regarding this definition, I'll break it down more simply: We are modeling the conditional probability P(Audio|Voice). If the model samples from this distribution for a Voice class not observed during training, it is by definition zero-shot. "Prediction" here is not a simple classification, but the estimation of this conditional probability distribution for a Voice class not observed during training. Providing reference audio to a model at inference-time is no different than including an AGENTS.md when interacting with an LLM. You're providing context, not updating the model weights.

woodson 10mo ago
This generic answer from Wikipedia is not very helpful in this context. Zero-shot voice cloning in TTS usually means that data of the target speaker you want the generated speech to sound like does not need to be included in the training data used to train the TTS models. In other words, you can provide an audio sample of the target speaker together with the text to be spoken to generate the audio that sounds like it was spoken by that speaker.
- coder543 10mo ago
  Why wouldn’t that be one-shot voice cloning? The concept of calling it zero shot doesn’t really make sense to me.
  - geocar 10mo ago
    So if you get your target to record (say) 1 hour of audio, that's a one-shot. If you didn't do that (because you have 100 hours of other people talking), that's zero-shots, no?
    - nateb2022 10mo ago
      > So if you get your target to record (say) 1 hour of audio, that's a one-shot. No, that would still be zero shot. Providing inference-time context (in this case, audio) is no different than giving a prompt to an LLM. Think of it as analogous to an AGENTS.md included in a prompt. You're not retraining the model, you're simply putting the rest of the prompt into context. If you actually stopped and fine-tuned the model weights on that single clip, that would be one-shot learning.
      - ImPostingOnHN 10mo ago
        > Providing inference-time context (in this case, audio) is no different than giving a prompt to an LLM. Right... And you have 0-shot prompts ("give me a list of animals"), 1-shot prompts ("give me a list of animals, for example: a cat"), 2-shot prompts ("give me a list of animals, for example: a cat; a dog"), etc. The "shot" refers to how many examples are provided to the LLM in the prompt, and have nothing to do with training or tuning, in every context I've ever seen.
        nateb2022 10mo ago
        > Right... And you have 0-shot prompts ("give me a list of animals"), 1-shot prompts ("give me a list of animals, for example: a cat"), 2-shot prompts ("give me a list of animals, for example: a cat; a dog"), etc. > The "shot" refers to how many examples are provided to the LLM in the prompt, and have nothing to do with training or tuning, in every context I've ever seen. In formal ML, "shot" refers to the number of samples available for a specific class during the training phase. You're describing a colloquial usage of the term found only in prompt engineering. You can't apply an LLMism to a voice cloning model where standard ML definitions apply.
- nateb2022 10mo ago
  > This generic answer from Wikipedia is not very helpful in this context. Actually, the general definition fits this context perfectly. In machine learning terms, a specific 'speaker' is simply a 'class.' Therefore, a model generating audio for a speaker it never saw during training is the exact definition of the Zero-Shot Learning problem setup: "a learner observes samples from classes which were not observed during training," as I quoted. Your explanation just rephrases the very definition you dismissed.
  - woodson 10mo ago
    From your definition: > a learner observes samples from classes which were not observed during training, and needs to predict the class that they belong to. That's not what happens in zero-shot voice cloning, which is why I dismissed your definition copied from Wikipedia.
    - nateb2022 10mo ago
      > That's not what happens in zero-shot voice cloning It is exactly what happens. You are confusing the task (classification vs. generation) with the learning paradigm (zero-shot). In the voice cloning context, the class is the speaker's voice (not observed during training), samples of which are generated by the machine learning model. The definition applies 1:1. During inference, it is predicting the conditional probability distribution of audio samples that belong to that unseen class. It is "predict[ing] the class that they belong to," which very same class was "not observed during training." You're getting hung up on the semantics.
      - woodson 10mo ago
        Jeez, OP asked what it means in this context (zero-shot voice cloning), where you quoted a generic definition copied from Wikipedia. I defined it concretely for this context. Don't take it as a slight, there is no need to get all argumentative.
numpad0 10mo ago
I think the point is it's not zero shot if a sample is needed. A system that require one sample is usually considered one-shot, or few-shot if it needs few, etc etc.

throwaway132448 10mo ago

The *-shot jargon is just in-crowd nonsense that has been meaningless since day one (or zero). Like Big O notation but even more arbitrary (as evidenced by all the answers to your comment).