3 ms·

I think you would agree that a GAN, or any generative model can only generate something in the same domain as what it was trained on. If you trained on mostly o

by p1esk 6y ago

I think you would agree that a GAN, or any generative model can only generate something in the same domain as what it was trained on. If you trained on mostly on human faces with a little bit of rabbits, it's not going to generate rabbits well. If you trained it on mostly on English text and a little bit on Mandarin, it's not going to generate good text in Mandarin. Same with sounds. Different languages use different sounds. If they use any generative model in their codec, they had to train it first, offline, on some dataset. They can't possibly train it equally well on all languages, so we should be able to tell the difference in quality when comparing English to more exotic languages.

ChuckMcM 6y ago
I agree with you 100%! This is where I am wondering: > If they use any generative model in their codec, they had to train it first, offline, on some dataset. One thing I'm wondering if they have a model that can be "retrained" on the fly. Let's assume for this discussion that you've got a model with 1024 weights in it. You train it on spoken text, all languages, just throw anything at it that is speech. That gets a you a generalized model that isn't specialized for any particular kind of speech and the results will be predictably mixed when you generate random speech from it. But if you take it, and ran a "mini" training system on just the sample of interest, so you have this general model, you digitize the speech, you run it through your trainer, now the generalized model is better at generating exactly this kind of speech agreed? So now you take the weights and generate a set of changes from the previous "generic" set, you bundle those changes in the header of the data you are sending and label them appropriately. Now you send only the data bits from the training set that were needed to activate those parts of the model that are updated. Your data product becomes (<model deltas>, <sound deltas>). What I'm wondering is this, if every digitization is used to train the model, and you can send the model deltas in a way that the receiver can incorporate those changes in a predictable way to its local model. Can you then send just the essential features of the digitized sound and get it to re-generate by the model on the other end (which has incorporated the model deltas you sent). Here is an analogy for how I'm thinking about this, and it can be completely wrong, just speculating. If you wanted to "transport" a human with the least number of bits you could simply take their DNA and their mental state and transmit THAT to a cloning facility. No need to digitize every scar, every bit of tissue, instead a model is used to regenerate the person and their 'state' is sent as state of mind. That is clearly science fiction, but some of the GAN models I've played with have this "feel" where they will produce reliably consistent results from the same seed. Not exact results necessarily, but very consistent. From that, and this article, I'm wondering if they figured out how to compute the 'seed' + 'initial conditions', given the model that will reproduce what was just digitized. If they have, then its a pretty amazing result.
- p1esk 6y ago
  What you described could work in principle, but in practice, "mini" training on a single sample is not likely to produce good results, unless the sample is very large. Also, this finetuning would most likely be quite resource intensive. I recall older speech recognition systems where they would ask you to read a specific text sample to adapt the model to your voice, so yes, this can work. If you can fit a large generative model (e.g. an rnn or a transformer) in the codec, you might be able to offer something like "prompt engineering" [1], where the weights of the model don't change, but the hidden state vectors are adjusted using the current input. So, using your analogy, weights would be DNA, and the hidden state vectors would be the "mental state". By talking to this person you adjust their mental state to hopefully steer the conversation in the right direction. [1] https://www.gwern.net/GPT-3#prompts-as-programming https://www.gwern.net/GPT-3#prompts-as-programming

by p1esk 6y ago

ChuckMcM 6y ago

I agree with you 100%! This is where I am wondering: > If they use any generative model in their codec, they had to train it first, offline, on some dataset. One thing I'm wondering if they have a model that can be "retrained" on the fly. Let's assume for this discussion that you've got a model with 1024 weights in it. You train it on spoken text, all languages, just throw anything at it that is speech. That gets a you a generalized model that isn't specialized for any particular kind of speech and the results will be predictably mixed when you generate random speech from it. But if you take it, and ran a "mini" training system on just the sample of interest, so you have this general model, you digitize the speech, you run it through your trainer, now the generalized model is better at generating exactly this kind of speech agreed? So now you take the weights and generate a set of changes from the previous "generic" set, you bundle those changes in the header of the data you are sending and label them appropriately. Now you send only the data bits from the training set that were needed to activate those parts of the model that are updated. Your data product becomes (<model deltas>, <sound deltas>). What I'm wondering is this, if every digitization is used to train the model, and you can send the model deltas in a way that the receiver can incorporate those changes in a predictable way to its local model. Can you then send just the essential features of the digitized sound and get it to re-generate by the model on the other end (which has incorporated the model deltas you sent). Here is an analogy for how I'm thinking about this, and it can be completely wrong, just speculating. If you wanted to "transport" a human with the least number of bits you could simply take their DNA and their mental state and transmit THAT to a cloning facility. No need to digitize every scar, every bit of tissue, instead a model is used to regenerate the person and their 'state' is sent as state of mind. That is clearly science fiction, but some of the GAN models I've played with have this "feel" where they will produce reliably consistent results from the same seed. Not exact results necessarily, but very consistent. From that, and this article, I'm wondering if they figured out how to compute the 'seed' + 'initial conditions', given the model that will reproduce what was just digitized. If they have, then its a pretty amazing result.

p1esk 6y ago
What you described could work in principle, but in practice, "mini" training on a single sample is not likely to produce good results, unless the sample is very large. Also, this finetuning would most likely be quite resource intensive. I recall older speech recognition systems where they would ask you to read a specific text sample to adapt the model to your voice, so yes, this can work. If you can fit a large generative model (e.g. an rnn or a transformer) in the codec, you might be able to offer something like "prompt engineering" [1], where the weights of the model don't change, but the hidden state vectors are adjusted using the current input. So, using your analogy, weights would be DNA, and the hidden state vectors would be the "mental state". By talking to this person you adjust their mental state to hopefully steer the conversation in the right direction. [1] https://www.gwern.net/GPT-3#prompts-as-programming https://www.gwern.net/GPT-3#prompts-as-programming