4 ms·
That's helpful! I've done a lot of work in audio synthesis, which is notoriously difficult measure. The gold-standard is human ratings of audio quality, but it
by sdenton4 3y ago
That's helpful!
I've done a lot of work in audio synthesis, which is notoriously difficult measure. The gold-standard is human ratings of audio quality, but it is tough to design good tests (easy to fatigue raters) and the iteration time waiting for results is quite long.
Instead, there's now some projects which use neural networks trained on human ratings to predict audio quality, such as ViSQoL: https://github.com/google/visqol https://github.com/google/visqol
This opens up fast iteration - scores going up generally corresponds to higher quality - followed by human testing at major milestones (eg, releasing a paper/model). VISQOL has a harder time comparing 'unrelated' models, IMO - ends up being not so great for comparison of different techniques, but excellent for measuring incremental improvement or catching regressions.
But, in the end, yes - you can use NN's to measure the quality of other NN's, so long as you're careful about it and make use of human raters from time to time as well.
The problem of test data getting into the training data seems to be an especially pernicious issue with LLM's, which isn't really arising in the audio synthesis space.
- blackkettle 3y agoI’ve started doing this with ASR hypotheses from colloquial spontaneous speech. It tends to have similar issues. Lots of shady human ground truth especially where addresses, alphanumeric sequences, repairs and repetitions and other essentially non read speech are concerned. The very large Whisper models are consistent in their transcription style and highly reliable as long as you pick strongly represented languages. And ChatGPT can do a very good job at comparing the linguistic coherence of hypotheses from multiple recognizers. Together these models can annotate, analyze and ingest far more data more consistently than human annotators at this point (at least in the best covered languages). We haven’t quite realized this as a community yet though, because the standard datasets we use for evaluation contain all these human inconsistencies. Wild times.
- 2c2c2c 3y agojust curious, are there any open models doing the opposite of audio synthesis? As in able to generate the stems for a song?
- edude03 3y agohttps://github.com/facebookresearch/demucs https://github.com/facebookresearch/demucs
- sdenton4 3y agoHa, I was working on neutral voice compression, which consists of an encoder which creates what the kids are calling tokens these days, and a decoder which synthesizes speech from the tokens. AudioLM puts a language model on top of the compression tokens, and thus can generate speech or other audio. There's piles of recent papers pushing that approach into music generation. Mulan is a name that comes to mind. Or maybe your interested in audio separation to get at the isolated instruments? There's lots of great work on that, as well. Like MixIT, which is an unsupervised audio separation system.