4 ms·
RealSinger
- telotortium 4y ago> Synthesizing high-quality singing voices from music scores is a challenging problem in music generation and has many practical applications. Samples generated by existing singing voice synthesis (SVS) systems can roughly reflect the lyrics, pitch and duration in a given score, but they fail to contain necessary details. In this paper, based on stochastic differential equations (SDE) we propose RealSinger to generate 22.05kHz ultra-realistic singing voice conditioned on a music score. Our RealSinger learns to find the stochastic process path from a source of white noise to the target singing voice manifold under the conditional music score, allowing to sing the music score while maintaining the local voice details of the target singer. During training, our model learns to accurately predict the direction of movement in the ambient Euclidean space onto the low-dimensional singing voice manifold. RealSinger's framework is very flexible. It can either generate intermediate feature representations of the singing voice, such as mel-spectrogram, or directly generate the final waveform, as in the end-to-end style which rectifies defects and accumulation errors introduced by two-stage connected singing synthesis systems. An extensive subjective and objective test on benchmark datasets shows significant gains in perceptual quality using RealSinger. The mean-opinion-scores (MOS) obtained with RealSinger are closer to those of the human singer's original high-fidelity singing voice than to those obtained with any state-of-the-art method.
- p1esk 4y agoI was wondering how is this different from a regular diffusion model. From the paper [0]: If the number of steps in the diffusion model tends to infinity, it becomes an SDE. So basically they came up with something that is like a continuous diffusion model. [0] https://openreview.net/pdf?id=ctnmrjv6lU5 https://openreview.net/pdf?id=ctnmrjv6lU5