6 ms·

Sinsy: HMM-based Singing Voice Synthesis System

48 points by ghosthamlet 8y ago

PaulHoule 8y ago
Looks like Hatsune Miku has a little sister now. I am fascinated with singing voice synthesis because it is an entirely different problem than normal voice synthesis. Try to get a normal voice synthesizer to sing and the results are usually awful, some of the obvious pitch shifting methods also don't work well because the pitch of the voice is a changing fundamental and harmonics that are filtered by formants in the head that don't change. (Just try improving your singing with a frequency counter and you will see that it doesn't work.)
- posterboy 8y ago
  > formants in the head Funny polysemy: The sound forming cavities of the sounding body. The form of a sound recognized by the sound processor, ie. the head.
- hatsunearu 8y ago
  To those who don't know Miku in the "scientific" perspective (I don't get to introduce her this way very often!) Hatsune Miku is a voicebank/sound profile for a software called Vocaloid, a singing voice synthesizer created by Yamaha. The voicebanks are prepared by recording a bunch of phonemes from a human, and appropriately packaging the tone qualities. Here's a whitepaper describing Vocaloid on the surface level: https://pdfs.semanticscholar.org/4606/eeaa5945a262639879cc8bd00b1cadd39a86.pdf https://pdfs.semanticscholar.org/4606/eeaa5945a262639879cc8b... I believe Vocaloid is pretty much state of the art in terms of fully synthesized singing, so it's a pretty interesting bar to compare to. Vocaloid lets the producer tweak a ton of parameters, like the familiar ADSR stuff, and a few different voicebanks (even for the same "character"). The cool thing is even with the same software, you can achieve really cool and diverse sets of aural atmospheres: (artistically using her 'roboticness'; the second part is a different vocaloid called IA, so you can hear how they sound different) https://www.youtube.com/watch?v=b0L6myhhmsk https://www.youtube.com/watch?v=b0L6myhhmsk (deep and somber) https://www.youtube.com/watch?v=gkIpRTq1S6A https://www.youtube.com/watch?v=gkIpRTq1S6A (getting close to 'human') https://www.youtube.com/watch?v=8kX5_69sO8Q https://www.youtube.com/watch?v=8kX5_69sO8Q The fact that you can create vastly different music with the same "voice" was pretty neat, and the Vocaloid scene at the height of its popularity was incredible--a flood of amateur producers started musicmaking with Vocaloid, got popular almost exclusively over the Japanese internet, and turned it into a day job. Vocaloid is kind of waning (:[) but it was, and still is, a great cultural icon of unique creative expression. That's the kind of scene singing voice synthesizers created, and it's a really interesting field to play in. I hope new machine learning advancements spice up the playing field.
  - PaulHoule 8y ago
    The absolutely best synthesis for ordinary voices is to generate a large voicebank from an individual. Hypothetically you could get away with a small number of phonemes but they change sounds when they are next to each other, in terms of cadence and tone, etc. With maybe 20 hours of good recordings and much more labor you could clone anybody's voice. Getting good results would require optimizing the choice of which sound to use where and making small time and pitch bends to get it perfect. (I'd like to clone Lester Holt and Rie Tanaka.)
pepijndevos 8y ago
It could be really useful to generate practice tracks for choirs with this.
homarp 8y ago
You can also look at eCantorix, a singing synthesis frontend for espeak https://github.com/ttm/ecantorix https://github.com/ttm/ecantorix
wodenokoto 8y ago
It is considerably better at Japanese than English or Chinese.

48 points by ghosthamlet 8y ago

Looks like Hatsune Miku has a little sister now. I am fascinated with singing voice synthesis because it is an entirely different problem than normal voice synthesis. Try to get a normal voice synthesizer to sing and the results are usually awful, some of the obvious pitch shifting methods also don't work well because the pitch of the voice is a changing fundamental and harmonics that are filtered by formants in the head that don't change. (Just try improving your singing with a frequency counter and you will see that it doesn't work.)

posterboy 8y ago
> formants in the head Funny polysemy: The sound forming cavities of the sounding body. The form of a sound recognized by the sound processor, ie. the head.
hatsunearu 8y ago
To those who don't know Miku in the "scientific" perspective (I don't get to introduce her this way very often!) Hatsune Miku is a voicebank/sound profile for a software called Vocaloid, a singing voice synthesizer created by Yamaha. The voicebanks are prepared by recording a bunch of phonemes from a human, and appropriately packaging the tone qualities. Here's a whitepaper describing Vocaloid on the surface level: https://pdfs.semanticscholar.org/4606/eeaa5945a262639879cc8bd00b1cadd39a86.pdf https://pdfs.semanticscholar.org/4606/eeaa5945a262639879cc8b... I believe Vocaloid is pretty much state of the art in terms of fully synthesized singing, so it's a pretty interesting bar to compare to. Vocaloid lets the producer tweak a ton of parameters, like the familiar ADSR stuff, and a few different voicebanks (even for the same "character"). The cool thing is even with the same software, you can achieve really cool and diverse sets of aural atmospheres: (artistically using her 'roboticness'; the second part is a different vocaloid called IA, so you can hear how they sound different) https://www.youtube.com/watch?v=b0L6myhhmsk https://www.youtube.com/watch?v=b0L6myhhmsk (deep and somber) https://www.youtube.com/watch?v=gkIpRTq1S6A https://www.youtube.com/watch?v=gkIpRTq1S6A (getting close to 'human') https://www.youtube.com/watch?v=8kX5_69sO8Q https://www.youtube.com/watch?v=8kX5_69sO8Q The fact that you can create vastly different music with the same "voice" was pretty neat, and the Vocaloid scene at the height of its popularity was incredible--a flood of amateur producers started musicmaking with Vocaloid, got popular almost exclusively over the Japanese internet, and turned it into a day job. Vocaloid is kind of waning (:[) but it was, and still is, a great cultural icon of unique creative expression. That's the kind of scene singing voice synthesizers created, and it's a really interesting field to play in. I hope new machine learning advancements spice up the playing field.
- PaulHoule 8y ago
  The absolutely best synthesis for ordinary voices is to generate a large voicebank from an individual. Hypothetically you could get away with a small number of phonemes but they change sounds when they are next to each other, in terms of cadence and tone, etc. With maybe 20 hours of good recordings and much more labor you could clone anybody's voice. Getting good results would require optimizing the choice of which sound to use where and making small time and pitch bends to get it perfect. (I'd like to clone Lester Holt and Rie Tanaka.)

pepijndevos 8y ago

It could be really useful to generate practice tracks for choirs with this.

homarp 8y ago

You can also look at eCantorix, a singing synthesis frontend for espeak https://github.com/ttm/ecantorix https://github.com/ttm/ecantorix

wodenokoto 8y ago

It is considerably better at Japanese than English or Chinese.