19 ms·

Mozilla releases the largest to-date public domain transcribed voice dataset

705 points by Vinnl 8y ago

The_Amp_Walrus 8y ago
I'm interested in using this for text to speech, rather than speech to text. Is wavenet still the state of the art for training on a dataset like this?
- 1ris 8y ago
  The biggest feature of this data set is that it includes lots of different accents and non-native speakers. This should improve voice recognition in these areas, but I'm not sure if it is that usful for voice generation.
sgc 8y ago
I love this. I do a lot of dictation on mobile devices for work, with middling, and perhaps more importantly, frustrating results (needless to say we are working on programming our way out of that hole). It is an area ripe for open source progress given the failure of larger companies with large proprietary data sets to make basic common sense decisions in their transcription algorithms, together with no way to provide impactful feedback. If anybody is interested, there is definitely a market for a more robust dictation library that can be integrated into apps and works offline. It just needs to be professional - e.g. allow for preferences including the ability to indicate a strong preference for standard language and grammar over all slang, not forcing Title Case for anything resembling a brand name, having a training mode for words and phrases of the user's choosing, blacklisting of certain word or phrase results which are false positives, and proper learning from user corrections during use so the tedium of correcting the same phrase 100 times disappears.
- 99052882514569 8y ago
  There's also a huge need for better transcription software in radiology. Existing ones are expensive and are just not good enough to be an actual time-saver. (A radiologist friend describes switching from a human medical transcriptionist to one of these "AI" software thingies as a cost-cutting measure by his hospital, or more accurately as a cost-offload measure. Hospital offloads salary, radiologist spends more time correcting stupid transcription mistakes for no extra pay).
  - MattSayar 8y ago
    I thought it would be faster to correct a transcript than type it up fresh. I can type pretty damn quickly, but still slower than the average person talks. I would rather follow along to an AI transcript and correct a few words than type it all new.
    - TheSpiceIsLife 8y ago
      Faster for who? The in the case described above the radiologist does none of the typing and none of the correction.
    - jcims 8y ago
      Problem is that transcription errors never have a misspelling and usually sound ok in yourself nene voice. You have to really pay attention to find them (reading the text backwards a sentence at a time sometimes helps with this)
      - derefr 8y ago
        Imagine a professional “assisted stenography” software package. Rather than automatic full transcription + proof-reading after the fact, it would transcribe “live” but with any words where its model gave a low-confidence output highlighted with the expectation that the human stenographer, listening to the audio at the same time the machine is, will type in the correct transcription (or just hit tab to accept, like in autocomplete.)
        jcims 8y ago
        I would love it if I could always see the top three possible interpretations of what I just said to my phone. It's interesting watching the speech recognition think about what's being said and flip words around once it has enough context.
  - chapium 8y ago
    Thats a pretty cynical view, not entirely unjustified. Transcription services are costly, but dictation software offers plenty of other benefits.
abakker 8y ago
I have a (probably dumb) question: why don't we just use audiobooks for this? There are thousands and thousands of hours where the transcripts were written, and then read aloud. Some of them are now public domain. I'm sure some validation would need to be done, but it seems like there would be an endless validation set there. Am I missing something obvious?
- applecrazy 8y ago
  Audiobook narrators enunciate (almost) perfectly. Real people don't.
- wetpaws 8y ago
  Pretty sure it wont be nearly enough both from the language variety point of view and the variety of voices (age, gender, tone, etc)
- jgehring 8y ago
  When training speech recognition systems you want to use data that closely matches your target domain. Models trained on audiobooks read by professionals will not perform very well for transcribing conversational or spontaneous speech or if there is background noise.
  - abakker 8y ago
    But, if I understand correctly, systems can be trained separately on "this is background noise" and then apply those filters first, and then work with cleaned audio, right? I've been using krisp.ai for a few weeks and it has been fantastic at doing exactly that in real-time. Regarding conversational speech, I get that. Books are definitely not conversational. I guess the next question though, would be: is the objective to build a model that understands all words, or conversational speech? <novice> It seems like transfer learning on a model trained on audiobooks and then conversations would still be a good path, right? </novice>
    - anoncake 8y ago
      > I guess the next question though, would be: is the objective to build a model that understands all words, or conversational speech? <novice> It seems like transfer learning on a model trained on audiobooks and then conversations would still be a good path, right? </novice> Understanding all words is not the problem. I don't know if it's universal, but frequently, a speech-to-text model is actually two models: A voice model (mapping raw audio to phonemes) and a language model (which models what the language looks like, i.e. what sentences are likely and which words exist). So if you want the STT system to understand novels, include novels in the training data for the language model. You can then combine it with a voice model suitable for conversational speech/the user's accent/background noise.
deleted 8y ago
[deleted]
echelon 8y ago
This is amazing! I've been using LJS trained models and then cross-training them to target speakers, but this looks like it may produce even higher quality results. I've previously implemented concatenative TTS using unit selection [1]. The quality is spotty, so I'm throwing it out and going with the ML approach, which produces higher fidelity voices even in my own experiments. My next steps are taking an end-to-end synthesis model and porting it to run cheaply on the CPU. Thanks so much for making this data available, Mozilla! You're helping democratize this technology for individual engineers and researchers that don't have Google's resources. [1] http://trumped.com http://trumped.com
vkaku 8y ago
Great. Is there an open voice assistant project yet? Or a privacy focussed one?
- intopieces 8y ago
  Yes! Here’s a recent HN discussion about itD https://news.ycombinator.com/item?id=19152561 https://news.ycombinator.com/item?id=19152561
- j88439h84 8y ago
  Naomi project
deleted 8y ago
[deleted]
16bytes 8y ago
It's great to see innovation in the space of open data. There have recently been a number of assertions that better quality ML data will outperform better ML algorithms, and this has certainly been true in my experience as well, especially in domains like speech recognition. There's going to be a long road to catch up to the big players, however. Even 15 years ago there were companies who were doing 1M minutes of labled voice data per year. The data gap between established players and newcomers to the market will continue to grow unless we invest in efforts like this.
- novaRom 8y ago
  Original Deep Speech 2 paper released few years ago mentioned a hundred thousand or two hundred thousand hours, but the amount of data has increased since then significantly. Still quite a lot of languages have very tiny datasets of transcribed data.
elektor 8y ago
My voice is in this dataset! For the past few months, I've been on the iOS CommonVoice app reading sentences out loud. It's great fun, I'd recommend it.
a3_nm 8y ago
I'm puzzled by the "You agree to not attempt to determine the identity of speakers in the Common Voice dataset". On some level it's a good idea to want to request this, but as the dataset is public-domain, isn't it going to get mirrored and retrieved by people who won't have to agree to anything? ...
- lucb1e 8y ago
  It has no legal meaning whatsoever since, indeed, it's public domain. Or maybe it can affect the downloader, but certainly not anyone who got the data from the downloader (without such promise). I think the point is to remind people that they aren't cool with it and that because you can (CC0) it doesn't mean you should.
Theodores 8y ago
In reading the comments here I see that some people do use voice with their hand rectangles for other purposes than speaking to someone. I never do even though there is some Google Assistant icon staring at me. If I accidentally start recording my voice by fumbling buttons then I instinctively try to stop it so I can continue pecking into the keyboard. Is it a generational thing and will future generations find typing into a search box as anachronistic as I find using a land line with a rotary dial? I am also British and therefore not as loud as some people in the English speaking world. Talking to my phone on the train would make me cringe. Clearly people like me will die off soon enough, however, is adoption a problem for these voice technology things? How do people get into changing habits from pecking at a keyboard to the evidently easier voice driven way of doing things? Is there one use case, e.g. in the car, where the habit of speaking to a gadget is learned?
- BenjiWiebe 8y ago
  For a while i had voice commands working/enabled on my phone, and it was nice being able to read messages and reply to them while commuting, without taking my hands off the wheel or my eyes from the road. Had the accuracy been better, I'd probably be using it yet.
- MattSayar 8y ago
  When I was one-handed for a while after surgery, I would use my phone's speech-to-text function often to send emails. Far faster than one-handed typing.
deleted 8y ago
[deleted]
robax 8y ago
It would be super rad if someone plugged this dataset into CMU Sphinx. Google is the only decent ASR and a competitive open source alternative would be awesome. A big thank you to Mozilla for this dataset.
microcolonel 8y ago
Do they have the audio in a different format than MP3? MP3 is really garbage and introduces weird biases for voice.
- microcolonel 8y ago
  As an update, in looking at the code it seems they hardcode the choice of MP3 for storage in the server. Also very weird that they serve the tarball of MP3s gzipped, which seems mostly pointless to me, as it amounts to a reduction of maybe ~4% for a tremendous amount of time spent uncompressing the tarball (which itself has a bunch of useless macOS-specific headers, three apiece on each file). Many of the MP3 files are literally just empty (zero size) or partially written (corrupt), it seems. I wonder if that issue comes from their choice to package the tarballs on macOS, or some underlying issue on the server side.
  - nshm 8y ago
    I think its on purpose to not give too much advantage to downloaders, there was a discussion long time ago, but they still continue with mp3. There are also other not so nice points: https://github.com/kaldi-asr/kaldi/issues/2141 https://github.com/kaldi-asr/kaldi/issues/2141
dabinat 8y ago
I’ve been contributing to Common Voice for several months now. If anyone else is thinking of making contributions, it’s worth mentioning that there are a lot more speakers than validators and English currently has a 1 year validation backlog, so new validators are more useful right now than new speakers.
- gdfasfklshg4 8y ago
  Interesting! I started doing some validation a while ago then stopped because I figured it would be the other way around and I wasn't prepared to speak. I will start validating again!
Wowfunhappy 8y ago
I was decided to spend 10-ish minutes validating voices, because why not. I did not hear any lines that were flat-out spoken incorrectly, at least as far as I could tell. However, I did come across a ton of really poor samples, to the point of being somewhat difficult to understand. Things like: • Really strong accents • Horrible, muffled microphones • Background noise • Super quiet • A couple "robotic" samples I legitimately think were generated via text-to-speech software All of these types of samples—save the last—constitute possible real-world scenarios. But, do they make for good training data? I very little about machine learning, but it makes logical sense to me that you'd want to teach the computer with "clean" data—something with a high signal-to-noise ratio which is as close to the "average" of the real world as possible. Is this completely wrong? Separately, they ought to provide this type of instruction on what to do with borderline samples. If I legitimately can't tell for sure whether a word was spoken correctly, what should I do?
- opportune 8y ago
  Yes, you do want the bad voice samples precisely because they correspond to actual input that a model might receive. A dataset with only clear samples would likely have a stronger "signal" overall meaning it might be easier for an academic testing a model to get higher test accuracy training on that data. The bad data makes the model more robust to non-ideal input types
  - XMPPwocky 8y ago
    In fact, it's not uncommon to take a dataset and deliberately, randomly distort it- for images, things like scaling, rotating, cropping, blurring, altering color balance and gamma, flipping, adding random noise... The idea is to make the model resistant to that "bad" input and effectively enlarge your dataset for free- if you have a picture of a cat, you can automatically also get loads of pictures that you know should still be classified as a cat- rotated 15 degrees clockwise, noisy (like in low-light conditions), the tip of its tail is out of frame, the camera's automatic white balance screws up... Also, the robotic samples may be real human voices mangled by LPC- Think a lossy VoIP call.
- 8y ago
stunt 8y ago
Sadly, this is an area that big companies are not contributing to keep their competitive advantage. Really glad to see Mozilla is trying to change that.
ShiftEnter 8y ago
Did anyone notice that the gender distributions are heavily skewed towards males? The website states a 10% for female for English and a stunning 2% for Dutch even.

705 points by Vinnl 8y ago

I'm interested in using this for text to speech, rather than speech to text. Is wavenet still the state of the art for training on a dataset like this?

1ris 8y ago
The biggest feature of this data set is that it includes lots of different accents and non-native speakers. This should improve voice recognition in these areas, but I'm not sure if it is that usful for voice generation.

sgc 8y ago

I love this. I do a lot of dictation on mobile devices for work, with middling, and perhaps more importantly, frustrating results (needless to say we are working on programming our way out of that hole). It is an area ripe for open source progress given the failure of larger companies with large proprietary data sets to make basic common sense decisions in their transcription algorithms, together with no way to provide impactful feedback. If anybody is interested, there is definitely a market for a more robust dictation library that can be integrated into apps and works offline. It just needs to be professional - e.g. allow for preferences including the ability to indicate a strong preference for standard language and grammar over all slang, not forcing Title Case for anything resembling a brand name, having a training mode for words and phrases of the user's choosing, blacklisting of certain word or phrase results which are false positives, and proper learning from user corrections during use so the tedium of correcting the same phrase 100 times disappears.

99052882514569 8y ago
There's also a huge need for better transcription software in radiology. Existing ones are expensive and are just not good enough to be an actual time-saver. (A radiologist friend describes switching from a human medical transcriptionist to one of these "AI" software thingies as a cost-cutting measure by his hospital, or more accurately as a cost-offload measure. Hospital offloads salary, radiologist spends more time correcting stupid transcription mistakes for no extra pay).
- MattSayar 8y ago
  I thought it would be faster to correct a transcript than type it up fresh. I can type pretty damn quickly, but still slower than the average person talks. I would rather follow along to an AI transcript and correct a few words than type it all new.
  - TheSpiceIsLife 8y ago
    Faster for who? The in the case described above the radiologist does none of the typing and none of the correction.
  - jcims 8y ago
    Problem is that transcription errors never have a misspelling and usually sound ok in yourself nene voice. You have to really pay attention to find them (reading the text backwards a sentence at a time sometimes helps with this)
    - derefr 8y ago
      Imagine a professional “assisted stenography” software package. Rather than automatic full transcription + proof-reading after the fact, it would transcribe “live” but with any words where its model gave a low-confidence output highlighted with the expectation that the human stenographer, listening to the audio at the same time the machine is, will type in the correct transcription (or just hit tab to accept, like in autocomplete.)
      - jcims 8y ago
        I would love it if I could always see the top three possible interpretations of what I just said to my phone. It's interesting watching the speech recognition think about what's being said and flip words around once it has enough context.
- chapium 8y ago
  Thats a pretty cynical view, not entirely unjustified. Transcription services are costly, but dictation software offers plenty of other benefits.

abakker 8y ago

I have a (probably dumb) question: why don't we just use audiobooks for this? There are thousands and thousands of hours where the transcripts were written, and then read aloud. Some of them are now public domain. I'm sure some validation would need to be done, but it seems like there would be an endless validation set there. Am I missing something obvious?

applecrazy 8y ago
Audiobook narrators enunciate (almost) perfectly. Real people don't.
wetpaws 8y ago
Pretty sure it wont be nearly enough both from the language variety point of view and the variety of voices (age, gender, tone, etc)
jgehring 8y ago
When training speech recognition systems you want to use data that closely matches your target domain. Models trained on audiobooks read by professionals will not perform very well for transcribing conversational or spontaneous speech or if there is background noise.
- abakker 8y ago
  But, if I understand correctly, systems can be trained separately on "this is background noise" and then apply those filters first, and then work with cleaned audio, right? I've been using krisp.ai for a few weeks and it has been fantastic at doing exactly that in real-time. Regarding conversational speech, I get that. Books are definitely not conversational. I guess the next question though, would be: is the objective to build a model that understands all words, or conversational speech? <novice> It seems like transfer learning on a model trained on audiobooks and then conversations would still be a good path, right? </novice>
  - anoncake 8y ago
    > I guess the next question though, would be: is the objective to build a model that understands all words, or conversational speech? <novice> It seems like transfer learning on a model trained on audiobooks and then conversations would still be a good path, right? </novice> Understanding all words is not the problem. I don't know if it's universal, but frequently, a speech-to-text model is actually two models: A voice model (mapping raw audio to phonemes) and a language model (which models what the language looks like, i.e. what sentences are likely and which words exist). So if you want the STT system to understand novels, include novels in the training data for the language model. You can then combine it with a voice model suitable for conversational speech/the user's accent/background noise.

deleted 8y ago

[deleted]

echelon 8y ago

This is amazing! I've been using LJS trained models and then cross-training them to target speakers, but this looks like it may produce even higher quality results. I've previously implemented concatenative TTS using unit selection [1]. The quality is spotty, so I'm throwing it out and going with the ML approach, which produces higher fidelity voices even in my own experiments. My next steps are taking an end-to-end synthesis model and porting it to run cheaply on the CPU. Thanks so much for making this data available, Mozilla! You're helping democratize this technology for individual engineers and researchers that don't have Google's resources. [1] http://trumped.com http://trumped.com

vkaku 8y ago

Great. Is there an open voice assistant project yet? Or a privacy focussed one?

intopieces 8y ago
Yes! Here’s a recent HN discussion about itD https://news.ycombinator.com/item?id=19152561 https://news.ycombinator.com/item?id=19152561
j88439h84 8y ago
Naomi project

deleted 8y ago

[deleted]

16bytes 8y ago

It's great to see innovation in the space of open data. There have recently been a number of assertions that better quality ML data will outperform better ML algorithms, and this has certainly been true in my experience as well, especially in domains like speech recognition. There's going to be a long road to catch up to the big players, however. Even 15 years ago there were companies who were doing 1M minutes of labled voice data per year. The data gap between established players and newcomers to the market will continue to grow unless we invest in efforts like this.

novaRom 8y ago
Original Deep Speech 2 paper released few years ago mentioned a hundred thousand or two hundred thousand hours, but the amount of data has increased since then significantly. Still quite a lot of languages have very tiny datasets of transcribed data.

elektor 8y ago

My voice is in this dataset! For the past few months, I've been on the iOS CommonVoice app reading sentences out loud. It's great fun, I'd recommend it.

a3_nm 8y ago

I'm puzzled by the "You agree to not attempt to determine the identity of speakers in the Common Voice dataset". On some level it's a good idea to want to request this, but as the dataset is public-domain, isn't it going to get mirrored and retrieved by people who won't have to agree to anything? ...

lucb1e 8y ago
It has no legal meaning whatsoever since, indeed, it's public domain. Or maybe it can affect the downloader, but certainly not anyone who got the data from the downloader (without such promise). I think the point is to remind people that they aren't cool with it and that because you can (CC0) it doesn't mean you should.

Theodores 8y ago

In reading the comments here I see that some people do use voice with their hand rectangles for other purposes than speaking to someone. I never do even though there is some Google Assistant icon staring at me. If I accidentally start recording my voice by fumbling buttons then I instinctively try to stop it so I can continue pecking into the keyboard. Is it a generational thing and will future generations find typing into a search box as anachronistic as I find using a land line with a rotary dial? I am also British and therefore not as loud as some people in the English speaking world. Talking to my phone on the train would make me cringe. Clearly people like me will die off soon enough, however, is adoption a problem for these voice technology things? How do people get into changing habits from pecking at a keyboard to the evidently easier voice driven way of doing things? Is there one use case, e.g. in the car, where the habit of speaking to a gadget is learned?

BenjiWiebe 8y ago
For a while i had voice commands working/enabled on my phone, and it was nice being able to read messages and reply to them while commuting, without taking my hands off the wheel or my eyes from the road. Had the accuracy been better, I'd probably be using it yet.
MattSayar 8y ago
When I was one-handed for a while after surgery, I would use my phone's speech-to-text function often to send emails. Far faster than one-handed typing.

deleted 8y ago

[deleted]

robax 8y ago

It would be super rad if someone plugged this dataset into CMU Sphinx. Google is the only decent ASR and a competitive open source alternative would be awesome. A big thank you to Mozilla for this dataset.

microcolonel 8y ago

Do they have the audio in a different format than MP3? MP3 is really garbage and introduces weird biases for voice.

microcolonel 8y ago
As an update, in looking at the code it seems they hardcode the choice of MP3 for storage in the server. Also very weird that they serve the tarball of MP3s gzipped, which seems mostly pointless to me, as it amounts to a reduction of maybe ~4% for a tremendous amount of time spent uncompressing the tarball (which itself has a bunch of useless macOS-specific headers, three apiece on each file). Many of the MP3 files are literally just empty (zero size) or partially written (corrupt), it seems. I wonder if that issue comes from their choice to package the tarballs on macOS, or some underlying issue on the server side.
- nshm 8y ago
  I think its on purpose to not give too much advantage to downloaders, there was a discussion long time ago, but they still continue with mp3. There are also other not so nice points: https://github.com/kaldi-asr/kaldi/issues/2141 https://github.com/kaldi-asr/kaldi/issues/2141

dabinat 8y ago

I’ve been contributing to Common Voice for several months now. If anyone else is thinking of making contributions, it’s worth mentioning that there are a lot more speakers than validators and English currently has a 1 year validation backlog, so new validators are more useful right now than new speakers.

gdfasfklshg4 8y ago
Interesting! I started doing some validation a while ago then stopped because I figured it would be the other way around and I wasn't prepared to speak. I will start validating again!

Wowfunhappy 8y ago

I was decided to spend 10-ish minutes validating voices, because why not. I did not hear any lines that were flat-out spoken incorrectly, at least as far as I could tell. However, I did come across a ton of really poor samples, to the point of being somewhat difficult to understand. Things like: • Really strong accents • Horrible, muffled microphones • Background noise • Super quiet • A couple "robotic" samples I legitimately think were generated via text-to-speech software All of these types of samples—save the last—constitute possible real-world scenarios. But, do they make for good training data? I very little about machine learning, but it makes logical sense to me that you'd want to teach the computer with "clean" data—something with a high signal-to-noise ratio which is as close to the "average" of the real world as possible. Is this completely wrong? Separately, they ought to provide this type of instruction on what to do with borderline samples. If I legitimately can't tell for sure whether a word was spoken correctly, what should I do?

opportune 8y ago
Yes, you do want the bad voice samples precisely because they correspond to actual input that a model might receive. A dataset with only clear samples would likely have a stronger "signal" overall meaning it might be easier for an academic testing a model to get higher test accuracy training on that data. The bad data makes the model more robust to non-ideal input types
- XMPPwocky 8y ago
  In fact, it's not uncommon to take a dataset and deliberately, randomly distort it- for images, things like scaling, rotating, cropping, blurring, altering color balance and gamma, flipping, adding random noise... The idea is to make the model resistant to that "bad" input and effectively enlarge your dataset for free- if you have a picture of a cat, you can automatically also get loads of pictures that you know should still be classified as a cat- rotated 15 degrees clockwise, noisy (like in low-light conditions), the tip of its tail is out of frame, the camera's automatic white balance screws up... Also, the robotic samples may be real human voices mangled by LPC- Think a lossy VoIP call.
8y ago

stunt 8y ago

Sadly, this is an area that big companies are not contributing to keep their competitive advantage. Really glad to see Mozilla is trying to change that.

ShiftEnter 8y ago

Did anyone notice that the gender distributions are heavily skewed towards males? The website states a 10% for female for English and a stunning 2% for Dutch even.