4 ms·

The goal of making a large, publicly available training corpus for ASR is incredibly admirable, but this approach is problematic. People speak entirely differen

by gok 7y ago

The goal of making a large, publicly available training corpus for ASR is incredibly admirable, but this approach is problematic. People speak entirely differently when reading from a script. Models trained on read speech (like LibriSpeech) generally don't perform well on spontaneous speech test sets (like Switchboard). Transcribing speech that was read from a script isn't a particularly interesting problem. This effort would be more interesting if it could collect speech data in a more specific domain, like web search queries.

cco 7y ago
Why do you view this as admirable? I view it similarly to creating a large cache of explosives and munitions and then giving them away for free. Sure a lot of people might have some fun with them on the weekend but the largest impact of this technology will be used against people, not for them.
- RussianCow 7y ago
  > the largest impact of this technology will be used against people, not for them How so?
- rpmisms 7y ago
  Many of the people here are optimistic about the use of tech for the betterment of mankind.
- staktrace 7y ago
  With your line of reasoning, would you prefer a world where all the munitions were held by megavillains, or a world where both megavillains and regular people had access to the munitions?
  - minikites 7y ago
    The latter strategy doesn't seem to work out that well for the USA because it results in regular people using the munitions on other regular people.
    - zenography 7y ago
      To defend themselves, hundreds of thousands to millions of times per year. https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3194685 https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3194685
      - williamdclt 7y ago
        > indicate that defensive uses of guns by crime victims are far more common than offensive uses by criminals That's literally the worst case scenario. Victims escalating the violence.
        zenography 7y ago
        Are you saying that people defending themselves is worse than killing?
lunixbochs 7y ago
I have many general English speech recognition models for wav2letter trained on ~3000 hours of non-switchboard audio including 1k from Common Voice, and on several datasets I have collected or found myself. Common Voice has optional fields for age and accent in the dataset, but I don't use them. I just toss everything into a single bin. I've had users with many accents (including e.g. thick UK, India, Germany), report that my 3k hour model including Common Voice performed significantly better for their accent than a librispeech-only model, and also much better than the older macOS speech recognition English models. (And how a model feels for a user in the real world is something you can't really get by just testing against switchboard) I can't afford switchboard, so I'm very glad that datasets like Common Voice, librispeech, and TED-LIUM exist. I hope at some point I can get the model running in a website, it really does feel pretty good to use. For Talon we developed a custom speech collection site, that has a lot of prompts we control: https://speech.talonvoice.com https://speech.talonvoice.com - I tried it out with the TIMIT prompt list initially, but right now I'm recording dense command-like speech, which I've found some of my more experienced users are able to say naturally/quickly, not like they're reading from a prompt. I don't have a separate verification process, because I've had a lot of success with a process that automatically prunes inputs that "obviously make the model much worse". The site is basically designed to record as fast as possible, with keyboard shortcuts to go to the next item and start/stop recording so you can almost record nonstop, which ends up being slightly less forced than just reading one sentence. The site is pretty reusable, I previously used it at noise.talonvoice.com to record "noise recognition" samples. Here's the source of the current speech site, if someone wants to spin up a common-voice-lite for a specific domain it's pretty easy (just need to run a python app somewhere): https://github.com/talonvoice/noise/tree/speech-dataset https://github.com/talonvoice/noise/tree/speech-dataset
- gok 7y ago
  > how a model feels for a user in the real world is something you can't really get by just testing against switchboard Fair but statistical systems are also extremely susceptible to placebo. I would be curious what a test set shows.
  - lunixbochs 7y ago
cnxhk 7y ago
They recently released a subset of LibriSpeech called LibriTTS which consists only clean speech.
- lunixbochs 7y ago
  Thanks for the notice! I hadn't seen this release yet.
- woodson 7y ago
  As the name implies, LibriTTS is meant for people (researchers, really) to develop text-to-speech (TTS) systems. The point isn't that LibriSpeech isn't clean enough. Rather, it's that conversational speech is very different from read speech (which is based on written text). Everything has an effect, even planning of utterances (think: hesitations, "uhm", "uh"), turn taking behaviors (think: how speakers negotiate taking turns), how speakers self-correct, phonetic convergence (think: speakers adapting their speech to be more similar to that of their interlocutor), and so on. The Common Voice data won't help with that, as it's read speech. It's far more expensive to collect conversational speech datasets, as transcription (or correction of automatic transcripts) involves a lot of manual labor.

by gok 7y ago

cco 7y ago

Why do you view this as admirable? I view it similarly to creating a large cache of explosives and munitions and then giving them away for free. Sure a lot of people might have some fun with them on the weekend but the largest impact of this technology will be used against people, not for them.

RussianCow 7y ago
> the largest impact of this technology will be used against people, not for them How so?
rpmisms 7y ago
Many of the people here are optimistic about the use of tech for the betterment of mankind.
staktrace 7y ago
With your line of reasoning, would you prefer a world where all the munitions were held by megavillains, or a world where both megavillains and regular people had access to the munitions?
- minikites 7y ago
  The latter strategy doesn't seem to work out that well for the USA because it results in regular people using the munitions on other regular people.
  - zenography 7y ago
    To defend themselves, hundreds of thousands to millions of times per year. https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3194685 https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3194685
    - williamdclt 7y ago
      > indicate that defensive uses of guns by crime victims are far more common than offensive uses by criminals That's literally the worst case scenario. Victims escalating the violence.
      - zenography 7y ago
        Are you saying that people defending themselves is worse than killing?

lunixbochs 7y ago

I have many general English speech recognition models for wav2letter trained on ~3000 hours of non-switchboard audio including 1k from Common Voice, and on several datasets I have collected or found myself. Common Voice has optional fields for age and accent in the dataset, but I don't use them. I just toss everything into a single bin. I've had users with many accents (including e.g. thick UK, India, Germany), report that my 3k hour model including Common Voice performed significantly better for their accent than a librispeech-only model, and also much better than the older macOS speech recognition English models. (And how a model feels for a user in the real world is something you can't really get by just testing against switchboard) I can't afford switchboard, so I'm very glad that datasets like Common Voice, librispeech, and TED-LIUM exist. I hope at some point I can get the model running in a website, it really does feel pretty good to use. For Talon we developed a custom speech collection site, that has a lot of prompts we control: https://speech.talonvoice.com https://speech.talonvoice.com - I tried it out with the TIMIT prompt list initially, but right now I'm recording dense command-like speech, which I've found some of my more experienced users are able to say naturally/quickly, not like they're reading from a prompt. I don't have a separate verification process, because I've had a lot of success with a process that automatically prunes inputs that "obviously make the model much worse". The site is basically designed to record as fast as possible, with keyboard shortcuts to go to the next item and start/stop recording so you can almost record nonstop, which ends up being slightly less forced than just reading one sentence. The site is pretty reusable, I previously used it at noise.talonvoice.com to record "noise recognition" samples. Here's the source of the current speech site, if someone wants to spin up a common-voice-lite for a specific domain it's pretty easy (just need to run a python app somewhere): https://github.com/talonvoice/noise/tree/speech-dataset https://github.com/talonvoice/noise/tree/speech-dataset

gok 7y ago
> how a model feels for a user in the real world is something you can't really get by just testing against switchboard Fair but statistical systems are also extremely susceptible to placebo. I would be curious what a test set shows.
- lunixbochs 7y ago

cnxhk 7y ago

They recently released a subset of LibriSpeech called LibriTTS which consists only clean speech.

lunixbochs 7y ago
Thanks for the notice! I hadn't seen this release yet.
woodson 7y ago
As the name implies, LibriTTS is meant for people (researchers, really) to develop text-to-speech (TTS) systems. The point isn't that LibriSpeech isn't clean enough. Rather, it's that conversational speech is very different from read speech (which is based on written text). Everything has an effect, even planning of utterances (think: hesitations, "uhm", "uh"), turn taking behaviors (think: how speakers negotiate taking turns), how speakers self-correct, phonetic convergence (think: speakers adapting their speech to be more similar to that of their interlocutor), and so on. The Common Voice data won't help with that, as it's read speech. It's far more expensive to collect conversational speech datasets, as transcription (or correction of automatic transcripts) involves a lot of manual labor.