4 ms·

I don't really see how the "old" methods were really less accessible. There were tools such as HTK, cmu sphinx, etc., or srilm for language modelling, each with

by woodson 12y ago

I don't really see how the "old" methods were really less accessible. There were tools such as HTK, cmu sphinx, etc., or srilm for language modelling, each with documentation and a large user base. Granted, a lot of fiddling is involved if one wants to use speaker adaptative training (MLLR, VTLN), feature transforms (HLDA, MLLT), MLP features (TANDEM), etc., but DNN approaches come with their own set of screws to tweak.. It's just hard to make something work really well for a specific use case; when contributors to an open-source project are all trying to scratch their own itch (make it work for their specific use [language, vocabulary, etc.]), the result may not be universally satisfying.

modeless 12y ago
The difference is that the old methods were large systems made up of many different pieces that all required a ton of domain knowledge specific to speech and language. Training DNNs requires a lot of knowledge about DNNs, but not nearly as much knowledge about speech. Knowledge of how to train DNNs is highly transferable between domains like speech and vision. Similarly, the actual code can be mostly shared as well; something like Theano would be just as suited to running speech nets as vision nets. I don't think we're quite there yet, but DNNs have the potential to replace every piece of the speech pipeline with one single net that gets audio samples on one side and spits out characters on the other. All those acronyms you mentioned (with many, many PhD theses behind them) will be irrelevant, in the same way that tons of previously successful specialized computer vision feature detectors (HoG, SIFT, SURF, etc) are now irrelevant to the state of the art in object recognition.
- kylebgorman 12y ago
  Your prediction that DNNs will replace much of the pipeline is very interesting to me, but I hypothesize that you're at least partially wrong. I predict that DNNs will impact early stages of the pipeline which operate on continuously valued inputs, but I am skeptical that that DNNs will ultimately be the best solution for late discrete processing (e.g., decoding, language modeling). That DNNs ever perform well in discrete classification tasks just tells me we haven't spent enough time feature-engineering.
  - modeless 12y ago
    It's the ability of DNNs to replace feature engineering that makes them interesting. They have completely obsoleted feature engineering in object recognition in just a few short years. Have you seen the latest DNN results in translation and image captioning? I think DNNs are quickly going to surpass the state of the art in language modeling.
- woodson 12y ago
  A lot of the methods I listed don't have anything to do with speech per se. They DO have something to do with how to use data to e.g. remove unwanted variability or achieve better class separation. What makes them old methods is that they were developed in the context of using Gaussian mixture models for modelling Hidden Markov model state output probabilities. As such you could perhaps apply them in classifying birdsong, gunshots, or whatever else (with varying success, of course..). I have no doubt that these methods and their acronyms will become irrelevant (perhaps they already are), but I guess some of the basic underlying ideas about variability will re-emerge in the training regime of DNNs. Sure, the algorithms (and implementations) for training DNNs are the same, but these ideas are incorporated in the preparation and handling of training data (compare that with augmentation, like creating translated images etc.).

by woodson 12y ago

modeless 12y ago

The difference is that the old methods were large systems made up of many different pieces that all required a ton of domain knowledge specific to speech and language. Training DNNs requires a lot of knowledge about DNNs, but not nearly as much knowledge about speech. Knowledge of how to train DNNs is highly transferable between domains like speech and vision. Similarly, the actual code can be mostly shared as well; something like Theano would be just as suited to running speech nets as vision nets. I don't think we're quite there yet, but DNNs have the potential to replace every piece of the speech pipeline with one single net that gets audio samples on one side and spits out characters on the other. All those acronyms you mentioned (with many, many PhD theses behind them) will be irrelevant, in the same way that tons of previously successful specialized computer vision feature detectors (HoG, SIFT, SURF, etc) are now irrelevant to the state of the art in object recognition.

kylebgorman 12y ago
Your prediction that DNNs will replace much of the pipeline is very interesting to me, but I hypothesize that you're at least partially wrong. I predict that DNNs will impact early stages of the pipeline which operate on continuously valued inputs, but I am skeptical that that DNNs will ultimately be the best solution for late discrete processing (e.g., decoding, language modeling). That DNNs ever perform well in discrete classification tasks just tells me we haven't spent enough time feature-engineering.
- modeless 12y ago
  It's the ability of DNNs to replace feature engineering that makes them interesting. They have completely obsoleted feature engineering in object recognition in just a few short years. Have you seen the latest DNN results in translation and image captioning? I think DNNs are quickly going to surpass the state of the art in language modeling.
woodson 12y ago
A lot of the methods I listed don't have anything to do with speech per se. They DO have something to do with how to use data to e.g. remove unwanted variability or achieve better class separation. What makes them old methods is that they were developed in the context of using Gaussian mixture models for modelling Hidden Markov model state output probabilities. As such you could perhaps apply them in classifying birdsong, gunshots, or whatever else (with varying success, of course..). I have no doubt that these methods and their acronyms will become irrelevant (perhaps they already are), but I guess some of the basic underlying ideas about variability will re-emerge in the training regime of DNNs. Sure, the algorithms (and implementations) for training DNNs are the same, but these ideas are incorporated in the preparation and handling of training data (compare that with augmentation, like creating translated images etc.).