3 ms·
There is also the transformer approach (eventually with local attention to bound latency), (like I'm doing in my project (Work in Progress) : https://github.com
by GistNoesis 8y ago
There is also the transformer approach (eventually with local attention to bound latency), (like I'm doing in my project (Work in Progress) : https://github.com/GistNoesis/Wisteria/blob/master/SpeechToText.md https://github.com/GistNoesis/Wisteria/blob/master/SpeechToT... ), though it's in the same line of thought as the convolutional CTC.
The RNN-T is a nice idea though, if I understand it correctly it's another approach to the alignment problem.
In CTC, you are generating sequence like TTTTHHHEE CCCAAATT, which mean that your language model must deal with these repetitions, and you can't train using text without repetitions. In RNN-T you are learning to advance the cursor on either audio sequence or text sequence so as to maintain alignment, kind of like you do when you merge-sort two sorted lists, therefore it outputs THE CAT, and you can use a standard language model.
- p1esk 8y agoWhat is "convolutional CTC"? Gated convolutions as LM is similar to RNN-T idea [1], but you have to deal with softmax, so I'm not sure how well this would work in practice, especially on a mobile processor. [1] https://arxiv.org/abs/1612.08083 https://arxiv.org/abs/1612.08083
- blcArmadillo 8y agoI'm guessing Connectionist Temporal Classification: https://www.cs.toronto.edu/~graves/icml_2006.pdf https://www.cs.toronto.edu/~graves/icml_2006.pdf
- JulianSlzr 8y agoWe explored self-attention + CTC in our ICASSP 2019 paper (https://arxiv.org/abs/1901.10055 https://arxiv.org/abs/1901.10055). Our implementation uses internal infra, so we're a ways from releasing code :( Hoping paper details suffice and help with the parameter search, and happy to respond to questions over e-mail. Would love to see an open-source implementation with local or directed attention built out!
- p1esk 8y agoVery interesting! Perhaps I should revise my original comment to “everyone is moving to transformers lately” :)