4 ms·

Viterbi algorithm, Baum Welch algorithm, forward-backward algorithm, beam search, etc, are all very closely related, all using basically the same kind of dynami

Viterbi algorithm, Baum Welch algorithm, forward-backward algorithm, beam search, etc, are all very closely related, all using basically the same kind of dynamic programming. Viterbi calculates the max and argmax, Baum Welch (forward-backward) calculates the sum (forward) and then also the posteriors (backward). The difference is just the max vs sum. And to get the argmax, you would just backtrack. Beam search is an approximation for the max and argmax. Btw, using automatic differentiation, just calculating the forward scores (sum) is enough, automatic differentiation will then automatically lead to the backward pass. This is not just for HMMs but it applies to many kinds of search. Naturally for CTC (can be seen as a simple special case of HMM), transducer, but also encoder-decoder or decoder-only (language) models, and more.

chilukrn 3y ago
Do you have a good reference(s) that starts from the basics all the way to CTC loss? The distill.pub article is good, but personally doesn't provide a good enough intuition...
- albertzeyer 3y ago
  You can study CTC in isolation, ignoring all the HMM background. That is how CTC was also originally introduced, by mostly ignoring any of the existing HMM literature. So e.g. look at the original CTC paper. But I think the distill.pub article (https://distill.pub/2017/ctc/ https://distill.pub/2017/ctc/) is also good. For studying HMMs, any speech recognition lecture should cover that. We teach that at RWTH Aachen University but I don't think there are public recordings. But probably you should find some other lectures online somewhere. You also find a lot of tutorials for Kaldi: https://kaldi-asr.org/ https://kaldi-asr.org/ Maybe check this book: https://www.microsoft.com/en-us/research/publication/automatic-speech-recognition-a-deep-learning-approach/ https://www.microsoft.com/en-us/research/publication/automat... The relation of CTC and HMM becomes intuitively clear once you get the concept of HMMs. Often in terms of speech recognition, it is all formulated as finite state automata (FSA) (or finite state transducer (FST), or weighted FST (WFST)), and the CTC FST just looks a bit different (simpler) than the traditional HMMs, but in all cases, you can think about having states with possible transitions. This is all mostly about the modeling. The training is more different. For CTC, you often calculate the log prob of the full sequence over all possible alignments directly, while for HMMs, people often use a fixed alignment, and calculate framewise cross entropy. I did some research on the relation of CTC training and HMM training: https://www-i6.informatik.rwth-aachen.de/publications/download/1035/ZeyerAlbertBeckEugenSchl%FCterRalfNeyHermann--CTCintheContextofGeneralizedFull-SumHMMTraining--2017.pdf https://www-i6.informatik.rwth-aachen.de/publications/downlo... Maybe my PhD thesis gives a good overview: https://www-i6.informatik.rwth-aachen.de/publications/download/1223/Zeyer--2022.pdf https://www-i6.informatik.rwth-aachen.de/publications/downlo...

chilukrn 3y ago

Do you have a good reference(s) that starts from the basics all the way to CTC loss? The distill.pub article is good, but personally doesn't provide a good enough intuition...

albertzeyer 3y ago
You can study CTC in isolation, ignoring all the HMM background. That is how CTC was also originally introduced, by mostly ignoring any of the existing HMM literature. So e.g. look at the original CTC paper. But I think the distill.pub article (https://distill.pub/2017/ctc/ https://distill.pub/2017/ctc/) is also good. For studying HMMs, any speech recognition lecture should cover that. We teach that at RWTH Aachen University but I don't think there are public recordings. But probably you should find some other lectures online somewhere. You also find a lot of tutorials for Kaldi: https://kaldi-asr.org/ https://kaldi-asr.org/ Maybe check this book: https://www.microsoft.com/en-us/research/publication/automatic-speech-recognition-a-deep-learning-approach/ https://www.microsoft.com/en-us/research/publication/automat... The relation of CTC and HMM becomes intuitively clear once you get the concept of HMMs. Often in terms of speech recognition, it is all formulated as finite state automata (FSA) (or finite state transducer (FST), or weighted FST (WFST)), and the CTC FST just looks a bit different (simpler) than the traditional HMMs, but in all cases, you can think about having states with possible transitions. This is all mostly about the modeling. The training is more different. For CTC, you often calculate the log prob of the full sequence over all possible alignments directly, while for HMMs, people often use a fixed alignment, and calculate framewise cross entropy. I did some research on the relation of CTC training and HMM training: https://www-i6.informatik.rwth-aachen.de/publications/download/1035/ZeyerAlbertBeckEugenSchl%FCterRalfNeyHermann--CTCintheContextofGeneralizedFull-SumHMMTraining--2017.pdf https://www-i6.informatik.rwth-aachen.de/publications/downlo... Maybe my PhD thesis gives a good overview: https://www-i6.informatik.rwth-aachen.de/publications/download/1223/Zeyer--2022.pdf https://www-i6.informatik.rwth-aachen.de/publications/downlo...