Great writeup! I like the detail, especially in Making a Great Speech To Text Model. It looks like their research is against an older wav2letter model (likely

Great writeup! I like the detail, especially in Making a Great Speech To Text Model. It looks like their research is against an older wav2letter model (likely the conv_glu model from the original 2018 Gated ConvNet paper). Facebook has released a lot of interesting architectures since then, my favorite of which is the Streaming ConvNet arch [1] Ideas from the post compared to my experience with wav2letter: 1. Model Stride - increased model stride to 8x Streaming ConvNet increased the stride to 8, while the original conv_glu model had stride 2 (cite: [2]), so Facebook agrees on this one. 2. Compact Regularized Networks - use separable convolutions, add skip connections, and attention modules Streaming ConvNets use Time-Depth Separable Convolutions, but no skip connections or attention. Facebook's "state of the art 2019" research tried a lot of architectures, the most accurate of which was a transformer model trained with attention and seq2seq criterion. However in my experience it was very large and slow to train. 3. Using Byte-Pair Encoding Streaming ConvNets model uses 10,000 word pieces, which is a similar thought (the token set FB uses is here: [4] but I've had better luck with a sentencepiece model trained on more text data) instead of a ~26 letter alphabet from the original conv_glu model. 4. Better Encoder The blog post conclusion here isn't very obvious, it seems to hand wave that you could maybe use a transformer architecture but doesn't go into specifics. 5. Balance Capacity I don't completely understand what they changed here but it seems useful. 6. Stabilize the Training in Different Domains, Balance Generalization When adding domains to wav2letter, I've basically trained entirely from scratch using all available data. I long for a catastrophic forgetting mitigation like EWC but for now I've had a lot of success by just making the data funnel bigger and more varied. Data matters quite a bit, in this post [5] I compare a model I trained on a wide variety of data, with Facebook's equivalent model as well as Facebook's State Of The Art model. The wide-variety model generalized significantly better than both of Facebook's examples that had only trained on audiobooks. 7. Make a very fast decoder. The beam search decoder in this branch [6] of Talon's wav2letter fork was at the time significantly faster than Facebook's decoder. It was hand optimized and even includes some code for threaded decoding (which I'm not using at the moment because single threaded is fast enough for my workload). On single threaded decoding it can hit a realtime factor in the ballpark of 0.01x, or 100 seconds of audio per cpu second. With my smallest model (a significantly reduced size conv_glu wav2letter model), which is also in the 0.01x ballpark for encoding, we can consistently hit around 0.02x end to end on a consumer CPU. To demonstrate, here is a random sample of debug timing information from my machine when recently decoding command phrases with wav2letter during real interactive use (the very long audio samples aren't the model's fault, I was testing a mic in a noisy environment): [audio]=1260.000ms [emit]=16.164ms (0.01x) [decode]=11.326ms (0.01x) [total]=27.490ms (0.02x) [audio]=1260.000ms [emit]=16.164ms (0.01x) [decode]=11.326ms (0.01x) [total]=27.490ms (0.02x) [audio]=930.000ms [emit]=12.853ms (0.01x) [decode]=3.716ms (0.00x) [total]=16.569ms (0.02x) [audio]=1080.000ms [emit]=14.936ms (0.01x) [decode]=3.296ms (0.00x) [total]=18.231ms (0.02x) [audio]=1020.000ms [emit]=12.976ms (0.01x) [decode]=10.058ms (0.01x) [total]=23.034ms (0.02x) [audio]=930.000ms [emit]=13.836ms (0.01x) [decode]=9.998ms (0.01x) [total]=23.834ms (0.03x) [audio]=1140.000ms [emit]=14.535ms (0.01x) [decode]=3.589ms (0.00x) [total]=18.124ms (0.02x) [audio]=1650.000ms [emit]=20.516ms (0.01x) [decode]=5.281ms (0.00x) [total]=25.797ms (0.02x) [audio]=3480.000ms [emit]=31.386ms (0.01x) [decode]=32.966ms (0.01x) [total]=64.352ms (0.02x) [audio]=29760.000ms [emit]=253.671ms (0.01x) [decode]=325.198ms (0.01x) [total]=578.869ms (0.02x) [audio]=8850.000ms [emit]=70.844ms (0.01x) [decode]=76.267ms (0.01x) [total]=147.111ms (0.02x) [audio]=960.000ms [emit]=13.924ms (0.01x) [decode]=8.135ms (0.01x) [total]=22.060ms (0.02x) [audio]=1770.000ms [emit]=19.760ms (0.01x) [decode]=18.612ms (0.01x) [total]=38.372ms (0.02x) [audio]=930.000ms [emit]=12.488ms (0.01x) [decode]=6.842ms (0.01x) [total]=19.330ms (0.02x) [audio]=1050.000ms [emit]=15.726ms (0.01x) [decode]=7.645ms (0.01x) [total]=23.371ms (0.02x) [audio]=930.000ms [emit]=13.899ms (0.01x) [decode]=8.714ms (0.01x) [total]=22.613ms (0.02x) [audio]=40410.000ms [emit]=292.916ms (0.01x) [decode]=289.109ms (0.01x) [total]=582.025ms (0.01x) [1] https://github.com/facebookresearch/wav2letter/tree/master/recipes/models/streaming_convnets https://github.com/facebookresearch/wav2letter/tree/master/r... [2] https://github.com/facebookresearch/wav2letter/issues/543#issuecomment-588062135 https://github.com/facebookresearch/wav2letter/issues/543#is... [3] https://github.com/facebookresearch/wav2letter/tree/master/recipes/models/sota/2019 https://github.com/facebookresearch/wav2letter/tree/master/r... [4] https://dl.fbaipublicfiles.com/wav2letter/streaming_convnets/librispeech/librispeech-train-all-unigram-10000.tokens https://dl.fbaipublicfiles.com/wav2letter/streaming_convnets... [5] https://github.com/facebookresearch/wav2letter/issues/577 https://github.com/facebookresearch/wav2letter/issues/577 [6] https://github.com/talonvoice/wav2letter/tree/decoder https://github.com/talonvoice/wav2letter/tree/decoder