These end-to-end speech recognition systems are very intriguing. One major limitation is that since they don't model phonetics, they have no great way to deal w

These end-to-end speech recognition systems are very intriguing. One major limitation is that since they don't model phonetics, they have no great way to deal with highly irregular orthography that doesn't show up in the training data. For example, there is no great way for the system to learn that the pronunciation "black" can be spelled "6LACK" sometimes. The paper on arXiv goes into how they deal with this. Basically they run a traditional WFST decoder over the output of the RNN-T to take spelling context into account. Still, it's impressive how far the neural system can get with no explicit lexicon or acoustic modeling in general.