Although this blog post focuses on learning small toy examples, I could see the approach scaling up to real problems in natural language processing. At one poi

Although this blog post focuses on learning small toy examples, I could see the approach scaling up to real problems in natural language processing. At one point, Google's own speech recognition system was built on a C++ finite state transducer library [1][2]. FSTs were used to represent the language model, reverse dictionary (map from phonemes to words), hidden markov models, and so on. Although FSTs were a common abstraction used at runtime, at training time, we still needed special-purpose ML algorithms to create models that were later "compiled" into FSTs. [3] So I could see how a general-purpose differentiable FSM, if scaled up, could actually produce a really powerful simplification -- you would just need one end-to-end learning algorithm to subsume all these various models. [1] https://www.openfst.org/twiki/bin/view/FST/WebHome https://www.openfst.org/twiki/bin/view/FST/WebHome [2] Much of this has obviously changed since the popularization of deep learning -- this is the 2011-era system. [3] A finite state transducer is a finite state machine with input and output labels, so that it translates between two regular languages