3 ms·

Thanks for your insightful comment. I'll read the links later. I had just assumed that RNNs were TC, didn't think of limitation put on by bounded precision sin

Thanks for your insightful comment. I'll read the links later. I had just assumed that RNNs were TC, didn't think of limitation put on by bounded precision since I assumed that any bounded precision could be compensated by growing memory module. So, after your comment, I went literature searching and I found this: https://papers.nips.cc/paper_files/paper/2021/hash/ef452c63f81d0105dd4486f775adec81-Abstract.html https://papers.nips.cc/paper_files/paper/2021/hash/ef452c63f... I haven't read it yet. But if it is true, then RNN would seem to be TC

nyrikki 2y ago
Note from the open review of that paper. > As discussed in the paper and pointed out by the reviewer, the growing memory module is non-differentiable, and so it cannot be trained directly by SGD. We acknowledge this observation. Two stack FSA/RNN are interesting, but as of now, not usable in practice.