4 ms·
Interesting that they're using RNN transducer. I thought everyone's moved to CNN lately.
by p1esk 8y ago
Interesting that they're using RNN transducer. I thought everyone's moved to CNN lately.
- pakl 8y agoFeedforward CNNs cannot tolerate as much noise and real-world variability as RNNs.
- p1esk 8y agoCitation?
- robbomacrae 8y agoThere is this new type of trolling where someone just puts in bare minimal counterarguments and asks for citations knowing full well it's labour intensive to do so. If your not trolling.. your not doing a good job of avoiding these comparisons. The onus should be on you here to provide citations first that dissprove the person your replying to.
- fwip 8y agoClosely related to sea-lioning, in my opinion.
- stunami 8y agoAbsolutely.
- deleted 8y ago[deleted]
- p1esk 8y agoI'm not trolling. The statement that RNN are somehow more noise-tolerant than CNNs does not make sense to me, and is not based on any literature about noise tolerance in NNs that I'm familiar with. Also, no arguments have been provided as to why this could possibly be the case.
- pakl 8y agoHi p1esk, RNNs can be more tolerant to noise because they can learn transient or dynamic attractors. If the inputs move an RNN into an attractor, small changes due to noise make little difference to the state. Recurrence can help with robustness in some other very important ways as well. Citations for this dates from the 80s and 90s. I don't know the best reference offhand. You could look at some old Hinton stuff if you're a fan. Lots published on this.
- p1esk 8y agoThat’s handwaving. The only way to find out if rnns are more robust to noise than cnns is to test them on the same task, with the same inputs and the same noise. Preferably using similar number of parameters and achieving similar accuracy before noise is applied. Then gradually increase the amount of noise, and compare the impact. Nothing like this has been published AFAIK. After you have the results of this experiment you can try to explain them with attractors and what not, but I would be surprised if there was much difference. Would make a good paper though!
- rocky1138 8y ago> The onus should be on you here to provide citations first that dissprove the person your replying to. Unless you're dealing with opinions, I disagree. The onus is on the person trying to give evidence to actually give evidence.
- taneq 8y agoThere was a post here 9 days ago discussing "intellectual DoS attacks" which basically work like this: https://news.ycombinator.com/item?id=19293036 https://news.ycombinator.com/item?id=19293036
- lawrenceyan 8y agoCNN's are only mentioned because of potential processing considerations as computationally, they are easier to deal with. But given the nature of speech recognition, which is so highly temporally correlated, it shouldn't be a surprise that a recurrent neural network would be used. This is pretty much exactly the purpose the RNN type of model architecture was designed for. Also if you haven't looked into the properties of how exactly a RNN Transducer functions, I highly recommend doing so. They help resolve a great deal of problems that traditional RNNs and CNNs are unable to deal with.
- p1esk 8y agoI take it you haven't been following the field lately, have you? It is surprising because convnets just work better for speech recognition ([1] is the latest SOA). I'm guessing gated convolutional LMs are slower than RNN transducers when deployed on mobile. Can someone confirm? [1] https://arxiv.org/abs/1812.06864 https://arxiv.org/abs/1812.06864
- vsef 8y agoThe paper you site gets equal to SOA performance on Wall Street Journal and LibriSpeech test sets, both of which are clean, read speech, and not at all representative of what a phone or assistant recognizer deals with. The convnet described there also does not perform streaming recognition. The primary reason to be interested in convnets for speech is computational parallelism, not because they have especially strong results for accuracy.
- p1esk 8y agoI'm curious why people think that recurrent architectures are somehow more noise-tolerant. Where did this come from?
- vsef 8y agoThey're not that I know of. But the paper cited is just showing relative parity or slight improvement on relatively toy examples. The claim that convnets are the clear winner for speech in general/what everyone is doing now is just not true. I work in the field, a more accurate summary would be that there are a number of viable architectures that currently get fairly similar accuracy, but that have other pros/cons with respect to streaming, memory use, parallelism, model size, integration with external language models and context, complexity of the decoder, friendliness to different types of hardware etc.
- pault 8y agoSorry to go on a tangent, but this is the first time I've heard the word "transducer" outside of a conversation about clojure. Is it the same concept?
- crdrost 8y agoNot quite. A Clojure transducer from A to B is pure function which takes As to lists of Bs. To apply it to a sequence of As you flatMap it, so you apply it to each A in sequence and get a sequence of Bs which you splice into your final resulting sequence. Maps and filters are special cases, for example const mapTrans = fn => function* (x) { yield fn(x); }; const filterTrans = predicate => function* (x) { if (predicate(x)) { yield x; } }; const dupeTrans = n => function* (x) { for (let i = 0; i < n; i++) { yield x; } }; Clojure just observed that an isomorphism of these functions under Church encoding looked like f<B> -> f<A> for a special parametric type f, so could be composed with ordinary function composition, albeit backwards. An RNN transducer is fundamentally three functions: One takes a list of recent As to some C1. Another takes a list of recent Bs to some C2. A final one takes a C1 and C2 to a B. Rather than mapping over each input independently the RNN-T is allowed to learn something about the relationship of recent outputs to the next output, and the relationship of nearby inputs. Clojure transducers thus have an order insensitivity that RNN transducers are allowed to be sensitive to.
- braindead_in 8y agoDeepSpeech 3 from Baidu also uses transducers. http://research.baidu.com/Blog/index-view?id=90 http://research.baidu.com/Blog/index-view?id=90.