4 ms·
Is that blowup perhaps an artifact of displaying the result as an expression, but where the result as an internal data structure is more compact, because of sha
by abecedarius 11y ago
Is that blowup perhaps an artifact of displaying the result as an expression, but where the result as an internal data structure is more compact, because of sharing? That's what I'd expect, although I haven't tried Theano yet.
That is, I'd expect the sin((x + x)) which appears twice to be just one internal node which is referenced twice, and so on.
- gh02t 11y agoThat's my understanding, it does not compute the expression directly unless you ask it to. Could be wrong though, I've only used it a few times. AD does involve symbolic differentiation of each individual instruction, but instead of keeping the symbolic expression it is replaced with the numeric value, so the whole expression is never computed. It isn't magic and there are cases where it blows up. For instance, there are two basic ways to implement it, forward and backward propagation, each of which is good for a certain situation that the other is terrible at.