3 ms·
One thing that is not entirely clear to me about constructions like these is the behavior of array indexing. I understand how to differentiate the assignment of
by whatshisface 8y ago
One thing that is not entirely clear to me about constructions like these is the behavior of array indexing. I understand how to differentiate the assignment of x+y into the variable z, but if I use X as an index into memory and then write Y at that location, the proper result of differentiation looks a lot less clear. One reason for that is my memory locations are discrete points, but X must be treated as if it was continuous, if functions over it are to be differentiable.
- radarsat1 8y agoMy understanding is that the trick to differentiable key-value stores is to treat the key as a probability over locations, and to treat the value as a weighted sum using these probabilities. From (pdf) https://aclweb.org/anthology/D16-1147 https://aclweb.org/anthology/D16-1147
- choeger 8y agoIntuitively you only use the value of X in that case, not the derivative. I don't know if that yields a consistent or useful language, though, since you'd also need to consider the reverse operation. In general I think that languages that need such a calculus (Modelica's functions come to mind) would show a lot of such complications when implements a rigorous description of the semantics.
- hjorthjort 8y agoFrom page 2: "The presented theoretical model enables analytic investigations of differentiable programs through algebraic tools [...]; i.e. the presented operators can take the same role as higher order functions in functional programming." So maybe we should think more in terms of lambda calculus and pure functional programming? There, you don't really write Y to memory location X. In section 5.5 they describe how they "generalize the notion of an iteration from the integers, n ∈ N, to the reals, x ∈ R, and consider fractional iterations."