4 ms·
Seems like it will be hard to get it to work in Tensorflow, because it needs to compute the gradient in an unusual way, which afaik don't play nice with the exi
by GistNoesis 8y ago
Seems like it will be hard to get it to work in Tensorflow, because it needs to compute the gradient in an unusual way, which afaik don't play nice with the existing architecture.
My guess is it will need some deep wizardry of the same kind as OpenAI gradient check-pointing.
Neural ODE, is a nice trick to reduce the memory usage to O(1) instead of O(nb timesteps). But the implementation cost and complexity cost probably mean we are better using gradient check-pointing on a forward dynamic and pay the memory cost.
It will also probably won't play well with noise.
Are there any implementation of it in tensorflow yet?
- jsinai 8y agoThere is a PyTorch implementation by the authors https://github.com/rtqichen/torchdiffeq https://github.com/rtqichen/torchdiffeq and there is a Julia implementation due: https://news.ycombinator.com/item?id=18676986#18691054 https://news.ycombinator.com/item?id=18676986#18691054. As for TensorFlow, we will have to wait and see if it can be done. The adjoint method is matrix multiplication, but the tricky part will also be integrating with numerical ODE solvers.
- deleted 8y ago[deleted]
- 317070 8y agoI have one in tf, so I can confirm it is possible.
- GistNoesis 8y agoCool. This mean that hopefully there will be some tf open source implementations in the future. I guess there is something I don't see. I'm intrigued, does your code run inside a single sess.run() so that it can be composed nicely? If so did you use a "special trick"?
- liuliu 8y agoThis is the first time I seriously read the paper and the code. Does this mean for ODE net, we are sharing weights for f? (even though we evaluate f at different y and t points)? If that is the case, seems you can implement a full static version of ODE net with tf.While.