6 ms·
Neural ODE's: Understanding how they model data
- quantumwoke 8y agoThe diagrams here really helped to explain neural ODEs in an intuitive fashion. Does anyone know the best library that implements them so I can play around in an iPyNB? :-)
- ChrisRackauckas 8y agohttps://news.ycombinator.com/item?id=18979541 https://news.ycombinator.com/item?id=18979541 shows (with code) how to build and train neural ODEs.
- jsinai 8y agoThanks. Looks fantastic. Also appreciate the feedback from parent.
- contravariant 8y ago> y(x) = f(x,y), y(x0) = y0 If this is supposed to be the ODE definition, shouldn't it be y'(x) = f(x,y)? Otherwise I don't quite understand the definition of 'f'.
- GistNoesis 8y agoSeems like it will be hard to get it to work in Tensorflow, because it needs to compute the gradient in an unusual way, which afaik don't play nice with the existing architecture. My guess is it will need some deep wizardry of the same kind as OpenAI gradient check-pointing. Neural ODE, is a nice trick to reduce the memory usage to O(1) instead of O(nb timesteps). But the implementation cost and complexity cost probably mean we are better using gradient check-pointing on a forward dynamic and pay the memory cost. It will also probably won't play well with noise. Are there any implementation of it in tensorflow yet?
- jsinai 8y agoThere is a PyTorch implementation by the authors https://github.com/rtqichen/torchdiffeq https://github.com/rtqichen/torchdiffeq and there is a Julia implementation due: https://news.ycombinator.com/item?id=18676986#18691054 https://news.ycombinator.com/item?id=18676986#18691054. As for TensorFlow, we will have to wait and see if it can be done. The adjoint method is matrix multiplication, but the tricky part will also be integrating with numerical ODE solvers.
- deleted 8y ago[deleted]
- 317070 8y agoI have one in tf, so I can confirm it is possible.
- GistNoesis 8y agoCool. This mean that hopefully there will be some tf open source implementations in the future. I guess there is something I don't see. I'm intrigued, does your code run inside a single sess.run() so that it can be composed nicely? If so did you use a "special trick"?
- liuliu 8y agoThis is the first time I seriously read the paper and the code. Does this mean for ODE net, we are sharing weights for f? (even though we evaluate f at different y and t points)? If that is the case, seems you can implement a full static version of ODE net with tf.While.
- briga 8y agoThanks for this--science needs more writers who are able to distill complex subjects into clear and readable form.
- jsinai 8y agoThanks, I appreciate the kind words.
- lucidrains 8y agoSecond this! Thank you for the wonderful explanation
- buboard 8y agoa quick intro to the concept is given by an author of the paper here: https://www.reddit.com/r/MachineLearning/comments/a65v5r/neural_ordinary_differential_equations_pdf/ebt11w6/ https://www.reddit.com/r/MachineLearning/comments/a65v5r/neu...
- madhadron 8y agoCan anyone comment on the relationship between this and the differential equations used by the connectionist neuroscientists back in the 1980's?