8 ms·

Senior author here, I'm happy to answer any questions. We just released source code: https://github.com/rtqichen/torchdiffeq https://github.com/rtqichen/torchd

Senior author here, I'm happy to answer any questions. We just released source code: https://github.com/rtqichen/torchdiffeq https://github.com/rtqichen/torchdiffeq . This includes PyTorch implementations of adaptive ODE solvers that can be differentiated through automatically. So you can mix and match these ODE solvers with any other differentiable model component. There's already been a bit of follow-up work, turning Continuous Normalizing Flows into a practical generative density model: https://arxiv.org/abs/1810.01367 https://arxiv.org/abs/1810.01367 And now we're mainly working on 1) Regularizing ODE nets to be faster to solve and 2) getting the time-series model to scale up and extend it to stochastic differential equations.

ssivark 8y ago
Could you place the work in context, and provide a simplified explanation for someone who understands math and ML, but is not familiar with the literature on normalizing flows and autoencoders? Thanks! I tried reading, but the abstract and introductory section were a little too terse for me :-)
- duvenaud 8y ago
  Sure thing. A few years ago, everyone switched their deep nets to "residual nets". Instead of building deep models like this: h1 = f1(x) h2 = f2(h1) h3 = f3(h2) h4 = f3(h3) y = f5(h4) They now build them like this: h1 = f1(x) + x h2 = f2(h1) + h1 h3 = f3(h2) + h2 h4 = f4(h3) + h3 y = f5(h4) + h4 Where f1, f2, etc are neural net layers. The idea is that it's easier to model a small change to an almost-correct answer than to output the whole improved answer at once. In the last couple of years a few different groups noticed that this looks like a primitive ODE solver (Euler's method) that solves the trajectory of a system by just taking small steps in the direction of the system dynamics and adding them up. They used this connection to propose things like better training methods. We just took this idea to its logical extreme: What if we _define_ a deep net as a continuously evolving system? So instead of updating the hidden units layer by layer, we define their derivative with respect to depth instead. We call this an ODE net. Now, we can use off-the-shelf adaptive ODE solvers to compute the final state of these dynamics, and call that the output of the neural network. This has drawbacks (it's slower to train) but lots of advantages too: We can loosen the numerical tolerance of the solver to make our nets faster at test time. We can also handle continuous-time models a lot more naturally. It turns out that there is also a simpler version of the change of variables formula (for density modeling) when you move to continuous time.
  - im3w1l 8y ago
    So one question about that. In h1 = f1(x) + x h2 = f2(h1) + h1 h3 = f3(h2) + h2 h4 = f4(h3) + h3 y = f5(h4) + h4 the functions are all different. But to see it as "a primitive ODE solver", then the functions should be the same? So if I understand correctly you have a different take on RNN's but not on deep residual nets in general?
    - gundeep59 8y ago
      They model the above as dh(t)/dt, generalizing the discrete case (the equations you wrote) to a continuous case. Check the eq 2 in the paper. The statement following the equation makes it clear that "Starting from the input layer h(0), we can define the output layer h(T) to be the solution to this ODE initial value problem at some time T". Here, as per my understanding, h(0) can be the input itself. The function f mentioned in the eq 2 is the RNN cell.
    - sampo 8y ago
      > then the functions should be the same? If we conceptually think that advancing from one neural net layer to the next one is the same as taking a time step with an ODE solver, then a bit more precise notation would be h1 = f(t=1,x) + x h2 = f(t=2,h1) + h1 h3 = f(t=3,h2) + h2 h4 = f(t=4,h3) + h3 y = f(t=5,h4) + h4 Now you can say that the function f is always the same, but it still can give very different values for Δh when evaluated at different time points.
      - soVeryTired 8y ago
        I do think it's misleading to compare the method to a general feed-forward network though, for two reasons. First, to preserve the analogy between eq. 1 and 2, the thetas in equation two should have their own dynamics, which should be learned. Second, even if Equation 1 doesn't allow it, in a general feed-forward network it's possible for the state to change dimension between layers. I don't see how that could happen with the continuous model. Neat paper, but it'd be nice if they had tied the analogy more explicitly to RNNs in the introduction.
- innagadadavida 8y ago
  From a software standpoint, will any of these ideas be ported to TensorFlow or is this very different?
ArtWomb 8y ago
Congrats on winning Best Paper ;) Is the Zero-G environment of deep space the ideal environment for continuous differential models?
- duvenaud 8y ago
  Thanks! If I understand your question, you're asking if the ODE time-series model we proposed works best in things like idealized frictionless or gravity-less setting? The answer is that we expect it to work (in principle) even for messy, non-physics-experiment-style situations. This because we don't model the system dynamics directly, but in a latent space that we learn. So even in a messy situation like health monitoring of humans, as long as the data are being driven mostly by hidden factors that are changing smoothly through time (like overall health, infections, hormones or whatever) the neural ODE can abstract away from the messy data. This is all still just in principle - there are a few technical challenges to scaling these models up. But nothing that looks insurmountable right now.
  - zygotic12 8y ago
    Stunning work. Please don't waste yourself in Finance - I bet your phone has already rung.
byt143 8y ago
How does this relate to previous work that's described NNets in the limit of infinite width as GPs?
- duvenaud 8y ago
  Ooo, great question. In both cases, we define a function using a sum over an infinite number of infinitesimal things. In the GP case, those things are random and independent of each other, so the central limit theorem applies and we get a simple Gaussian. In the ODE case, the infinitesimals are deterministic and depend on the previous ones in the sequence, so the final answer is deterministic and impossible to compute exactly in general. You could also use a GP to model the dynamics of an ODE, and this was done recently: [https://arxiv.org/abs/1803.04303 https://arxiv.org/abs/1803.04303] although the drawback was that they couldn't train the model by backpropagation.
diego898 8y ago
Hello! Can you comment on how this relates to: https://arxiv.org/pdf/1804.00779.pdf https://arxiv.org/pdf/1804.00779.pdf
- duvenaud 8y ago
  Good question. The model developed in that paper, (neural autoregressive flows) is a discrete-layered architecture. It's a member of the Normalizing Flows family of models. Normalizing flows define a parametric density by transforming a sample from a Gaussian by a series of transformations: z0 ~ Normal(0, I) z1 = f1(z0) z2 = f2(z1) z3 = f3(z2) x = f4(z3) and they use the change of variables formula to compute p(x): log p(x) = log p(z0) + log det | df/dz0 | In our paper, we propose a continuous-time version of normalizing flows, called Continuous Normalizing Flows. We derived a continuous-time version of the change of variables formula: dlogp(z(t)) = -trace(df/dz) Anyways, the confusion is probably that both models are called flows. We wish that Normalizing Flows had used a different name, so that we could save the word "flow" for continuous-time transformations, but it's too late for that :) Neural autoregressive flows are powerful density models, but are computationally costly to sample from. Continuous normalizing flows cost about the same to evaluate densities and to sample. We compared the two in a follow-up paper: [https://arxiv.org/abs/1810.01367 https://arxiv.org/abs/1810.01367]
joe_the_user 8y ago
What results do you have on state of the art metrics? Also, any outline of how one would go about training one of your systems? What kind of dataset does it do best with?
- duvenaud 8y ago
  None in this paper, which is just showing proofs of concept. But our follow-up paper does have SOTA density modeling among the class of efficiently-sample-able generative models. This is because the design of continuous flow models is less constrained than for discrete flows. But other than that, it's still early days for this whole model class. You train these models in the same way you train a neural network, with stochastic gradient descent. For supervised learning, its main benefit is extra flexibility in the speed/precision tradeoff. For time-series problems, we expect this will ultimately allow us to handle data that's collected at irregular intervals - but we haven't yet moved past the prototype stage in that setting.
iandanforth 8y ago
Several recent papers take steps to isolate weights from updates during training to prevent catastrophic forgetting. In the ODE formulation is there a way to do something similar?
- duvenaud 8y ago
  Interesting question. There might be, but that's more a question of fitting parameters. This paper was about a different way to set up a parametric model, that can be trained in the usual way. I don't think the fact that internally it uses ODEs changes its susceptibility to catastrophic forgetting in the online learning setting.
ratman 8y ago
Hi, 1. What is a black box ode solver? (It sounds like a proprietary tool but that doesn't make sense and googling didn't bring up anything useful) 2. Have you found any areas where there is a significant difference in the quality of the results using this method rather then the normal discrete method? I am sorry if these questions are a bit bellow you, or if you already answered them in the paper.
- theSage 8y ago
  Black box means you have no access to how a system actually does some task. All you know is that given an input the system will provide the output. ODE solver is an ordinary differential equation solver. For example [a] [a] : https://docs.scipy.org/doc/scipy/reference/generated/scipy.integrate.ode.html https://docs.scipy.org/doc/scipy/reference/generated/scipy.i...
- duvenaud 8y ago
  1. As the other reply said, we just mean an ODE solver whose internal operations are a detail we don't need to worry about. 2. This paper only had proofs of concepts and toy demos. But our follow-up paper, FFJORD [https://arxiv.org/abs/1810.01367 https://arxiv.org/abs/1810.01367] we used the continuous normalizing flows to get SOTA efficiently sample-able density models.
stealthcat 8y ago
How is this different from spiking neural networks (SNN)? Seems like easier and simplified (e.g. no synapse) and densely connected version of SNN, with less control on inter neuron connections. Last time, I read work to port SNN (with ODE solver) to Pytorch https://github.com/Hananel-Hazan/bindsnet https://github.com/Hananel-Hazan/bindsnet https://www.frontiersin.org/articles/10.3389/fninf.2018.00089/full https://www.frontiersin.org/articles/10.3389/fninf.2018.0008...
- duvenaud 8y ago
  My understanding is that the advantage of something like a SNN is mainly in custom-hardware ultra-lower-power applications. But it adds a lot of complexity and makes training a lot more difficult, compared to something like ODE nets or standard discrete neural nets. Looking at the links you provided, they don't appear to train the network dynamics, only the feedout weights. In principle you could differentiate through the ODE solvers used in the software package linked to train the dynamics, but as far as I know we were the first people to release an open-source set of differentiable solvers using reverse-mode adjoint sensitivities (the most scalable variant of autodiff for training scalar losses). [https://github.com/rtqichen/torchdiffeq https://github.com/rtqichen/torchdiffeq] In the paper we talk a bit about how you can now more easily train Possion process likelihoods, which might make training SNNs easier. I'm not sure.
  - ssfrr 8y ago
    > as far as I know we were the first people to release an open-source set of differentiable solvers using reverse-mode adjoint sensitivities Would the DifferentialEquations.jl work in the Julia community qualify? As I understand it for the pure Julia solvers you can run them through autodiff to differentiate with respect to the parameters. Chris Rackauckas has talked a lot about how cool it is that he can take his solvers that were not written to be AD-aware and use somebody else’s AD package that’s not specialized for diff eqs, and combined you get differentialable diff eqs.
    - ChrisRackauckas 8y ago
ExpectMax 8y ago
Thanks for this awesome peace of research! I'm really looking forward to further developments in the field :) I have two small questions regarding the paper: 1. When comparing to Normalizing Flows (planar flows), in Section 4.1, how were these fitted in the Maximum Likelihood Training section? If I understand it correctly NF's don't have a closed form inverse, s.t. ML training should not be possible. 2. Do you encounter any issues regarding stability during training? Other Flow based approaches such as Glow use certain tricks to ensure that the Flow initial reduces to an identity transform, to increase stability and ensure reliable convergence.
- duvenaud 8y ago
  1. Great question! You're correct that standard NF isn't efficiently invertible. CNF is, and we wanted a fair comparison. So for this experiment, we reversed the direction that NF transforms the data, so that it goes from the data to the latent space. Training this way means that you can't use the resulting model as a generator, but it at least let us compare likelihoods with CNF for this paper. 2. We had to set the error tolerance relatively small during training to keep the gradients stable. I don't think we used any fancy initialization tricks, but to be honest I have to ask Ricky Chen and Will Grathwohl, who ran all the FFJORD experiments.
rkevingibson 8y ago
I'm a little late to this thread, but wanted to thank you for the paper! I found it interesting enough to blog about it (https://rkevingibson.github.io/blog/neural-networks-as-ordinary-differential-equations/ https://rkevingibson.github.io/blog/neural-networks-as-ordin...). I wonder if you've given any thought to generalizing to fractional differential equations? My intuition tells me that the dynamics that you're learning are "local" in the sense that the ODE solvers depend only on the current state (and maybe some recent history), whereas learning the dynamics of a fractional system could give the system a larger "history" in the case of your time-series models.
wjhuggins 8y ago
Would it be possible to apply the kind of reasoning you used to a discretization of space also? One could imagine starting with a convolutional neural network that is also a residual network (I may be butchering the proper terminology here) and taking the limit of an infinitely fine discretization in space as well as time to arrive at a PDE instead of an ODE. Would the adjoint method approach that you used for backpropagation work in this case?
bionhoward 8y ago
Would it be possible to apply ODE nets to Universal [Solomonoff] Induction Program [Levin] Search Tensor Factorization ? Bion Howard (a biologist) medium.com/bit-pharma bitpharma.com
MielMe 8y ago
Was the backwards integration stable? It would be surprising from a mathematical perspective of ODEs.

ssivark 8y ago

Could you place the work in context, and provide a simplified explanation for someone who understands math and ML, but is not familiar with the literature on normalizing flows and autoencoders? Thanks! I tried reading, but the abstract and introductory section were a little too terse for me :-)

duvenaud 8y ago
Sure thing. A few years ago, everyone switched their deep nets to "residual nets". Instead of building deep models like this: h1 = f1(x) h2 = f2(h1) h3 = f3(h2) h4 = f3(h3) y = f5(h4) They now build them like this: h1 = f1(x) + x h2 = f2(h1) + h1 h3 = f3(h2) + h2 h4 = f4(h3) + h3 y = f5(h4) + h4 Where f1, f2, etc are neural net layers. The idea is that it's easier to model a small change to an almost-correct answer than to output the whole improved answer at once. In the last couple of years a few different groups noticed that this looks like a primitive ODE solver (Euler's method) that solves the trajectory of a system by just taking small steps in the direction of the system dynamics and adding them up. They used this connection to propose things like better training methods. We just took this idea to its logical extreme: What if we _define_ a deep net as a continuously evolving system? So instead of updating the hidden units layer by layer, we define their derivative with respect to depth instead. We call this an ODE net. Now, we can use off-the-shelf adaptive ODE solvers to compute the final state of these dynamics, and call that the output of the neural network. This has drawbacks (it's slower to train) but lots of advantages too: We can loosen the numerical tolerance of the solver to make our nets faster at test time. We can also handle continuous-time models a lot more naturally. It turns out that there is also a simpler version of the change of variables formula (for density modeling) when you move to continuous time.
- im3w1l 8y ago
  So one question about that. In h1 = f1(x) + x h2 = f2(h1) + h1 h3 = f3(h2) + h2 h4 = f4(h3) + h3 y = f5(h4) + h4 the functions are all different. But to see it as "a primitive ODE solver", then the functions should be the same? So if I understand correctly you have a different take on RNN's but not on deep residual nets in general?
  - gundeep59 8y ago
    They model the above as dh(t)/dt, generalizing the discrete case (the equations you wrote) to a continuous case. Check the eq 2 in the paper. The statement following the equation makes it clear that "Starting from the input layer h(0), we can define the output layer h(T) to be the solution to this ODE initial value problem at some time T". Here, as per my understanding, h(0) can be the input itself. The function f mentioned in the eq 2 is the RNN cell.
  - sampo 8y ago
    > then the functions should be the same? If we conceptually think that advancing from one neural net layer to the next one is the same as taking a time step with an ODE solver, then a bit more precise notation would be h1 = f(t=1,x) + x h2 = f(t=2,h1) + h1 h3 = f(t=3,h2) + h2 h4 = f(t=4,h3) + h3 y = f(t=5,h4) + h4 Now you can say that the function f is always the same, but it still can give very different values for Δh when evaluated at different time points.
    - soVeryTired 8y ago
      I do think it's misleading to compare the method to a general feed-forward network though, for two reasons. First, to preserve the analogy between eq. 1 and 2, the thetas in equation two should have their own dynamics, which should be learned. Second, even if Equation 1 doesn't allow it, in a general feed-forward network it's possible for the state to change dimension between layers. I don't see how that could happen with the continuous model. Neat paper, but it'd be nice if they had tied the analogy more explicitly to RNNs in the introduction.
innagadadavida 8y ago
From a software standpoint, will any of these ideas be ported to TensorFlow or is this very different?

ArtWomb 8y ago

Congrats on winning Best Paper ;) Is the Zero-G environment of deep space the ideal environment for continuous differential models?

duvenaud 8y ago
Thanks! If I understand your question, you're asking if the ODE time-series model we proposed works best in things like idealized frictionless or gravity-less setting? The answer is that we expect it to work (in principle) even for messy, non-physics-experiment-style situations. This because we don't model the system dynamics directly, but in a latent space that we learn. So even in a messy situation like health monitoring of humans, as long as the data are being driven mostly by hidden factors that are changing smoothly through time (like overall health, infections, hormones or whatever) the neural ODE can abstract away from the messy data. This is all still just in principle - there are a few technical challenges to scaling these models up. But nothing that looks insurmountable right now.
- zygotic12 8y ago
  Stunning work. Please don't waste yourself in Finance - I bet your phone has already rung.

byt143 8y ago

How does this relate to previous work that's described NNets in the limit of infinite width as GPs?

duvenaud 8y ago
Ooo, great question. In both cases, we define a function using a sum over an infinite number of infinitesimal things. In the GP case, those things are random and independent of each other, so the central limit theorem applies and we get a simple Gaussian. In the ODE case, the infinitesimals are deterministic and depend on the previous ones in the sequence, so the final answer is deterministic and impossible to compute exactly in general. You could also use a GP to model the dynamics of an ODE, and this was done recently: [https://arxiv.org/abs/1803.04303 https://arxiv.org/abs/1803.04303] although the drawback was that they couldn't train the model by backpropagation.

diego898 8y ago

Hello! Can you comment on how this relates to: https://arxiv.org/pdf/1804.00779.pdf https://arxiv.org/pdf/1804.00779.pdf

duvenaud 8y ago
Good question. The model developed in that paper, (neural autoregressive flows) is a discrete-layered architecture. It's a member of the Normalizing Flows family of models. Normalizing flows define a parametric density by transforming a sample from a Gaussian by a series of transformations: z0 ~ Normal(0, I) z1 = f1(z0) z2 = f2(z1) z3 = f3(z2) x = f4(z3) and they use the change of variables formula to compute p(x): log p(x) = log p(z0) + log det | df/dz0 | In our paper, we propose a continuous-time version of normalizing flows, called Continuous Normalizing Flows. We derived a continuous-time version of the change of variables formula: dlogp(z(t)) = -trace(df/dz) Anyways, the confusion is probably that both models are called flows. We wish that Normalizing Flows had used a different name, so that we could save the word "flow" for continuous-time transformations, but it's too late for that :) Neural autoregressive flows are powerful density models, but are computationally costly to sample from. Continuous normalizing flows cost about the same to evaluate densities and to sample. We compared the two in a follow-up paper: [https://arxiv.org/abs/1810.01367 https://arxiv.org/abs/1810.01367]

joe_the_user 8y ago

What results do you have on state of the art metrics? Also, any outline of how one would go about training one of your systems? What kind of dataset does it do best with?

duvenaud 8y ago
None in this paper, which is just showing proofs of concept. But our follow-up paper does have SOTA density modeling among the class of efficiently-sample-able generative models. This is because the design of continuous flow models is less constrained than for discrete flows. But other than that, it's still early days for this whole model class. You train these models in the same way you train a neural network, with stochastic gradient descent. For supervised learning, its main benefit is extra flexibility in the speed/precision tradeoff. For time-series problems, we expect this will ultimately allow us to handle data that's collected at irregular intervals - but we haven't yet moved past the prototype stage in that setting.

iandanforth 8y ago

Several recent papers take steps to isolate weights from updates during training to prevent catastrophic forgetting. In the ODE formulation is there a way to do something similar?

duvenaud 8y ago
Interesting question. There might be, but that's more a question of fitting parameters. This paper was about a different way to set up a parametric model, that can be trained in the usual way. I don't think the fact that internally it uses ODEs changes its susceptibility to catastrophic forgetting in the online learning setting.

ratman 8y ago

Hi, 1. What is a black box ode solver? (It sounds like a proprietary tool but that doesn't make sense and googling didn't bring up anything useful) 2. Have you found any areas where there is a significant difference in the quality of the results using this method rather then the normal discrete method? I am sorry if these questions are a bit bellow you, or if you already answered them in the paper.

theSage 8y ago
Black box means you have no access to how a system actually does some task. All you know is that given an input the system will provide the output. ODE solver is an ordinary differential equation solver. For example [a] [a] : https://docs.scipy.org/doc/scipy/reference/generated/scipy.integrate.ode.html https://docs.scipy.org/doc/scipy/reference/generated/scipy.i...
duvenaud 8y ago
1. As the other reply said, we just mean an ODE solver whose internal operations are a detail we don't need to worry about. 2. This paper only had proofs of concepts and toy demos. But our follow-up paper, FFJORD [https://arxiv.org/abs/1810.01367 https://arxiv.org/abs/1810.01367] we used the continuous normalizing flows to get SOTA efficiently sample-able density models.

stealthcat 8y ago

How is this different from spiking neural networks (SNN)? Seems like easier and simplified (e.g. no synapse) and densely connected version of SNN, with less control on inter neuron connections. Last time, I read work to port SNN (with ODE solver) to Pytorch https://github.com/Hananel-Hazan/bindsnet https://github.com/Hananel-Hazan/bindsnet https://www.frontiersin.org/articles/10.3389/fninf.2018.00089/full https://www.frontiersin.org/articles/10.3389/fninf.2018.0008...

duvenaud 8y ago
My understanding is that the advantage of something like a SNN is mainly in custom-hardware ultra-lower-power applications. But it adds a lot of complexity and makes training a lot more difficult, compared to something like ODE nets or standard discrete neural nets. Looking at the links you provided, they don't appear to train the network dynamics, only the feedout weights. In principle you could differentiate through the ODE solvers used in the software package linked to train the dynamics, but as far as I know we were the first people to release an open-source set of differentiable solvers using reverse-mode adjoint sensitivities (the most scalable variant of autodiff for training scalar losses). [https://github.com/rtqichen/torchdiffeq https://github.com/rtqichen/torchdiffeq] In the paper we talk a bit about how you can now more easily train Possion process likelihoods, which might make training SNNs easier. I'm not sure.
- ssfrr 8y ago
  > as far as I know we were the first people to release an open-source set of differentiable solvers using reverse-mode adjoint sensitivities Would the DifferentialEquations.jl work in the Julia community qualify? As I understand it for the pure Julia solvers you can run them through autodiff to differentiate with respect to the parameters. Chris Rackauckas has talked a lot about how cool it is that he can take his solvers that were not written to be AD-aware and use somebody else’s AD package that’s not specialized for diff eqs, and combined you get differentialable diff eqs.
  - ChrisRackauckas 8y ago

ExpectMax 8y ago

Thanks for this awesome peace of research! I'm really looking forward to further developments in the field :) I have two small questions regarding the paper: 1. When comparing to Normalizing Flows (planar flows), in Section 4.1, how were these fitted in the Maximum Likelihood Training section? If I understand it correctly NF's don't have a closed form inverse, s.t. ML training should not be possible. 2. Do you encounter any issues regarding stability during training? Other Flow based approaches such as Glow use certain tricks to ensure that the Flow initial reduces to an identity transform, to increase stability and ensure reliable convergence.

duvenaud 8y ago
1. Great question! You're correct that standard NF isn't efficiently invertible. CNF is, and we wanted a fair comparison. So for this experiment, we reversed the direction that NF transforms the data, so that it goes from the data to the latent space. Training this way means that you can't use the resulting model as a generator, but it at least let us compare likelihoods with CNF for this paper. 2. We had to set the error tolerance relatively small during training to keep the gradients stable. I don't think we used any fancy initialization tricks, but to be honest I have to ask Ricky Chen and Will Grathwohl, who ran all the FFJORD experiments.

rkevingibson 8y ago

I'm a little late to this thread, but wanted to thank you for the paper! I found it interesting enough to blog about it (https://rkevingibson.github.io/blog/neural-networks-as-ordinary-differential-equations/ https://rkevingibson.github.io/blog/neural-networks-as-ordin...). I wonder if you've given any thought to generalizing to fractional differential equations? My intuition tells me that the dynamics that you're learning are "local" in the sense that the ODE solvers depend only on the current state (and maybe some recent history), whereas learning the dynamics of a fractional system could give the system a larger "history" in the case of your time-series models.

wjhuggins 8y ago

Would it be possible to apply the kind of reasoning you used to a discretization of space also? One could imagine starting with a convolutional neural network that is also a residual network (I may be butchering the proper terminology here) and taking the limit of an infinitely fine discretization in space as well as time to arrive at a PDE instead of an ODE. Would the adjoint method approach that you used for backpropagation work in this case?

bionhoward 8y ago

Would it be possible to apply ODE nets to Universal [Solomonoff] Induction Program [Levin] Search Tensor Factorization ? Bion Howard (a biologist) medium.com/bit-pharma bitpharma.com

MielMe 8y ago

Was the backwards integration stable? It would be surprising from a mathematical perspective of ODEs.