duvenaud

12 ms·

91.
by duvenaud 6y ago
I think everyone who uses probabilities to make decisions agrees that all our models are flawed in many ways. But how else do you propose to weigh different possibilities? As for your proposal for how to spend money, I agree those are frui
92.
by duvenaud 8y ago
> But with a given Jacobian function there's no finite differences or forward mode. Right, but instantiating an entire Jacobian is always going to scale at least quadratically with time. The point I was trying to make is that the e
93.
by duvenaud 8y ago
Thanks for the detailed remarks, and for pointing out that my statement above about the novelty of our implementation was wrong - CasADi uses the same algorithm as was released in 2013. My apologies. Joel Andersson pointed this out to us
94.
by duvenaud 8y ago
Well, you can differentiate through the operations of ODE solvers written in any framework that has autodiff. But the whole idea of adjoint sensitivity analysis is to not differentiate through the operations of the solver, to save memory a
95.
by duvenaud 8y ago
Whoops, I just realized that I failed to qualify that statement properly. CasADi [ https://web.casadi.org/ ], released in 2013, uses adjoint sensitivities to compute gradients through ODE solutions using symbolic (but still
96.
by duvenaud 8y ago
It does, but that's just a coincidence. That's a visualization of the latent space for the toy time-series problem in our paper that has to learn to model spirals of different sizes and orientations.
97.
by duvenaud 8y ago
Ah, that would stop certain dimensions from changing, but the output would still be the same size.
98.
by duvenaud 8y ago
Section 7 is titled "Related Work".
99.
by duvenaud 8y ago
1. Great question! You're correct that standard NF isn't efficiently invertible. CNF is, and we wanted a fair comparison. So for this experiment, we reversed the direction that NF transforms the data, so that it goes from the d
100.
by duvenaud 8y ago
My understanding is that the advantage of something like a SNN is mainly in custom-hardware ultra-lower-power applications. But it adds a lot of complexity and makes training a lot more difficult, compared to something like ODE nets or sta
101.
by duvenaud 8y ago
None in this paper, which is just showing proofs of concept. But our follow-up paper does have SOTA density modeling among the class of efficiently-sample-able generative models. This is because the design of continuous flow models is les
102.
by duvenaud 8y ago
Interesting question. There might be, but that's more a question of fitting parameters. This paper was about a different way to set up a parametric model, that can be trained in the usual way. I don't think the fact that intern
103.
by duvenaud 8y ago
1. As the other reply said, we just mean an ODE solver whose internal operations are a detail we don't need to worry about. 2. This paper only had proofs of concepts and toy demos. But our follow-up paper, FFJORD [ https://
104.
by duvenaud 8y ago
I did a CS undergrad at the University of Manitoba. Then took some time off to do a startup and was in the army reserves. Then when to UBC to do an MSc in CS + Stats. My Phd was officially in the made-up-sounding subject of "Informa
105.
by duvenaud 8y ago
The comparison we make is to residual networks, which I think is valid. First, we do parameterize a theta that changes with time, using a hypernet. But this is equivalent to the way sampo wrote the equations above - you can just feed time
106.
by duvenaud 8y ago
Good question. The model developed in that paper, (neural autoregressive flows) is a discrete-layered architecture. It's a member of the Normalizing Flows family of models. Normalizing flows define a parametric density by transformi
107.
by duvenaud 8y ago
Ooo, great question. In both cases, we define a function using a sum over an infinite number of infinitesimal things. In the GP case, those things are random and independent of each other, so the central limit theorem applies and we get a
108.
by duvenaud 8y ago
The closest thing to a continuous-time RNN that exists are Neural Hawkes Processes [ https://arxiv.org/abs/1612.09328 ]. They use a different model where observing the system necessarily changes its state, which is natu
109.
by duvenaud 8y ago
Sure thing. A few years ago, everyone switched their deep nets to "residual nets". Instead of building deep models like this: h1 = f1(x) h2 = f2(h1) h3 = f3(h2) h4 = f3(h3) y = f5(h4) They now build them like thi
110.
by duvenaud 8y ago
Thanks! If I understand your question, you're asking if the ODE time-series model we proposed works best in things like idealized frictionless or gravity-less setting? The answer is that we expect it to work (in principle) even for m
111.
by duvenaud 8y ago
Senior author here, I'm happy to answer any questions. We just released source code: https://github.com/rtqichen/torchdiffeq . This includes PyTorch implementations of adaptive ODE solvers that can be differentiat
112.
by duvenaud 10y ago
That diagram is just meant to show what type the inputs and outputs of the neural network are. I'm not a chemist myself, but the other two authors are.
113.
by duvenaud 10y ago
Yes, the datasets were the real limiting factor for this project, and as you point out, reactions depend on many more things than their reagents. Hopefully this work will inspire someone to build a better dataset of reaction setups and out
114.
by duvenaud 10y ago
Thank you! I guess all you need to do to get chemistry on hacker news is add neural nets to it. I honestly don't know much about synthesis or the state of computationally assisted inverse design of molecules, except that it seems like
115.
by duvenaud 10y ago
Thanks for the kind words! I would agree that the function we're trying to learn is very complicated, with many exceptions. But that only improves our comparative advantage, at least compared to novice chemists. We might also be abl
116.
by duvenaud 10y ago
Good points. However, even if chemists already know how to predict reactions, giving this knowledge to a machine will allow a much faster search over possible synthetic routes. I agree that reaction outcomes depend on many other factors be
117.
by duvenaud 10y ago
One of the authors here. Ask us anything! This paper is just a first step - what we'd really like to use this for is designing recipes for synthesizing new molecules. I would also be remiss if I didn't link to a closely-related p
118.
by duvenaud 10y ago
Thank you very much, Max! To answer your questions: - As you say, jointly training the autoencoder for reconstruction and prediction accuracy would be easy to code, but it might be tricky to get the tradeoff right since we usually have rel
119.
by duvenaud 10y ago
One of the authors of the original paper https://arxiv.org/abs/1610.02415 here. From a machine learning point of view, we simply glued together two techniques: text autoencoders and Bayesian optimization. That is, we t
120.
by duvenaud 11y ago
As far as I understand, the adjoint method computes gradients for functions that obey hard constraints (such as fluid solvers), and the main advantage is that it avoids differentiating through iterative constraint satisfaction procedures. I

More ›