7 ms·

I don't mean to mean but: what is surprising about any of this ? Joseph Fourier's solution to the heat-equation (linear diffusion) was in fact the origin of th

I don't mean to mean but: what is surprising about any of this ? Joseph Fourier's solution to the heat-equation (linear diffusion) was in fact the origin of the FT. The high-freq coefficients decay (as -t^2 IIRC) in there; the reverse is also known to be "unstable" (numerically, and is singular from the equillibrium). More over, the reformulation doesn't immediately reveal some computational speedup, or a better alternative formulation (which is usually a measure of how valuable it is epistemically). (Edit: note that Heat-equation is more akin to the Fokker-Planck eqn, not actual Diffusion as an SDE as is used in Diffusion models).

ackbar03 2y ago
I think what's interesting about it is the inter-relation between different disciplines and how the ideas are connected. The connection between the heat-equation and the generative diffusion models we see to day, and its relation to the Fourier Transform would not have been immediately obvious to me.
joaogui1 2y ago
I mean you didn't mention autoregressive models anywhere in your comment, whereas the post is about the connection between diffusion and autoregressive modelling. Also it's a blog post, if it has figured out a speed-up or improved method it would probably have been a paper
aDyslecticCrow 2y ago
> What is surprising about any of this? Connections between fields drive new ideas. And this has especially been the case for recent AI progress. With the speed at which the field is moving, ideas that are obvious to some still have a significant chance of not being tried yet. Just as the connection between the Kalman filter and RNN models or the significant similarities between back-propagation and the whole field of control theory. If it's truly not surprising, then that's just another reason to try it out if nobody else has. Does everything always need to be immediately "useful"?

ackbar03 2y ago

I think what's interesting about it is the inter-relation between different disciplines and how the ideas are connected. The connection between the heat-equation and the generative diffusion models we see to day, and its relation to the Fourier Transform would not have been immediately obvious to me.

joaogui1 2y ago

I mean you didn't mention autoregressive models anywhere in your comment, whereas the post is about the connection between diffusion and autoregressive modelling. Also it's a blog post, if it has figured out a speed-up or improved method it would probably have been a paper

aDyslecticCrow 2y ago

> What is surprising about any of this? Connections between fields drive new ideas. And this has especially been the case for recent AI progress. With the speed at which the field is moving, ideas that are obvious to some still have a significant chance of not being tried yet. Just as the connection between the Kalman filter and RNN models or the significant similarities between back-propagation and the whole field of control theory. If it's truly not surprising, then that's just another reason to try it out if nobody else has. Does everything always need to be immediately "useful"?