3 ms·
> 21. That every expression graph built from differentiable elementary functions and producing a scalar output has a gradient that can itself be written as an e
by stevenhuang 11mo ago
> 21. That every expression graph built from differentiable elementary functions and producing a scalar output has a gradient that can itself be written as an expression graph, and furthermore that the latter expression graph is always the same size as the first one and is easy to find, and thus that it’s possible to fit very large expression graphs to data.
> 22. That, eerily, biological life and biological intelligence does not appear to make use of that property of expression graphs.
Claim 22 is interesting. I can believe that it isn't immediately apparent because biological life is too complex (putting it mildly), but is that the extent of it?
- kragen 11mo agoWe haven't found anything in nature that resembles reverse-mode automatic differentiation, either in evolution or in neuroscience.
- mountainreason 11mo agoWhat would that look like. Is there an implementation of the finite difference method in biology somehow?
- kragen 11mo agoI'd be surprised.
- mountainreason 11mo agoYeah thanks, me too. So I'm wondering what point 22 is getting at. How could that even be possible?
- kragen 11mo agoBiology can use discrete methods; we have discrete neurons (and other cells), discrete chemical species like ATP, discrete amino acids, discrete genes made of discrete nucleotides, etc.
- mountainreason 11mo agoFur sure. But I don't get how that leads to an expectation that our biology is computing and utilizing derivatives in a computational graph. Point 22 seems to imply that the other finds it notable that that isn't happening.
- mountainreason 11mo agoAuthor*