6 ms·

This says nothing new. This idea was around since the 80s, if not earlier, and used extensively for automatic differentiation textbook. But, for an audience th

This says nothing new. This idea was around since the 80s, if not earlier, and used extensively for automatic differentiation textbook. But, for an audience that seems to think it rides on the frontier of knowledge and that we have just discoveres things (when, in fact, it's our ignorance of earlier research that is the reason that we are re-discovering them), such a medium post might be like a drug :D

touisteur 3y ago
The interesting thing, to me is how this taskgraph representation of neural networks has taken off AND been optimized to hell, just looking at the techs behind tensorrt, jax, openvino, tvm,finn and the whole hell these tools make the graph go through to wring interesting performance, is amazing. For once, we almost standardised on one (or 2, or 3, relatively transpilable) simpler language/DSLs, and a whole industry jumped on optimizing them is rarely heard of, except for gcc and llvm. This actual 'at last a new reduced semantic that we can maybe optimized better' has such huge effort (and success) behind is the amazing part, for me.
garganzol 3y ago
Sometimes simple (and old) ideas start to click and ignite a tornado when the timing is right. While many may argue that the notion of a computational graph was associated with neural networks (NN) from the very beginning, this post is quite novel. It sheds the light on how NN and the general computation theory are actually interconnected, it is basically the same thing. For me, it was an instant blast: the computational graph reminds me a typical intermediate representation (IR) tree of a textbook compiler. And because it is a formal graph, all mathematical benefits of the graph theory suddenly start to click. For instance, the graph theory formalizes the definitions of cyclic directed graphs versus acyclic directed graphs. If we imagine for a moment that a graph vertex represents a unit of computation, and an edge represents a data flow, it immediately starts to resonate with Lambda calculus. It quickly becomes evident that when the graph is cyclic, it represents a recursion in terms of Lambda calculus and thus becomes Turing-complete. If computational graph is acyclic, Turing completeness is not achievable. If you are going to say that this is an obvious and well-known observation - I will be surprised, because it is not. And all that enlightenment was possible thanks to this article combined with a bit of knowledge about graphs, compilers, and lambda calculus. (It just so happened that I'm relatively well-versed in those topics due to a professional involvement.) --- If we continue to formalize this observation further, we may soon find a formal proof that a program P and a neural network NN are equivalent: P ~= NN Both have inputs (I) and outputs (O): O = P(I) O = NN(I) Both perform a computation by calculating output O for a given input I. The Turing-completeness observation and the direct mapping to and from the Lambda calculus will give us a way to translate an arbitrary program P to an equivalent neural network NN: P -> NN But the most intriguing part is that the inverse operation also becomes available: NN -> P Congratulations. We have just found a way to translate a working neural network NN to a formal program P written in a programming language L. What are consequences of that? The most obvious one is that we can now create a software that will be able to automatically translate a trained neural network NN to a working deterministic program P written in a programming language of choice and vice versa. We have just made a small step towards an imaginary AI system that can work as good as a professional software developer. Basically, one day we will be able to create a software-based software developer, a skilled one. And the level of skills will 100% surpass human abilities one day because Turing-completeness guarantees the unboundedness. The steps of NN -> P and P -> NN translation may be performed by the neural network itself. It seems that we already started to see that with ChatGPT 3. --- As you can see, one simple observation allowed us to precisely calculate the future for many years ahead. And this is why I love the math so much.
- IIAOPSW 3y ago
  I had an idea, perhaps crackpot, that will maybe catch fire for you too. What if the graph structure of the natural language parse tree was directly reflected in the literal biological neural network. What if the reason for natural language grammar is that it directly reflects the way it is processed under the hood.
- bjourne 3y ago
  You might be interested in the universal approximation theorem.

touisteur 3y ago

The interesting thing, to me is how this taskgraph representation of neural networks has taken off AND been optimized to hell, just looking at the techs behind tensorrt, jax, openvino, tvm,finn and the whole hell these tools make the graph go through to wring interesting performance, is amazing. For once, we almost standardised on one (or 2, or 3, relatively transpilable) simpler language/DSLs, and a whole industry jumped on optimizing them is rarely heard of, except for gcc and llvm. This actual 'at last a new reduced semantic that we can maybe optimized better' has such huge effort (and success) behind is the amazing part, for me.

garganzol 3y ago

Sometimes simple (and old) ideas start to click and ignite a tornado when the timing is right. While many may argue that the notion of a computational graph was associated with neural networks (NN) from the very beginning, this post is quite novel. It sheds the light on how NN and the general computation theory are actually interconnected, it is basically the same thing. For me, it was an instant blast: the computational graph reminds me a typical intermediate representation (IR) tree of a textbook compiler. And because it is a formal graph, all mathematical benefits of the graph theory suddenly start to click. For instance, the graph theory formalizes the definitions of cyclic directed graphs versus acyclic directed graphs. If we imagine for a moment that a graph vertex represents a unit of computation, and an edge represents a data flow, it immediately starts to resonate with Lambda calculus. It quickly becomes evident that when the graph is cyclic, it represents a recursion in terms of Lambda calculus and thus becomes Turing-complete. If computational graph is acyclic, Turing completeness is not achievable. If you are going to say that this is an obvious and well-known observation - I will be surprised, because it is not. And all that enlightenment was possible thanks to this article combined with a bit of knowledge about graphs, compilers, and lambda calculus. (It just so happened that I'm relatively well-versed in those topics due to a professional involvement.) --- If we continue to formalize this observation further, we may soon find a formal proof that a program P and a neural network NN are equivalent: P ~= NN Both have inputs (I) and outputs (O): O = P(I) O = NN(I) Both perform a computation by calculating output O for a given input I. The Turing-completeness observation and the direct mapping to and from the Lambda calculus will give us a way to translate an arbitrary program P to an equivalent neural network NN: P -> NN But the most intriguing part is that the inverse operation also becomes available: NN -> P Congratulations. We have just found a way to translate a working neural network NN to a formal program P written in a programming language L. What are consequences of that? The most obvious one is that we can now create a software that will be able to automatically translate a trained neural network NN to a working deterministic program P written in a programming language of choice and vice versa. We have just made a small step towards an imaginary AI system that can work as good as a professional software developer. Basically, one day we will be able to create a software-based software developer, a skilled one. And the level of skills will 100% surpass human abilities one day because Turing-completeness guarantees the unboundedness. The steps of NN -> P and P -> NN translation may be performed by the neural network itself. It seems that we already started to see that with ChatGPT 3. --- As you can see, one simple observation allowed us to precisely calculate the future for many years ahead. And this is why I love the math so much.

IIAOPSW 3y ago
I had an idea, perhaps crackpot, that will maybe catch fire for you too. What if the graph structure of the natural language parse tree was directly reflected in the literal biological neural network. What if the reason for natural language grammar is that it directly reflects the way it is processed under the hood.
bjourne 3y ago
You might be interested in the universal approximation theorem.