5 ms·
An algebraic generalization for graph and tensor-based neural networks [pdf]
- eddieone 9y agoSearch the CGP section, that blew my mind and it has become more rare these days. Reminds me of Nexus, a science fiction book.
- cs702 9y agoTL;DR: an early effort to establish a 'standard language' for representing and symbolically manipulating (e.g., transforming) every possible neural net that could ever be constructed with graphs and tensors. Very cool to see this sort of thing.
- amelius 9y agoTwo questions: Does this work also allow cyclic structures such as recurrent neural networks (RNNs)? Does this work generalize to graphs in general?
- cs702 9y agoIn principle it should, because these guys are using matrices to represent network structures, and every finite graph (cyclic or acyclic, directed or undirected) can be represented by a matrix. Examples: https://en.wikipedia.org/wiki/Adjacency_matrix#Examples https://en.wikipedia.org/wiki/Adjacency_matrix#Examples
- deepnotderp 9y agoCool work, now hopefully we can stop using increasingly arbitrary notation in papers... ;)
- macawfish 9y agoOut of curiosity, which notation are you referring to?
- v-yadli 9y agoHe's probably referring to the problem that is presented in the paper: nobody uses standard notations because there's no such thing! :) Edit: For example, how do you say 'one layer'? W_i? L_i?
- deepnotderp 9y agoYup! And half the papers mean W_i to mean the weights for the layer. And then there are those who say screw mathematics completely and use the Hadamard or Kronecker product symbol to denote convolution, which has caused me no end of trouble in the past.
- macawfish 9y agoAhh, I get it now!
- flor1s 9y agoIt seems like "arbitrary" notation is also a problem in other fields of mathematics such as Linear Algebra - perhaps depending on the background of the author in math or CS. In Linear Algebra every text book has their own favorite notation for matrices, row/column vectors (special letter such as c for column vectors, roman letters, roman letters with an arrow on top) and scalars (greek letters or the letter c).
- gbrown 9y agoNow add the distinction between random and nonrandom.
- jokoon 9y agoI stopped trying to learn about machine learning in general because mathematical notations are always used everywhere in ML. I think it's odd that programming languages and code have not replaced math. I mean I agree that you need math for ML, but that doesn't mean you can't show what you do with pseudo code. Math is fine to prove things or simplify a formula, but when it involves computing, why not use code instead? It's okay to use math in physics, but when you deal with computing and data, I think it's a little misplaced. Not to mention that the language of mathematics is poorly defined. I mean I would love to learn math by learning its syntax.
- minflynn 9y agoI used a very similar idea to create an indirect coding for neuroevolution that was successful in solving some tasks. It's based on a generalization of K^2 Trees and shares the same ideas of modularity and hierarchy.
- cbennett 9y agoThat sounds really interesting. What tasks did you attempt? If you have open-sourced this in any sense, would enjoy to check it out and/or contribute.
- zitterbewegung 9y agoSo what does this let you do other than having a standard notation for both graph and tensor NN?
- gleenn 9y agoI've heard training using some tools is easier than others. This might allow you to experiment on one tool and then easily port your commonly formatted model to a more production like system, for performance or whatever other reasons. Seems like compatibility between these things would be pretty useful
- xaedes 9y agoSee page 22: " Modular Neuroevolution • We are currently working on a neuroevolution framework based on Cartesian genetic programming (CGP) and existing work on CGP-based ANNs. • The algebraic framework introduced in this work will be the basis for the genetic representation and operators. • Evolved networks will be a mix of de novo evolved modules and existing modules in the form of ANN layers, relational, and functional programs. • The representation will be based on a mapping between algebraic expressions and a recursive, modular adjacency structure. "