Really agree with the extra simplicity in the ReLU case. Liwen Zhang, Gregory Naitzat and Lek-Heng Lim showed last year that "the family of such neural networks

Really agree with the extra simplicity in the ReLU case. Liwen Zhang, Gregory Naitzat and Lek-Heng Lim showed last year that "the family of such neural networks is equivalent to the family of tropical rational maps", where rational functions are quotient of polynomials and "tropical" is in the context of Tropical Geometry, where instead of the typical "plus, times" ring one uses a "min, plus" ring that has somewhat unexpected applications. For instance, with its module theory one can calculate minimum costs paths in graphs just as one does for reachability starting with the adjacency matrix of a graph over a boolean semiring. arXiv:1805.07091v1