Truth! Without nonlinearity, each layer is essentially just a matrix multiplication. You could just as well use a single equivalent layer that represents the pr

Truth! Without nonlinearity, each layer is essentially just a matrix multiplication. You could just as well use a single equivalent layer that represents the product of the layers' matrices. Nonlinearity lets a single node partition the input space into two (fuzzy) equivalence classes. That's powerful stuff.