3 ms·
Slightly unrelated, but does anyone think we'll ever discover a closed-form solution to neural network optimization? If you think about it, the network is known
by rp1 5y ago
Slightly unrelated, but does anyone think we'll ever discover a closed-form solution to neural network optimization? If you think about it, the network is known, the initial weights are known, the training data is known, and the objective function is known. All of the work spent "training" is spent computing the solution to an optimization problem that could (theoretically) have a closed-form solution.
- Nevermark 5y agoSince we can't find general closed form solutions for simpler equations such as 5th order polynomials, general closed form NN solvers (vs optimizers) do not look likely. However, constraints on the general class of polynomials of 5th order and higher can result in subclasses that are solvable in closed form. So there may be a long tail of closed form solvable classes of NN architectures waiting to be discovered. Whether non-trivial closed-form subclasses of neural networks exist, and whether any of those classes is actually useful ... seems an area worth exploring.
- archibaldJ 5y agoDigressing back to qc.. If you look at how Shor's algorithm works, it's more like manipulating some kind of embeded values (that we have no direct access to) eg via Hadamard gates and eventually collapsing the entanglement into the solution. So maybe we can have something closed-form-ish that solves NN by arranging some q gates some way