> It seems like quite a paradox to build something but to not know how it actually works and yet it works. That's because of the "magic" of gradient descent.

> It seems like quite a paradox to build something but to not know how it actually works and yet it works. That's because of the "magic" of gradient descent. You fill your neural network with completely random weights. But because of the way you've defined the math, you can tell how each individual weight will affect the value output at the other end; and specifically, you an take the derivative. So when the output is "wrong", you say, "would increasing this weight or decreasing have gotten me closer to the correct answer"? If increasing the node would have gotten you closer, you increase it a bit; if decreasing it would have gotten you closer you decrease it a bit. The result is that although we program the gradient descent algorithm, we don't directly program the actual circuits that the weights contain. Rather, the nodes "converge" into weights which end up implementing complex circuitry that was not explicitly programmed.