4 ms·

I’m not saying gradient descent was empirically discovered, I’m saying that its use in machine learning (or elsewhere, I suppose) is itself a form of empiricism

I’m not saying gradient descent was empirically discovered, I’m saying that its use in machine learning (or elsewhere, I suppose) is itself a form of empiricism in that what it is is essentially a repeated observe/measure error/adjust cycle.

skydhash 7d ago
> I’m not saying gradient descent was empirically discovered, I’m saying that its use in machine learning is itself a form of empiricism. A repeated observe/adjust-based-on-data cycle The data is the input, the output is to generally find the lowest amount of a loss function. It’s a greedy approach because brute forcing is inefficient. It’s no more empirical than a greedy algorithm for scheduling.
- theptip 7d ago
  > It’s no more empirical than a greedy algorithm for scheduling. Right, GP is drawing a distinction between search, ie mechanical exploration of a space, with understanding, ie having a map of the territory such that you don’t need trial and error.
dwattttt 7d ago
> its use in machine learning (or elsewhere, I suppose) is itself a form of empiricism in that what it is is essentially a repeated observe/measure error/adjust cycle. "Empiricism" implies that the technique is based on observable, but not mathematically proven foundations. If a problem space is convex, gradient descent is guaranteed to converge to a global optimal solution, regardless of whether you know the exact formulation of the space. Applying it when you don't understand if a space is convex is another question, but that's not a fault of gradient descent.