3 ms·
For the first section, where he uses a Hessian and finds he really only needs one step for linear regression -- isn't he basically doing a newton method, and it
by abeppu 7y ago
For the first section, where he uses a Hessian and finds he really only needs one step for linear regression -- isn't he basically doing a newton method, and it works more or less perfectly b/c with the normal squared error term (L2 norm) his problem is exactly quadratic? I haven't unpacked his math carefully but I think (quasi)newton methods are often worth a look if (a) your dimension is small enough that the the D^2 hessian is ok, and (b) your data is small enough that you can afford to us exact derivatives rather than mini-batch estimates.
- nil-sec 7y agoActually there is no need for gradient decent at all if you have a linear regression problem, because you can find the min/max exactly by inverting your weight-matrix (pseudo-inverse if degenerate). Additionally, second order optimisation, i.e. using the Hessian in addition to the gradient is a well studied problem in the literature. My understanding is that it is in general not worth it because calculating the Hessian is very expensive (see e.g. https://arxiv.org/abs/2002.09018 https://arxiv.org/abs/2002.09018).
- James_Henry 7y agoIt seems to me that gradient descent would be, often, a better solution than the moore-penrose inverse. Is it not more efficient in most cases?
- oivey 7y agoOne important example is large, sparse systems/matrices. The pseudo-inverse requires computing an inversion that can be very expensive due to the size of the matrix. That matrix generally will also no longer be sparse, so it will be expensive to store and work with.
- zwaps 7y agoI always thought you‘d use Gram Schmitz for linear regression https://en.m.wikipedia.org/wiki/Gram–Schmidt_process https://en.m.wikipedia.org/wiki/Gram–Schmidt_process
- deleted 7y ago[deleted]