3 ms·
How important is optimization? Like a lot of engineers I know, I took a few required courses in undergrad on linear algebra and stats. But I've never studied op
by mr_overalls 8y ago
How important is optimization? Like a lot of engineers I know, I took a few required courses in undergrad on linear algebra and stats. But I've never studied optimization. And I see it come up all the time on lists of central theory for ML & AI. . .
Is there a classic textbook on the subject? Are there any free online courses that are considered good?
https://www.coursera.org/courses?query=optimization https://www.coursera.org/courses?query=optimization
- snakeboy 8y agoNocedal & Wright [1] is generally considered the definitive text for reference, at least. Very clear and complete from what I recall. [1] https://www.springer.com/us/book/9780387303031 https://www.springer.com/us/book/9780387303031
- Puer 8y agoConvex Optimization by Lieven Vandenberghe and Stephen P. Boyd was the one my professors always recommended and I think it's excellent.
- deleted 8y ago[deleted]
- Eridrus 8y agoOptimization as a tool is important and widely used, but... almost everything grabbing headlines uses some form of SGD + Momentum. Very little of the actual progress comes from better optimization.
- tnecniv 8y agoOptimization can be pretty useful. Most stats / ML problems are posed as minimizing a function subject to some constraints. Depending on your problem, you might be able to exploit special structures to solve problems faster than just doing gradient descent. If you know linear algebra and stats, you'll be fine getting through an optimization book. Boyd's book is canonical at this point, but might be hard to get through. Before you get to actually optimizing anything, you need to make your way through some chapters on convex analysis with little application.