6 ms·

Algebra, Topology, Differential Calculus, and Optimization Theory [pdf]

98 points by azhenley 4y ago

quanto 4y ago
This is a 2000-page monograph on those topics in the title. I was hoping that the monograph goes deeper into optimization theory, but it seems that, while the monograph builds up to optimization, it only covers select topics in optimization. The title says it is "for computer science and machine learning." As such, deeper mathematical works in optimization theory understandably do not seem to be the intent behind the monograph.
- rhymer 4y ago
  Just curious what are the deeper optimization topics you were hoping to see in the monograph?
  - zekrioca 4y ago
    Not OP, but probably Convex Optimization [1]. [1] https://web.stanford.edu/~boyd/cvxbook/bv_cvxbook.pdf https://web.stanford.edu/~boyd/cvxbook/bv_cvxbook.pdf
    - Bimos 4y ago
      Or maybe branch-and-bound? Useful but not that mathematical.
    - chisquared 4y ago
      Judging from the TOC, everything in Part I of Boyd and Vandenberghe appears in some form in the monograph, so I don't think that's it.
  - jpfr 4y ago
    Shameless plug: I hold a lecture on "Optimization Methods for Machine Learning" that goes deep into convex optimization. Video recordings from the corona period (+ slides) are available here: https://www.youtube.com/playlist?list=PLdkTDauaUnQpzuOCZyUUZc0lxf4-PXNR5 https://www.youtube.com/playlist?list=PLdkTDauaUnQpzuOCZyUUZ...
    - cashsterling 4y ago
      Thank you... I'm going to check out your videos and slides.
noelwelsh 4y ago
I'm confused as to the intended audience for this. It's not a textbook. In the bits I looked at it doesn't attempt to teach the material, just banging out definitions and theorems. Also, the table of contents doesn't have hyperlinks to the actual content, which makes navigation a massive annoyance. Similarly, there aren't hyperlinks to the references. An impressive amount of work went into this, but why?
- deleted 4y ago
  [deleted]
- c-c-c-c-c 4y ago
  It's just a reference, I wouldn't view it as teaching material. Its quite useful to look things up in.
  - fleetwoodsnack 4y ago
    Yep, this is an extended survey of interrelated concepts organized according to increasing complexity. Similar to textbooks in the organizational principle, but differentiated in substance in that it assumes familiarity with the underlying topics and focuses on the relationships between the concepts rather than the concepts themselves. Good for practitioners I think.
- nicklaf 4y ago
  The book is still in progress [0]. For example, the introduction has yet to be written, which presumably would answer some of these questions about the intended audience. The author also has books [1] on tangentially related subjects, several with a good amount of overlap in material. [0] https://www.cis.upenn.edu/~jean/gbooks/geomath.html https://www.cis.upenn.edu/~jean/gbooks/geomath.html [1] https://www.cis.upenn.edu/~jean/gbooks/home.html https://www.cis.upenn.edu/~jean/gbooks/home.html
- whoevercares 4y ago
  I feel this is good foundation in CS and interested into more advanced ML and math. At least for me I feel excited when reading the toc since I have touched many of those concept over the years but never went deeper
ngcc_hk 4y ago
After some dive into quantum computing I found that unless you have Python/lisp simulation all these does not get into this old programmer mind.
- jshaqaw 4y ago
  Same here. I’ve been having fun learning a lot of math from first principles in recent years but it’s 1000 times easier when presented in notation/framework I already have internalized from code. I’ve found some of the work in type theory a solid bridge between these worlds as a sort of Rosetta Stone. Works like HoTT are challenging and require (me) to keep learning other areas along the way to fill in knowledge gaps but it provides me with a common framework to grok math concepts in line with computational intuitions without sacrificing rigor.
Rokid 4y ago
Remotely related - If you want to get a feel for topology optimization, the section of Topology Optimization at my university made a couple of cool little apps for exploring the concept in relation to structural mechanics: https://www.topopt.mek.dtu.dk/apps-and-software https://www.topopt.mek.dtu.dk/apps-and-software.
nextaccountic 4y ago
Chapter 6, at page 163, says that direct sums can't be built directly from the cartesian product because, instead of defining ordered pairs (a, b) != (b, a), they are unordered, so E ∐ F = F ∐ E However, direct sums in general are defined as being ordered, see https://math.stackexchange.com/questions/2694051/what-is-the-direct-sum https://math.stackexchange.com/questions/2694051/what-is-the... and https://en.wikipedia.org/wiki/Direct_sum https://en.wikipedia.org/wiki/Direct_sum How to make sense of that? Actually I'm confused because Wikipedia says "The direct sum of two abelian groups A A and B B is another abelian group A ⊕ B consisting of the ordered pairs (a, b) where a ∈ A and b ∈ B . To add ordered pairs, we define the sum (a, b) + (c, d) = (a + c, b + d)", and, while vector spaces have a little more structure than abelian groups, the definition should match up; and this definition seems just wrong, since it implies that direct sums are just cartesian products! PS: in my understanding, direct sums in math are analogous to sum types in programming (like in Haskell, data X = One Float | Another Float Float, X is a direct sum between ℝ and ℝ²), or better yet, anonymous sum types like the polymorphic variants of OCaml.
- bmacho 4y ago
  I believe that this upenn text doesn't use the 2 variable direct sum operation at all (which is not commutative), but abuses its notation to denote the direct sum of an indexed family. (But then he can't talk about E ∐ F which was his original goal :/)
  - nextaccountic 4y ago
    > denote the direct sum of an indexed family. Indeed he says so, but yes he keeps using E ∐ F Why is the direct sum of an indexed family different in terms of commutativity?
    - bmacho 4y ago
      The direct sum of an indexed family is not a function of two variables, so it is meaningless to talk about its commutativity.
- 4y ago

98 points by azhenley 4y ago

This is a 2000-page monograph on those topics in the title. I was hoping that the monograph goes deeper into optimization theory, but it seems that, while the monograph builds up to optimization, it only covers select topics in optimization. The title says it is "for computer science and machine learning." As such, deeper mathematical works in optimization theory understandably do not seem to be the intent behind the monograph.

rhymer 4y ago
Just curious what are the deeper optimization topics you were hoping to see in the monograph?
- zekrioca 4y ago
  Not OP, but probably Convex Optimization [1]. [1] https://web.stanford.edu/~boyd/cvxbook/bv_cvxbook.pdf https://web.stanford.edu/~boyd/cvxbook/bv_cvxbook.pdf
  - Bimos 4y ago
    Or maybe branch-and-bound? Useful but not that mathematical.
  - chisquared 4y ago
    Judging from the TOC, everything in Part I of Boyd and Vandenberghe appears in some form in the monograph, so I don't think that's it.
- jpfr 4y ago
  Shameless plug: I hold a lecture on "Optimization Methods for Machine Learning" that goes deep into convex optimization. Video recordings from the corona period (+ slides) are available here: https://www.youtube.com/playlist?list=PLdkTDauaUnQpzuOCZyUUZc0lxf4-PXNR5 https://www.youtube.com/playlist?list=PLdkTDauaUnQpzuOCZyUUZ...
  - cashsterling 4y ago
    Thank you... I'm going to check out your videos and slides.

noelwelsh 4y ago

I'm confused as to the intended audience for this. It's not a textbook. In the bits I looked at it doesn't attempt to teach the material, just banging out definitions and theorems. Also, the table of contents doesn't have hyperlinks to the actual content, which makes navigation a massive annoyance. Similarly, there aren't hyperlinks to the references. An impressive amount of work went into this, but why?

deleted 4y ago
[deleted]
c-c-c-c-c 4y ago
It's just a reference, I wouldn't view it as teaching material. Its quite useful to look things up in.
- fleetwoodsnack 4y ago
  Yep, this is an extended survey of interrelated concepts organized according to increasing complexity. Similar to textbooks in the organizational principle, but differentiated in substance in that it assumes familiarity with the underlying topics and focuses on the relationships between the concepts rather than the concepts themselves. Good for practitioners I think.
nicklaf 4y ago
The book is still in progress [0]. For example, the introduction has yet to be written, which presumably would answer some of these questions about the intended audience. The author also has books [1] on tangentially related subjects, several with a good amount of overlap in material. [0] https://www.cis.upenn.edu/~jean/gbooks/geomath.html https://www.cis.upenn.edu/~jean/gbooks/geomath.html [1] https://www.cis.upenn.edu/~jean/gbooks/home.html https://www.cis.upenn.edu/~jean/gbooks/home.html
whoevercares 4y ago
I feel this is good foundation in CS and interested into more advanced ML and math. At least for me I feel excited when reading the toc since I have touched many of those concept over the years but never went deeper

ngcc_hk 4y ago

After some dive into quantum computing I found that unless you have Python/lisp simulation all these does not get into this old programmer mind.

jshaqaw 4y ago
Same here. I’ve been having fun learning a lot of math from first principles in recent years but it’s 1000 times easier when presented in notation/framework I already have internalized from code. I’ve found some of the work in type theory a solid bridge between these worlds as a sort of Rosetta Stone. Works like HoTT are challenging and require (me) to keep learning other areas along the way to fill in knowledge gaps but it provides me with a common framework to grok math concepts in line with computational intuitions without sacrificing rigor.

Rokid 4y ago

Remotely related - If you want to get a feel for topology optimization, the section of Topology Optimization at my university made a couple of cool little apps for exploring the concept in relation to structural mechanics: https://www.topopt.mek.dtu.dk/apps-and-software https://www.topopt.mek.dtu.dk/apps-and-software.

nextaccountic 4y ago

Chapter 6, at page 163, says that direct sums can't be built directly from the cartesian product because, instead of defining ordered pairs (a, b) != (b, a), they are unordered, so E ∐ F = F ∐ E However, direct sums in general are defined as being ordered, see https://math.stackexchange.com/questions/2694051/what-is-the-direct-sum https://math.stackexchange.com/questions/2694051/what-is-the... and https://en.wikipedia.org/wiki/Direct_sum https://en.wikipedia.org/wiki/Direct_sum How to make sense of that? Actually I'm confused because Wikipedia says "The direct sum of two abelian groups A A and B B is another abelian group A ⊕ B consisting of the ordered pairs (a, b) where a ∈ A and b ∈ B . To add ordered pairs, we define the sum (a, b) + (c, d) = (a + c, b + d)", and, while vector spaces have a little more structure than abelian groups, the definition should match up; and this definition seems just wrong, since it implies that direct sums are just cartesian products! PS: in my understanding, direct sums in math are analogous to sum types in programming (like in Haskell, data X = One Float | Another Float Float, X is a direct sum between ℝ and ℝ²), or better yet, anonymous sum types like the polymorphic variants of OCaml.

bmacho 4y ago
I believe that this upenn text doesn't use the 2 variable direct sum operation at all (which is not commutative), but abuses its notation to denote the direct sum of an indexed family. (But then he can't talk about E ∐ F which was his original goal :/)
- nextaccountic 4y ago
  > denote the direct sum of an indexed family. Indeed he says so, but yes he keeps using E ∐ F Why is the direct sum of an indexed family different in terms of commutativity?
  - bmacho 4y ago
    The direct sum of an indexed family is not a function of two variables, so it is meaningless to talk about its commutativity.
4y ago