5 ms·

The "math" used in ML/AI papers is usually just a sort of 'whiteboard math' which is a domain-specific mishmash of linear algebra, calculus, set theory and stat

The "math" used in ML/AI papers is usually just a sort of 'whiteboard math' which is a domain-specific mishmash of linear algebra, calculus, set theory and statistics. If you could find a book just going through the relevant bits you wouldnt really have to "learn math again", it can be translated into english straightforwardly -- very very few ML papers relevant to industry have extended proofs, etc. that require eg., even being able to differentiate anything yourself. 90% of it is: here's the domain (ie., type) of our variables, here's the formula of our functions, we're taking a weighted average with some inner products involved. It might sound like a lot of math, but it's really all doable in semester-1 of an undergrad course, were it focused enough.

programjames 3y ago
Normalizing flows and neural ODEs are becoming a bigger thing, and they do involve some more heavy maths.
- jurgenaut23 3y ago
  Yes, and if you look at neural operators, then you're in for something that even math majors need to put a bit of effort into.
  - programjames 3y ago
    Oh, I read that paper about a year ago. I feel like it could have been explained better, but the concepts aren't that hard if you've taken numerical methods.
sjducb 3y ago
I think you don’t realise how much math you know. I’ve heard of set theory, but never studied it.
- mjburgess 3y ago
  sure, but you dont need to understand actual set theory -- it's just notation for mostly obvious programming stuff, R means float, Z means int R^2 is actually notation from linear algebra, but here it means a point is 2 floats with a measure of distance between points etc. A lot of this could just be given in a "crib sheet" for tech people, and you'd get 80% of it straightaway. It's years of work to understand this notation as used by the professional domains it was invented by, but it's a couple weeks for most "good, technical, software engineers" -- since they arent really using notation in much more than superficial ways. Of course there are hard mathematical papers, etc. but they're rare in ML/AI mainstream papers; and not something most would read. If you need to be able to read, eg., some sort of adv. statistical time series research in economic modelling, you'd already have the background to do that. If all you want is to be able to read 90% of the popular papers, the notation in them is just syntax sugar for things you could state easily in english or python
  - roenxi 3y ago
    > sure, but you dont need to understand actual set theory How would you know? You sound like you understand set theory. Ironically, a body needs a rather extensive grounding in the theory and practice of set theory to look at something and think "I don't need to use the tools of set theory here" with deserved confidence. A lot of maths is about having a huge repertoire of tricks that you know won't help solve the problem at hand. Saves weeks of fruitless attempts. But I'll bet you didn't notice the things you know not to do. That does matter for interpreting these papers.
    - aleph_minus_one 3y ago
      > > sure, but you dont need to understand actual set theory > How would you know? You sound like you understand set theory. Easy: most mathematicians don't really care about set theory (beyond the fact that it exists and is the standard foundation). Understanding set theory goes deep. For example: why does the Banach-Tarski paradox exist in ZFC (C: Axiom of Choice (https://en.wikipedia.org/wiki/Axiom_of_choice https://en.wikipedia.org/wiki/Axiom_of_choice)), but not in ZFD (D: Axiom of Determinacy (https://en.wikipedia.org/wiki/Axiom_of_determinacy https://en.wikipedia.org/wiki/Axiom_of_determinacy))? Or another one: why does the statement "every field has an algebraic closure" not hold in ZF (but in ZFC)? Never heard of this stuff despite having studied mathematics? Just like I said: most mathematicians don't really care about set theory.
zelphirkalt 3y ago
While I think, that you may be right, that most papers are not about proofs, and that higher math is mostly about proofs, I still think it is math and important math at that. Perhaps it is a link in the chain between mathematics and engineering skills. The proof centered math is indeed the difficult one. I struggled with a lot at university and was completely unprepared for that finishing school. So I think I have a good idea about the difference in difficulty.
DeathArrow 3y ago
>If you could find a book just going through the relevant bits you wouldnt really have to "learn math again" Hard to find a book or a course doing applied math just for AI and ML. I was interested in learning the math needed to understand quantitative analysis and I was lucky to find some resources doing just that.
vehicles2b 3y ago
We probably think fluently of these mathematical structures in tangible, real world terms. Still it requires some practice translating to the mathematical abstractions, and still further practice thinking in terms of the abstractions altogether. I don’t know how relevant it is that the textbook content can be covered in a single semester.
danielmarkbruce 3y ago
This is definitely true. It's largely just notation for communication. I just paste sections of dense math from AI/ML papers into chatgpt and it explains it. Almost none of it is complicated. It's just really awful notation.

programjames 3y ago

Normalizing flows and neural ODEs are becoming a bigger thing, and they do involve some more heavy maths.

jurgenaut23 3y ago
Yes, and if you look at neural operators, then you're in for something that even math majors need to put a bit of effort into.
- programjames 3y ago
  Oh, I read that paper about a year ago. I feel like it could have been explained better, but the concepts aren't that hard if you've taken numerical methods.

sjducb 3y ago

I think you don’t realise how much math you know. I’ve heard of set theory, but never studied it.

mjburgess 3y ago
sure, but you dont need to understand actual set theory -- it's just notation for mostly obvious programming stuff, R means float, Z means int R^2 is actually notation from linear algebra, but here it means a point is 2 floats with a measure of distance between points etc. A lot of this could just be given in a "crib sheet" for tech people, and you'd get 80% of it straightaway. It's years of work to understand this notation as used by the professional domains it was invented by, but it's a couple weeks for most "good, technical, software engineers" -- since they arent really using notation in much more than superficial ways. Of course there are hard mathematical papers, etc. but they're rare in ML/AI mainstream papers; and not something most would read. If you need to be able to read, eg., some sort of adv. statistical time series research in economic modelling, you'd already have the background to do that. If all you want is to be able to read 90% of the popular papers, the notation in them is just syntax sugar for things you could state easily in english or python
- roenxi 3y ago
  > sure, but you dont need to understand actual set theory How would you know? You sound like you understand set theory. Ironically, a body needs a rather extensive grounding in the theory and practice of set theory to look at something and think "I don't need to use the tools of set theory here" with deserved confidence. A lot of maths is about having a huge repertoire of tricks that you know won't help solve the problem at hand. Saves weeks of fruitless attempts. But I'll bet you didn't notice the things you know not to do. That does matter for interpreting these papers.
  - aleph_minus_one 3y ago
    > > sure, but you dont need to understand actual set theory > How would you know? You sound like you understand set theory. Easy: most mathematicians don't really care about set theory (beyond the fact that it exists and is the standard foundation). Understanding set theory goes deep. For example: why does the Banach-Tarski paradox exist in ZFC (C: Axiom of Choice (https://en.wikipedia.org/wiki/Axiom_of_choice https://en.wikipedia.org/wiki/Axiom_of_choice)), but not in ZFD (D: Axiom of Determinacy (https://en.wikipedia.org/wiki/Axiom_of_determinacy https://en.wikipedia.org/wiki/Axiom_of_determinacy))? Or another one: why does the statement "every field has an algebraic closure" not hold in ZF (but in ZFC)? Never heard of this stuff despite having studied mathematics? Just like I said: most mathematicians don't really care about set theory.

zelphirkalt 3y ago

While I think, that you may be right, that most papers are not about proofs, and that higher math is mostly about proofs, I still think it is math and important math at that. Perhaps it is a link in the chain between mathematics and engineering skills. The proof centered math is indeed the difficult one. I struggled with a lot at university and was completely unprepared for that finishing school. So I think I have a good idea about the difference in difficulty.

DeathArrow 3y ago

>If you could find a book just going through the relevant bits you wouldnt really have to "learn math again" Hard to find a book or a course doing applied math just for AI and ML. I was interested in learning the math needed to understand quantitative analysis and I was lucky to find some resources doing just that.

vehicles2b 3y ago

We probably think fluently of these mathematical structures in tangible, real world terms. Still it requires some practice translating to the mathematical abstractions, and still further practice thinking in terms of the abstractions altogether. I don’t know how relevant it is that the textbook content can be covered in a single semester.

danielmarkbruce 3y ago

This is definitely true. It's largely just notation for communication. I just paste sections of dense math from AI/ML papers into chatgpt and it explains it. Almost none of it is complicated. It's just really awful notation.