7 ms·

Pen and Paper Exercises in Machine Learning (2022)

413 points by ibobev 2y ago

antipaul 2y ago
So who among current ML practitioners building “useful” ML could solve some of these? _Should they_ be able to?
- jerrygenser 2y ago
  Nope, i don't think they should or need to be able to. These exercises are useful for mathematical maturity which results in intuition needed to develop novel algorithms or low level optimizations. Not needed to use existing train and deploy ML algorithms in general.
- danielmarkbruce 2y ago
  Depends on your definition of "ML practitioner", "building" and "ML". Look at the section on optimization - some people have an extremely good grasp of this and it helps them mentally iterate through possible loss functions and possible ways to update parameters and what can go wrong.
- psyklic 2y ago
  Good news -- if you're not interested in extending state-of-the-art and simply want to call APIs, you don't have to learn ML deeply.
- biotechbio 2y ago
  I am curious about the same thing. I worked as a ML engineer for several years and have a couple of degrees in the field. Skimming over the document, I recognized almost everything but I would not be able to recall many of these topics if asked without context, although at one time I might have been able to. What are others' general level of recall for this stuff? Am I a charlatan who never was very good at math or is it just expected that you will forget these things in time if you're not using them regularly?
- grandempire 2y ago
  Some people see studying as a chore and want to learn the minimum to get the job done. Others find it insightful and fun and enjoy doing problems and reading material. Both approaches make contributions and can lead to success, but in different ways.
kingkongjaffa 2y ago
complete with solutions, beautiful, thank you for sharing! I'd be interested in more of these pen and paper exercises, if there is such a term, for other topics.
- blackbear_ 2y ago
  Not sure which other topics you mean, but "1000 exercises in probability" should keep you busy for a while (one can find the PDF online). For other math oriented riddles, check out "The colossal book of short puzzles and problems" and "The art and craft of problem solving"
axpy906 2y ago
Love it.
simojo 2y ago
Very neat! Reminds me of Tom Yeh's "AI By Hand" exercises [0]. [0] https://www.byhand.ai/ https://www.byhand.ai/
- Sysreq2 2y ago
  This is what I was expecting. Very much appreciated. OP’s paper is good - but I sort of feel like it’s singing to the choir. It’s a great resource if you already know the material.
S4M 2y ago
Looks neat! My only criticism would be that the solutions are given right after the questions so I couldn't help to read the answer of one question before thinking it through by myself.
plants 2y ago
This is really neat! I work in machine learning but still feel imposter syndrome with my foundations with math (specifically linear algebra and matrix/tensor operations). Does anyone have any more good resources for problem sets with an emphasis on deep learning foundational skills? I find I learn best if I do a bit of hands-on work every day (and if I can learn things from multiple teachers’ perspectives)
lucasoshiro 2y ago
Seems to be cool, but, one of thing that most annoys me on studying machine learning is that I may dive as deep as it is possible in theory, but I can't see how it connects to the practice, i. e. how it makes me choose the correct number of neurons in a layer, how many layers, the activation functions, if I should use a neural network or other techniques, and so on... If someone have something explaining that I'll be grateful
- godelski 2y ago
  Just curious, how "deep" have you gone into the theory? What resources have you used? How strong is your math background? Unfortunately a lot of the theory does require some heavy mathematics, the type you won't see in a typical undergraduate degree even for more math heavy subjects like physics. Topics such as differential geometry, metric theory, set theory, abstract algebra, and high dimensional statistics. But I do promise that the theory helps and can build some very strong intuition. It is also extremely important that you have a deep understanding of what these mathematical operations are doing. It does look like this exercise book is trying to build that intuition, but I haven't read it in depth. I can say it is a good start, but only the very beginning of the theory journey. There is a long road ahead beyond this. > how it makes me choose the correct number of neurons in a layer, how many layers, Take a look at the Whitney embedding theorem. While this isn't a precise answer, it'll help you gain some intuition about the minimal number of parameters you need (and the VGG paper will help you understand width vs depth). In a transformer, the MLP layer post attention scales up 4x the dimensions before coming down, which allows for untangling any knots in the data. While 2x is the minimum, 4x creates a smoother landscape and so the problem can be solved more easily. Some of this is discussed in paper (Schaeffer, Miranda, and Koyejo) that counters the famous Emergent Abilities paper by Wei et al. This should be discussed early on in ML courses when discussing problems like XOR or the concentric circle. These problems are difficult because in their natural dimension you cannot draw a hyperplane discriminating them, but by increasing the dimensionality of the problem you can. This fact is usually mentioned in intro ML courses but I'm not aware of one that contains more details such as a discussion of the Whitney embedding theorem that allow you to better generalize the concepts here. > the activation functions There's a very short video I like that visualizes Gelu[0], even using the concentric circles! The channel has a lot of other visualizations that will really benefit your intuition. You may see where the differential geometry background can provide benefits. Understanding how to manipulate manifolds is critical to understanding what these networks are doing to the data. Unfortunately these visualizations will not benefit you once you scale beyond 3D as weird things happen in high dimensions, even as low as 10[1]. A lot of visual intuition goes out the window and this often leads people to either completely abandon it or make erroneous assumptions (no, your friend cannot visualize 4D objects[2,3] and that image you see of a tesseract is quite misleading). The activation functions provide non-linearity to the networks. A key ingredient missing from the preceptron model. Remember that with the universal approximation theorem you can approximate any smooth, Lipschitz-continuious function, over a closed boundary. You can, in simple cases, relate this to Riemann Summation, but you are using smooth "bump functions" instead of rectangles. I'm being fairly hand-wavy here on purpose because this is not precise but there are relationships to be found here. This is a HN comment, I have to overly simplify. Also remember that a linear layer without an activation can only perform Affine Transformations. That is, after all, what a matrix multiplication is capable of (another oversimplification). The learning curve is quite steep and there's a big jump from the common "it's just GMMs" or "it's just linear algebra" that is commonly claimed[4]. There is a lot of depth here, and unfortunately due to the hype there is a lot of stuff that says "deep" or "advanced mathematics" but it is important to remember that these terms are extremely relative. What is deep to one person is shallow to another. But if it isn't going beyond calculus, you are going to struggle, and I am extremely empathetic to that. But again, I do promise that there is a lot of insight to be gained by digging into the mathematics. There is benefit to doing things the hard way. I won't try to convince you that it is easy or that there isn't a lot of noise surrounding the topic, because that'd be a lie. If it were easy, ML systems wouldn't be "black boxes"![5] I would also encourage you to learn some meta physics. Something like Ian Hacking's representing and Intervening is a good start. There are limitations to what can be understand through experimentation alone, famously illustrated in Dyson's recounting of then Fermi rejected his paper[6]. There is a common misunderstanding of the saying "with 4 parameters I can fit an elephant and with 5 I can make it wiggle its trunk." [6] can help provide a better understanding to this, but we truly do need to understand the limitation of empirical studies. Science relies on the combination of empirical studies and theory. They are no good without the other. This is because science is about creating causal models, so one must be quite careful and be extremely nuanced when doing any form of evaluation. The subtle details can easily trick you. [0] https://www.youtube.com/watch?v=uiB97cPEVxM https://www.youtube.com/watch?v=uiB97cPEVxM [1] https://www.penzba.co.uk/cgi-bin/PvsNP.py?SpikeySpheres https://www.penzba.co.uk/cgi-bin/PvsNP.py?SpikeySpheres [2] https://www.youtube.com/shorts/_n7TMDnYdVY https://www.youtube.com/shorts/_n7TMDnYdVY [3] https://www.youtube.com/watch?v=FfiQBvcdFG0 https://www.youtube.com/watch?v=FfiQBvcdFG0 [4] https://news.ycombinator.com/item?id=43418334 https://news.ycombinator.com/item?id=43418334 [5] I actually dislike this term. It is better to say that they are opaque. A black box would imply that we have zero insights. But in reality we can see everything going on inside, it is just extremely difficult to interpret. We also do have some understanding, so the interpretation isn't impenetrable. [6] https://www.youtube.com/watch?v=hV41QEKiMlM https://www.youtube.com/watch?v=hV41QEKiMlM
dang 2y ago
Discussed at the time: Pen and paper exercises in machine learning (2021) - https://news.ycombinator.com/item?id=31913057 https://news.ycombinator.com/item?id=31913057 - June 2022 (55 comments)
imranq 2y ago
If someone could turn these into an adaptive Khan Academy style app, that would be incredible
- mathandsurf 2y ago
  Just curious for you or anyone else, what would make such an app compelling for you to use? And maybe not one that's just aimed at learning the content of this document, but if you'd like to think more broadly, an app aimed at helping you learn and retain things that you're currently interested in, studying, etc. For these machine learning problems specifically, feel like there are so many people that would greatly benefit from having some form of spaced repetitive practice (as you mention like the adaptive Khan Academy style app), or some other easy-to-use format. I just wonder what other features people would want that would make them want to use something like this over learning with other resources (e.g., YouTube videos, reading books, etc.)
  - imranq 2y ago
    There are already some resources like that like: leetgpu.com https://github.com/srush/GPU-Puzzles https://github.com/srush/GPU-Puzzles For me its about a sense of progress, like in chess you can have an ELO score. Or in Duolingo theres a roadmap. If there were levels to this you could get more confident in your abilities. Right now the levels are basically bachelors, masters, and PhD. Coarse and expensive
FilosofumRex 2y ago
Funny how mathematicians always try to sneak their linear algebra and matrix theory into ML. If you didn't know any better, you'd think academicians had invented LLMs and are the experts to be consulted with. If anything academicians and theoreticians held ML back and forced generations of grad students doing symbolic proofs, like in this example, just because computational techniques were too lowbrow for them.
- nophunphil 2y ago
  Interesting. Can you share an example of this?
- Abishek_Muthian 2y ago
  Interesting perspective, Would you have recommendations for resources which prioritizes "computational techniques" over "symbolic proofs"?
- grandempire 2y ago
  If you want to contribute to ML and not just use existing techniques, math skills are the most important limiter. Who do you know making contributions who isn’t fluent in linear algebra? Also why are you summarizing the entire field as “LLMs”?
  - Matthyze 2y ago
    Are math skills really? Most aspects of deep learning don't require a deep understanding of mathematics to understand. Backprop, convolution, attention, recurrent networks, skip connections, GANs, RL, GNNs, etc. can all be stood with only simple calculus and linear algebra. I understand that the theoretical motivation for models is often more math-heavy, but I'm skeptical that motivations need always be mathematical in nature.
    - thecleaner 2y ago
      Every MLE who didnt study Math really likes to downplay its importance. Yeah you dont need measure theoretic probability, but you need a grasp of Lin Alg to structure your computations better. Remember the normalization that we do in attention ? That has a math justification. So I guess yeah academics did have a role in building LLMs. I mean computer scientists really do like to pretend like they invented the whole field. Whereas in reality the average OS, compilers, networks class has nothing to do with core ML. But of course are also important and these barbs dont get us anywhere.
BeetleB 2y ago
Isn't arxiv meant for research level papers? Surprised to see this hosted there.

413 points by ibobev 2y ago

antipaul 2y ago

So who among current ML practitioners building “useful” ML could solve some of these? _Should they_ be able to?

jerrygenser 2y ago
Nope, i don't think they should or need to be able to. These exercises are useful for mathematical maturity which results in intuition needed to develop novel algorithms or low level optimizations. Not needed to use existing train and deploy ML algorithms in general.
danielmarkbruce 2y ago
Depends on your definition of "ML practitioner", "building" and "ML". Look at the section on optimization - some people have an extremely good grasp of this and it helps them mentally iterate through possible loss functions and possible ways to update parameters and what can go wrong.
psyklic 2y ago
Good news -- if you're not interested in extending state-of-the-art and simply want to call APIs, you don't have to learn ML deeply.
biotechbio 2y ago
I am curious about the same thing. I worked as a ML engineer for several years and have a couple of degrees in the field. Skimming over the document, I recognized almost everything but I would not be able to recall many of these topics if asked without context, although at one time I might have been able to. What are others' general level of recall for this stuff? Am I a charlatan who never was very good at math or is it just expected that you will forget these things in time if you're not using them regularly?
grandempire 2y ago
Some people see studying as a chore and want to learn the minimum to get the job done. Others find it insightful and fun and enjoy doing problems and reading material. Both approaches make contributions and can lead to success, but in different ways.

kingkongjaffa 2y ago

complete with solutions, beautiful, thank you for sharing! I'd be interested in more of these pen and paper exercises, if there is such a term, for other topics.

blackbear_ 2y ago
Not sure which other topics you mean, but "1000 exercises in probability" should keep you busy for a while (one can find the PDF online). For other math oriented riddles, check out "The colossal book of short puzzles and problems" and "The art and craft of problem solving"

axpy906 2y ago

Love it.

simojo 2y ago

Very neat! Reminds me of Tom Yeh's "AI By Hand" exercises [0]. [0] https://www.byhand.ai/ https://www.byhand.ai/

Sysreq2 2y ago
This is what I was expecting. Very much appreciated. OP’s paper is good - but I sort of feel like it’s singing to the choir. It’s a great resource if you already know the material.

S4M 2y ago

Looks neat! My only criticism would be that the solutions are given right after the questions so I couldn't help to read the answer of one question before thinking it through by myself.

plants 2y ago

This is really neat! I work in machine learning but still feel imposter syndrome with my foundations with math (specifically linear algebra and matrix/tensor operations). Does anyone have any more good resources for problem sets with an emphasis on deep learning foundational skills? I find I learn best if I do a bit of hands-on work every day (and if I can learn things from multiple teachers’ perspectives)

lucasoshiro 2y ago

Seems to be cool, but, one of thing that most annoys me on studying machine learning is that I may dive as deep as it is possible in theory, but I can't see how it connects to the practice, i. e. how it makes me choose the correct number of neurons in a layer, how many layers, the activation functions, if I should use a neural network or other techniques, and so on... If someone have something explaining that I'll be grateful

godelski 2y ago
Just curious, how "deep" have you gone into the theory? What resources have you used? How strong is your math background? Unfortunately a lot of the theory does require some heavy mathematics, the type you won't see in a typical undergraduate degree even for more math heavy subjects like physics. Topics such as differential geometry, metric theory, set theory, abstract algebra, and high dimensional statistics. But I do promise that the theory helps and can build some very strong intuition. It is also extremely important that you have a deep understanding of what these mathematical operations are doing. It does look like this exercise book is trying to build that intuition, but I haven't read it in depth. I can say it is a good start, but only the very beginning of the theory journey. There is a long road ahead beyond this. > how it makes me choose the correct number of neurons in a layer, how many layers, Take a look at the Whitney embedding theorem. While this isn't a precise answer, it'll help you gain some intuition about the minimal number of parameters you need (and the VGG paper will help you understand width vs depth). In a transformer, the MLP layer post attention scales up 4x the dimensions before coming down, which allows for untangling any knots in the data. While 2x is the minimum, 4x creates a smoother landscape and so the problem can be solved more easily. Some of this is discussed in paper (Schaeffer, Miranda, and Koyejo) that counters the famous Emergent Abilities paper by Wei et al. This should be discussed early on in ML courses when discussing problems like XOR or the concentric circle. These problems are difficult because in their natural dimension you cannot draw a hyperplane discriminating them, but by increasing the dimensionality of the problem you can. This fact is usually mentioned in intro ML courses but I'm not aware of one that contains more details such as a discussion of the Whitney embedding theorem that allow you to better generalize the concepts here. > the activation functions There's a very short video I like that visualizes Gelu[0], even using the concentric circles! The channel has a lot of other visualizations that will really benefit your intuition. You may see where the differential geometry background can provide benefits. Understanding how to manipulate manifolds is critical to understanding what these networks are doing to the data. Unfortunately these visualizations will not benefit you once you scale beyond 3D as weird things happen in high dimensions, even as low as 10[1]. A lot of visual intuition goes out the window and this often leads people to either completely abandon it or make erroneous assumptions (no, your friend cannot visualize 4D objects[2,3] and that image you see of a tesseract is quite misleading). The activation functions provide non-linearity to the networks. A key ingredient missing from the preceptron model. Remember that with the universal approximation theorem you can approximate any smooth, Lipschitz-continuious function, over a closed boundary. You can, in simple cases, relate this to Riemann Summation, but you are using smooth "bump functions" instead of rectangles. I'm being fairly hand-wavy here on purpose because this is not precise but there are relationships to be found here. This is a HN comment, I have to overly simplify. Also remember that a linear layer without an activation can only perform Affine Transformations. That is, after all, what a matrix multiplication is capable of (another oversimplification). The learning curve is quite steep and there's a big jump from the common "it's just GMMs" or "it's just linear algebra" that is commonly claimed[4]. There is a lot of depth here, and unfortunately due to the hype there is a lot of stuff that says "deep" or "advanced mathematics" but it is important to remember that these terms are extremely relative. What is deep to one person is shallow to another. But if it isn't going beyond calculus, you are going to struggle, and I am extremely empathetic to that. But again, I do promise that there is a lot of insight to be gained by digging into the mathematics. There is benefit to doing things the hard way. I won't try to convince you that it is easy or that there isn't a lot of noise surrounding the topic, because that'd be a lie. If it were easy, ML systems wouldn't be "black boxes"![5] I would also encourage you to learn some meta physics. Something like Ian Hacking's representing and Intervening is a good start. There are limitations to what can be understand through experimentation alone, famously illustrated in Dyson's recounting of then Fermi rejected his paper[6]. There is a common misunderstanding of the saying "with 4 parameters I can fit an elephant and with 5 I can make it wiggle its trunk." [6] can help provide a better understanding to this, but we truly do need to understand the limitation of empirical studies. Science relies on the combination of empirical studies and theory. They are no good without the other. This is because science is about creating causal models, so one must be quite careful and be extremely nuanced when doing any form of evaluation. The subtle details can easily trick you. [0] https://www.youtube.com/watch?v=uiB97cPEVxM https://www.youtube.com/watch?v=uiB97cPEVxM [1] https://www.penzba.co.uk/cgi-bin/PvsNP.py?SpikeySpheres https://www.penzba.co.uk/cgi-bin/PvsNP.py?SpikeySpheres [2] https://www.youtube.com/shorts/_n7TMDnYdVY https://www.youtube.com/shorts/_n7TMDnYdVY [3] https://www.youtube.com/watch?v=FfiQBvcdFG0 https://www.youtube.com/watch?v=FfiQBvcdFG0 [4] https://news.ycombinator.com/item?id=43418334 https://news.ycombinator.com/item?id=43418334 [5] I actually dislike this term. It is better to say that they are opaque. A black box would imply that we have zero insights. But in reality we can see everything going on inside, it is just extremely difficult to interpret. We also do have some understanding, so the interpretation isn't impenetrable. [6] https://www.youtube.com/watch?v=hV41QEKiMlM https://www.youtube.com/watch?v=hV41QEKiMlM

dang 2y ago

Discussed at the time: Pen and paper exercises in machine learning (2021) - https://news.ycombinator.com/item?id=31913057 https://news.ycombinator.com/item?id=31913057 - June 2022 (55 comments)

imranq 2y ago

If someone could turn these into an adaptive Khan Academy style app, that would be incredible

mathandsurf 2y ago
Just curious for you or anyone else, what would make such an app compelling for you to use? And maybe not one that's just aimed at learning the content of this document, but if you'd like to think more broadly, an app aimed at helping you learn and retain things that you're currently interested in, studying, etc. For these machine learning problems specifically, feel like there are so many people that would greatly benefit from having some form of spaced repetitive practice (as you mention like the adaptive Khan Academy style app), or some other easy-to-use format. I just wonder what other features people would want that would make them want to use something like this over learning with other resources (e.g., YouTube videos, reading books, etc.)
- imranq 2y ago
  There are already some resources like that like: leetgpu.com https://github.com/srush/GPU-Puzzles https://github.com/srush/GPU-Puzzles For me its about a sense of progress, like in chess you can have an ELO score. Or in Duolingo theres a roadmap. If there were levels to this you could get more confident in your abilities. Right now the levels are basically bachelors, masters, and PhD. Coarse and expensive

FilosofumRex 2y ago

Funny how mathematicians always try to sneak their linear algebra and matrix theory into ML. If you didn't know any better, you'd think academicians had invented LLMs and are the experts to be consulted with. If anything academicians and theoreticians held ML back and forced generations of grad students doing symbolic proofs, like in this example, just because computational techniques were too lowbrow for them.

nophunphil 2y ago
Interesting. Can you share an example of this?
Abishek_Muthian 2y ago
Interesting perspective, Would you have recommendations for resources which prioritizes "computational techniques" over "symbolic proofs"?
grandempire 2y ago
If you want to contribute to ML and not just use existing techniques, math skills are the most important limiter. Who do you know making contributions who isn’t fluent in linear algebra? Also why are you summarizing the entire field as “LLMs”?
- Matthyze 2y ago
  Are math skills really? Most aspects of deep learning don't require a deep understanding of mathematics to understand. Backprop, convolution, attention, recurrent networks, skip connections, GANs, RL, GNNs, etc. can all be stood with only simple calculus and linear algebra. I understand that the theoretical motivation for models is often more math-heavy, but I'm skeptical that motivations need always be mathematical in nature.
  - thecleaner 2y ago
    Every MLE who didnt study Math really likes to downplay its importance. Yeah you dont need measure theoretic probability, but you need a grasp of Lin Alg to structure your computations better. Remember the normalization that we do in attention ? That has a math justification. So I guess yeah academics did have a role in building LLMs. I mean computer scientists really do like to pretend like they invented the whole field. Whereas in reality the average OS, compilers, networks class has nothing to do with core ML. But of course are also important and these barbs dont get us anywhere.

BeetleB 2y ago

Isn't arxiv meant for research level papers? Surprised to see this hosted there.