7 ms·

You don't need to be more good in math than in high school. AI is a chain of functions and you derive over those to get to the loss-function (gradient) to tell

You don't need to be more good in math than in high school. AI is a chain of functions and you derive over those to get to the loss-function (gradient) to tell you which parameters to change to get a better result (simplified!). Now this structure of functions is different in each implementations, but the type of function is quite similar - even though a large model will combine billions of those nodes and weights. Those visualizations tell you f.e. that some models connect neurons back to ones earlier in the chain to better remember a state. But the activation function is usually a weight and threshold. KAN changes the functions on the edges to more sophisticated ones than just "multiply by 0.x" and uses known physical formulas that you can actually explain to a human instead of the result coming from 100x different weights which tell you nothing. The language models we use currently may map how your brain works, but how strong the neurons are connected and to which others does not tell you anything. Instead a computer can chain different functions like you would chain a normal work task and explain each step to you / combine those learned routines on different tasks. I am by no means an expert in this field, but i do a lot of category theory, especially for the reason that i wanted a more explainable neuron network. So take my pov with a grain of salt, but please don't be discouraged to learn this. If you can program a little and remember some calculus you can definitely grasp these concepts after learning the vocabulary!

godelski 2y ago
> You don't need to be more good in math than in high school. I'm very tired of this... it needs to stop as it literally hinders ML progress 1) I know one (ONE) person who took multivariate calculus in high school. They did so by going to the local community college. I know zero people who took linear algebra. I just checked the listing of my old high school. Over a decade later neither multivariate calculus nor linear algebra is offered. 2) There's something I like to tell my students You don't need math to train a good model, but you do need to know math to know why your model is wrong. I'm sure many here recognize the reference[0], but being able to make a model that performs successfully on a test set[1] is not always meaningful. For example, about a year ago I was working a very big tech firm and increased their model's capacity on customer data by over 200% with a model that performed worse on their "test set". No additional data was used, nor did I make any changes to the architecture. Figure that out without math. (note, I was able to predict poor generalization performance PRIOR to my changes and accurately predict my model's significantly higher generalization performance) 3) Math isn't just writing calculations down. That's part of it -- a big part -- but the concepts are critical. And to truly understand those concepts, you at some point need to do these calculations. Because at the end of the day, math is a language[2]. 4) Just because the simplified view is not mathematically intensive does not mean math isn't important nor does it mean there isn't extremely complex mathematics under the hood. You're only explaining the mathematics in a simple way that is only about the updating process. There's a lot more to ML. And this should obviously be true since we consider them "black boxes"[3]. A lack of interpretability is not due to an immutable law, but due to our lack of understanding of a highly complex system. Yes, maybe each action in that system is simple, but if that meant the system as a whole was simple then I welcome you to develop a TOE for physics. Emergence is useful but also a pain in the ass[4]. [0] https://en.wikipedia.org/wiki/All_models_are_wrong https://en.wikipedia.org/wiki/All_models_are_wrong [1] For one, this is more accurately called a validation set. Test sets are held out. No more tuning. You're done. This is self-referential to my point. [2] If you want to fight me on this, at least demonstrate to me you have taken an abstract algebra course and understand ideals and rings. Even better if axioms and set theory. I accept other positions, but too many argue from the basis of physics without understanding the difference between a physics and physics. Just because math is the language of physics does not mean math (or even physics) is inherently an objective principle (physics is a model). [3] I hate this term. They are not black, but they are opaque. Which is to say that there is _some_ transparency. [4] I am using the term "emergence" in the way a physicist would, not what you've seen in an ML paper. Why? Well read point 4 again starting at footnote [3].
- Onavo 2y ago
  > 1) I know one (ONE) person who took multivariate calculus in high school. Unless you are specifically dealing with intractable Bayesian integral problems, the multivariate calculus involved in NNs are primarily differentiation, not integration. The fun problems like boundary conditions and Stokes/Green that makes up the meat of multivariable calculus don't truly apply when you are dealing with differentiation only. In other words you only need the parts of calc 2/3 that can be taught in an afternoon, not the truly difficult parts. > I'm sure many here recognize the reference[0], but being able to make a model that performs successfully on a test set[1] is not always meaningful. (sic) ...[2] If you want to fight me on this, at least demonstrate to me you have taken an abstract algebra course and understand ideals and rings. Even better if axioms and set theory. Doesn't matter, if it creates value, it is sufficiently correct for all intents and purposes. Pray tell me how discrete math and abstract algebra has anything to do with day to day ML research. If you want to appeal to physics sure, plenty of Ising models, energy functions, and belief propagation in ML but you have lost all credibility bringing up discrete math. Again those correlation tests you use to fact check your model are primarily linear frequentist models. Most statistics practitioners outside of graduate research will just be plugging formulas, not doing research level proofs. > Just because the simplified view is not mathematically intensive does not mean math isn't important nor does it mean there isn't extremely complex mathematics under the hood. You're only explaining the mathematics in a simple way that is only about the updating process. There's a lot more to ML. Are you sure? The traditional linear algebra (and similar) models never (or rarely) outperformed neural networks, except perhaps on efficiency, absent hardware acceleration and all other things being equal. A flapping bird wing is beautiful from a bioengineering point of view but the aerospace industry is powered by dumb (mostly) static airfoils. Just because something is elegant doesn't mean it solves problems. A scaled up CNN is about as boring a NN can get, yet it beats the pants off all those traditional computer vision algorithms that I am sure contain way more "discrete math and abstract algebra". That being said, more knowledge is always a good thing, but I am not naive enough to believe that ML research can only be advanced by people with "mathematical maturity". It's still in the highly empirical stage where we experimentation (regardless of whether it's guided by mathematical intuition) dominates. I have seen plenty of interesting ML results from folks who don't know what ELBOs and KL divergences are.
woolion 2y ago
If you do a lot of category theory, you most likely have a high "mathematical maturity" (Terry Tao spoke about this) Even if the math is fairly basic, you need to understand what is important where, which function could be replaced, etc. With mathematical maturity you realize how some details are not really significant while they take a lot mental space when you don't have it. It's part of the progression.
- Krei-se 2y ago
  That's true, but i mostly want to encourage coders to learn the vocabulary, because even at one or two years of coding experience you apply most of the rules you need for this (you explained this term way better than i could). In physics i had a lot of trouble in the beginning with the new symbols that where introduced, but they just shorten longer functions. Also chaining output of one function into another or combining them is something most coders do every day and it's written a bit different in math, but the same. What i was missing was someone telling me: "This xyz basically is what you do here with zyx every day and already on a complex level like spacetime. You can do this!". Certainly no teacher can know every specialty of their students, but usually people uncertain about their abilities are the ones that are so deep in their field that they mistake others not understanding them as being behind - when they just miss the vocabulary to exchange ideas. Last month i started with spinors and on one slide one term was named with a handful of different identifiers - same in category theory. The terms and names come from different fields and it can already be an eye-opener to some coder that a vector is just an array or that a scalar is a simple number, etc. The education system and environment may have given up on coders that are sitting in some company struggling with themselves even though they do a lot of good work and i'm always trying to reach those as i know how isolating it can be.
- Krei-se 2y ago
  Oh i forget: Many thanks for the Terry Tao mentioning, i will certainly have a look!
byteknight 2y ago
We went to different highschools unfortunately.
- Krei-se 2y ago
  Yeah, i figured that after the replies lol. I'm from Germany, it's mostly US here i guess. Back to topic: One thing that bugged me in school was learning only the derivative rules like n^x => x*n^(x-1) and mostly skipping on using limits to do it in general. https://d2l.ai/chapter_preliminaries/calculus.html https://d2l.ai/chapter_preliminaries/calculus.html <-- See Chapter 2.4.1 For AI you also need to know the power and chain rules mostly, but if you know that you are already way ahead of the bootcampers. At 16 i coded a 3D-engine on my graphical calculator and got stuck at arbitrary rotations around a point - but at mid 30s learned you need 4 dimensions and quaternions lol. So never be hard on yourself if you have to learn something that others know at any other point in their lives. Don't hesitate to drop me a mail if you want! I mostly hang around on #coreboot too!

godelski 2y ago

> You don't need to be more good in math than in high school. I'm very tired of this... it needs to stop as it literally hinders ML progress 1) I know one (ONE) person who took multivariate calculus in high school. They did so by going to the local community college. I know zero people who took linear algebra. I just checked the listing of my old high school. Over a decade later neither multivariate calculus nor linear algebra is offered. 2) There's something I like to tell my students You don't need math to train a good model, but you do need to know math to know why your model is wrong. I'm sure many here recognize the reference[0], but being able to make a model that performs successfully on a test set[1] is not always meaningful. For example, about a year ago I was working a very big tech firm and increased their model's capacity on customer data by over 200% with a model that performed worse on their "test set". No additional data was used, nor did I make any changes to the architecture. Figure that out without math. (note, I was able to predict poor generalization performance PRIOR to my changes and accurately predict my model's significantly higher generalization performance) 3) Math isn't just writing calculations down. That's part of it -- a big part -- but the concepts are critical. And to truly understand those concepts, you at some point need to do these calculations. Because at the end of the day, math is a language[2]. 4) Just because the simplified view is not mathematically intensive does not mean math isn't important nor does it mean there isn't extremely complex mathematics under the hood. You're only explaining the mathematics in a simple way that is only about the updating process. There's a lot more to ML. And this should obviously be true since we consider them "black boxes"[3]. A lack of interpretability is not due to an immutable law, but due to our lack of understanding of a highly complex system. Yes, maybe each action in that system is simple, but if that meant the system as a whole was simple then I welcome you to develop a TOE for physics. Emergence is useful but also a pain in the ass[4]. [0] https://en.wikipedia.org/wiki/All_models_are_wrong https://en.wikipedia.org/wiki/All_models_are_wrong [1] For one, this is more accurately called a validation set. Test sets are held out. No more tuning. You're done. This is self-referential to my point. [2] If you want to fight me on this, at least demonstrate to me you have taken an abstract algebra course and understand ideals and rings. Even better if axioms and set theory. I accept other positions, but too many argue from the basis of physics without understanding the difference between a physics and physics. Just because math is the language of physics does not mean math (or even physics) is inherently an objective principle (physics is a model). [3] I hate this term. They are not black, but they are opaque. Which is to say that there is _some_ transparency. [4] I am using the term "emergence" in the way a physicist would, not what you've seen in an ML paper. Why? Well read point 4 again starting at footnote [3].

Onavo 2y ago
> 1) I know one (ONE) person who took multivariate calculus in high school. Unless you are specifically dealing with intractable Bayesian integral problems, the multivariate calculus involved in NNs are primarily differentiation, not integration. The fun problems like boundary conditions and Stokes/Green that makes up the meat of multivariable calculus don't truly apply when you are dealing with differentiation only. In other words you only need the parts of calc 2/3 that can be taught in an afternoon, not the truly difficult parts. > I'm sure many here recognize the reference[0], but being able to make a model that performs successfully on a test set[1] is not always meaningful. (sic) ...[2] If you want to fight me on this, at least demonstrate to me you have taken an abstract algebra course and understand ideals and rings. Even better if axioms and set theory. Doesn't matter, if it creates value, it is sufficiently correct for all intents and purposes. Pray tell me how discrete math and abstract algebra has anything to do with day to day ML research. If you want to appeal to physics sure, plenty of Ising models, energy functions, and belief propagation in ML but you have lost all credibility bringing up discrete math. Again those correlation tests you use to fact check your model are primarily linear frequentist models. Most statistics practitioners outside of graduate research will just be plugging formulas, not doing research level proofs. > Just because the simplified view is not mathematically intensive does not mean math isn't important nor does it mean there isn't extremely complex mathematics under the hood. You're only explaining the mathematics in a simple way that is only about the updating process. There's a lot more to ML. Are you sure? The traditional linear algebra (and similar) models never (or rarely) outperformed neural networks, except perhaps on efficiency, absent hardware acceleration and all other things being equal. A flapping bird wing is beautiful from a bioengineering point of view but the aerospace industry is powered by dumb (mostly) static airfoils. Just because something is elegant doesn't mean it solves problems. A scaled up CNN is about as boring a NN can get, yet it beats the pants off all those traditional computer vision algorithms that I am sure contain way more "discrete math and abstract algebra". That being said, more knowledge is always a good thing, but I am not naive enough to believe that ML research can only be advanced by people with "mathematical maturity". It's still in the highly empirical stage where we experimentation (regardless of whether it's guided by mathematical intuition) dominates. I have seen plenty of interesting ML results from folks who don't know what ELBOs and KL divergences are.

woolion 2y ago

If you do a lot of category theory, you most likely have a high "mathematical maturity" (Terry Tao spoke about this) Even if the math is fairly basic, you need to understand what is important where, which function could be replaced, etc. With mathematical maturity you realize how some details are not really significant while they take a lot mental space when you don't have it. It's part of the progression.

Krei-se 2y ago
That's true, but i mostly want to encourage coders to learn the vocabulary, because even at one or two years of coding experience you apply most of the rules you need for this (you explained this term way better than i could). In physics i had a lot of trouble in the beginning with the new symbols that where introduced, but they just shorten longer functions. Also chaining output of one function into another or combining them is something most coders do every day and it's written a bit different in math, but the same. What i was missing was someone telling me: "This xyz basically is what you do here with zyx every day and already on a complex level like spacetime. You can do this!". Certainly no teacher can know every specialty of their students, but usually people uncertain about their abilities are the ones that are so deep in their field that they mistake others not understanding them as being behind - when they just miss the vocabulary to exchange ideas. Last month i started with spinors and on one slide one term was named with a handful of different identifiers - same in category theory. The terms and names come from different fields and it can already be an eye-opener to some coder that a vector is just an array or that a scalar is a simple number, etc. The education system and environment may have given up on coders that are sitting in some company struggling with themselves even though they do a lot of good work and i'm always trying to reach those as i know how isolating it can be.
Krei-se 2y ago
Oh i forget: Many thanks for the Terry Tao mentioning, i will certainly have a look!

byteknight 2y ago

We went to different highschools unfortunately.

Krei-se 2y ago
Yeah, i figured that after the replies lol. I'm from Germany, it's mostly US here i guess. Back to topic: One thing that bugged me in school was learning only the derivative rules like n^x => x*n^(x-1) and mostly skipping on using limits to do it in general. https://d2l.ai/chapter_preliminaries/calculus.html https://d2l.ai/chapter_preliminaries/calculus.html <-- See Chapter 2.4.1 For AI you also need to know the power and chain rules mostly, but if you know that you are already way ahead of the bootcampers. At 16 i coded a 3D-engine on my graphical calculator and got stuck at arbitrary rotations around a point - but at mid 30s learned you need 4 dimensions and quaternions lol. So never be hard on yourself if you have to learn something that others know at any other point in their lives. Don't hesitate to drop me a mail if you want! I mostly hang around on #coreboot too!