42 ms·

I interview a lot of developers for ML positions at our company. The first red flag is always a lack of math. Candidates who come in with API-level competence i

by dxbydt 11y ago

I interview a lot of developers for ML positions at our company. The first red flag is always a lack of math. Candidates who come in with API-level competence ie. can implement an ML algo using this,that or the other API, without any understanding of some basic math behind it, always fare poorly. Atleast in ML, not having an understanding of math is pretty much like claiming expertise in riding a bicycle by watching 100s of bicycle videos on youtube without ever riding a bicycle yourself. This isn't the case with say, front-end web dev, where nobody really cares whether what's under the hood is php or jquery or elegantly handcrafted elm as long as the webpage looks good and functions as advertised. You can get a lot of mileage by hiring someone who can "make webpages" without knowing what tools they use to make that webpage. Once you get traction etc. you'll probably rewrite all of the cruft :) With ML, if you hire somebody who can "create decision tree in python mllib" without knowing the first thing about what entropy is & how to compute entropy ( a real candidate, unfortunately ), you are simply inflicting a lot of pain on yourself and your customers. Suppose such a person is deciding whether or not to give you a loan, and he decides to construct a decision tree. He'll happily throw in your zip code and credit card number into the mix, not realizing that those two features have super high entropy but the tree will have serious overfitting issues ie. the tree will simply not generalize to unseen data. He won't realize these things because he won't know what entropy is in the first place, since he only thinks of decision trees in terms of some black box that comes out of some ML api.

NegatioN 11y ago
I get that math is an important aspect, but being able to realise that throwing in those variables would create entropy sounds like more of a common-sense thing, no? I guess I should still brush up on math though, it seems.
Hockenbrizzle 11y ago
A lack of mathematical intuition is a serious problem for many people from engineering to biology to economics. It certainly plagued me throughout my engineering bachelors studies and is something I continually work to get better at. In my opinion, physics students learn the best framework for thinking and get a very good mathematical intuition. For example, here's a problem from an introductory QM book that really threw me for a loop when I was studying: A needle of length L is dropped at random onto a sheet of paper ruled with parallel lines a distance L apart. What is the probability that the needle will cross a line?
- maged 11y ago
  > A needle of length L is dropped at random onto a sheet of paper ruled with parallel lines a distance L apart. What is the probability that the needle will cross a line? Thickness of line is needed right? Otherwise P approaches 100% as thickness approaches 0?
  - davmre 11y ago
    The lines are infinitely thin. Equivalently you can imagine the paper is divided into regions of width L, and the question is whether the needle will cross a region boundary (https://en.wikipedia.org/wiki/Buffon's_needle https://en.wikipedia.org/wiki/Buffon's_needle).
    - nonbel 11y ago
      I don't think that page explains it very well, but have poor math background. I imagined notebook paper with horizontal lines spaced L apart and then the needle dropping at any angle. When the needle is vertical the probability it cross a line is 1, when horizontal it is zero. The length of the needle L is the hypotenuse of a triangle. If we call the angle from horizontal x, the "height" of the needle can be anywhere within h=Lsin(x) for x between 0 and pi/2. The "lines" are like a sample of a point from a uniform distribution U with width L, and h is an interval inside U. The probability a number sampled from a distribution of width L will fall within interval h is h/L. Substituting for h gives p(cross|x) = sin(x). Then assuming the needle is equally likely to drop at any angle, for any one angle theta we get probability density p(theta=x) = 1/(pi/2-0)= 2/pi. The probability the needle drops at angle x AND crosses a line is the product of p(theta=x)p(cross|x)= (2/pi)sin(x). As mentioned, x can range between 0 and pi/2. To get the probability the needle drops at angle x1 OR x2 OR x3, etc and cross we need to sum all these. So take the integral of (2/pi)sin(x) from 0:pi/2. This gives 2/pi.
- theworstshill 11y ago
  99%?
- ihartley 11y ago
andreasvc 11y ago
Not to detract from your point that math is important, but in that example proper methodology (e.g., cross-validation), proper feature engineering, and especially domain knowledge are probably even more important. You can be aware of the strengths & weaknesses of different machine learning algorithms without being intimately familiar with the math. Ideally, ML methods are not treated as a "black boxes", but some aspects are inherently black box, even if you do know the math (e.g., parameter tuning).
jzymbaluk 11y ago
As an undergrad interested in machine learning, what areas of math specifically should I be focusing on?
- dxbydt 11y ago
  1.math prereqs at UW:http://courses.washington.edu/css490/2012.Winter/lecture_slides/02_math_essentials.pdf http://courses.washington.edu/css490/2012.Winter/lecture_sli... 2. math prereqs at UCSC:http://people.ucsc.edu/~praman1/static/pub/math-for-ml.pdf http://people.ucsc.edu/~praman1/static/pub/math-for-ml.pdf 3. math prereqs at UMD: https://www.umiacs.umd.edu/~hal/courses/2013S_ML/math4ml.pdf https://www.umiacs.umd.edu/~hal/courses/2013S_ML/math4ml.pdf
- gtani 11y ago
  here's a list of open content (free pdf) texts that have appendixes or review chapters - UML - http://goodfeli.github.io/dlbook/ http://goodfeli.github.io/dlbook/ - 3 books: Barber, MacKay, and Rasmussen/Williams from this list: https://www.reddit.com/r/MachineLearning/comments/1jeawf/machine_learning_books/ https://www.reddit.com/r/MachineLearning/comments/1jeawf/mac...
mailshanx 11y ago
>> He'll happily throw in your zip code and credit card number into the mix, not realizing that those two features have super high entropy You will be surprised: you can make significant gains by including the zip code - i've seen that happen in a competitive setting. Where you live probably contains some signal about your credit worthiness. Having said that, of course it doesn't make sense to simply feed the raw zip code to the tree. An appropriate encoding (most people would use a one-hot encoding, though there exist better ones) of the zip code will be key to extracting signal in a robust way. >> ... the tree will have serious overfitting issues Isn't it an almost standard practise now to use an ensemble of trees, such as a random forest? Decision trees have long been known to be prone to overfitting.

by dxbydt 11y ago

NegatioN 11y ago

I get that math is an important aspect, but being able to realise that throwing in those variables would create entropy sounds like more of a common-sense thing, no? I guess I should still brush up on math though, it seems.

Hockenbrizzle 11y ago

A lack of mathematical intuition is a serious problem for many people from engineering to biology to economics. It certainly plagued me throughout my engineering bachelors studies and is something I continually work to get better at. In my opinion, physics students learn the best framework for thinking and get a very good mathematical intuition. For example, here's a problem from an introductory QM book that really threw me for a loop when I was studying: A needle of length L is dropped at random onto a sheet of paper ruled with parallel lines a distance L apart. What is the probability that the needle will cross a line?

maged 11y ago
> A needle of length L is dropped at random onto a sheet of paper ruled with parallel lines a distance L apart. What is the probability that the needle will cross a line? Thickness of line is needed right? Otherwise P approaches 100% as thickness approaches 0?
- davmre 11y ago
  The lines are infinitely thin. Equivalently you can imagine the paper is divided into regions of width L, and the question is whether the needle will cross a region boundary (https://en.wikipedia.org/wiki/Buffon's_needle https://en.wikipedia.org/wiki/Buffon's_needle).
  - nonbel 11y ago
    I don't think that page explains it very well, but have poor math background. I imagined notebook paper with horizontal lines spaced L apart and then the needle dropping at any angle. When the needle is vertical the probability it cross a line is 1, when horizontal it is zero. The length of the needle L is the hypotenuse of a triangle. If we call the angle from horizontal x, the "height" of the needle can be anywhere within h=Lsin(x) for x between 0 and pi/2. The "lines" are like a sample of a point from a uniform distribution U with width L, and h is an interval inside U. The probability a number sampled from a distribution of width L will fall within interval h is h/L. Substituting for h gives p(cross|x) = sin(x). Then assuming the needle is equally likely to drop at any angle, for any one angle theta we get probability density p(theta=x) = 1/(pi/2-0)= 2/pi. The probability the needle drops at angle x AND crosses a line is the product of p(theta=x)p(cross|x)= (2/pi)sin(x). As mentioned, x can range between 0 and pi/2. To get the probability the needle drops at angle x1 OR x2 OR x3, etc and cross we need to sum all these. So take the integral of (2/pi)sin(x) from 0:pi/2. This gives 2/pi.
theworstshill 11y ago
99%?
ihartley 11y ago

andreasvc 11y ago

Not to detract from your point that math is important, but in that example proper methodology (e.g., cross-validation), proper feature engineering, and especially domain knowledge are probably even more important. You can be aware of the strengths & weaknesses of different machine learning algorithms without being intimately familiar with the math. Ideally, ML methods are not treated as a "black boxes", but some aspects are inherently black box, even if you do know the math (e.g., parameter tuning).

jzymbaluk 11y ago

As an undergrad interested in machine learning, what areas of math specifically should I be focusing on?

dxbydt 11y ago
1.math prereqs at UW:http://courses.washington.edu/css490/2012.Winter/lecture_slides/02_math_essentials.pdf http://courses.washington.edu/css490/2012.Winter/lecture_sli... 2. math prereqs at UCSC:http://people.ucsc.edu/~praman1/static/pub/math-for-ml.pdf http://people.ucsc.edu/~praman1/static/pub/math-for-ml.pdf 3. math prereqs at UMD: https://www.umiacs.umd.edu/~hal/courses/2013S_ML/math4ml.pdf https://www.umiacs.umd.edu/~hal/courses/2013S_ML/math4ml.pdf
gtani 11y ago
here's a list of open content (free pdf) texts that have appendixes or review chapters - UML - http://goodfeli.github.io/dlbook/ http://goodfeli.github.io/dlbook/ - 3 books: Barber, MacKay, and Rasmussen/Williams from this list: https://www.reddit.com/r/MachineLearning/comments/1jeawf/machine_learning_books/ https://www.reddit.com/r/MachineLearning/comments/1jeawf/mac...

mailshanx 11y ago

>> He'll happily throw in your zip code and credit card number into the mix, not realizing that those two features have super high entropy You will be surprised: you can make significant gains by including the zip code - i've seen that happen in a competitive setting. Where you live probably contains some signal about your credit worthiness. Having said that, of course it doesn't make sense to simply feed the raw zip code to the tree. An appropriate encoding (most people would use a one-hot encoding, though there exist better ones) of the zip code will be key to extracting signal in a robust way. >> ... the tree will have serious overfitting issues Isn't it an almost standard practise now to use an ensemble of trees, such as a random forest? Decision trees have long been known to be prone to overfitting.