5 ms·

I'm really not so amused by the bloating of words like "machine learning through polynomial regression". People have been doing polynomial regression for centur

I'm really not so amused by the bloating of words like "machine learning through polynomial regression". People have been doing polynomial regression for centuries (almost as old as linear regression) and how does it become a new thing under the umbrella of machine learning? And not to mention, normal equation is vastly superior to gradient descent in solving out the coefficients for polynomial regression. (The criticism is not directed to OP but to the field in general.)

ced 7y ago
It's an introductory article. "Complex subject" through "simple example" is a compelling formula, no?
fizixer 7y ago
'through' in the title connects with 'introduction' not 'machine learning'. e.g., introduction to space-shuttle flying, through flight simulator on a gaming console.
curiousgal 7y ago
"Machine Learning" these days is people rediscovering statistics. It's always amusing how stats are underappreciated. I was at a career fair the other day talking to a manager at HSBC, he was surprised to know that a stats curriculum includes stochastic calculus...
- mistymountains 7y ago
  >manager >HSBC two strikes fam.
- solveit 7y ago
  I'm surprised too. Do you actually mean that most stats majors learn stochastic calculus or just that it's possible to take stochastic calculus? As a pure mathematician, my experience is that stochastic calculus was only taught to very upper-level undergraduates or graduate students because of its measure theory prerequisite. And given the number of maths majors that graduate without ever having taken a course in measure theory, I would be surprised if many stats majors actually learned stochastic calculus. Unless you guys did what the physicists do and just taught the theorems without going too deeply into the foundations, I guess?
  - curiousgal 7y ago
    I should have specified that this took place in France. Engineering students, are first introduced to probability during the CPGE (intensive math, physics, economics, 2 or 3 year track). Said introduction is based on measure theory (sigma algebras, Lebesgue integral, etc). Once at a statistics engineering school, all of the stats courses are based on probability and measure theory. Then you get into martingales and Markov chains and the rest of the stochastic processes follow along. I think that many people don't realize that Statistics is heavily based on probability, so basically anything "stochastic" shouldn't be that far fetched.
throwlaplace 7y ago
>normal equation is vastly superior to gradient descent in solving out the coefficients for polynomial regression Completely depends on the size of your data set. I would not want to pseudo-invert a design matrix > 1000 samples.
- curiousgal 7y ago
  You don't need to invert the matrix. You want to solve X' X Beta = X' Y for Beta. Computationally, it is faster to solve the equations one by one. https://www.johndcook.com/blog/2010/01/19/dont-invert-that-matrix/ https://www.johndcook.com/blog/2010/01/19/dont-invert-that-m...
  - nuclearnice1 7y ago
    Isn’t John D Cook arguing for iterative methods there? It does not read as a defense of “using the normal equations” or direct methods. In fact, the reference to sparsity makes you think John May be referring to SGD or a similar iterative method. Obviously, any solution you get solves the normal equation so it’s unclear what the parent comment meant by using the normal equation is superior to SGD.
    - macleginn 7y ago
      Stochastic gradient descent only approximates the solution, and it may do that by looking at a single equation (or a small batch of them) at a time, without ever bothering with the whole system.
      - nuclearnice1 7y ago
        Is it approximating the gradient or approximating the solution? If it’s the gradient, doesn’t it have the real gradient in expectation, so you can run as many iterations to get epsilon precision up to what your machine will support?
    - salty_biscuits 7y ago
      It'd be something like conjugate gradient, not sgd. Unless A itself is too big to fit in memory.
gumby 7y ago
I think you have it backwards: this is saying "hey, polynomial regression isn't fancy, so by standing on it you can see what machine learning is all about.
rickdeveloper 7y ago
As others have mentioned I did not intend to bloat by using the word 'machine learning.' Polynomial regression is taught as first concept in many machine learning courses so I figured it might be a good way to introduce newcomers to the field. Many of the posts currently available only show how to make another neural network for MNIST, but do not cover the background theory. I know that polynomial regression is a concept of statistics, but I would argue most of machine learning is too. That is the reason I choose this concept as an intro to machine learning (and consequently statistics). Thank you for your feedback. I really appreciate it. I have added a paragraph on the normal equation (should be online anytime) and why you might/might not want to use it.
anthony_doan 7y ago
We had an American Statistical Association workshop for the local Southern California chapter recently. The speaker for the workshop mentioned he had to often change Logistic regression wikipedia page back to statistic technique from Machine Learning because people keep on changing it to ML. At the same time ASA have release their 2019 report about data science and the cross road statistic is at. It's an interesting intersection ML/data science & Statistic is having at this point. https://www.amstat.org/ASA/News/Statistics-at-a-Crossroads-Recommendations-Are-Released.aspx https://www.amstat.org/ASA/News/Statistics-at-a-Crossroads-R... The report also give statistic version of their definition of statistic vs machine learning and what each field emphasize on. I think often people are just really unclear on statistic and ML field because they never sat down to think about the what define each field. It also end up getting political and people being defensive. But I think at least this report give a statistic view point.

ced 7y ago

It's an introductory article. "Complex subject" through "simple example" is a compelling formula, no?

fizixer 7y ago

'through' in the title connects with 'introduction' not 'machine learning'. e.g., introduction to space-shuttle flying, through flight simulator on a gaming console.

curiousgal 7y ago

"Machine Learning" these days is people rediscovering statistics. It's always amusing how stats are underappreciated. I was at a career fair the other day talking to a manager at HSBC, he was surprised to know that a stats curriculum includes stochastic calculus...

mistymountains 7y ago
>manager >HSBC two strikes fam.
solveit 7y ago
I'm surprised too. Do you actually mean that most stats majors learn stochastic calculus or just that it's possible to take stochastic calculus? As a pure mathematician, my experience is that stochastic calculus was only taught to very upper-level undergraduates or graduate students because of its measure theory prerequisite. And given the number of maths majors that graduate without ever having taken a course in measure theory, I would be surprised if many stats majors actually learned stochastic calculus. Unless you guys did what the physicists do and just taught the theorems without going too deeply into the foundations, I guess?
- curiousgal 7y ago
  I should have specified that this took place in France. Engineering students, are first introduced to probability during the CPGE (intensive math, physics, economics, 2 or 3 year track). Said introduction is based on measure theory (sigma algebras, Lebesgue integral, etc). Once at a statistics engineering school, all of the stats courses are based on probability and measure theory. Then you get into martingales and Markov chains and the rest of the stochastic processes follow along. I think that many people don't realize that Statistics is heavily based on probability, so basically anything "stochastic" shouldn't be that far fetched.

throwlaplace 7y ago

>normal equation is vastly superior to gradient descent in solving out the coefficients for polynomial regression Completely depends on the size of your data set. I would not want to pseudo-invert a design matrix > 1000 samples.

curiousgal 7y ago
You don't need to invert the matrix. You want to solve X' X Beta = X' Y for Beta. Computationally, it is faster to solve the equations one by one. https://www.johndcook.com/blog/2010/01/19/dont-invert-that-matrix/ https://www.johndcook.com/blog/2010/01/19/dont-invert-that-m...
- nuclearnice1 7y ago
  Isn’t John D Cook arguing for iterative methods there? It does not read as a defense of “using the normal equations” or direct methods. In fact, the reference to sparsity makes you think John May be referring to SGD or a similar iterative method. Obviously, any solution you get solves the normal equation so it’s unclear what the parent comment meant by using the normal equation is superior to SGD.
  - macleginn 7y ago
    Stochastic gradient descent only approximates the solution, and it may do that by looking at a single equation (or a small batch of them) at a time, without ever bothering with the whole system.
    - nuclearnice1 7y ago
      Is it approximating the gradient or approximating the solution? If it’s the gradient, doesn’t it have the real gradient in expectation, so you can run as many iterations to get epsilon precision up to what your machine will support?
  - salty_biscuits 7y ago
    It'd be something like conjugate gradient, not sgd. Unless A itself is too big to fit in memory.

gumby 7y ago

I think you have it backwards: this is saying "hey, polynomial regression isn't fancy, so by standing on it you can see what machine learning is all about.

rickdeveloper 7y ago

As others have mentioned I did not intend to bloat by using the word 'machine learning.' Polynomial regression is taught as first concept in many machine learning courses so I figured it might be a good way to introduce newcomers to the field. Many of the posts currently available only show how to make another neural network for MNIST, but do not cover the background theory. I know that polynomial regression is a concept of statistics, but I would argue most of machine learning is too. That is the reason I choose this concept as an intro to machine learning (and consequently statistics). Thank you for your feedback. I really appreciate it. I have added a paragraph on the normal equation (should be online anytime) and why you might/might not want to use it.

anthony_doan 7y ago

We had an American Statistical Association workshop for the local Southern California chapter recently. The speaker for the workshop mentioned he had to often change Logistic regression wikipedia page back to statistic technique from Machine Learning because people keep on changing it to ML. At the same time ASA have release their 2019 report about data science and the cross road statistic is at. It's an interesting intersection ML/data science & Statistic is having at this point. https://www.amstat.org/ASA/News/Statistics-at-a-Crossroads-Recommendations-Are-Released.aspx https://www.amstat.org/ASA/News/Statistics-at-a-Crossroads-R... The report also give statistic version of their definition of statistic vs machine learning and what each field emphasize on. I think often people are just really unclear on statistic and ML field because they never sat down to think about the what define each field. It also end up getting political and people being defensive. But I think at least this report give a statistic view point.