4 ms·

No. All the textbooks know that polynomials of high degree are numerically dangerous and you need to be careful when handling them. The articles examples only

No. All the textbooks know that polynomials of high degree are numerically dangerous and you need to be careful when handling them. The articles examples only work because the interval 0 to 1 (or -1 to 1) were chosen. For whatever reason the author does not point that out or even acknowledges the fact that had he chosen a larger interval the limitations of floating point arithmetic would have ruined the argument he was trying to make. 10^100 is a very large number and numerically difficult to treat. For whatever reason the author pretends this is not a valid reason to be cautious about high degree polynomials.

heisenzombie 2y ago
He seems reasonably explicit about this: "" This means that when using polynomial features, the data must be normalized to lie in an interval. It can be done using min-max scaling, computing empirical quantiles, or passing the feature through a sigmoid. But we should avoid the use of polynomials on raw un-normalized features. ""
- constantcrying 2y ago
  No. This paragraph has nothing to do with numerics. It is about the fact that continuous functions can not be approximated globally by polynomials. So you need to restrict to intervals for reasons of mathematical theory. This is totally unrelated to the numerical issues, which are nowhere even acknowledged.
  - vient 2y ago
    But what's the point in acknowledging numerical issues outside of [-1,1] if polynomials do not even work there, as author explicitly notes?
    - constantcrying 2y ago
      All polynomials "work" globally. That some polynomials form an orthonormal basis over certain intervals is essentially irrelevant. The author does not address the single most important reason why high degrees of polynomials are dangerous. Which is pretty insane to be honest, obviously to be honest you have to at least mention why people tell you to be cautious about high degree polynomials AND point out why your examples circumvent the problem. Anything else is extremely dishonest and misleading.
  - sheepscreek 2y ago
    Is there a mathematical proof or basis to back what you’re saying?
    - constantcrying 2y ago
      That polynomials do not approximate continuous functions globally?? That is pretty obvious. Consider that every polynomial grows arbitrarily large, but not every continuous function does.
    - gthompson512 2y ago
      There is a simple corollary to Stone-Weierstrass that extends to infinite intervals, but requires the use of rational functions.
hervature 2y ago
Neural network training is harder when the input range is allowed to deviate from [-1, 1]. The only reason why it sometimes works for neural networks is because the first layer has a chance to normalize it.
jansan 2y ago
If you have a set of points, why can't you just map them to an interval [0, 1] before fitting the polynomial?
toxigun 2y ago
Actually they aren't. You never compute high powers of the argument when working with specialized bases. You use the recursive formula that both the Bernstein basis and the orthogonal polynomial bases are endowed with. This is implemented in numpy, so you don't have to do anything yourself. Just call, for example, np.polynomial.legendre.legvander to get the features for the Legendre basis. And a basis orthogonal over [-1,1] is easily made orthogonal over arbitrary interval. Take p_i to be the i-th legendre polynomial, then the basis composed of q_i(x)=p_i(2(x-a)/(b-a)-1) is orthogonal over [a,b]. Each q_i is itself a polynomial of degree i, but you never use its coefficients explicitly. There is an entire library for computing with polynomial apptoximants of functions over arbitrary intervala using orthogonal polynomials - Chebfun. The entire scientific and spectral differential equations community knows there are no numerical issues working with high degree polynomials over arbitrary intervals. The ML community just hasn't caught up.
- constantcrying 2y ago
  >The entire scientific and spectral differential equations community knows there are no numerical issues working with high degree polynomials over arbitrary intervals. This is totally wrong. Of course there are enormous numerical problems with high degree polynomials. Computing large powers of large numbers is enormously unstable and needs to be avoided under basically all circumstances, that is what makes dealing with those polynomials difficult and cautioning people against this is obvious correct. What you described are the ways to deal with those problems. But this isn't what the article does. My problem with the article is the following: - Does not mention the most important issue with high degree polynomials, namely their numerical instability. - Gives examples, but does not explain why they circumvent the numerical problems at all. The most important choice made (the interval of 0 to 1) is portrayed as essentially arbitrary, not an absolute necessity. - Concludes that there are no problems with high degree polynomials, based on the fact that the experiments worked. Not based on the fact that the actual issues were circumvented, leaving the reader with a totally wrong impression of the issue. This is terrible scholarship and makes for a terrible article. Not acknowledging the issue is a terrible thing to do and not explain why seemingly arbitrary choices are extremely important is another terrible thing to do. The whole article fails at actually portraying the issue at all. To be clear, I am not saying that this approximation does not work or that with appropriate scaling and the right polynomials these issues can't be mostly circumvented. Or that high degree polynomials "in general" are incalculable. I am saying that this article completely fails to say what the issue is and why the examples work.

heisenzombie 2y ago

He seems reasonably explicit about this: "" This means that when using polynomial features, the data must be normalized to lie in an interval. It can be done using min-max scaling, computing empirical quantiles, or passing the feature through a sigmoid. But we should avoid the use of polynomials on raw un-normalized features. ""

constantcrying 2y ago
No. This paragraph has nothing to do with numerics. It is about the fact that continuous functions can not be approximated globally by polynomials. So you need to restrict to intervals for reasons of mathematical theory. This is totally unrelated to the numerical issues, which are nowhere even acknowledged.
- vient 2y ago
  But what's the point in acknowledging numerical issues outside of [-1,1] if polynomials do not even work there, as author explicitly notes?
  - constantcrying 2y ago
    All polynomials "work" globally. That some polynomials form an orthonormal basis over certain intervals is essentially irrelevant. The author does not address the single most important reason why high degrees of polynomials are dangerous. Which is pretty insane to be honest, obviously to be honest you have to at least mention why people tell you to be cautious about high degree polynomials AND point out why your examples circumvent the problem. Anything else is extremely dishonest and misleading.
- sheepscreek 2y ago
  Is there a mathematical proof or basis to back what you’re saying?
  - constantcrying 2y ago
    That polynomials do not approximate continuous functions globally?? That is pretty obvious. Consider that every polynomial grows arbitrarily large, but not every continuous function does.
  - gthompson512 2y ago
    There is a simple corollary to Stone-Weierstrass that extends to infinite intervals, but requires the use of rational functions.

hervature 2y ago

Neural network training is harder when the input range is allowed to deviate from [-1, 1]. The only reason why it sometimes works for neural networks is because the first layer has a chance to normalize it.

jansan 2y ago

If you have a set of points, why can't you just map them to an interval [0, 1] before fitting the polynomial?

toxigun 2y ago

Actually they aren't. You never compute high powers of the argument when working with specialized bases. You use the recursive formula that both the Bernstein basis and the orthogonal polynomial bases are endowed with. This is implemented in numpy, so you don't have to do anything yourself. Just call, for example, np.polynomial.legendre.legvander to get the features for the Legendre basis. And a basis orthogonal over [-1,1] is easily made orthogonal over arbitrary interval. Take p_i to be the i-th legendre polynomial, then the basis composed of q_i(x)=p_i(2(x-a)/(b-a)-1) is orthogonal over [a,b]. Each q_i is itself a polynomial of degree i, but you never use its coefficients explicitly. There is an entire library for computing with polynomial apptoximants of functions over arbitrary intervala using orthogonal polynomials - Chebfun. The entire scientific and spectral differential equations community knows there are no numerical issues working with high degree polynomials over arbitrary intervals. The ML community just hasn't caught up.

constantcrying 2y ago
>The entire scientific and spectral differential equations community knows there are no numerical issues working with high degree polynomials over arbitrary intervals. This is totally wrong. Of course there are enormous numerical problems with high degree polynomials. Computing large powers of large numbers is enormously unstable and needs to be avoided under basically all circumstances, that is what makes dealing with those polynomials difficult and cautioning people against this is obvious correct. What you described are the ways to deal with those problems. But this isn't what the article does. My problem with the article is the following: - Does not mention the most important issue with high degree polynomials, namely their numerical instability. - Gives examples, but does not explain why they circumvent the numerical problems at all. The most important choice made (the interval of 0 to 1) is portrayed as essentially arbitrary, not an absolute necessity. - Concludes that there are no problems with high degree polynomials, based on the fact that the experiments worked. Not based on the fact that the actual issues were circumvented, leaving the reader with a totally wrong impression of the issue. This is terrible scholarship and makes for a terrible article. Not acknowledging the issue is a terrible thing to do and not explain why seemingly arbitrary choices are extremely important is another terrible thing to do. The whole article fails at actually portraying the issue at all. To be clear, I am not saying that this approximation does not work or that with appropriate scaling and the right polynomials these issues can't be mostly circumvented. Or that high degree polynomials "in general" are incalculable. I am saying that this article completely fails to say what the issue is and why the examples work.