4 ms·

Fun fact: when working on Herbie (http://herbie.uwplse.org http://herbie.uwplse.org), our automated tool for reducing floating-point error by rearranging your m

Fun fact: when working on Herbie (http://herbie.uwplse.org http://herbie.uwplse.org), our automated tool for reducing floating-point error by rearranging your mathematical expressions, we found that fast-math often undid Herbie's improvements. In a sense, Herbie and fast-math are opposites: one makes code more accurate (sometimes slower, sometimes faster), while the other makes code faster (sometimes less accurate, sometimes more).

simonbyrne 5y ago
Herbie is a great tool, especially for teaching.
named-user 5y ago
The docs for fast math literally say this "This option is not turned on by any -O option besides -Ofast since it can result in incorrect output for programs that depend on an exact implementation of IEEE or ISO rules/specifications for math functions. It may, however, yield faster code for programs that do not require the guarantees of these specifications." So what did you expect? I was taught in my undergrad class not to use fast math if you ever wanted accuracy. I don't understand how you missed this?
- pavpanchekha 5y ago
  Hi, I noticed you're a new user, so in that spirit this is not the kind of comment that is well-received on HN. It reads as arrogant and condescending, though I assume that was not your intention. Since I publish regularly in this area, the condescension feels especially off-base. To answer the question in good faith: while the documentation clearly states that fast-math can reduce accuracy, in isolation the accuracy changes are often small, and moreover in isolation the accuracy changes often look positive. So I was surprised, when evaluating Herbie, that it frequently caused large, negative changes in accuracy, specifically by undoing changes made to improve accuracy.
  - deleted 5y ago
    [deleted]
Quekid5 5y ago
That's a real "fun fact" kind of thing. Love it. Of course, the 'real' solution is actual Numerical Analysis (as I'm sure you know) to keep the error properly bounded, but it's really interesting to have a sort of middle ground which just stabilizes the math locally... which might be good enough. Other fun fact: Numerical Analysis is a thing that's really hard to imagine unless you happen to be introduced to it during an education. It's so obviously a thing once you've heard of it, but REALLY hard to come up with ex nihilo.
kazinator 5y ago
If you have a program which finds some order of items in order to optimize something, and then you introduce some confounding technology which scrambles the order of the items afterward, that's indistinguishable from introducing bugs into that optimizing program; the output no longer implements the optimization requirements that the program tries to ensure. I don't see how Herbie's accuracy improvements could not be undone, if Herbie's output is fed to a back-end which doesn't preserve Herbie's order of operations as Herbie requires and depends on.
shoo 5y ago
thank you for sharing the link to Herbie, that looks like a useful tool. If I follow at high level, it looks like Herbie is trying to rewrite expressions to minimise error without runtime performance constraints. Are there alternative tools that focus on rewriting code to maximise performance while keeping error below some configurable bound? i guess compilers are generally focused on the latter problem, perhaps without giving the user much control over the degree of error they are willing to tolerate.
- mattpharr 5y ago
  > Are there alternative tools that focus on rewriting code to maximise performance while keeping error below some configurable bound? There are! See followup work by @pavpanchekha and others on "Pherbie", which finds a set of Pareto-optimal rewritings of a program so that it's possible to trade-off error and performance: https://ztatlock.net/pubs/2021-arith-pherbie/paper.pdf https://ztatlock.net/pubs/2021-arith-pherbie/paper.pdf.
  - shoo 5y ago
    Fantastic! This appears to be available in the command line version of herbie with the `--pareto` flag: > Enables multi-objective improvement. Herbie will attempt to simultaneously optimize for both accuracy and expression cost. Rather than generating a single "ideal" output expression, Herbie will generate many output expressions. This mode is still considered experimental. This will take a long time to run. We recommend timeouts measured in hours.
  - adgjlsfhk1 5y ago
    I've generally been disappointed by ferbie when I've tried it, but this does look really cool. I would love it if this provides a path to making it easier to get good tradeoffs between precision and accuracy.

simonbyrne 5y ago

Herbie is a great tool, especially for teaching.

named-user 5y ago

The docs for fast math literally say this "This option is not turned on by any -O option besides -Ofast since it can result in incorrect output for programs that depend on an exact implementation of IEEE or ISO rules/specifications for math functions. It may, however, yield faster code for programs that do not require the guarantees of these specifications." So what did you expect? I was taught in my undergrad class not to use fast math if you ever wanted accuracy. I don't understand how you missed this?

pavpanchekha 5y ago
Hi, I noticed you're a new user, so in that spirit this is not the kind of comment that is well-received on HN. It reads as arrogant and condescending, though I assume that was not your intention. Since I publish regularly in this area, the condescension feels especially off-base. To answer the question in good faith: while the documentation clearly states that fast-math can reduce accuracy, in isolation the accuracy changes are often small, and moreover in isolation the accuracy changes often look positive. So I was surprised, when evaluating Herbie, that it frequently caused large, negative changes in accuracy, specifically by undoing changes made to improve accuracy.
- deleted 5y ago
  [deleted]

Quekid5 5y ago

That's a real "fun fact" kind of thing. Love it. Of course, the 'real' solution is actual Numerical Analysis (as I'm sure you know) to keep the error properly bounded, but it's really interesting to have a sort of middle ground which just stabilizes the math locally... which might be good enough. Other fun fact: Numerical Analysis is a thing that's really hard to imagine unless you happen to be introduced to it during an education. It's so obviously a thing once you've heard of it, but REALLY hard to come up with ex nihilo.

kazinator 5y ago

If you have a program which finds some order of items in order to optimize something, and then you introduce some confounding technology which scrambles the order of the items afterward, that's indistinguishable from introducing bugs into that optimizing program; the output no longer implements the optimization requirements that the program tries to ensure. I don't see how Herbie's accuracy improvements could not be undone, if Herbie's output is fed to a back-end which doesn't preserve Herbie's order of operations as Herbie requires and depends on.

shoo 5y ago

thank you for sharing the link to Herbie, that looks like a useful tool. If I follow at high level, it looks like Herbie is trying to rewrite expressions to minimise error without runtime performance constraints. Are there alternative tools that focus on rewriting code to maximise performance while keeping error below some configurable bound? i guess compilers are generally focused on the latter problem, perhaps without giving the user much control over the degree of error they are willing to tolerate.

mattpharr 5y ago
> Are there alternative tools that focus on rewriting code to maximise performance while keeping error below some configurable bound? There are! See followup work by @pavpanchekha and others on "Pherbie", which finds a set of Pareto-optimal rewritings of a program so that it's possible to trade-off error and performance: https://ztatlock.net/pubs/2021-arith-pherbie/paper.pdf https://ztatlock.net/pubs/2021-arith-pherbie/paper.pdf.
- shoo 5y ago
  Fantastic! This appears to be available in the command line version of herbie with the `--pareto` flag: > Enables multi-objective improvement. Herbie will attempt to simultaneously optimize for both accuracy and expression cost. Rather than generating a single "ideal" output expression, Herbie will generate many output expressions. This mode is still considered experimental. This will take a long time to run. We recommend timeouts measured in hours.
- adgjlsfhk1 5y ago
  I've generally been disappointed by ferbie when I've tried it, but this does look really cool. I would love it if this provides a path to making it easier to get good tradeoffs between precision and accuracy.