4 ms·

Floats Don't Agree with Themselves

37 points by cremer 5mo ago

fefa4ka 5mo ago
[dead]
adampunk 5mo ago
Makes you wish everyone agreed on extended precision!
- cremer 5mo ago
  careful what you wish for, x87's extended precision is what wrote the original bug: 80 bits in registers,64 on a spill, so the value depended on register pressure ARM and WASM dropped it for a reason, and more bits would not help anyway, a sign is one bit, any rounding step that disagrees at the last bit flips it
jmalicki 5mo ago
I love seeing a Shewchuk citation other than my ML background of learning conjugate gradient! He is truly a great educator!
- cremer 5mo ago
  his predicates paper opens with "Computational geometers despise floating-point arithmetic" same trick as the CG title: write the sentence a frustrated reader would write, then aren it.. if you like those the Triangle paper is the third one in the same key
- LegionMammal978 5mo ago
  Yeah, in general, this is a problem that people have spent a lot of time thinking about; while floating-point numbers can be finicky, they're what you have to work with if you have inputs at multiple scales. (Meanwhile, I wonder why it's a fair bit harder to look up Ozaki et al.'s optimized version [0] compared to Shewchuk's original paper [1], unless perhaps later authors have found it to be no improvement at all.) [0] https://www.tuhh.de/ti3/paper/rump/OzBueOgOiRu15.pdf https://www.tuhh.de/ti3/paper/rump/OzBueOgOiRu15.pdf [1] https://people.eecs.berkeley.edu/~jrs/papers/robust-predicates.pdf https://people.eecs.berkeley.edu/~jrs/papers/robust-predicat...
stuaxo 5mo ago
Not sure if this was written with AI assistance of not, but I've become allergic to linguistic triples as LLMs use them so much, reading "Same code. Same input. Different answer" makes me not want to read the rest.
- jan_Inkepa 5mo ago
  a tricolon crescens ^^ https://latinaadvitam.blogspot.com/2007/02/poetry-device-of-day-tricolon-crescens.html https://latinaadvitam.blogspot.com/2007/02/poetry-device-of-...
  - gobdovan 5mo ago
    I think the problem is the mismatch between the intended evocative tone of tricolon crescens and the triviality of a 'computer quirk' in the grand scheme of things. "Use figures of speech, but don't sling them around like monkey shit" - my literature teacher.
- thecaio 5mo ago
  Same. I now tend to simply abandon writing when I see those tell tale signs
- mr_mitm 5mo ago
  I also got turned off after the very first paragraph. Too bad, because the content itself might be fine, but the AI language has become repulsive.
- weberer 5mo ago
  Its weird that almost every AI generated blurb tends to have that. Usually followed by "Its not just X, its Y". I wonder where they picked that up from.
  - Jtarii 5mo ago
    It sounds authoritative so people would respond positively to it when presented with it during training.
sampo 5mo ago
I wish the blog would reveal the values of the 3 floats that make their cross_sign(A, B, C) to give different results in different platforms.
- LegionMammal978 5mo ago
  At least the Rust compiler (TFA's project is written in Rust) tries to configure LLVM specifically to avoid these discrepancies, and to treat all basic floating-point operations exactly as written with round-to-nearest behavior [0]. It does not have any of the -ffast-math options that the author('s LLM) is panicking about. The main caveat is that on x86 targets without SSE2, LLVM is deeply wired to use the x87 instructions without attempting to emulate the IEEE overflow/underflow behavior [1]. So perhaps it could be possible to exhibit a discrepancy, but only by compiling for i586 and an ancient target-cpu. It's very doubtful that this was the cause of the original client vs. server issue in TFA's introduction. [0] https://rust-lang.github.io/rfcs/3514-float-semantics.html https://rust-lang.github.io/rfcs/3514-float-semantics.html [1] https://github.com/rust-lang/rust/issues/114479 https://github.com/rust-lang/rust/issues/114479
tyilo 5mo ago
If the original code was written in Rust, then I don't think the Rust compiler is allowed to do any of these "optimizations" of rewriting floating point expressions.
- josefx 5mo ago
  The C compiler also isn't allowed to do all of them. However some people use fast-math or compilers that default to fast-math to break the rules. Some older targets also may use the 80 bit fpu, which is its own mess and any sane compiler will default to properly sized SSE instructions instead.
- pfortuny 5mo ago
  IIRC it also depends on the CPU's storage and capabilities.
- j16sdiz 5mo ago
  On x86, LLVM does not follow strict IEEE 754 behavior all the time.
antiquark 5mo ago
That's why they always teach: "never compare floats for equality." Or maybe they don't teach that anymore, I dunno. See link for the Fundamental Axiom of Floating Point Arithmetic: All floating point arithmetic operations are exact up to a relative error of epsilon_machine. https://www.johnbcoughlin.com/posts/floating-point-axiom/ https://www.johnbcoughlin.com/posts/floating-point-axiom/
- cjfd 5mo ago
  On the other hand, if you store a small integer in a float it is generally reliable to compare to it. E.g., setting a float to zero and comparing whether the float is zero.
- TomMasz 5mo ago
  I certainly teach that. When we work problems involving money, I always recommend students use integers for cents and only convert to dollars and cents when they have to print them.
saagarjha 5mo ago
> It was IEEE 754 working as advertised. The standard pins down the storage format. It does not pin down behavior. IEEE 754 absolutely does pin down behavior.
- nnevatie 5mo ago
  You don't understand. In the modern AI-infested World, it is more important to make grandioese claims with em-dashes or periods working as hype-catalyst, than it is to be correct.
- j16sdiz 5mo ago
  This article is full of LLM-induced inconsistency. That being said, many hardware instruction does not follow strict IEEE 754 behavior.
lexicality 5mo ago
> -ffast-math default, often on without intent Ah yes -ffast-math, also known as -fincorrect-math
- yxhuvud 5mo ago
  It makes sense in certain contexts, but it is batshit insane that it is enabled on a program (including any linked stuff) level. It would have made sense as numbers with separate types or perhaps even if you marked a region in code and it applied only to code in that region. Then the consequences of enabling it could be understood in ways that could be safe and usable (eg enabling it for a hot loop where the numerical ranges are well defined and safe in context). But no, instead it is a fat shotgun.
IshKebab 5mo ago
This seems to be AI bullshit. You can easily get deterministic floats that give the same results everywhere. Just don't use -ffast-math.
- LegionMammal978 5mo ago
  Alas, compilers have historically fudged the behavior on some 32-bit targets, e.g., x86 targets without SSE2 [0], so you'd have to go all the way to soft-float implementations if you really want guarantees. But these targets are rare, and very unlikely to be the cause of the original issue. (And NaN payloads in general are a mess across platforms and ABIs, but that can be solved by treating all NaNs as identical.) [0] https://github.com/rust-lang/rust/issues/114479 https://github.com/rust-lang/rust/issues/114479
happytoexplain 5mo ago
> It was IEEE 754 working as advertised. The standard pins down the storage format. It does not pin down behavior. Huh? This is just false, isn't it? AFAIK, it pins down behavior, and the differences come from things like changing the order of operations (i.e. accumulating rounding error in a different order). I'm not an expert, though.
- gnabgib 5mo ago
  It does not pin down the behaviour.. just a storage format. See the NaNs (many representations, unspecified bits), the zeros (both), subnormals, the multiple rounding rules (no behaviour required). https://en.wikipedia.org/wiki/IEEE_754 https://en.wikipedia.org/wiki/IEEE_754
exabrial 5mo ago
Once I understood that floating point arithmetic was just a fast useful approximation of actual numbers, it changed my thinking of when to use them. Visualizing of floats as a arbitrary sample of numbers along the number line that you were allowed to choose from is a good way to figure out whether or not you should be using them. Much like financial calculations, for this use case, exact representations of inputs are needed. An approximation of what the input might be isn’t useful. One thing if you want point out though: there is a difference between reproduceability and accuracy. The Java Virtual Machine (>=17, or strictfp) on every processor arch, OS, glibc, etc guarantees strict reproduceability for basic operations on floats. some operations Math (pow, cos, log, etc) package are allowed to differ within a tiny precision window. If you need absolute reproduceability, you can use StrictMath, which gives us an interesting property for a library like this: you could use floats, and it would be a bit reproducible on every platform and software stack combination and be deterministic, everywhere. It would not have absolute integer math accuracy however. Whether that is still useful is up to you.

37 points by cremer 5mo ago

fefa4ka 5mo ago

[dead]

adampunk 5mo ago

Makes you wish everyone agreed on extended precision!

cremer 5mo ago
careful what you wish for, x87's extended precision is what wrote the original bug: 80 bits in registers,64 on a spill, so the value depended on register pressure ARM and WASM dropped it for a reason, and more bits would not help anyway, a sign is one bit, any rounding step that disagrees at the last bit flips it

jmalicki 5mo ago

I love seeing a Shewchuk citation other than my ML background of learning conjugate gradient! He is truly a great educator!

cremer 5mo ago
his predicates paper opens with "Computational geometers despise floating-point arithmetic" same trick as the CG title: write the sentence a frustrated reader would write, then aren it.. if you like those the Triangle paper is the third one in the same key
LegionMammal978 5mo ago
Yeah, in general, this is a problem that people have spent a lot of time thinking about; while floating-point numbers can be finicky, they're what you have to work with if you have inputs at multiple scales. (Meanwhile, I wonder why it's a fair bit harder to look up Ozaki et al.'s optimized version [0] compared to Shewchuk's original paper [1], unless perhaps later authors have found it to be no improvement at all.) [0] https://www.tuhh.de/ti3/paper/rump/OzBueOgOiRu15.pdf https://www.tuhh.de/ti3/paper/rump/OzBueOgOiRu15.pdf [1] https://people.eecs.berkeley.edu/~jrs/papers/robust-predicates.pdf https://people.eecs.berkeley.edu/~jrs/papers/robust-predicat...

stuaxo 5mo ago

Not sure if this was written with AI assistance of not, but I've become allergic to linguistic triples as LLMs use them so much, reading "Same code. Same input. Different answer" makes me not want to read the rest.

jan_Inkepa 5mo ago
a tricolon crescens ^^ https://latinaadvitam.blogspot.com/2007/02/poetry-device-of-day-tricolon-crescens.html https://latinaadvitam.blogspot.com/2007/02/poetry-device-of-...
- gobdovan 5mo ago
  I think the problem is the mismatch between the intended evocative tone of tricolon crescens and the triviality of a 'computer quirk' in the grand scheme of things. "Use figures of speech, but don't sling them around like monkey shit" - my literature teacher.
thecaio 5mo ago
Same. I now tend to simply abandon writing when I see those tell tale signs
mr_mitm 5mo ago
I also got turned off after the very first paragraph. Too bad, because the content itself might be fine, but the AI language has become repulsive.
weberer 5mo ago
Its weird that almost every AI generated blurb tends to have that. Usually followed by "Its not just X, its Y". I wonder where they picked that up from.
- Jtarii 5mo ago
  It sounds authoritative so people would respond positively to it when presented with it during training.

sampo 5mo ago

I wish the blog would reveal the values of the 3 floats that make their cross_sign(A, B, C) to give different results in different platforms.

LegionMammal978 5mo ago
At least the Rust compiler (TFA's project is written in Rust) tries to configure LLVM specifically to avoid these discrepancies, and to treat all basic floating-point operations exactly as written with round-to-nearest behavior [0]. It does not have any of the -ffast-math options that the author('s LLM) is panicking about. The main caveat is that on x86 targets without SSE2, LLVM is deeply wired to use the x87 instructions without attempting to emulate the IEEE overflow/underflow behavior [1]. So perhaps it could be possible to exhibit a discrepancy, but only by compiling for i586 and an ancient target-cpu. It's very doubtful that this was the cause of the original client vs. server issue in TFA's introduction. [0] https://rust-lang.github.io/rfcs/3514-float-semantics.html https://rust-lang.github.io/rfcs/3514-float-semantics.html [1] https://github.com/rust-lang/rust/issues/114479 https://github.com/rust-lang/rust/issues/114479

tyilo 5mo ago

If the original code was written in Rust, then I don't think the Rust compiler is allowed to do any of these "optimizations" of rewriting floating point expressions.

josefx 5mo ago
The C compiler also isn't allowed to do all of them. However some people use fast-math or compilers that default to fast-math to break the rules. Some older targets also may use the 80 bit fpu, which is its own mess and any sane compiler will default to properly sized SSE instructions instead.
pfortuny 5mo ago
IIRC it also depends on the CPU's storage and capabilities.
j16sdiz 5mo ago
On x86, LLVM does not follow strict IEEE 754 behavior all the time.

antiquark 5mo ago

That's why they always teach: "never compare floats for equality." Or maybe they don't teach that anymore, I dunno. See link for the Fundamental Axiom of Floating Point Arithmetic: All floating point arithmetic operations are exact up to a relative error of epsilon_machine. https://www.johnbcoughlin.com/posts/floating-point-axiom/ https://www.johnbcoughlin.com/posts/floating-point-axiom/

cjfd 5mo ago
On the other hand, if you store a small integer in a float it is generally reliable to compare to it. E.g., setting a float to zero and comparing whether the float is zero.
TomMasz 5mo ago
I certainly teach that. When we work problems involving money, I always recommend students use integers for cents and only convert to dollars and cents when they have to print them.

saagarjha 5mo ago

> It was IEEE 754 working as advertised. The standard pins down the storage format. It does not pin down behavior. IEEE 754 absolutely does pin down behavior.

nnevatie 5mo ago
You don't understand. In the modern AI-infested World, it is more important to make grandioese claims with em-dashes or periods working as hype-catalyst, than it is to be correct.
j16sdiz 5mo ago
This article is full of LLM-induced inconsistency. That being said, many hardware instruction does not follow strict IEEE 754 behavior.

lexicality 5mo ago

> -ffast-math default, often on without intent Ah yes -ffast-math, also known as -fincorrect-math

yxhuvud 5mo ago
It makes sense in certain contexts, but it is batshit insane that it is enabled on a program (including any linked stuff) level. It would have made sense as numbers with separate types or perhaps even if you marked a region in code and it applied only to code in that region. Then the consequences of enabling it could be understood in ways that could be safe and usable (eg enabling it for a hot loop where the numerical ranges are well defined and safe in context). But no, instead it is a fat shotgun.

IshKebab 5mo ago

This seems to be AI bullshit. You can easily get deterministic floats that give the same results everywhere. Just don't use -ffast-math.

LegionMammal978 5mo ago
Alas, compilers have historically fudged the behavior on some 32-bit targets, e.g., x86 targets without SSE2 [0], so you'd have to go all the way to soft-float implementations if you really want guarantees. But these targets are rare, and very unlikely to be the cause of the original issue. (And NaN payloads in general are a mess across platforms and ABIs, but that can be solved by treating all NaNs as identical.) [0] https://github.com/rust-lang/rust/issues/114479 https://github.com/rust-lang/rust/issues/114479

happytoexplain 5mo ago

> It was IEEE 754 working as advertised. The standard pins down the storage format. It does not pin down behavior. Huh? This is just false, isn't it? AFAIK, it pins down behavior, and the differences come from things like changing the order of operations (i.e. accumulating rounding error in a different order). I'm not an expert, though.

gnabgib 5mo ago
It does not pin down the behaviour.. just a storage format. See the NaNs (many representations, unspecified bits), the zeros (both), subnormals, the multiple rounding rules (no behaviour required). https://en.wikipedia.org/wiki/IEEE_754 https://en.wikipedia.org/wiki/IEEE_754

exabrial 5mo ago

Once I understood that floating point arithmetic was just a fast useful approximation of actual numbers, it changed my thinking of when to use them. Visualizing of floats as a arbitrary sample of numbers along the number line that you were allowed to choose from is a good way to figure out whether or not you should be using them. Much like financial calculations, for this use case, exact representations of inputs are needed. An approximation of what the input might be isn’t useful. One thing if you want point out though: there is a difference between reproduceability and accuracy. The Java Virtual Machine (>=17, or strictfp) on every processor arch, OS, glibc, etc guarantees strict reproduceability for basic operations on floats. some operations Math (pow, cos, log, etc) package are allowed to differ within a tiny precision window. If you need absolute reproduceability, you can use StrictMath, which gives us an interesting property for a library like this: you could use floats, and it would be a bit reproducible on every platform and software stack combination and be deterministic, everywhere. It would not have absolute integer math accuracy however. Whether that is still useful is up to you.