4 ms·

> Key Takeaways: > [...] > Floating-point arithmetic is hard. I have successfully avoided FP code for most of my career. At this point, I consider the domain

> Key Takeaways: > [...] > Floating-point arithmetic is hard. I have successfully avoided FP code for most of my career. At this point, I consider the domain sophisticated enough to be an independent skill on someone's resume.

RhysU 5y ago
Just this week I watched someone discover that computing summary statistics in 32-bit on a large dataset is a bad idea. The computer science curricula needs to incorporate more computational science. It's a shame to charge someone tens of thousands of USD and to not warn them that floating point has some obvious footcanons.
- bigiain 5y ago
  > Just this week I watched someone discover that computing summary statistics in 32-bit on a large dataset is a bad idea. The computer science curricula needs to incorporate more computational science. Sadly, I suspect too many "computer science" courses have turned into "vocational coding" courses, and now those people are computing summary statistics on large datasets in Javascript...
- bqmjjx0kac 5y ago
  Could you shed some light on what they did wrong, and what would be a better way to do it?
  - cellularmitosis 5y ago
    not OP, but the hint is in “computing summary statistics in 32-bit on a large dataset”. A large dataset means lots of values, maybe we can assume the number of values is way bigger than any individual value. Perhaps think of McDonalds purchases nation-wide: billions of values but each value is probably less than $10. The simplest summary statistic would be a grand total (sum). If you have a good mental model of floats, you immediately see the problem! The mental model of floats which I use is 1) floats are not numbers, they are buckets, and 2) as you get further away from zero, the buckets get bigger. So let’s say you are calculating the sum, and it is already at 1 billion, and the next purchase is $3.57. You take 1 billion, you add 3.57 to it, and you get... 1 billion. And this happens for all of the rest of the purchases as well. Remember: 1 billion is not a number, it is a bucket, and it turns out that when you are that far away from zero, the size of the bucket is 64. So 3.57 is simply not big enough to reach the next bucket.
    - RhysU 5y ago
      Well explained! All of the later contributions to the sum are effectively ignored or their contributions severely damaged in 32-bit because the "buckets" are big. It was precisely this problem. The individual had done all data preparation/normalization in 32-bit because the model training used 32-bit on the GPU. It's a very reasonable mistake if one hasn't been exposed to floating point woes. I was pleased to see that the individual ultimately caught it when observing that 2 libraries disagreed about the mean. Computing a 64-bit mean was enough. Compensated (i.e. Kahan) summation would have worked too.
    - bqmjjx0kac 5y ago
      Thanks for the explanation!
user3939382 5y ago
There are libraries that offer more appropriate ways of dealing with it, but last time I ran into a FP-related bug (something to do with parsing xlsx into MySQL) I fixed it quickly by converting everything to strings and doing some unholy procedure on them. It worked but it wasn’t my proudest moment as a programmer.
- tomrod 5y ago
  I wish to learn a better way. FP is sure to byte again and again.
  - marcosdumay 5y ago
    Well, manually floating the point in a string is sure to bite again and again too, but way more frequently than in binary. There is actually no better way, if you try to calculate over the reals (with computers or whatever you want), you are prone to be bitten. Once in a while there's an article about intervalar algebra on HN, those are a great opportunity to just nod positively and remember all of the flaws of intervalar algebra I got to learn on my school's physics labs. (And yeah, those flaws do fit some problems better than FP, but not all.)
    - rini17 5y ago
      Pity the rational numbers (fractions) did not catch on. Of course it has flaws too but bit easier to grasp. And it handles important cases like 1/3 or 1/10 exactly.
pvg 5y ago
I consider the domain sophisticated enough to be an independent skill It's been a whole field with its own patron saint for a quite a while, take a look at https://en.wikipedia.org/wiki/William_Kahan https://en.wikipedia.org/wiki/William_Kahan
exporectomy 5y ago
As long as you're using it to represent what could be physical measurements of real-valued quantities, it's nearly impossible to go wrong. Problems happen when you want stupendous precision or human readability. Numerically unstable algorithms are a problem too but again, intuitively so if you think of the numbers as physical measurements.
- RhysU 5y ago
  > it's nearly impossible to go wrong It's a matter of time if one doesn't know to look for numerically stable algorithms. Or if one thinks performance merits dropping stability. https://github.com/RhysU/ar/issues/3 https://github.com/RhysU/ar/issues/3 was an old saga in that vein.
- seoaeu 5y ago
  Notably, this is only true of 64-bit floats. Sticking to 32-bit floats saves memory and sometimes are faster to compute with, but you can absolutely run into precision problems with them. When tracking time, you'll only have millisecond precision for under 5 hours. When representing spacial coordinates, positions on the Earth will only be precise to a handful of meters.
- opheliate 5y ago
  So you have problems if you want a precise answer, you want to display your answer, or if you want to use any of a large number of useful algorithms? That sounds like it’s quite easy to go wrong.
  - exporectomy 5y ago
    You can't want a precise answer from physical measurements unless you don't know how to measure things. Display should be done with libraries, and numerical instability makes algorithms basically useless, so you pretty much have to be inventing it yourself.
- necheffa 5y ago
  I do a lot of floating point math at work and constantly run into problems either from someone else's misunderstanding, my own misunderstanding, or we just moved to a new microarchitecture and CPU dispatch hits a little different manifesting itself as rounding error to write off (public safety industry).

RhysU 5y ago

Just this week I watched someone discover that computing summary statistics in 32-bit on a large dataset is a bad idea. The computer science curricula needs to incorporate more computational science. It's a shame to charge someone tens of thousands of USD and to not warn them that floating point has some obvious footcanons.

bigiain 5y ago
> Just this week I watched someone discover that computing summary statistics in 32-bit on a large dataset is a bad idea. The computer science curricula needs to incorporate more computational science. Sadly, I suspect too many "computer science" courses have turned into "vocational coding" courses, and now those people are computing summary statistics on large datasets in Javascript...
bqmjjx0kac 5y ago
Could you shed some light on what they did wrong, and what would be a better way to do it?
- cellularmitosis 5y ago
  not OP, but the hint is in “computing summary statistics in 32-bit on a large dataset”. A large dataset means lots of values, maybe we can assume the number of values is way bigger than any individual value. Perhaps think of McDonalds purchases nation-wide: billions of values but each value is probably less than $10. The simplest summary statistic would be a grand total (sum). If you have a good mental model of floats, you immediately see the problem! The mental model of floats which I use is 1) floats are not numbers, they are buckets, and 2) as you get further away from zero, the buckets get bigger. So let’s say you are calculating the sum, and it is already at 1 billion, and the next purchase is $3.57. You take 1 billion, you add 3.57 to it, and you get... 1 billion. And this happens for all of the rest of the purchases as well. Remember: 1 billion is not a number, it is a bucket, and it turns out that when you are that far away from zero, the size of the bucket is 64. So 3.57 is simply not big enough to reach the next bucket.
  - RhysU 5y ago
    Well explained! All of the later contributions to the sum are effectively ignored or their contributions severely damaged in 32-bit because the "buckets" are big. It was precisely this problem. The individual had done all data preparation/normalization in 32-bit because the model training used 32-bit on the GPU. It's a very reasonable mistake if one hasn't been exposed to floating point woes. I was pleased to see that the individual ultimately caught it when observing that 2 libraries disagreed about the mean. Computing a 64-bit mean was enough. Compensated (i.e. Kahan) summation would have worked too.
  - bqmjjx0kac 5y ago
    Thanks for the explanation!

user3939382 5y ago

There are libraries that offer more appropriate ways of dealing with it, but last time I ran into a FP-related bug (something to do with parsing xlsx into MySQL) I fixed it quickly by converting everything to strings and doing some unholy procedure on them. It worked but it wasn’t my proudest moment as a programmer.

tomrod 5y ago
I wish to learn a better way. FP is sure to byte again and again.
- marcosdumay 5y ago
  Well, manually floating the point in a string is sure to bite again and again too, but way more frequently than in binary. There is actually no better way, if you try to calculate over the reals (with computers or whatever you want), you are prone to be bitten. Once in a while there's an article about intervalar algebra on HN, those are a great opportunity to just nod positively and remember all of the flaws of intervalar algebra I got to learn on my school's physics labs. (And yeah, those flaws do fit some problems better than FP, but not all.)
  - rini17 5y ago
    Pity the rational numbers (fractions) did not catch on. Of course it has flaws too but bit easier to grasp. And it handles important cases like 1/3 or 1/10 exactly.

pvg 5y ago

I consider the domain sophisticated enough to be an independent skill It's been a whole field with its own patron saint for a quite a while, take a look at https://en.wikipedia.org/wiki/William_Kahan https://en.wikipedia.org/wiki/William_Kahan

exporectomy 5y ago

As long as you're using it to represent what could be physical measurements of real-valued quantities, it's nearly impossible to go wrong. Problems happen when you want stupendous precision or human readability. Numerically unstable algorithms are a problem too but again, intuitively so if you think of the numbers as physical measurements.

RhysU 5y ago
> it's nearly impossible to go wrong It's a matter of time if one doesn't know to look for numerically stable algorithms. Or if one thinks performance merits dropping stability. https://github.com/RhysU/ar/issues/3 https://github.com/RhysU/ar/issues/3 was an old saga in that vein.
seoaeu 5y ago
Notably, this is only true of 64-bit floats. Sticking to 32-bit floats saves memory and sometimes are faster to compute with, but you can absolutely run into precision problems with them. When tracking time, you'll only have millisecond precision for under 5 hours. When representing spacial coordinates, positions on the Earth will only be precise to a handful of meters.
opheliate 5y ago
So you have problems if you want a precise answer, you want to display your answer, or if you want to use any of a large number of useful algorithms? That sounds like it’s quite easy to go wrong.
- exporectomy 5y ago
  You can't want a precise answer from physical measurements unless you don't know how to measure things. Display should be done with libraries, and numerical instability makes algorithms basically useless, so you pretty much have to be inventing it yourself.
necheffa 5y ago
I do a lot of floating point math at work and constantly run into problems either from someone else's misunderstanding, my own misunderstanding, or we just moved to a new microarchitecture and CPU dispatch hits a little different manifesting itself as rounding error to write off (public safety industry).