4 ms·

One of the many problems you get from assuming that floating point "numbers" are a field (or ring for ints), when they aren't even associative, let alone commut

by mjhay 4y ago

One of the many problems you get from assuming that floating point "numbers" are a field (or ring for ints), when they aren't even associative, let alone commutative. I hope in 100 years we have a practical representation for the reals without all these slings and arrows. Floats are more-or-less the best minimal message length encoding if you have no prior information, but I think there could be better encodings for most purposes, especially to prohibit many cases of numerical issues.

dahart 4y ago
Interesting thought! Does a rational representation do any better by field standards? https://iquilezles.org/articles/floatingbar/ https://iquilezles.org/articles/floatingbar/ What would it take to have a practical representation that’s better? I’m not well versed in the fields and rings theories in math, but it seems like the fundamental problem here is that floats have a finite number of bits, which means they are forced to round the results. That rounding is what causes a very large portion of all the trouble, it is what breaks associativity & commutativity. Since there is no practical way to have an unlimited number of bits, I wonder if a practical field of floats cannot exist in general?
- mjhay 4y ago
  I know rational representations (with bignums) can work better for some purposes (albeit not high-perf numerical ones), but the rub is, like you say, you're trying to represent an uncountably infinite set with finite memory, so everything is going to be some kind of space/time/fidelity tradeoff. The only concrete finite fields are the finite fields of size p^k, where p is prime. I'm kind of thinking that there must be some representations that at least can shove off the breaking of associativity, commutativity, etc more into the background. It's probably more of a computer science and usability problem, but hopefully backed up by enough theory to provide some guarantees. You can make a "free" field as a programming interface (like in the OP) - and the floats would then be a leaky implementation of that interface. I think the question is how to make an implementation that better captures what we talk about when we talk about real numbers, while having good enough performance for big numerical tasks, and avoiding common numerical problems. Definitely not easy, and I think application dependent to a great extent - a place where one really has to be able to glide up and down levels of abstraction.
  - bodhiandphysics 4y ago
    The thing about floating point numbers is that while they're not associative... they are approximately associative (mod inf and nan). In other words (ab)c = a(bc) + e where e is is a number much smaller than max(a,b,c). This means that we can measure quite easily how bad an operation is in floating points, using what's called the condition number, and there are many many good ways of either estimating or bounding the condition number. So floating point numbers are easy to analyze, though they require a bit more machinery than most programmers get in intro to computer science. Really floats are the right way to do calculations involving reals on the computer. Of course, if you're not actually using reals, you shouldn't use them!
    - mjhay 4y ago
      Yeah the condition number helps in simple cases (or when solving single linear systems), but things rapidly spiral once you start getting into solving stiff ODEs, etc - to the point that only numerical specialists can judge it. I feel like there ought to be ways of dealing with these issues in a more fine-grained manner with better guardrails - something like a numeric-aware typesystem or automatic algorithmic scheme generator. Like I said though, maybe in 100 years.
      - bodhiandphysics 4y ago
        stiffness has nothing to do with fp numbers... stiff equations are just as hard to solve using arbitrary precision bigints since the issue is the limits of approximations of the integral.
        mjhay 4y ago
        That's getting back to my point though - the integral is over measurable sets of reals, and we are approximating that with finite-difference floats. This is a complex problem with many guises.
  - deleted 4y ago
    [deleted]
fatneckbeard 4y ago
there is no practical representation of the reals because their theory is based on infinite numbers and computers are finite. aka you are never going to store sqrt 2 in a register. the closest you can get is symbolic algebra packages. but at the end you still have to come down from infinite land and actually have an ASCII decimal representation because you cannot cut a board at sqrt2 feet long or fill your car with sqrt2 gallons of gasoline. or pay pi dollars for a pizza and expect your bank to keep a list of all the different transcendental atomic values you have used for payments over the years. "Your account has 17 pi plus 45 sqrt2 plus 97 e minus the golden ratio" is never going to work very well. one thing that i think would help is to put non-real mathematics into the mathematics curriculum and stop handwaving about how a computer does real numbers.. (it doesnt). teach discrete math, modular arithmetic, prime number theory, at the same time as pre-calculus.

by mjhay 4y ago

dahart 4y ago

Interesting thought! Does a rational representation do any better by field standards? https://iquilezles.org/articles/floatingbar/ https://iquilezles.org/articles/floatingbar/ What would it take to have a practical representation that’s better? I’m not well versed in the fields and rings theories in math, but it seems like the fundamental problem here is that floats have a finite number of bits, which means they are forced to round the results. That rounding is what causes a very large portion of all the trouble, it is what breaks associativity & commutativity. Since there is no practical way to have an unlimited number of bits, I wonder if a practical field of floats cannot exist in general?

mjhay 4y ago
I know rational representations (with bignums) can work better for some purposes (albeit not high-perf numerical ones), but the rub is, like you say, you're trying to represent an uncountably infinite set with finite memory, so everything is going to be some kind of space/time/fidelity tradeoff. The only concrete finite fields are the finite fields of size p^k, where p is prime. I'm kind of thinking that there must be some representations that at least can shove off the breaking of associativity, commutativity, etc more into the background. It's probably more of a computer science and usability problem, but hopefully backed up by enough theory to provide some guarantees. You can make a "free" field as a programming interface (like in the OP) - and the floats would then be a leaky implementation of that interface. I think the question is how to make an implementation that better captures what we talk about when we talk about real numbers, while having good enough performance for big numerical tasks, and avoiding common numerical problems. Definitely not easy, and I think application dependent to a great extent - a place where one really has to be able to glide up and down levels of abstraction.
- bodhiandphysics 4y ago
  The thing about floating point numbers is that while they're not associative... they are approximately associative (mod inf and nan). In other words (ab)c = a(bc) + e where e is is a number much smaller than max(a,b,c). This means that we can measure quite easily how bad an operation is in floating points, using what's called the condition number, and there are many many good ways of either estimating or bounding the condition number. So floating point numbers are easy to analyze, though they require a bit more machinery than most programmers get in intro to computer science. Really floats are the right way to do calculations involving reals on the computer. Of course, if you're not actually using reals, you shouldn't use them!
  - mjhay 4y ago
    Yeah the condition number helps in simple cases (or when solving single linear systems), but things rapidly spiral once you start getting into solving stiff ODEs, etc - to the point that only numerical specialists can judge it. I feel like there ought to be ways of dealing with these issues in a more fine-grained manner with better guardrails - something like a numeric-aware typesystem or automatic algorithmic scheme generator. Like I said though, maybe in 100 years.
    - bodhiandphysics 4y ago
      stiffness has nothing to do with fp numbers... stiff equations are just as hard to solve using arbitrary precision bigints since the issue is the limits of approximations of the integral.
      - mjhay 4y ago
        That's getting back to my point though - the integral is over measurable sets of reals, and we are approximating that with finite-difference floats. This is a complex problem with many guises.
- deleted 4y ago
  [deleted]

fatneckbeard 4y ago

there is no practical representation of the reals because their theory is based on infinite numbers and computers are finite. aka you are never going to store sqrt 2 in a register. the closest you can get is symbolic algebra packages. but at the end you still have to come down from infinite land and actually have an ASCII decimal representation because you cannot cut a board at sqrt2 feet long or fill your car with sqrt2 gallons of gasoline. or pay pi dollars for a pizza and expect your bank to keep a list of all the different transcendental atomic values you have used for payments over the years. "Your account has 17 pi plus 45 sqrt2 plus 97 e minus the golden ratio" is never going to work very well. one thing that i think would help is to put non-real mathematics into the mathematics curriculum and stop handwaving about how a computer does real numbers.. (it doesnt). teach discrete math, modular arithmetic, prime number theory, at the same time as pre-calculus.