5 ms·

Lots of the math in games and ML assumes you’re operating in the reals. For example, the optimization in neural networks. Integers and fixed point won’t work.

• bwestergard 6y ago
  Floats have a totally different algebra than the reals, though.
  • adwn 6y ago
    Only in the strictly mathematical sense. To a first approximation, floating-point numbers behave like real numbers, and that's good enough for many use-cases (not all, though). Similarly, fixed-width integers have a totally different algebra than true integers, and yet they're immensely useful.
    • legulere 6y ago
      fixed-width integers have exactly the same semantics as true integers bar overflow, which is a very easy to understand concept. Floating-point numbers behave like a physicist would do calculations rounding at every step, but doing everything in binary. Or hexadecimal if you prefer, it's equivalent, but still difficult to get your head around to. Then there additionally is + and - 0 and + and - infinity, and several settings for how rounding is done exactly.
    • bwestergard 6y ago
      "To a first approximation, floating-point numbers behave like real numbers" This is incorrect. To count as an approximation, there have to be some accuracy bounds. This is impossible to define, as the reals don't have the same cardinality as any floating point number system. Now, for many interesting and useful real-valued functions, a float-valued function can be defined that is an approximation. But there is no general way to map a function defined on the reals to a function defined on the floats with a known accuracy.
  • oivey 6y ago
    Yeah, but closer to the reals than fixed point or integers. That’s kind of the point of the standard.
• sltkr 6y ago
  Doesn't ML work fine with fixed point numbers? I don't think there is a particular use case for very large or very small numbers.
  • sharpneli 6y ago
    It's the exact opposite. It really loves large and small numbers, the precision of a number doesn't matter that much That's the reason why they introduced https://en.wikipedia.org/wiki/Bfloat16_floating-point_format https://en.wikipedia.org/wiki/Bfloat16_floating-point_format It's similar to normal half precision float but the exponent is the same as with 32bit float and mantissa has been sacrificed to get room for it.
  • oivey 6y ago
    Referring to ML is so vague that it’s not particularly meaningful. The answer in general is no. There’s been some work showing neural networks can function with low precision floating point, but that’s still pretty different than fixed point.
  • PeterisP 6y ago
    ML optimization algorithms generally do require proper handling of very,very small numbers for e.g. calculation of gradients. There's a bunch of work on discretization of models after you've done optimizing them - that works, you can get high-performance inference with low-bit fixed point numbers; but that's after the learning has been done using proper floating point (however, you don't necessarily need high accuracy floating point, e.g. single precision may work betted than double precision simply because it's twice less bytes to copy over).