4 ms·

> The idea here is to have all numbers be treated as reals (in the mathematical sense) No. You can't represent, compare etc. most real numbers, so they aren't

> The idea here is to have all numbers be treated as reals (in the mathematical sense) No. You can't represent, compare etc. most real numbers, so they aren't any good for computational purposes. Practically usable number types include integers and things that have sufficiently number-like behaviour (e.g. fields) and can be represented with a finite number of integers: rational numbers, rational numbers with field extensions, vector spaces, finite sets, intervals, etc. In a language for grammars you can probably stop at integers; general purpose programming languages might want more (for instance, Python has rational numbers) but advanced number types can usually be implemented with libraries.

kstenerud 4y ago
I think maybe I'm not understanding you correctly... In most cases for binary formats an integer type will be expected for 99% of cases. Something like `uint(8,a/b)` will only generate an alternates set within the realm of integers, so a rule containing such an expression as this could only match when this particular byte matches a/b, which can only happen when a/b results in an integer from 0 to 255. For all a/b where this isn't true, the rule will not match. But support for matching float encodings is still needed, such as for `reading = float(32,0~1);`, which would happily match an ieee754 32-bit binary float approximation of 0.3 or 0.8 or anything from 0 to 1 that can be represented in this float format.
- HelloNurse 4y ago
  You are talking about the limited precision numbers in data, not about appropriate numbers for use in your language to express counts, repetitions, aggregate sizes etc. They are two related but categorically different kinds of entity, like screws and screwdrivers or sheet music and audio files. Grammar rules aren't supposed to work "for 99% of cases" depending on complex invisible constraints such as whether finite precision causes floating point arithmetic to depart from exact integer arithmetic.
  - kstenerud 4y ago
    I'm not saying that grammar rules are supposed to work for only 99% of cases; that comment was referring to the fact that most binary data of interest for binding (counts etc) will be stored in the document as integers. But I'm still not understanding what you're saying. All counts, repetitions, aggregate sizes etc are going to come from limited precision data in the document being examined (except in cases where you're passing in a numeric literal). In some cases these values will be used directly to produce expressions via functions and repetition, and in others they will be used in calculations that are then passed to functions or repetition to produce expressions. The grammar rules only match expressions, which are bit patterns, not numbers. Some of those bit patterns are arrived at through functions such as `float()`, taking a number range to give an expression composed of a set of possible (ieee754 binary) bit patterns. The bit patterns that the `float(32,1~10)` expression will match are far more numerous than the bit patterns that the `uint(32,1~10)` expression will match (0x00000001, 0x00000002, ... 0x0000000a), even though they both are taking in the same numeric range (each picks out only the values it can represent as discrete bit patterns from the range). `uint(32,1.5)` is a malformed grammar because it violates the invariant of `uint()`, whereas `uint(32,2)` is fine (giving an expression of the big endian bit pattern 00000000000000000000000000000010).
    - HelloNurse 4y ago
      The problem is slightly subtler and, I need to reiterate, at the grammar level. Suppose you define some constants a, b, c, etc. with integer values. First of all, it isn't obvious that these constants do not overflow the mantissa size of whatever floating point type you process them as; but we can generously assume that they are representable exactly. Then you use these fake integers to compute something that should be an actual integer, for example the number of occurrences of something as (a*b)/c in one place and (a/c)*b in another place (probably obfuscated by layers of variables and very simple computations that would be harmless with exact arithmetic), and the two values might differ (and they might be both wrong). So without even getting into the bit patterns of valid "example" values you are unable to specify your grammar reliably. Floating point arithmetic in the grammar is an unnecessary nightmare that you are inflicting to your users.

kstenerud 4y ago

I think maybe I'm not understanding you correctly... In most cases for binary formats an integer type will be expected for 99% of cases. Something like `uint(8,a/b)` will only generate an alternates set within the realm of integers, so a rule containing such an expression as this could only match when this particular byte matches a/b, which can only happen when a/b results in an integer from 0 to 255. For all a/b where this isn't true, the rule will not match. But support for matching float encodings is still needed, such as for `reading = float(32,0~1);`, which would happily match an ieee754 32-bit binary float approximation of 0.3 or 0.8 or anything from 0 to 1 that can be represented in this float format.

HelloNurse 4y ago
You are talking about the limited precision numbers in data, not about appropriate numbers for use in your language to express counts, repetitions, aggregate sizes etc. They are two related but categorically different kinds of entity, like screws and screwdrivers or sheet music and audio files. Grammar rules aren't supposed to work "for 99% of cases" depending on complex invisible constraints such as whether finite precision causes floating point arithmetic to depart from exact integer arithmetic.
- kstenerud 4y ago
  I'm not saying that grammar rules are supposed to work for only 99% of cases; that comment was referring to the fact that most binary data of interest for binding (counts etc) will be stored in the document as integers. But I'm still not understanding what you're saying. All counts, repetitions, aggregate sizes etc are going to come from limited precision data in the document being examined (except in cases where you're passing in a numeric literal). In some cases these values will be used directly to produce expressions via functions and repetition, and in others they will be used in calculations that are then passed to functions or repetition to produce expressions. The grammar rules only match expressions, which are bit patterns, not numbers. Some of those bit patterns are arrived at through functions such as `float()`, taking a number range to give an expression composed of a set of possible (ieee754 binary) bit patterns. The bit patterns that the `float(32,1~10)` expression will match are far more numerous than the bit patterns that the `uint(32,1~10)` expression will match (0x00000001, 0x00000002, ... 0x0000000a), even though they both are taking in the same numeric range (each picks out only the values it can represent as discrete bit patterns from the range). `uint(32,1.5)` is a malformed grammar because it violates the invariant of `uint()`, whereas `uint(32,2)` is fine (giving an expression of the big endian bit pattern 00000000000000000000000000000010).
  - HelloNurse 4y ago
    The problem is slightly subtler and, I need to reiterate, at the grammar level. Suppose you define some constants a, b, c, etc. with integer values. First of all, it isn't obvious that these constants do not overflow the mantissa size of whatever floating point type you process them as; but we can generously assume that they are representable exactly. Then you use these fake integers to compute something that should be an actual integer, for example the number of occurrences of something as (a*b)/c in one place and (a/c)*b in another place (probably obfuscated by layers of variables and very simple computations that would be harmless with exact arithmetic), and the two values might differ (and they might be both wrong). So without even getting into the bit patterns of valid "example" values you are unable to specify your grammar reliably. Floating point arithmetic in the grammar is an unnecessary nightmare that you are inflicting to your users.