4 ms·

The part that was most surprising to me was how much the performance of serializing floating-point numbers has improved, even just in the past decade [1]. [1]

The part that was most surprising to me was how much the performance of serializing floating-point numbers has improved, even just in the past decade [1]. [1] https://github.com/jk-jeon/dragonbox?tab=readme-ov-file#performance https://github.com/jk-jeon/dragonbox?tab=readme-ov-file#perf...

jameshart 1y ago
Roundtripping IEEE floating point values via conversion to decimal UTF-8 strings and back is a ridiculously fragile process, too, not just slow. The difference between which values are precisely representable in binary and which are precisely representable in decimal means small errors can creep in.
- gugagore 1y ago
  You don't have to precisely represent the float in decimal. You just have to have each float have a unique decimal representation, which you can guarantee if you include enough digits: 9 for 32-bit floats, and 17 for 64-bit floats. https://randomascii.wordpress.com/2012/02/11/they-sure-look-equal/ https://randomascii.wordpress.com/2012/02/11/they-sure-look-...
  - jameshart 1y ago
    And you need to trust that whoever is generating the JSON you’re consuming, or will consume the JSON you generate, is using a library which agrees about what those representations round to.
    - jk-jeon 1y ago
      Note that the consumer side doesn't really have a lot of ambiguity. You just read the number, compute its precise value as written, round it to the closest binary representation with banker's rounding. You do anything other than this only under very special circumstances. Virtually all ambiguity lies on the producer side, which can be cleared out by using any of the formatting algorithms with the roundtripping-guarantee. EDIT: If you're talking about decimal->binary->decimal round-tripping, it's a completely different story though.
    - kccqzy 1y ago
      JSON itself doesn't mandate that IEEE754 numbers be used.
      - jameshart 1y ago
        Indeed - you could be serializing to or from JSON where the in-memory representation you're aiming for is actually a floating point decimal. JSON doesn't care.
      - Waterluvian 1y ago
        This is one of those really common misunderstandings in my experience. Indeed JSON doesn’t encode any specific precision at all. It’s just a decimal number of any length you possibly want, knowing that parsing libraries will likely decode it into something like IEEE754. This is why libraries like Python’s json will let you give it a custom parser, if, say, you wanted a Decimal object for numbers.
      - MobiusHorizons 1y ago
        Like it or not, but json data types are inherently linked to the primatives available in JavaScript. You can, of course write JSON that can’t be handled with the native types available in JavaScript, but the native parser will always deserialize to a native type. Until very recently all numbers were iee754 doubles in JavaScript, although arbitrary precision bignums do exist now. So the defacto precision limit of a number in JSON that needs to be compatible is an IEE754. If you control your clients you can do whatever you want though.
        jameshart 1y ago
        The standard definitely limits what precision you should expect to be handled. But how JSON numbers are handled by different parsers might surprise you. This blog post actually does a good job of detailing the subtleties and the choices made in a few standard languages and libraries: https://github.com/bterlson/blog/blob/main/content/blog/what-is-a-json-number.md https://github.com/bterlson/blog/blob/main/content/blog/what... I think one particular surprise is that C# and Java standard parsers both use openAPI schema hints that a piece of data is of type ‘number’ to map the value to a decimal floating point type, not a binary one.
- jk-jeon 1y ago
  A way to achieve perfect round-tripping was proposed back in 1990, by Steele and White (and likely they are not the first ones who came up with a similar idea). I guess their proposal probably wasn't extremely popular at least until 2000's, compared to more classical `printf`-like rounding methods, but it seems many languages and platforms these days do provide such round-tripping formatting algorithms as the default option. So, I guess nowadays roundtripping isn't that hard, unless people do something sophisticated without really understanding what they're doing.
  - lifthrasiir 1y ago
    I do think the OP was worrying about such people. Now a performant and correctly rounded JSON library is reasonably common, but it was not the case a decade ago (I think).
  - kccqzy 1y ago
    Interesting! I didn't know about Steele and White's 1990 method. I did however remember the Burger and Dybvig's method from 1996.
  - ape4 1y ago
    "How to print floating-point numbers accurately" by Steele & White https://dl.acm.org/doi/10.1145/93548.93559 https://dl.acm.org/doi/10.1145/93548.93559
- kccqzy 1y ago
  Most languages in use (such as Python) have solved this problem ages ago. Take any floating point value other than NaN, convert it to string and convert the string back. It will compare exactly equal. Not only that, they are able to produce the shortest string representation.
  - jameshart 1y ago
    Maybe 'ridiculously fragile' is the wrong word. Perhaps 'needlessly' fragile would be better. The point is that it takes application of algorithms that need to be provably correctly implemented on both ends of any JSON serialization/deserialization. And if one implantation can roundtrip its own floating point values, that's great - but JSON is an interop format, so does it roundtrip if you send it to another system and back? It's just an unnecessary layer of complexity that binary floating point serializers do not have to worry about.
  - aardvark179 1y ago
    True, but many of them have had bugs in printing or parsing such numbers, and once those creep in they can cause real long term problems. I remember having to maintain alternative datums and projections in GIS software because of a parser error that had been introduced in the late 80s.

jameshart 1y ago

Roundtripping IEEE floating point values via conversion to decimal UTF-8 strings and back is a ridiculously fragile process, too, not just slow. The difference between which values are precisely representable in binary and which are precisely representable in decimal means small errors can creep in.

gugagore 1y ago
You don't have to precisely represent the float in decimal. You just have to have each float have a unique decimal representation, which you can guarantee if you include enough digits: 9 for 32-bit floats, and 17 for 64-bit floats. https://randomascii.wordpress.com/2012/02/11/they-sure-look-equal/ https://randomascii.wordpress.com/2012/02/11/they-sure-look-...
- jameshart 1y ago
  And you need to trust that whoever is generating the JSON you’re consuming, or will consume the JSON you generate, is using a library which agrees about what those representations round to.
  - jk-jeon 1y ago
    Note that the consumer side doesn't really have a lot of ambiguity. You just read the number, compute its precise value as written, round it to the closest binary representation with banker's rounding. You do anything other than this only under very special circumstances. Virtually all ambiguity lies on the producer side, which can be cleared out by using any of the formatting algorithms with the roundtripping-guarantee. EDIT: If you're talking about decimal->binary->decimal round-tripping, it's a completely different story though.
  - kccqzy 1y ago
    JSON itself doesn't mandate that IEEE754 numbers be used.
    - jameshart 1y ago
      Indeed - you could be serializing to or from JSON where the in-memory representation you're aiming for is actually a floating point decimal. JSON doesn't care.
    - Waterluvian 1y ago
      This is one of those really common misunderstandings in my experience. Indeed JSON doesn’t encode any specific precision at all. It’s just a decimal number of any length you possibly want, knowing that parsing libraries will likely decode it into something like IEEE754. This is why libraries like Python’s json will let you give it a custom parser, if, say, you wanted a Decimal object for numbers.
    - MobiusHorizons 1y ago
      Like it or not, but json data types are inherently linked to the primatives available in JavaScript. You can, of course write JSON that can’t be handled with the native types available in JavaScript, but the native parser will always deserialize to a native type. Until very recently all numbers were iee754 doubles in JavaScript, although arbitrary precision bignums do exist now. So the defacto precision limit of a number in JSON that needs to be compatible is an IEE754. If you control your clients you can do whatever you want though.
      - jameshart 1y ago
        The standard definitely limits what precision you should expect to be handled. But how JSON numbers are handled by different parsers might surprise you. This blog post actually does a good job of detailing the subtleties and the choices made in a few standard languages and libraries: https://github.com/bterlson/blog/blob/main/content/blog/what-is-a-json-number.md https://github.com/bterlson/blog/blob/main/content/blog/what... I think one particular surprise is that C# and Java standard parsers both use openAPI schema hints that a piece of data is of type ‘number’ to map the value to a decimal floating point type, not a binary one.
jk-jeon 1y ago
A way to achieve perfect round-tripping was proposed back in 1990, by Steele and White (and likely they are not the first ones who came up with a similar idea). I guess their proposal probably wasn't extremely popular at least until 2000's, compared to more classical `printf`-like rounding methods, but it seems many languages and platforms these days do provide such round-tripping formatting algorithms as the default option. So, I guess nowadays roundtripping isn't that hard, unless people do something sophisticated without really understanding what they're doing.
- lifthrasiir 1y ago
  I do think the OP was worrying about such people. Now a performant and correctly rounded JSON library is reasonably common, but it was not the case a decade ago (I think).
- kccqzy 1y ago
  Interesting! I didn't know about Steele and White's 1990 method. I did however remember the Burger and Dybvig's method from 1996.
- ape4 1y ago
  "How to print floating-point numbers accurately" by Steele & White https://dl.acm.org/doi/10.1145/93548.93559 https://dl.acm.org/doi/10.1145/93548.93559
kccqzy 1y ago
Most languages in use (such as Python) have solved this problem ages ago. Take any floating point value other than NaN, convert it to string and convert the string back. It will compare exactly equal. Not only that, they are able to produce the shortest string representation.
- jameshart 1y ago
  Maybe 'ridiculously fragile' is the wrong word. Perhaps 'needlessly' fragile would be better. The point is that it takes application of algorithms that need to be provably correctly implemented on both ends of any JSON serialization/deserialization. And if one implantation can roundtrip its own floating point values, that's great - but JSON is an interop format, so does it roundtrip if you send it to another system and back? It's just an unnecessary layer of complexity that binary floating point serializers do not have to worry about.
- aardvark179 1y ago
  True, but many of them have had bugs in printing or parsing such numbers, and once those creep in they can cause real long term problems. I remember having to maintain alternative datums and projections in GIS software because of a parser error that had been introduced in the late 80s.