7 ms·

> So, yes, instead of saying that "e" equals "65537", you're saying that "e" equals "AQAB". Aren't you glad you did those extra steps? Oh JSON. For those unfa

> So, yes, instead of saying that "e" equals "65537", you're saying that "e" equals "AQAB". Aren't you glad you did those extra steps? Oh JSON. For those unfamiliar with the reason here, it’s that JSON parsers cannot be relied upon to treat numbers properly. Is 4723476276172647362476274672164762476438 a valid JSON number? Yes, of course it is. What will a JSON parser due with it? Silently truncate it to a 64-bit or 63-bit integer, or a float, probably or if you’re very lucky emit an error (a good JSON decoder written in a sane language like Common Lisp would of course just return the number, but few of us are so lucky). So the only way to reliably get large integers into and out of JSON is to encode them as something else. Base64-encoded big-endian bytes is not a terrible choice. Silently doing the wrong thing is the root of many security errors, so it not wrong to treat every number in the protocol this way. Of course, then one loses the readability of JSON. JSON is better than XML, but it really isn’t great. Canonical S-expressions would have been far preferable, but for whatever reason the world didn’t go that way.

drob518 1y ago
Seems like a large integer can always be communicated as a vector of byte values in some specific endian order, which is easier to deal with than Base64 since a JSON parser will at least convert the byte value from text to binary for you. But yea, as a Clojure guy sexprs or EDN would be much better.
matja 1y ago
Aren't JSON parsers technically not following the standard if they don't reliably store a number that is not representable by a IEEE754 double precision float? It's a shame JSON parsers usually default to performance rather than correctness, by using bignums for numbers.
- q3k 1y ago
  Have a read through RFC7159 or 8259 and despair. > This specification allows implementations to set limits on the range and precision of numbers accepted JSON is a terrible interoperability standard.
  - matja 1y ago
    So a JSON parser that cannot store a 2 is technically compliant? :(
    - q3k 1y ago
      Yep. Or one that parses it into a 7 :)
      - kevingadd 1y ago
        I once debugged a production issue that boiled down to "A PCI compliance .dll was messing with floating point flags, causing the number 4 to unserialize as 12"
        xeromal 1y ago
        That sounds awful. lol
      - chasd00 1y ago
        > Or one that parses it into a 7 :) if it's known and acceptable that LLMs can hallucinate arguments to an API then i don't see how this isn't perfectly acceptable behavior either.
    - reichstein 1y ago
      JSON is a text format. A parser must recognize the text `2` as a valid production of the JSON number grammar. Converting that text to _any_ kind of numerical value is outside the scope of the specification. (At least the JSON.org specification, the RFC tries to say more.) As a textural format, when you use it for data interchange between different platforms, you should ensure that the endpoints agree on the _interpretation_, otherwise they won't see the same data. Again outside of the scope of the JSON specification.
      - tsimionescu 1y ago
        A JSON parser has to check if a numeric value is actually numeric - the JSON {"a" : 123456789} is valid, but {"a" : 12345678f} is not. Per the RFC, a standards-compliant JSON parser can also refuse {"a": 123456789} if it considers the number is too large.
- kens 1y ago
  > Aren't JSON parsers technically not following the standard if they don't reliably store a number that is not representable by a IEEE754 double precision float? That sentence has four negations and I honestly can't figure out what it means.
  - School-Cotton 1y ago
    “The standard technically requires that JSON parsers reliably store numbers, even those that are not representable by an IEEE double”. (It seems this claim is not true, but at least that’s what the sentence means.)
  - alterom 1y ago
    >> Aren't JSON parsers technically not following the standard if they don't reliably store a number that is not representable by a IEEE754 double precision float? >That sentence has four negations and I honestly can't figure out what it means. This example is halfway as bad as the one Orwell gives in my favorite essay, "Politics the the English Language"¹. Compare and contrast: >I am not, indeed, sure whether it is not true to say that the Milton who once seemed not unlike a seventeenth-century Shelley had not become, out of an experience ever more bitter in each year, more alien (sic) to the founder of that Jesuit sect which nothing could induce him to tolerate. Orwell has much to say about either. _____ ¹https://www.orwellfoundation.com/the-orwell-foundation/orwell/essays-and-other-works/politics-and-the-english-language/ https://www.orwellfoundation.com/the-orwell-foundation/orwel...
    - NooneAtAll3 1y ago
      that Orwell quote can be saved a lot by proper punctuation I am not, indeed, sure*,* whether it is not true to say that the Milton *(*who once seemed not unlike a seventeenth-century Shelley*)* had not become *-* out of an experience *-* ever more bitter in each year, more alien (sic) to the founder of that Jesuit sect*,* which nothing could induce him to tolerate.
      - alterom 1y ago
em-bee 1y ago
as someone who started the s-expression task on rosettacode.org, i approve. if you need an s-expression parser for your language, look here https://rosettacode.miraheze.org/wiki/S-expressions https://rosettacode.miraheze.org/wiki/S-expressions (the canonical url is https://rosettacode.org/wiki/S-expressions https://rosettacode.org/wiki/S-expressions but they have DNS issues right now)
tempodox 1y ago
“Worse is better” is still having ravaging success.
marcosdumay 1y ago
What I don't understand is why you (and a lot of other people) just expect S-expression parsers to not have the exact same problems.
- 01HNNWZ0MV43FF 1y ago
  I think they mean that Common Lisp has bigints by default
  - ryukafalz 1y ago
    As do Scheme and most other Lisps I'm familiar with, and integers/floats are typically specified to be distinct. I think we'd all be better off if that were true of JSON as well. I'd be happy to use s-expressions instead :) Though to GP's point, I suppose we might then end up with JS s-expression parsers that still treat ints and floats interchangeably.
    - petre 1y ago
      And in addition to that are unable to distingush between a string "42" and a number 42.
- deleted 1y ago
  [deleted]
- eadmund 1y ago
  Because canonical S-expressions don’t have numbers, just atoms (i.e., byte sequences) and lists. It is up to the using code to interpret "34" as the string "34" or the number 34 or the number 13,108 or the number 13,363, which is part of the protocol being used. In most instances, the byte sequence is probably a decimal number. Now, S-expressions as used for programming languages such as Lisp do have numbers, but again Lisp has bignums. As for parsers of Lisp S-expressions written in other languages: if they want to comply with the standard, they need to support bignums.
  - its-summertime 1y ago
ncruces 1y ago
Go can decode numbers losslessly as strings: https://pkg.go.dev/encoding/json#Number https://pkg.go.dev/encoding/json#Number json.Number is (almost) my “favorite” arbitrary decimal: https://github.com/ncruces/decimal?tab=readme-ov-file#decimal-arithmetic https://github.com/ncruces/decimal?tab=readme-ov-file#decima... I'm half joking, but I'm not sure why S-expressions would be better here. There are LISPs that don't do arbitrary precision math.
- mise_en_place 1y ago
  For actual SERDES, JSON becomes very brittle. It's better to use something like protobuf or cap'n'proto for such cases.
- eadmund 1y ago
  > Go can decode numbers losslessly as strings: https://pkg.go.dev/encoding/json#Number https://pkg.go.dev/encoding/json#Number Yup, and if you’re using JSON in Go you really do need to be using Number exclusively. Anything else will lead to pain. > I'm half joking, but I'm not sure why S-expressions would be better here. There are LISPs that don't do arbitrary precision math. Sure, but I’m referring specifically to https://www.ietf.org/archive/id/draft-rivest-sexp-13.html https://www.ietf.org/archive/id/draft-rivest-sexp-13.html, which only has lists and bytes, and so number are always just strings and it’s up to the program to interpret them.
kangalioo 1y ago
But what's wrong with sending the number as a string? `"65537"` instead of `"AQAB"`
- ayende 1y ago
  Too likely that this would not work because silent conversion to number along the way
  - iforgotpassword 1y ago
    Then just prefixing it with an underscore or any random letter would've been fine but of course base64 encoding the binary representation in base 64 makes you look so much smarter.
- comex 1y ago
  The question is how best to send the modulus, which is a much larger integer. For the reasons below, I'd argue that base64 is better. And if you're sending the modulus in base64, you may as well use the same approach for the exponent sent along with it. For RSA-4096, the modulus is 4096 bits = 512 bytes in binary, which (for my test key) is 684 characters in base64 or 1233 characters in decimal. So the base64 version is much smaller. Base64 is also more efficient to deal with. An RSA implementation will typically work with the numbers in binary form, so for the base64 encoding you just need to convert the bytes, which is a simple O(n) transformation. Converting the number between binary and decimal, on the other hand, is O(n^2) if done naively, or O(some complicated expression bigger than n log n) if done optimally. Besides computational complexity, there's also implementation complexity. Base conversion is an algorithm that you normally don't have to implement as part of an RSA implementation. You might argue that it's not hard to find some library to do base conversion for you. Some programming languages even have built-in bigint types. But you typically want to avoid using general-purpose bigint implementations for cryptography. You want to stick to cryptographic libraries, which typically aim to make all operations constant-time to avoid timing side channels. Indeed, the apparent ease-of-use of decimal would arguably be a bad thing since it would encourage implementors to just use a standard bigint type to carry the values around. You could argue that the same concern applies to base64, but it should be relatively safe to use a naive implementation of base64, since it's going to be a straightforward linear scan over the bytes with less room for timing side channels (though not none).
JackSlateur 1y ago
Is this ok ? Python 3.13.3 (main, May 21 2025, 07:49:52) [GCC 14.2.0] on linux Type "help", "copyright", "credits" or "license" for more information. >>> import json >>> json.loads('47234762761726473624762746721647624764380000000000000000000000000000000000000000000') 47234762761726473624762746721647624764380000000000000000000000000000000000000000000
- jazzyjackson 1y ago
  yes, python falls into the sane language category with arbitrary-precision arithmetic
  - faresahmed 1y ago
    Not so much, >>> s="1"+"0"*4300 >>> json.loads(s) ... ValueError: Exceeds the limit (4300 digits) for integer string conversion: value has 4301 digits; use sys.set_int_max_str_digits() to increase the limit This was done to prevent DoS attacks 3 years ago and have been backported to at least CPython 3.9 as it was considered a CVE. Relevant discussion: https://news.ycombinator.com/item?id=32753235 https://news.ycombinator.com/item?id=32753235 Your sibling comment suggests using decimal.Decimal which handles parsing >4300 digit numbers (by default).
    - lifthrasiir 1y ago
      This should be interpreted as a stop-gap measure before a subquadratic algorithm can be adopted. Take a look at _pylong.py in new enough CPython.
- teddyh 1y ago
  I prefer >> import json, decimal >> j = "47234762761726473624762746721647624764380000000000000000000000000000000000000000000" >> json.loads(j, parse_float=decimal.Decimal, parse_int=decimal.Decimal) Decimal('47234762761726473624762746721647624764380000000000000000000000000000000000000000000') This way you avoid this problem: >> import json >> j = "0.47234762761726473624762746721647624764380000000000000000000000000000000000000000000" >> json.loads(j) 0.47234762761726473 And instead can get: >> import json, decimal >> j = "0.47234762761726473624762746721647624764380000000000000000000000000000000000000000000" >> json.loads(j, parse_float=decimal.Decimal, parse_int=decimal.Decimal) Decimal('0.47234762761726473624762746721647624764380000000000000000000000000000000000000000000')
cortesoft 1y ago
> Canonical S-expressions would have been far preferable, but for whatever reason the world didn’t go that way. I feel like not understanding why JSON won out is being intentionally obtuse. JSON can easily be hand written, edited, and read for most data. Canonical S-expressions are not as easy to read and much harder to write by hand; having to prefix every atom with a length makes is very tedious to write by hand. If you have a JSON object you want to hand edit, you can just type... for an Canonical S-expression, you have to count how many characters you are typing/deleting, and then update the prefix. You might not think the ability to hand generate, read, and edit is important, but I am pretty sure that is a big reason JSON has won in the end. Oh, and the Ruby JSON parser handles that large number just fine.
- eadmund 1y ago
  > I feel like not understanding why JSON won out is being intentionally obtuse. I didn’t feel like my comment was the right place to shill for an alternative, but rather to complain about JSON. But since you raise it. > JSON can easily be hand written, edited, and read for most data. So can canonical S-expressions! > Canonical S-expressions are not as easy to read and much harder to write by hand; having to prefix every atom with a length makes is very tedious to write by hand. Which is why the advanced representation exists. I contend that this: (urn:ietf:params:acme:error:malformed (detail "Some of the identifiers requested were rejected") (subproblems ((urn:ietf:params:acme:error:malformed (detail "Invalid underscore in DNS name \"_example.org\"") (identifier (dns _example.org))) (urn:ietf:params:acme:error:rejectedIdentifier (detail "This CA will not issue for \"example.net\"") (identifier (dns example.net)))))) is far easier to read than this (the first JSON in RFC 8555): { "type": "urn:ietf:params:acme:error:malformed", "detail": "Some of the identifiers requested were rejected", "subproblems": [ { "type": "urn:ietf:params:acme:error:malformed", "detail": "Invalid underscore in DNS name \"_example.org\"", "identifier": { "type": "dns", "value": "_example.org" } }, { "type": "urn:ietf:params:acme:error:rejectedIdentifier", "detail": "This CA will not issue for \"example.net\"", "identifier": { "type": "dns", "value": "example.net" } } ] } > for an Canonical S-expression, you have to count how many characters you are typing/deleting, and then update the prefix. As you can see, no you do not.
TZubiri 1y ago
It feels like malpractice to use json in encryption
- red_admiral 1y ago
  Sadly JWT and friends are "standard". In theory the representation and the data are independent and you can marshal and unmarshal correctly. In practice, "alg:none" is a headache and everyone involved should be ashamed.
mindcrime 1y ago
JSON is better than XML, but it really isn’t great. JSON doesn't even support comments, c'mon. I mean, it's handy for some things, but I don't know if I'd say "JSON is better than XML" in any universal sense. I still go by the old saw "use the right tool for the job at hand". In some cases maybe it's JSON. In others XML. In others S-Exprs encoded in EBCDIC or something. Whatever works...
- deepsun 1y ago
  Yup, imagine if HTML was JSON-like, not XML-like.
supermatt 1y ago
As you said - it’s not really a problem with the JSON structure and format itself, but the underlying parser, which is specifically designed to map to the initial js types. There are parsers that don’t have this problem, but then the JSON itself is not portable. The problem with your solution is that it’s also not portable for the same reason (it’s not part of the standard), and the reason that it wasn’t done that way in the first place is because it wouldn’t map to those initial js types! FYI, you can easily work around this by using replacer and revivers that are part of the standards for stringify and parse and treat numbers differently. But again, the json isn’t portable to places without those replacer/revivers. I.e, the real problem is treating something that looks like json as json by using standards compliant json parsers - not the apparent structure of the format itself. You could fix this problem in an instant by calling it something other than JSON, but people will see it and still use a JSON parser because it looks like JSON, not because it is JSON.
- zelphirkalt 1y ago
  Isn't the actual problem that it is supposed to map to JS types, which are badly designed, and thus being infectious for other ecosystems, that don't have these defects?
tsimionescu 1y ago
This seems like a just-so story. Your explanation could make some sense if we were comparing {"e" : "AQAB"} to {"e" : 65537}, but there is no reason why that should be the alternative. The JSON {"e" : "65537"} will be read precisely the same way by any JSON parser out there. Converting the string "65537" to the number 65537 is exactly as easy (or hard), but certainly unambiguous, as converting the string "AQAB" to the same number. Of course, if you're doing this in JS and have reasons to think the resulting number may be larger than the precision of a double, you have a huge problem either way. Just as you would if you were writing this in C and thought the number may be larger than what can fit in a long long. But that's true regardless of how you represent it in JSON.
- deepsun 1y ago
  Some parsers, like PHP, may treat 65537 and "65537" the same. Room for vulnerability.
  - int_19h 1y ago
    Why would they do so? It's semantically distinct JSON, even JS itself treats it differently?
    - hiciu 1y ago
      It's PHP. Handling numbers in PHP is complicated enough that a reasonable person would not trust it by default. https://www.php.net/manual/en/language.types.numeric-strings.php https://www.php.net/manual/en/language.types.numeric-strings...
      - int_19h 1y ago
        I know that PHP will treat a string as if it were a number if you try to use it in a context where number is expected; JS does the same thing. But why would that affect JSON deserialization in a way that makes numbers and strings indistinguishable in principle (causing the loss of precision as described here)?
    - 1y ago
mnahkies 1y ago
I'm still haunted by a bug caused by the JSON serializer our C# apps were using emitting bigints as JSON numbers, only for the JavaScript consumers to mangle them silently. Kinda blows my mind that the accepted behavior is to just overflow and not raise an exception. I try to stick to strings for anything that's not a 32 bit int now.
fulafel 1y ago
Is the correct number implementation really the exception? The first 2 json decoders I just tried (Python & Clojure) worked correctly with that example.
rendaw 1y ago
Canonical S-expressions don't have an object/mapping type, which means you can't have generic tooling unambiguously perform certain common operations like data merges.
rr808 1y ago
> JSON is better than XML hard disagree on that one.
- motorest 1y ago
  [flagged]
  - rr808 1y ago
    You're welcome! I hope it was helpful.
zubspace 1y ago
Wouldn't it just solve a whole lot of problems if we could just add optional type declarations to json? It seems so simple and obvious that I'm kinda dumbfounded that this is not a thing yet. Most of the time you would not need it, but it would prevent the parser from making a wrong guess in all those edge cases. Probably there are types not every parser/language can accept, but at least it could throw a meaningful error instead of guessing or even truncating the value.
- ivanbakel 1y ago
  I doubt that would fix the issue. The real cause is that programmers mostly deal in fixed-size integers, and that’s how they think of integer values, since those are the concepts their languages provide. If you’re going to write a JSON library for your favourite programming language, you’re going to reach for whatever ints are the default, regardless of what the specs or type hints suggest. Haskell’s Aeson library is one of the few exceptions I’ve seen, since it only parses numbers to ‘Scientific’s (essentially a kind of bigint for rationals.) This makes the API very safe, but also incredibly annoying to use if you want to just munge some integers, since you’re forced to handle the error case of the unbounded values not fitting in your fixed-size integer values. Most programmers likely simply either don’t consider that case, or don’t want to have to deal with it, so bad JSON libraries are the default.
- movpasd 1y ago
  This is actually a deliberate design choice, which the breathtakingly short JSON standard explains quite well [0]. The designers deliberately didn't introduce any semantics and pushes all that to the implementors. I think this is a defensible design goal. If you introduce semantics, you're sure to annoy someone. There's an element of "worse is better" here [1]. JSON overtook XML exactly because it's so simple and solves for the social element of communication between disparate projects with wildly different philosophies, like UNIX byte-oriented I/O streams, or like the C calling conventions. --- [0] https://ecma-international.org/publications-and-standards/standards/ecma-404/ https://ecma-international.org/publications-and-standards/st... [1] https://en.wikipedia.org/wiki/Worse_is_better https://en.wikipedia.org/wiki/Worse_is_better
josephg 1y ago
The funny thing about this is that JavaScript the language has had support for BigIntegers for many years at this point. You can just write 123n for a bigint of 123. JSON could easily be extended to support them - but there’s no standards body with the authority to make a change like that. So we’re probably stuck with json as-is forever. I really hope something better comes along that we can all agree on before I die of old age. While we’re at it, I’d also love a way to embed binary data in json. And a canonical way to represent dates. And comments. And I’d like a sane, consistent way to express sum types. And sets and maps (with non string keys) - which JavaScript also natively supports. Sigh.
- aapoalas 1y ago
  It's more a problem of support and backwards compatibility. JSON and parsers for it are so ubiquitous, and the spec completely lacks any versioning support, that adding a feature would be a breaking change of horrible magnitude, on nearly all levels of the modern software infrastructure stack. I wouldn't be surprised if some CPUs might break from that :D JSON is a victim of its success: it has become too big to fail, and too big to improve.
- Sammi 1y ago
  There are easy workarounds to getting bigints in JSON: https://github.com/GoogleChromeLabs/jsbi/issues/30#issuecomment-1694399308 https://github.com/GoogleChromeLabs/jsbi/issues/30#issuecomm...
  - josephg 1y ago
    Sure; and I can encode maps and sets as entry lists. Binary data as strings and so on. But I don’t want to. I shouldn’t have to. The fact remains that json doesn’t have native support for any of this stuff. I want something json-like which supports all this stuff natively. I don’t want to have to figure out if some binary data is base64 encoded or hex encoded or whatever, and hack around jackson or serde or javascript to encode and decode my objects properly. Features like this should be built in.
    - 1y ago
ownedthx 1y ago
The numerical issues here are due to JavaScript, not JSON.
llm_nerd 1y ago
>JSON is better than XML JSON is still hack garbage compared to XML from the turn of the millennia. Like most dominant tech standards, JSON took hold purely because many developers are intellectually lazy and it was easier to slam some sloppy JSON together than to understand XML. XML with XSD, XPath and XQuery is simply a glorious combination.

drob518 1y ago

Seems like a large integer can always be communicated as a vector of byte values in some specific endian order, which is easier to deal with than Base64 since a JSON parser will at least convert the byte value from text to binary for you. But yea, as a Clojure guy sexprs or EDN would be much better.

matja 1y ago

Aren't JSON parsers technically not following the standard if they don't reliably store a number that is not representable by a IEEE754 double precision float? It's a shame JSON parsers usually default to performance rather than correctness, by using bignums for numbers.

q3k 1y ago
Have a read through RFC7159 or 8259 and despair. > This specification allows implementations to set limits on the range and precision of numbers accepted JSON is a terrible interoperability standard.
- matja 1y ago
  So a JSON parser that cannot store a 2 is technically compliant? :(
  - q3k 1y ago
    Yep. Or one that parses it into a 7 :)
    - kevingadd 1y ago
      I once debugged a production issue that boiled down to "A PCI compliance .dll was messing with floating point flags, causing the number 4 to unserialize as 12"
      - xeromal 1y ago
        That sounds awful. lol
    - chasd00 1y ago
      > Or one that parses it into a 7 :) if it's known and acceptable that LLMs can hallucinate arguments to an API then i don't see how this isn't perfectly acceptable behavior either.
  - reichstein 1y ago
    JSON is a text format. A parser must recognize the text `2` as a valid production of the JSON number grammar. Converting that text to _any_ kind of numerical value is outside the scope of the specification. (At least the JSON.org specification, the RFC tries to say more.) As a textural format, when you use it for data interchange between different platforms, you should ensure that the endpoints agree on the _interpretation_, otherwise they won't see the same data. Again outside of the scope of the JSON specification.
    - tsimionescu 1y ago
      A JSON parser has to check if a numeric value is actually numeric - the JSON {"a" : 123456789} is valid, but {"a" : 12345678f} is not. Per the RFC, a standards-compliant JSON parser can also refuse {"a": 123456789} if it considers the number is too large.
kens 1y ago
> Aren't JSON parsers technically not following the standard if they don't reliably store a number that is not representable by a IEEE754 double precision float? That sentence has four negations and I honestly can't figure out what it means.
- School-Cotton 1y ago
  “The standard technically requires that JSON parsers reliably store numbers, even those that are not representable by an IEEE double”. (It seems this claim is not true, but at least that’s what the sentence means.)
- alterom 1y ago
  >> Aren't JSON parsers technically not following the standard if they don't reliably store a number that is not representable by a IEEE754 double precision float? >That sentence has four negations and I honestly can't figure out what it means. This example is halfway as bad as the one Orwell gives in my favorite essay, "Politics the the English Language"¹. Compare and contrast: >I am not, indeed, sure whether it is not true to say that the Milton who once seemed not unlike a seventeenth-century Shelley had not become, out of an experience ever more bitter in each year, more alien (sic) to the founder of that Jesuit sect which nothing could induce him to tolerate. Orwell has much to say about either. _____ ¹https://www.orwellfoundation.com/the-orwell-foundation/orwell/essays-and-other-works/politics-and-the-english-language/ https://www.orwellfoundation.com/the-orwell-foundation/orwel...
  - NooneAtAll3 1y ago
    that Orwell quote can be saved a lot by proper punctuation I am not, indeed, sure*,* whether it is not true to say that the Milton *(*who once seemed not unlike a seventeenth-century Shelley*)* had not become *-* out of an experience *-* ever more bitter in each year, more alien (sic) to the founder of that Jesuit sect*,* which nothing could induce him to tolerate.
    - alterom 1y ago

em-bee 1y ago

as someone who started the s-expression task on rosettacode.org, i approve. if you need an s-expression parser for your language, look here https://rosettacode.miraheze.org/wiki/S-expressions https://rosettacode.miraheze.org/wiki/S-expressions (the canonical url is https://rosettacode.org/wiki/S-expressions https://rosettacode.org/wiki/S-expressions but they have DNS issues right now)

tempodox 1y ago

“Worse is better” is still having ravaging success.

marcosdumay 1y ago

What I don't understand is why you (and a lot of other people) just expect S-expression parsers to not have the exact same problems.

01HNNWZ0MV43FF 1y ago
I think they mean that Common Lisp has bigints by default
- ryukafalz 1y ago
  As do Scheme and most other Lisps I'm familiar with, and integers/floats are typically specified to be distinct. I think we'd all be better off if that were true of JSON as well. I'd be happy to use s-expressions instead :) Though to GP's point, I suppose we might then end up with JS s-expression parsers that still treat ints and floats interchangeably.
  - petre 1y ago
    And in addition to that are unable to distingush between a string "42" and a number 42.
deleted 1y ago
[deleted]
eadmund 1y ago
Because canonical S-expressions don’t have numbers, just atoms (i.e., byte sequences) and lists. It is up to the using code to interpret "34" as the string "34" or the number 34 or the number 13,108 or the number 13,363, which is part of the protocol being used. In most instances, the byte sequence is probably a decimal number. Now, S-expressions as used for programming languages such as Lisp do have numbers, but again Lisp has bignums. As for parsers of Lisp S-expressions written in other languages: if they want to comply with the standard, they need to support bignums.
- its-summertime 1y ago

ncruces 1y ago

Go can decode numbers losslessly as strings: https://pkg.go.dev/encoding/json#Number https://pkg.go.dev/encoding/json#Number json.Number is (almost) my “favorite” arbitrary decimal: https://github.com/ncruces/decimal?tab=readme-ov-file#decimal-arithmetic https://github.com/ncruces/decimal?tab=readme-ov-file#decima... I'm half joking, but I'm not sure why S-expressions would be better here. There are LISPs that don't do arbitrary precision math.

mise_en_place 1y ago
For actual SERDES, JSON becomes very brittle. It's better to use something like protobuf or cap'n'proto for such cases.
eadmund 1y ago
> Go can decode numbers losslessly as strings: https://pkg.go.dev/encoding/json#Number https://pkg.go.dev/encoding/json#Number Yup, and if you’re using JSON in Go you really do need to be using Number exclusively. Anything else will lead to pain. > I'm half joking, but I'm not sure why S-expressions would be better here. There are LISPs that don't do arbitrary precision math. Sure, but I’m referring specifically to https://www.ietf.org/archive/id/draft-rivest-sexp-13.html https://www.ietf.org/archive/id/draft-rivest-sexp-13.html, which only has lists and bytes, and so number are always just strings and it’s up to the program to interpret them.

kangalioo 1y ago

But what's wrong with sending the number as a string? `"65537"` instead of `"AQAB"`

ayende 1y ago
Too likely that this would not work because silent conversion to number along the way
- iforgotpassword 1y ago
  Then just prefixing it with an underscore or any random letter would've been fine but of course base64 encoding the binary representation in base 64 makes you look so much smarter.
comex 1y ago
The question is how best to send the modulus, which is a much larger integer. For the reasons below, I'd argue that base64 is better. And if you're sending the modulus in base64, you may as well use the same approach for the exponent sent along with it. For RSA-4096, the modulus is 4096 bits = 512 bytes in binary, which (for my test key) is 684 characters in base64 or 1233 characters in decimal. So the base64 version is much smaller. Base64 is also more efficient to deal with. An RSA implementation will typically work with the numbers in binary form, so for the base64 encoding you just need to convert the bytes, which is a simple O(n) transformation. Converting the number between binary and decimal, on the other hand, is O(n^2) if done naively, or O(some complicated expression bigger than n log n) if done optimally. Besides computational complexity, there's also implementation complexity. Base conversion is an algorithm that you normally don't have to implement as part of an RSA implementation. You might argue that it's not hard to find some library to do base conversion for you. Some programming languages even have built-in bigint types. But you typically want to avoid using general-purpose bigint implementations for cryptography. You want to stick to cryptographic libraries, which typically aim to make all operations constant-time to avoid timing side channels. Indeed, the apparent ease-of-use of decimal would arguably be a bad thing since it would encourage implementors to just use a standard bigint type to carry the values around. You could argue that the same concern applies to base64, but it should be relatively safe to use a naive implementation of base64, since it's going to be a straightforward linear scan over the bytes with less room for timing side channels (though not none).

JackSlateur 1y ago

Is this ok ? Python 3.13.3 (main, May 21 2025, 07:49:52) [GCC 14.2.0] on linux Type "help", "copyright", "credits" or "license" for more information. >>> import json >>> json.loads('47234762761726473624762746721647624764380000000000000000000000000000000000000000000') 47234762761726473624762746721647624764380000000000000000000000000000000000000000000

jazzyjackson 1y ago
yes, python falls into the sane language category with arbitrary-precision arithmetic
- faresahmed 1y ago
  Not so much, >>> s="1"+"0"*4300 >>> json.loads(s) ... ValueError: Exceeds the limit (4300 digits) for integer string conversion: value has 4301 digits; use sys.set_int_max_str_digits() to increase the limit This was done to prevent DoS attacks 3 years ago and have been backported to at least CPython 3.9 as it was considered a CVE. Relevant discussion: https://news.ycombinator.com/item?id=32753235 https://news.ycombinator.com/item?id=32753235 Your sibling comment suggests using decimal.Decimal which handles parsing >4300 digit numbers (by default).
  - lifthrasiir 1y ago
    This should be interpreted as a stop-gap measure before a subquadratic algorithm can be adopted. Take a look at _pylong.py in new enough CPython.
teddyh 1y ago
I prefer >> import json, decimal >> j = "47234762761726473624762746721647624764380000000000000000000000000000000000000000000" >> json.loads(j, parse_float=decimal.Decimal, parse_int=decimal.Decimal) Decimal('47234762761726473624762746721647624764380000000000000000000000000000000000000000000') This way you avoid this problem: >> import json >> j = "0.47234762761726473624762746721647624764380000000000000000000000000000000000000000000" >> json.loads(j) 0.47234762761726473 And instead can get: >> import json, decimal >> j = "0.47234762761726473624762746721647624764380000000000000000000000000000000000000000000" >> json.loads(j, parse_float=decimal.Decimal, parse_int=decimal.Decimal) Decimal('0.47234762761726473624762746721647624764380000000000000000000000000000000000000000000')

cortesoft 1y ago

> Canonical S-expressions would have been far preferable, but for whatever reason the world didn’t go that way. I feel like not understanding why JSON won out is being intentionally obtuse. JSON can easily be hand written, edited, and read for most data. Canonical S-expressions are not as easy to read and much harder to write by hand; having to prefix every atom with a length makes is very tedious to write by hand. If you have a JSON object you want to hand edit, you can just type... for an Canonical S-expression, you have to count how many characters you are typing/deleting, and then update the prefix. You might not think the ability to hand generate, read, and edit is important, but I am pretty sure that is a big reason JSON has won in the end. Oh, and the Ruby JSON parser handles that large number just fine.

eadmund 1y ago
> I feel like not understanding why JSON won out is being intentionally obtuse. I didn’t feel like my comment was the right place to shill for an alternative, but rather to complain about JSON. But since you raise it. > JSON can easily be hand written, edited, and read for most data. So can canonical S-expressions! > Canonical S-expressions are not as easy to read and much harder to write by hand; having to prefix every atom with a length makes is very tedious to write by hand. Which is why the advanced representation exists. I contend that this: (urn:ietf:params:acme:error:malformed (detail "Some of the identifiers requested were rejected") (subproblems ((urn:ietf:params:acme:error:malformed (detail "Invalid underscore in DNS name \"_example.org\"") (identifier (dns _example.org))) (urn:ietf:params:acme:error:rejectedIdentifier (detail "This CA will not issue for \"example.net\"") (identifier (dns example.net)))))) is far easier to read than this (the first JSON in RFC 8555): { "type": "urn:ietf:params:acme:error:malformed", "detail": "Some of the identifiers requested were rejected", "subproblems": [ { "type": "urn:ietf:params:acme:error:malformed", "detail": "Invalid underscore in DNS name \"_example.org\"", "identifier": { "type": "dns", "value": "_example.org" } }, { "type": "urn:ietf:params:acme:error:rejectedIdentifier", "detail": "This CA will not issue for \"example.net\"", "identifier": { "type": "dns", "value": "example.net" } } ] } > for an Canonical S-expression, you have to count how many characters you are typing/deleting, and then update the prefix. As you can see, no you do not.

TZubiri 1y ago

It feels like malpractice to use json in encryption

red_admiral 1y ago
Sadly JWT and friends are "standard". In theory the representation and the data are independent and you can marshal and unmarshal correctly. In practice, "alg:none" is a headache and everyone involved should be ashamed.

mindcrime 1y ago

JSON is better than XML, but it really isn’t great. JSON doesn't even support comments, c'mon. I mean, it's handy for some things, but I don't know if I'd say "JSON is better than XML" in any universal sense. I still go by the old saw "use the right tool for the job at hand". In some cases maybe it's JSON. In others XML. In others S-Exprs encoded in EBCDIC or something. Whatever works...

deepsun 1y ago
Yup, imagine if HTML was JSON-like, not XML-like.

supermatt 1y ago

As you said - it’s not really a problem with the JSON structure and format itself, but the underlying parser, which is specifically designed to map to the initial js types. There are parsers that don’t have this problem, but then the JSON itself is not portable. The problem with your solution is that it’s also not portable for the same reason (it’s not part of the standard), and the reason that it wasn’t done that way in the first place is because it wouldn’t map to those initial js types! FYI, you can easily work around this by using replacer and revivers that are part of the standards for stringify and parse and treat numbers differently. But again, the json isn’t portable to places without those replacer/revivers. I.e, the real problem is treating something that looks like json as json by using standards compliant json parsers - not the apparent structure of the format itself. You could fix this problem in an instant by calling it something other than JSON, but people will see it and still use a JSON parser because it looks like JSON, not because it is JSON.

zelphirkalt 1y ago
Isn't the actual problem that it is supposed to map to JS types, which are badly designed, and thus being infectious for other ecosystems, that don't have these defects?

tsimionescu 1y ago

This seems like a just-so story. Your explanation could make some sense if we were comparing {"e" : "AQAB"} to {"e" : 65537}, but there is no reason why that should be the alternative. The JSON {"e" : "65537"} will be read precisely the same way by any JSON parser out there. Converting the string "65537" to the number 65537 is exactly as easy (or hard), but certainly unambiguous, as converting the string "AQAB" to the same number. Of course, if you're doing this in JS and have reasons to think the resulting number may be larger than the precision of a double, you have a huge problem either way. Just as you would if you were writing this in C and thought the number may be larger than what can fit in a long long. But that's true regardless of how you represent it in JSON.

deepsun 1y ago
Some parsers, like PHP, may treat 65537 and "65537" the same. Room for vulnerability.
- int_19h 1y ago
  Why would they do so? It's semantically distinct JSON, even JS itself treats it differently?
  - hiciu 1y ago
    It's PHP. Handling numbers in PHP is complicated enough that a reasonable person would not trust it by default. https://www.php.net/manual/en/language.types.numeric-strings.php https://www.php.net/manual/en/language.types.numeric-strings...
    - int_19h 1y ago
      I know that PHP will treat a string as if it were a number if you try to use it in a context where number is expected; JS does the same thing. But why would that affect JSON deserialization in a way that makes numbers and strings indistinguishable in principle (causing the loss of precision as described here)?
  - 1y ago

mnahkies 1y ago

I'm still haunted by a bug caused by the JSON serializer our C# apps were using emitting bigints as JSON numbers, only for the JavaScript consumers to mangle them silently. Kinda blows my mind that the accepted behavior is to just overflow and not raise an exception. I try to stick to strings for anything that's not a 32 bit int now.

fulafel 1y ago

Is the correct number implementation really the exception? The first 2 json decoders I just tried (Python & Clojure) worked correctly with that example.

rendaw 1y ago

Canonical S-expressions don't have an object/mapping type, which means you can't have generic tooling unambiguously perform certain common operations like data merges.

rr808 1y ago

> JSON is better than XML hard disagree on that one.

motorest 1y ago
[flagged]
- rr808 1y ago
  You're welcome! I hope it was helpful.

zubspace 1y ago

Wouldn't it just solve a whole lot of problems if we could just add optional type declarations to json? It seems so simple and obvious that I'm kinda dumbfounded that this is not a thing yet. Most of the time you would not need it, but it would prevent the parser from making a wrong guess in all those edge cases. Probably there are types not every parser/language can accept, but at least it could throw a meaningful error instead of guessing or even truncating the value.

ivanbakel 1y ago
I doubt that would fix the issue. The real cause is that programmers mostly deal in fixed-size integers, and that’s how they think of integer values, since those are the concepts their languages provide. If you’re going to write a JSON library for your favourite programming language, you’re going to reach for whatever ints are the default, regardless of what the specs or type hints suggest. Haskell’s Aeson library is one of the few exceptions I’ve seen, since it only parses numbers to ‘Scientific’s (essentially a kind of bigint for rationals.) This makes the API very safe, but also incredibly annoying to use if you want to just munge some integers, since you’re forced to handle the error case of the unbounded values not fitting in your fixed-size integer values. Most programmers likely simply either don’t consider that case, or don’t want to have to deal with it, so bad JSON libraries are the default.
movpasd 1y ago
This is actually a deliberate design choice, which the breathtakingly short JSON standard explains quite well [0]. The designers deliberately didn't introduce any semantics and pushes all that to the implementors. I think this is a defensible design goal. If you introduce semantics, you're sure to annoy someone. There's an element of "worse is better" here [1]. JSON overtook XML exactly because it's so simple and solves for the social element of communication between disparate projects with wildly different philosophies, like UNIX byte-oriented I/O streams, or like the C calling conventions. --- [0] https://ecma-international.org/publications-and-standards/standards/ecma-404/ https://ecma-international.org/publications-and-standards/st... [1] https://en.wikipedia.org/wiki/Worse_is_better https://en.wikipedia.org/wiki/Worse_is_better

josephg 1y ago

The funny thing about this is that JavaScript the language has had support for BigIntegers for many years at this point. You can just write 123n for a bigint of 123. JSON could easily be extended to support them - but there’s no standards body with the authority to make a change like that. So we’re probably stuck with json as-is forever. I really hope something better comes along that we can all agree on before I die of old age. While we’re at it, I’d also love a way to embed binary data in json. And a canonical way to represent dates. And comments. And I’d like a sane, consistent way to express sum types. And sets and maps (with non string keys) - which JavaScript also natively supports. Sigh.

aapoalas 1y ago
It's more a problem of support and backwards compatibility. JSON and parsers for it are so ubiquitous, and the spec completely lacks any versioning support, that adding a feature would be a breaking change of horrible magnitude, on nearly all levels of the modern software infrastructure stack. I wouldn't be surprised if some CPUs might break from that :D JSON is a victim of its success: it has become too big to fail, and too big to improve.
Sammi 1y ago
There are easy workarounds to getting bigints in JSON: https://github.com/GoogleChromeLabs/jsbi/issues/30#issuecomment-1694399308 https://github.com/GoogleChromeLabs/jsbi/issues/30#issuecomm...
- josephg 1y ago
  Sure; and I can encode maps and sets as entry lists. Binary data as strings and so on. But I don’t want to. I shouldn’t have to. The fact remains that json doesn’t have native support for any of this stuff. I want something json-like which supports all this stuff natively. I don’t want to have to figure out if some binary data is base64 encoded or hex encoded or whatever, and hack around jackson or serde or javascript to encode and decode my objects properly. Features like this should be built in.
  - 1y ago

ownedthx 1y ago

The numerical issues here are due to JavaScript, not JSON.

llm_nerd 1y ago

>JSON is better than XML JSON is still hack garbage compared to XML from the turn of the millennia. Like most dominant tech standards, JSON took hold purely because many developers are intellectually lazy and it was easier to slam some sloppy JSON together than to understand XML. XML with XSD, XPath and XQuery is simply a glorious combination.