5 ms·

A stricter subset of JSON doesn't interest me without more specification surrounding how numerics should be decoded. How do I decode 18446744073709551615? What

by nly 9y ago

A stricter subset of JSON doesn't interest me without more specification surrounding how numerics should be decoded. How do I decode 18446744073709551615? What is the behaviour of a decoder that can't handle numeric types of that size? Should it cause a parse error? Should it truncate to 1.84467e+19? Should it overflow to 4294967295? Presumably if you want a canonical textual representation, the parse error is the only acceptable solution.

d33 9y ago
> What is the behaviour of a decoder that can't handle numeric types of that size? Is this actually relevant? It sounds like an implementation detail to me and it seems to boil down to using arbitrary size bigint. If you can't handle a bigint because it's too big for your machine, it sounds like a problem with the system, not parser.
- Spivak 9y ago
  We can play this game all day. What if I pass an integer that takes 32GiB of memory to store? Also the system's fault if the resources aren't available?
  - d33 9y ago
    I'd say so. This is kind of a security question - the data is valid from the protocol's perspective, it's the contents that are malicious. It's up to the developer to decide how to handle that - if it's super important to be able to process such big numbers, modify the parser to handle that (for example by storing them on the HDD). Same goes for another serialization format: bencoding. It supports arbitrarily nested data structures and most implementations are recursive, which can easily result in stack overflow if you mess up the implementation. But I wouldn't blame the standard - that's the price you pay for flexibility. At the end of the day, you need to sanitize your input anyway and keep your edge cases in mind. Same goes with XML bombs...
    - rmrfrmrf 9y ago
      I would think the JS part of JSON would preclude any massive integers. It’s still an IEEE floating point at the end of the day, so your max is (2^53)-1
  - bastawhiz 9y ago
    If you managed to load a JSON string into memory that contains a number that takes 32GiB of memory to store after parsing, I don't think you're worrying much about resources.
- nly 9y ago
  The behaviour of the encoder and decoder are just as important to specify as the bytes across the wire. This is the whole problem with JSON in the first place.
colanderman 9y ago
Decode it as a number with exactly that value as the numeric type the application requests. If the number is out of range, indicate that to the application (which should already have logic to handle numbers out of the range it cares about). If the application doesn't request that you decide it, don't. Leave it in some internal format which does not lose precision. Limit yourself only to memory limits the application requests. It's no different than decoding a string...
- nly 9y ago
  That's great, but most JSON libraries don't give you that level of access... they just decode in to a bunch of nested maps/dicts consisting of your languages native data types. So one day you swap out your JSON library and the behaviour of your application silently changes.
  - jandrese 9y ago
    Shouldn't they return a list of warnings/errors on the parse? Just blindly accepting a machine parse of some random input data is how you crash programs with bad input.
  - colanderman 9y ago
    Use a better JSON library :) (There are competent JSON implementations, though they're few and far between. PostgreSQL's is the only good example I can point to off the top of my head.)
    - chrismorgan 9y ago
      Here’s what Rust’s `json` crate does for the `Number` type: https://docs.rs/json/0.11.12/json/number/struct.Number.html https://docs.rs/json/0.11.12/json/number/struct.Number.html It stores it as positive, mantissa and exponent, allowing you to pull it apart in that way if you need to. Meanwhile, normal usage is via JsonValue (https://docs.rs/json/0.11.12/json/enum.JsonValue.html https://docs.rs/json/0.11.12/json/enum.JsonValue.html) which allows you to attempt to convert it into an f64, i32, u8, &c. as you desire, returning `None` if it doesn’t fit inside that type (e.g. -7 won’t go into an i32, 576 won’t go into a u8). By what’s being talked about here, that looks to be a competent JSON implementation.
seagreen 9y ago
For context, let's start with what JSON is. All the JSON spec really lets you do is say "this sequence of code points is JSON", or "this sequence isn't". It includes a few interoperability suggestions for how you might avoid JSON that will be hard to interpret, but it's extremely stark on parsing and generation. For a pretty epic summary of this, see this Reddit comment: https://www.reddit.com/r/programming/comments/59htn7/parsing_json_is_a_minefield/d98qxtj/ https://www.reddit.com/r/programming/comments/59htn7/parsing... You have a comment downthread about how inadequate this is, which I totally agree with: > The behaviour of the encoder and decoder are just as important to specify as the bytes across the wire. This is the whole problem with JSON in the first place. Unfortunately since the goal of Son is to be a subset of JSON, there's not much we can do about encoding/decoding within Son's spec. Instead I hope to build a sane foundation on which other tools can be built. Limiting the scope of Son seems like a good way to go about this. In this case, that means not specifying ranges for numbers, so that future tools such as schemas have as much flexibility as possible (for instance to include arbitrary precision numbers). EDIT: I'm mixing together two different things in my last paragraph. I don't want Son to know about Floats, Doubles, etc since JSON doesn't. That doesn't mean that we couldn't specify a range on number size though. The reason that is left out is to allow maximum flexibility for other tools to build on Son -- I'm trying to drop extraneous parts of JSON, but as few useful parts as possible. This needs more elaboration within the project's docs.
- ghettoimp 9y ago
  "In this case, that means not specifying ranges for numbers, so that future tools such as schemas have as much flexibility as possible (for instance to include arbitrary precision numbers)." If the allowed range of numbers isn't specified, then when you emit data in this format, you can't be sure it will be read correctly on the other end...
  - seagreen 9y ago
    This is definitely a problem, but the place to solve it isn't Son. Son is intended to be a very simple project that grabs all the clear wins. Do we really need both `0` and `-0`, stuff like that. There are too many different things JSON Numbers can represent for us to have a clear strategy for all them in Son. Int64s, doubles, floats, arbitrary precision numbers (eg https://hackage.haskell.org/package/scientific-0.3.5.2 https://hackage.haskell.org/package/scientific-0.3.5.2), etc. That makes this a good for for the schema layer (eg JSON Schema or whatever you're using), not the base specification layer.

by nly 9y ago

d33 9y ago

> What is the behaviour of a decoder that can't handle numeric types of that size? Is this actually relevant? It sounds like an implementation detail to me and it seems to boil down to using arbitrary size bigint. If you can't handle a bigint because it's too big for your machine, it sounds like a problem with the system, not parser.

Spivak 9y ago
We can play this game all day. What if I pass an integer that takes 32GiB of memory to store? Also the system's fault if the resources aren't available?
- d33 9y ago
  I'd say so. This is kind of a security question - the data is valid from the protocol's perspective, it's the contents that are malicious. It's up to the developer to decide how to handle that - if it's super important to be able to process such big numbers, modify the parser to handle that (for example by storing them on the HDD). Same goes for another serialization format: bencoding. It supports arbitrarily nested data structures and most implementations are recursive, which can easily result in stack overflow if you mess up the implementation. But I wouldn't blame the standard - that's the price you pay for flexibility. At the end of the day, you need to sanitize your input anyway and keep your edge cases in mind. Same goes with XML bombs...
  - rmrfrmrf 9y ago
    I would think the JS part of JSON would preclude any massive integers. It’s still an IEEE floating point at the end of the day, so your max is (2^53)-1
- bastawhiz 9y ago
  If you managed to load a JSON string into memory that contains a number that takes 32GiB of memory to store after parsing, I don't think you're worrying much about resources.
nly 9y ago
The behaviour of the encoder and decoder are just as important to specify as the bytes across the wire. This is the whole problem with JSON in the first place.

colanderman 9y ago

Decode it as a number with exactly that value as the numeric type the application requests. If the number is out of range, indicate that to the application (which should already have logic to handle numbers out of the range it cares about). If the application doesn't request that you decide it, don't. Leave it in some internal format which does not lose precision. Limit yourself only to memory limits the application requests. It's no different than decoding a string...

nly 9y ago
That's great, but most JSON libraries don't give you that level of access... they just decode in to a bunch of nested maps/dicts consisting of your languages native data types. So one day you swap out your JSON library and the behaviour of your application silently changes.
- jandrese 9y ago
  Shouldn't they return a list of warnings/errors on the parse? Just blindly accepting a machine parse of some random input data is how you crash programs with bad input.
- colanderman 9y ago
  Use a better JSON library :) (There are competent JSON implementations, though they're few and far between. PostgreSQL's is the only good example I can point to off the top of my head.)
  - chrismorgan 9y ago
    Here’s what Rust’s `json` crate does for the `Number` type: https://docs.rs/json/0.11.12/json/number/struct.Number.html https://docs.rs/json/0.11.12/json/number/struct.Number.html It stores it as positive, mantissa and exponent, allowing you to pull it apart in that way if you need to. Meanwhile, normal usage is via JsonValue (https://docs.rs/json/0.11.12/json/enum.JsonValue.html https://docs.rs/json/0.11.12/json/enum.JsonValue.html) which allows you to attempt to convert it into an f64, i32, u8, &c. as you desire, returning `None` if it doesn’t fit inside that type (e.g. -7 won’t go into an i32, 576 won’t go into a u8). By what’s being talked about here, that looks to be a competent JSON implementation.

seagreen 9y ago

For context, let's start with what JSON is. All the JSON spec really lets you do is say "this sequence of code points is JSON", or "this sequence isn't". It includes a few interoperability suggestions for how you might avoid JSON that will be hard to interpret, but it's extremely stark on parsing and generation. For a pretty epic summary of this, see this Reddit comment: https://www.reddit.com/r/programming/comments/59htn7/parsing_json_is_a_minefield/d98qxtj/ https://www.reddit.com/r/programming/comments/59htn7/parsing... You have a comment downthread about how inadequate this is, which I totally agree with: > The behaviour of the encoder and decoder are just as important to specify as the bytes across the wire. This is the whole problem with JSON in the first place. Unfortunately since the goal of Son is to be a subset of JSON, there's not much we can do about encoding/decoding within Son's spec. Instead I hope to build a sane foundation on which other tools can be built. Limiting the scope of Son seems like a good way to go about this. In this case, that means not specifying ranges for numbers, so that future tools such as schemas have as much flexibility as possible (for instance to include arbitrary precision numbers). EDIT: I'm mixing together two different things in my last paragraph. I don't want Son to know about Floats, Doubles, etc since JSON doesn't. That doesn't mean that we couldn't specify a range on number size though. The reason that is left out is to allow maximum flexibility for other tools to build on Son -- I'm trying to drop extraneous parts of JSON, but as few useful parts as possible. This needs more elaboration within the project's docs.

ghettoimp 9y ago
"In this case, that means not specifying ranges for numbers, so that future tools such as schemas have as much flexibility as possible (for instance to include arbitrary precision numbers)." If the allowed range of numbers isn't specified, then when you emit data in this format, you can't be sure it will be read correctly on the other end...
- seagreen 9y ago
  This is definitely a problem, but the place to solve it isn't Son. Son is intended to be a very simple project that grabs all the clear wins. Do we really need both `0` and `-0`, stuff like that. There are too many different things JSON Numbers can represent for us to have a clear strategy for all them in Son. Int64s, doubles, floats, arbitrary precision numbers (eg https://hackage.haskell.org/package/scientific-0.3.5.2 https://hackage.haskell.org/package/scientific-0.3.5.2), etc. That makes this a good for for the schema layer (eg JSON Schema or whatever you're using), not the base specification layer.