6 ms·

> In conclusion, JSON is not a data format you can rely on blindly. That was definitely not my take-away from the article. More like "JSON is not a data forma

> In conclusion, JSON is not a data format you can rely on blindly. That was definitely not my take-away from the article. More like "JSON is not a data format you can rely on blindly if you are using an esoteric edge-case and/or an alpha-stage parsing library." I haven't ever run into a single JSON issue that wasn't due to my own fat fingers or trying to serialize data that would have been better suited to something like BSON.

Tepix 10y ago
You are not considering a hostile environment like the internet where an attacker will use edge cases to get unforeseen results.
- inimino 10y ago
  The takeaway is "JSON parsing can fail and possibly crash" but I don't think that should be news.
- mi100hael 10y ago
  I couldn't really care less if someone POSTs some garbage JSON that results in them getting a 500 response. Better than someone POSTing an XML bomb and affecting other peoples' requests. Please enlighten me if you know of a serialization format with libraries for all common languages that lacks any gotchas or edge cases.
  - firethief 10y ago
    If all you're doing is parsing their JSON for it's own sake, it's just a 500; but that's the boring case. Consider what happens when typical web code is interacting with the JSON parser.
  - Someone 10y ago
    All software has edges, so edge cases are unavoidable. The best you can do is: - interpret the spec to the letter. - for every fragment of a statement you write, consider whether it might conceivably go wrong, and handle those cases (in the simplest matter because 'handling' means writing code, and that code, too, needs to go through this process). For example, a json parser must be prepared to handle missing values, extremely long keys and values (integers may have thousands of digits, think long about the question whether 64 bits always is enough for storing a string length, etc.), etc. - if you are truly paranoid, have very stringent security requirements, or expect to be heavily attacked, run the parser in a separate process. - fuzz your implementation.
    - gmfawcett 10y ago
      > think long about the question whether 64 bits always is enough for storing a string length, etc.), etc. I'm struggling to think of any realistic scenario where this isn't true!
      - Someone 10y ago
        So do I, but you should still consciously decide whether to add an overflow check. Let's do a quick estimate: one can read in the order of 2^24 bytes/second from disk. A day has on the order of 2^16 seconds, so that's 2^40 bytes/day. => You will need 2^24 days to read 2^64 bytes. I think that's around 50k years. That an attacker will try to generate a buffer overflow this way is a risk I would take, even if I thought the hardware had room to store that string. The only way I can foresee a real risk is when an optimizer can optimize away the computation of a string whose length it is asked to compute by an attacker. That's still very much far-fetched, and if no string gets allocated it's hard to see how it could become a security issue, but it could be a reason to be extra careful, for example when providing online access to a C++ compiler, with its template metaprogramming capabilities.
empath75 10y ago
Yeah, basically the json parsers and generators and the programs that use them seem to have settled on implementations that mostly work together.
jerf 10y ago
The sum totality of all the issues raised in that post is not an esoteric edge case, even if each individual element is an esoteric edge case. If you haven't encountered any of them in your real code yet, there's two basic possibilities. Either you aren't using JSON very hard at all... or you have encountered them and you just didn't realize it. You will, sooner or later. I'm not saying JSON is bad. Personally I think the sloppiness that this post enumerates is part of its success, and part of why it has so much support in so many languages. The more you nail down the semantics, the harder such widespread support gets. Right now you get a nice subset of the JSON spec that works in a huge variety of languages, where the JSON support for a given language usually converts the JSON into something fairly native for the language. This wouldn't be possible if you nailed down the semantics as hard as something like Protobuf does. For instance, a ton of statically-typed languages will let you encode and decode integers into JSON. But JSON doesn't have an integer type. Strict JSON support ought to forbid integers. But it's sloppiness lets us all just sort of ignore that and get on with life. If you have a need for precision, avoid JSON. And people probably generally need more precision than they realize and probably ought to reach for JSON a bit less often than they do. But on the other hand, the whole thing does mostly work, right? That can't be ignored.
- inimino 10y ago
  JSON doesn't have an integer type, but it certainly supports integers. Within, obviously, implementation defined limits. I'm with you up to "if you need precision, avoid JSON". Actually JSON is fine for the kinds of precision most use cases require, and when it isn't, you probably know it.
  - lobster_johnson 10y ago
    JSON's number support is a source of problems, because the standard is so informal [1]: JSON is agnostic about numbers. ... JSON instead offers only the representation of numbers that humans use: a sequence of digits This poses a problem for some languages, and tends to break things. You would expect encode(decode(string)) == string, but languages deal with numbers differently. For example, in Go, if you decode into a map[string]interface{}, you will get float64 by default. If you decode "42" and then encode it back to JSON, you'll get "42.0". (This is the reason Go has a special type you can use, json.Number, that preserves the value as its original string value.) This mostly causes issues for code that needs to be data-structure-agnostic (in the Go case: decoding into a interface{}, rather than a struct with "json:" tags), but there are edges cases where you can get a surprise. For example, since all numbers are technically both integers and floats, something like {"visitorCount": 42.0} is perfectly valid, and a client has to know to coerce the number into an integer if it wants to deal with it sanely, even though the meaning of that number might be nonsensical if treated as a float. [1] http://www.ecma-international.org/publications/files/ECMA-ST/ECMA-404.pdf http://www.ecma-international.org/publications/files/ECMA-ST...
wtetzner 10y ago
> In conclusion, JSON is not a data format you can rely on blindly. Seems like an odd thing to say in general. Perhaps "JSON parsers are not libraries you can rely on blindly", but that seems true of everything.
romaniv 10y ago
If there are two servers/services that communicate via JSON and they use different parsers, these types of issues can lead to rather nasty problems. Even if both parties fail gracefully on their own. This gets even worse if your software is an integration layer between two services you do not control.

Tepix 10y ago

You are not considering a hostile environment like the internet where an attacker will use edge cases to get unforeseen results.

inimino 10y ago
The takeaway is "JSON parsing can fail and possibly crash" but I don't think that should be news.
mi100hael 10y ago
I couldn't really care less if someone POSTs some garbage JSON that results in them getting a 500 response. Better than someone POSTing an XML bomb and affecting other peoples' requests. Please enlighten me if you know of a serialization format with libraries for all common languages that lacks any gotchas or edge cases.
- firethief 10y ago
  If all you're doing is parsing their JSON for it's own sake, it's just a 500; but that's the boring case. Consider what happens when typical web code is interacting with the JSON parser.
- Someone 10y ago
  All software has edges, so edge cases are unavoidable. The best you can do is: - interpret the spec to the letter. - for every fragment of a statement you write, consider whether it might conceivably go wrong, and handle those cases (in the simplest matter because 'handling' means writing code, and that code, too, needs to go through this process). For example, a json parser must be prepared to handle missing values, extremely long keys and values (integers may have thousands of digits, think long about the question whether 64 bits always is enough for storing a string length, etc.), etc. - if you are truly paranoid, have very stringent security requirements, or expect to be heavily attacked, run the parser in a separate process. - fuzz your implementation.
  - gmfawcett 10y ago
    > think long about the question whether 64 bits always is enough for storing a string length, etc.), etc. I'm struggling to think of any realistic scenario where this isn't true!
    - Someone 10y ago
      So do I, but you should still consciously decide whether to add an overflow check. Let's do a quick estimate: one can read in the order of 2^24 bytes/second from disk. A day has on the order of 2^16 seconds, so that's 2^40 bytes/day. => You will need 2^24 days to read 2^64 bytes. I think that's around 50k years. That an attacker will try to generate a buffer overflow this way is a risk I would take, even if I thought the hardware had room to store that string. The only way I can foresee a real risk is when an optimizer can optimize away the computation of a string whose length it is asked to compute by an attacker. That's still very much far-fetched, and if no string gets allocated it's hard to see how it could become a security issue, but it could be a reason to be extra careful, for example when providing online access to a C++ compiler, with its template metaprogramming capabilities.

empath75 10y ago

Yeah, basically the json parsers and generators and the programs that use them seem to have settled on implementations that mostly work together.

jerf 10y ago

The sum totality of all the issues raised in that post is not an esoteric edge case, even if each individual element is an esoteric edge case. If you haven't encountered any of them in your real code yet, there's two basic possibilities. Either you aren't using JSON very hard at all... or you have encountered them and you just didn't realize it. You will, sooner or later. I'm not saying JSON is bad. Personally I think the sloppiness that this post enumerates is part of its success, and part of why it has so much support in so many languages. The more you nail down the semantics, the harder such widespread support gets. Right now you get a nice subset of the JSON spec that works in a huge variety of languages, where the JSON support for a given language usually converts the JSON into something fairly native for the language. This wouldn't be possible if you nailed down the semantics as hard as something like Protobuf does. For instance, a ton of statically-typed languages will let you encode and decode integers into JSON. But JSON doesn't have an integer type. Strict JSON support ought to forbid integers. But it's sloppiness lets us all just sort of ignore that and get on with life. If you have a need for precision, avoid JSON. And people probably generally need more precision than they realize and probably ought to reach for JSON a bit less often than they do. But on the other hand, the whole thing does mostly work, right? That can't be ignored.

inimino 10y ago
JSON doesn't have an integer type, but it certainly supports integers. Within, obviously, implementation defined limits. I'm with you up to "if you need precision, avoid JSON". Actually JSON is fine for the kinds of precision most use cases require, and when it isn't, you probably know it.
- lobster_johnson 10y ago
  JSON's number support is a source of problems, because the standard is so informal [1]: JSON is agnostic about numbers. ... JSON instead offers only the representation of numbers that humans use: a sequence of digits This poses a problem for some languages, and tends to break things. You would expect encode(decode(string)) == string, but languages deal with numbers differently. For example, in Go, if you decode into a map[string]interface{}, you will get float64 by default. If you decode "42" and then encode it back to JSON, you'll get "42.0". (This is the reason Go has a special type you can use, json.Number, that preserves the value as its original string value.) This mostly causes issues for code that needs to be data-structure-agnostic (in the Go case: decoding into a interface{}, rather than a struct with "json:" tags), but there are edges cases where you can get a surprise. For example, since all numbers are technically both integers and floats, something like {"visitorCount": 42.0} is perfectly valid, and a client has to know to coerce the number into an integer if it wants to deal with it sanely, even though the meaning of that number might be nonsensical if treated as a float. [1] http://www.ecma-international.org/publications/files/ECMA-ST/ECMA-404.pdf http://www.ecma-international.org/publications/files/ECMA-ST...

wtetzner 10y ago

> In conclusion, JSON is not a data format you can rely on blindly. Seems like an odd thing to say in general. Perhaps "JSON parsers are not libraries you can rely on blindly", but that seems true of everything.

romaniv 10y ago

If there are two servers/services that communicate via JSON and they use different parsers, these types of issues can lead to rather nasty problems. Even if both parties fail gracefully on their own. This gets even worse if your software is an integration layer between two services you do not control.