7 ms·

Gigabytes per second can be a worrying statistic. It suggests that benchmarks would be parsing massive json files rather than the small ones that real-world app

Gigabytes per second can be a worrying statistic. It suggests that benchmarks would be parsing massive json files rather than the small ones that real-world applications deal with. However this library maintains roughly constant throughput for both small (eg 300 byte) and large documents, if it’s benchmarks are accurate.

userbinator 6y ago
Indeed, I feel like much of this is self-inflicted and could be avoided completely if only more developers would learn to use far more compact and efficient binary formats instead, where parsing (if it could even be called that) is just a natural consequence of reading the data itself. One of my favourite examples of this is numerical values, which is notably called out in the performance results for this library as being one of the slowest paths. In a textual format you need to read digits and accumulate them, and JSON allows floating-point which is even more complex to correctly parse. In a binary format, you read n bytes and interpret it as an integer or a floating-point value directly. One machine instruction instead of dozens if not hundreds or more. I see such inefficiencies so often that I wonder if most developers even know how to read/write binary formats anymore... "The fastest way to do something is to not do it at all."
- mumblemumble 6y ago
  > In a binary format, you read n bytes and interpret it as an integer or a floating-point value directly. IF it's a raw binary format. Other formats, such as protocol buffers, use packed representations in order to save space.
  - CoolGuySteve 6y ago
    This always struck me as being dumber than streaming the whole file through lz4 or something similar. At least by separating the compression from the encoding you have a choice of when decompression vs decoding will happen.
    - lalaithion 6y ago
      Does lz4 really have low enough overhead that it can compress a single integer better than having a predetermined compression scheme for small numbers? Sure, most protos are not a single integer, but i'd doubt it if the average proto message was more than 1 kb.
      - cbsmith 6y ago
        lz4 is kind of amazing, but yes, it does not really help when compressing a single integer.
    - mumblemumble 6y ago
      For certain inputs, compression doesn't necessarily work out so well, because of the overhead introduced by the compression format itself. For example, "compressing" a small datagram may actually make it larger. I also believe (but am not prepared to prove) that packed representations can, under the right circumstances, evade some information theoretic limits that constrain lossless compression, by engaging in a little honest cheating. Since you're using a purpose-specific format instead of creating a general-purpose compression scheme, you don't have to record every single bit of information in the blob itself. You can use a side channel - in the form of the format specification itself - to record some information. Which is why you see a packed representation (perhaps with optional compression) in things like protocol buffers, whereas a format like Parquet that's meant to store large volumes of self-describing data will just go straight for compression.
    - cbsmith 6y ago
      Compression algos come with their own disadvantages. Varint encoding exploits the normal distribution/Benford's law type reality that the vast majority of fixed point numbers tend to be closer to zero than their upper bounds. As fast as lz4 is, encoding & decoding varint is way, way faster. There are other advantages too... lz4 works at the block level, rather than the field level, so you have effects from surrounding fields and the need to decode all the fields/records at once. You also need a certain size block for it to really be useful... There are cases where what you are saying makes a lot of sense, and for that you have things like FlatBuffers & CapnProto.
  - tjchear 6y ago
    And if the source architecture and destination architecture use identical endian-ness.
- beached_whale 6y ago
  Fun fact, JSON's number type isn't floating point. It may map to it, but the limits on the numbers that are representible are defined by the encoder and decoder.
  - deleted 6y ago
    [deleted]
  - cbsmith 6y ago
    Which creates this lovely reality where you effectively have numbers that can be accurately represented in IEEE float but can't be represented accurately in decimal notation, and numbers that can be represented accurately in decimal notation but not in IEEE float. All of those cases are dicey with JSON.
    - beached_whale 6y ago
      Curious, what number is representable in IEEE float but not as a decimal?
      - nayuki 6y ago
        +Infinity, -Infinity, NaN (quiet, signaling, miscellaneous).
        beached_whale 6y ago
        Well, those are all error conditions and not a number. NaN's explicitly say they aren't, and +-Inf may or may not be infinity, all we know is that it is outside the representable range of IEEE floating point.
        derefr 6y ago
        IMHO, code should check for floating-point overflow just like it checks for integer overflow. If your inputs are finite and your output is infinite, that's similar to your inputs being positive and your output being negative: you should check for it and emit an error instead. Then infinity (like negative numbers) can have an actual semantic meaning, rather than just being the detritus of an error condition.
- zwieback 6y ago
  Having grown up with 8bit home computers everything about the web seems insanely inefficient. But text formats do have undeniable upside. Best would be more portable efficient formats like protobuf.
- smolder 6y ago
  This is why I love using serde bincode serialization. I can even deal with serde bincode binary blobs inside rust based wasm on the front end. It can save a ton of bytes on the wire, used appropriately.
- alfalfasprout 6y ago
  Unfortunately, binary formats still have plenty of issues too like usually requiring schemas (msgpack doesn't but it's MUCH slower than something with a schema like flatbuffers). They also are fairly clunky for handling nested structured data (eg; arrays of maps).
- nonbirithm 6y ago
  It might be worth noting that, by the words of Douglas Crockford, JSON wasn't even at first specified, it was "discovered."[0] As in, a small group of people at a startup were wanting an object interchange format they could use in the browser in the early 2000s, found that JavaScript object literals worked good enough, and went with that. It only became a standard after the fact. If the data you have requires more thought as to how to store it than "JavaScript's object notation is good enough," then perhaps a specialized binary format would work better. Then again, nearly all APIs only accept and receive JSON. JSON is the one object notation that gets all the attention, so most maintainers will focus their time making optimizations for reading or writing JSON. And if you don't like XML or YAML for some reason, then JSON is tempting to use as a config file format, although it critically lacks comments or heredocs. [0] http://www.json.org/fatfree.html http://www.json.org/fatfree.html
rightbyte 6y ago
I wonder if the lib moves computations and validation for access time. The graph over mb/s for different json-string sizes seem completely bogus. Small sizes should give more overhead and be slower but the line is totally flat. https://github.com/simdjson/simdjson/blob/master/doc/growing.png https://github.com/simdjson/simdjson/blob/master/doc/growing...
- dan-robertson 6y ago
  Well it claims to do validation. I think it perhaps doesn’t fully build json values into fast-to-access data structures but I don’t think this is unique amongst fast json parsers, or indeed necessary. It seems reasonably believable that it could maintain high throughput if it has minimal setup and keeps everything linear.
  - rightbyte 6y ago
    I tested it and it seems to do validation before access. The lib doesn't do double keys properly though, and returns the first result instead of the last. If you don't scan to the end each time at look-up and do no expensive hash map-backed tree I understand why it is faster at parsing ...
    - jsnell 6y ago
      What do you mean by "properly"? The original json.org "spec" is silent on the matter of non-unique keys, and RFC 7159 says: > When the names within an object are not unique, the behavior of software that receives such an object is unpredictable.
      - rightbyte 6y ago
        Oh, in that case it is fine then. I vaguely remembered that it had to have the last value according to the specification when keys are duplicate. I meant "properly" as in "according to the JSON specification". The lib is really nice to use from 5 minutes of testing.
    - deleted 6y ago
nly 6y ago
A few days ago, at work, I transcoded a 13GB file from a proprietary binary format in to a 190GB JSON file... just so I could use jq[0]. This was a small file by our standards. Performance matters [0] https://stedolan.github.io/jq/ https://stedolan.github.io/jq/
- smt88 6y ago
  I've done this same thing (advanced text searches from a proprietary format), but I just inserted it into a local SQLite database instead. Is there a reason JSON/jq made more sense for your purposes than SQLite (or even a local Mongo or Postgres DB, if you wanted to stick with JSON)?
  - remexre 6y ago
    pipes are really nice, jq lets you use them on json; compactness of syntax is another plus
  - nly 6y ago
    The data structures are often nested, and can change quite often, so spending time normalizing all of those in to columns and tables wouldn't make a lot of sense. I suppose on the human level the more important reason is I often get requests from non-techies in the company to see snippets from this data. They can read JSON just fine (Well, most of them... a few still insist on using Excel, so I have to flatten the JSON to csv with jq)
    - smt88 6y ago
      > spending time normalizing all of those in to columns and tables wouldn't make a lot of sense I agree with you (depending on the use-case, of course). I was talking about dumping the data into one of those databases (SQLite, Postgres, or MongoDB) and using their JSON querying functionality.
      - derefr 6y ago
        If you're really talking about streaming a JSON Lines file into a SQL DB as a series of binary JSON documents, that might make a lot of sense. If, however, you're really talking about a single JSON document, then you're going to be generating that 190GB intermediate JSON file either way. Plugging it into a database to query it just seems like an extra step for little benefit. (It's basically a variant on the question of whether you'll get better whole-pipeline latency from a data warehouse, or a data lake — which has no general answer.)
fulafel 6y ago
JSON is used to replace CSV in many applications such as logging / event streams - http://jsonlines.org/ http://jsonlines.org/
- hardwaresofton 6y ago
  There's also ndjson[0], personally I feel like I don't see jsonlines/ndjson in the wild enough -- people still aren't convinced of json logging (or doing stuff like using Content-Type to deliver ndjson/jsonlines over certain endpoints) in the codebases I run into. [0]: http://ndjson.org/ http://ndjson.org/
  - secondcoming 6y ago
    Using json for logging or event streams is fine until your cloud provider hands you your bill for cross zone or region traffic.
  - sixothree 6y ago
    FHIR Bulk Data Extract is using ndjson as a requirement. https://build.fhir.org/ig/HL7/bulk-data/ https://build.fhir.org/ig/HL7/bulk-data/
  - fulafel 6y ago
    Oh, so there were two different formats, what's the difference?
    - hardwaresofton 6y ago
      Funnily enough, issue #1 on ndjson[0] is this same question. There's a fantastic response with a table too[1]. [0]: https://github.com/ndjson/ndjson.github.io/issues/1 https://github.com/ndjson/ndjson.github.io/issues/1 [1]: https://github.com/ndjson/ndjson.github.io/issues/1#issuecomment-109935996 https://github.com/ndjson/ndjson.github.io/issues/1#issuecom...
      - fulafel 6y ago
sradman 6y ago
Gigabytes per second has been a common metric used to compare column store performance (compressed bitmap indexes), a sweet spot for SIMD. Daniel Lemire is involved with both Roaring Bitmaps [1] and EWAH [2]. It is real-world in terms of exposing I/O bottlenecks, if they exist in the architecture (compared to GPU, for example). I suspect it is the default/natural metric these researchers use rather than an exercise in cherry picking results. The underlying question is a good one: when does it make sense to adopt a SIMD library for JSON parsing? [1] https://roaringbitmap.org/ https://roaringbitmap.org/ [2] https://github.com/lemire/javaewah https://github.com/lemire/javaewah
jonstewart 6y ago
Many “real-world” applications deal with massive json files.
- jfkebwjsbx 6y ago
  Yeah, although one can argue they are badly designed if they do so from the performance/energy perspective.
  - jonstewart 6y ago
    Of course, but there are many situations in which one must deal with the input one is given.
cbsmith 6y ago
I'm not sure why you'd think that, but yes there is no special optimizations for files of size above about 512-bits IIRC.

userbinator 6y ago

Indeed, I feel like much of this is self-inflicted and could be avoided completely if only more developers would learn to use far more compact and efficient binary formats instead, where parsing (if it could even be called that) is just a natural consequence of reading the data itself. One of my favourite examples of this is numerical values, which is notably called out in the performance results for this library as being one of the slowest paths. In a textual format you need to read digits and accumulate them, and JSON allows floating-point which is even more complex to correctly parse. In a binary format, you read n bytes and interpret it as an integer or a floating-point value directly. One machine instruction instead of dozens if not hundreds or more. I see such inefficiencies so often that I wonder if most developers even know how to read/write binary formats anymore... "The fastest way to do something is to not do it at all."

mumblemumble 6y ago
> In a binary format, you read n bytes and interpret it as an integer or a floating-point value directly. IF it's a raw binary format. Other formats, such as protocol buffers, use packed representations in order to save space.
- CoolGuySteve 6y ago
  This always struck me as being dumber than streaming the whole file through lz4 or something similar. At least by separating the compression from the encoding you have a choice of when decompression vs decoding will happen.
  - lalaithion 6y ago
    Does lz4 really have low enough overhead that it can compress a single integer better than having a predetermined compression scheme for small numbers? Sure, most protos are not a single integer, but i'd doubt it if the average proto message was more than 1 kb.
    - cbsmith 6y ago
      lz4 is kind of amazing, but yes, it does not really help when compressing a single integer.
  - mumblemumble 6y ago
    For certain inputs, compression doesn't necessarily work out so well, because of the overhead introduced by the compression format itself. For example, "compressing" a small datagram may actually make it larger. I also believe (but am not prepared to prove) that packed representations can, under the right circumstances, evade some information theoretic limits that constrain lossless compression, by engaging in a little honest cheating. Since you're using a purpose-specific format instead of creating a general-purpose compression scheme, you don't have to record every single bit of information in the blob itself. You can use a side channel - in the form of the format specification itself - to record some information. Which is why you see a packed representation (perhaps with optional compression) in things like protocol buffers, whereas a format like Parquet that's meant to store large volumes of self-describing data will just go straight for compression.
  - cbsmith 6y ago
    Compression algos come with their own disadvantages. Varint encoding exploits the normal distribution/Benford's law type reality that the vast majority of fixed point numbers tend to be closer to zero than their upper bounds. As fast as lz4 is, encoding & decoding varint is way, way faster. There are other advantages too... lz4 works at the block level, rather than the field level, so you have effects from surrounding fields and the need to decode all the fields/records at once. You also need a certain size block for it to really be useful... There are cases where what you are saying makes a lot of sense, and for that you have things like FlatBuffers & CapnProto.
- tjchear 6y ago
  And if the source architecture and destination architecture use identical endian-ness.
beached_whale 6y ago
Fun fact, JSON's number type isn't floating point. It may map to it, but the limits on the numbers that are representible are defined by the encoder and decoder.
- deleted 6y ago
  [deleted]
- cbsmith 6y ago
  Which creates this lovely reality where you effectively have numbers that can be accurately represented in IEEE float but can't be represented accurately in decimal notation, and numbers that can be represented accurately in decimal notation but not in IEEE float. All of those cases are dicey with JSON.
  - beached_whale 6y ago
    Curious, what number is representable in IEEE float but not as a decimal?
    - nayuki 6y ago
      +Infinity, -Infinity, NaN (quiet, signaling, miscellaneous).
      - beached_whale 6y ago
        Well, those are all error conditions and not a number. NaN's explicitly say they aren't, and +-Inf may or may not be infinity, all we know is that it is outside the representable range of IEEE floating point.
        derefr 6y ago
        IMHO, code should check for floating-point overflow just like it checks for integer overflow. If your inputs are finite and your output is infinite, that's similar to your inputs being positive and your output being negative: you should check for it and emit an error instead. Then infinity (like negative numbers) can have an actual semantic meaning, rather than just being the detritus of an error condition.
zwieback 6y ago
Having grown up with 8bit home computers everything about the web seems insanely inefficient. But text formats do have undeniable upside. Best would be more portable efficient formats like protobuf.
smolder 6y ago
This is why I love using serde bincode serialization. I can even deal with serde bincode binary blobs inside rust based wasm on the front end. It can save a ton of bytes on the wire, used appropriately.
alfalfasprout 6y ago
Unfortunately, binary formats still have plenty of issues too like usually requiring schemas (msgpack doesn't but it's MUCH slower than something with a schema like flatbuffers). They also are fairly clunky for handling nested structured data (eg; arrays of maps).
nonbirithm 6y ago
It might be worth noting that, by the words of Douglas Crockford, JSON wasn't even at first specified, it was "discovered."[0] As in, a small group of people at a startup were wanting an object interchange format they could use in the browser in the early 2000s, found that JavaScript object literals worked good enough, and went with that. It only became a standard after the fact. If the data you have requires more thought as to how to store it than "JavaScript's object notation is good enough," then perhaps a specialized binary format would work better. Then again, nearly all APIs only accept and receive JSON. JSON is the one object notation that gets all the attention, so most maintainers will focus their time making optimizations for reading or writing JSON. And if you don't like XML or YAML for some reason, then JSON is tempting to use as a config file format, although it critically lacks comments or heredocs. [0] http://www.json.org/fatfree.html http://www.json.org/fatfree.html

rightbyte 6y ago

I wonder if the lib moves computations and validation for access time. The graph over mb/s for different json-string sizes seem completely bogus. Small sizes should give more overhead and be slower but the line is totally flat. https://github.com/simdjson/simdjson/blob/master/doc/growing.png https://github.com/simdjson/simdjson/blob/master/doc/growing...

dan-robertson 6y ago
Well it claims to do validation. I think it perhaps doesn’t fully build json values into fast-to-access data structures but I don’t think this is unique amongst fast json parsers, or indeed necessary. It seems reasonably believable that it could maintain high throughput if it has minimal setup and keeps everything linear.
- rightbyte 6y ago
  I tested it and it seems to do validation before access. The lib doesn't do double keys properly though, and returns the first result instead of the last. If you don't scan to the end each time at look-up and do no expensive hash map-backed tree I understand why it is faster at parsing ...
  - jsnell 6y ago
    What do you mean by "properly"? The original json.org "spec" is silent on the matter of non-unique keys, and RFC 7159 says: > When the names within an object are not unique, the behavior of software that receives such an object is unpredictable.
    - rightbyte 6y ago
      Oh, in that case it is fine then. I vaguely remembered that it had to have the last value according to the specification when keys are duplicate. I meant "properly" as in "according to the JSON specification". The lib is really nice to use from 5 minutes of testing.
  - deleted 6y ago

nly 6y ago

A few days ago, at work, I transcoded a 13GB file from a proprietary binary format in to a 190GB JSON file... just so I could use jq[0]. This was a small file by our standards. Performance matters [0] https://stedolan.github.io/jq/ https://stedolan.github.io/jq/

smt88 6y ago
I've done this same thing (advanced text searches from a proprietary format), but I just inserted it into a local SQLite database instead. Is there a reason JSON/jq made more sense for your purposes than SQLite (or even a local Mongo or Postgres DB, if you wanted to stick with JSON)?
- remexre 6y ago
  pipes are really nice, jq lets you use them on json; compactness of syntax is another plus
- nly 6y ago
  The data structures are often nested, and can change quite often, so spending time normalizing all of those in to columns and tables wouldn't make a lot of sense. I suppose on the human level the more important reason is I often get requests from non-techies in the company to see snippets from this data. They can read JSON just fine (Well, most of them... a few still insist on using Excel, so I have to flatten the JSON to csv with jq)
  - smt88 6y ago
    > spending time normalizing all of those in to columns and tables wouldn't make a lot of sense I agree with you (depending on the use-case, of course). I was talking about dumping the data into one of those databases (SQLite, Postgres, or MongoDB) and using their JSON querying functionality.
    - derefr 6y ago
      If you're really talking about streaming a JSON Lines file into a SQL DB as a series of binary JSON documents, that might make a lot of sense. If, however, you're really talking about a single JSON document, then you're going to be generating that 190GB intermediate JSON file either way. Plugging it into a database to query it just seems like an extra step for little benefit. (It's basically a variant on the question of whether you'll get better whole-pipeline latency from a data warehouse, or a data lake — which has no general answer.)

fulafel 6y ago

JSON is used to replace CSV in many applications such as logging / event streams - http://jsonlines.org/ http://jsonlines.org/

hardwaresofton 6y ago
There's also ndjson[0], personally I feel like I don't see jsonlines/ndjson in the wild enough -- people still aren't convinced of json logging (or doing stuff like using Content-Type to deliver ndjson/jsonlines over certain endpoints) in the codebases I run into. [0]: http://ndjson.org/ http://ndjson.org/
- secondcoming 6y ago
  Using json for logging or event streams is fine until your cloud provider hands you your bill for cross zone or region traffic.
- sixothree 6y ago
  FHIR Bulk Data Extract is using ndjson as a requirement. https://build.fhir.org/ig/HL7/bulk-data/ https://build.fhir.org/ig/HL7/bulk-data/
- fulafel 6y ago
  Oh, so there were two different formats, what's the difference?
  - hardwaresofton 6y ago
    Funnily enough, issue #1 on ndjson[0] is this same question. There's a fantastic response with a table too[1]. [0]: https://github.com/ndjson/ndjson.github.io/issues/1 https://github.com/ndjson/ndjson.github.io/issues/1 [1]: https://github.com/ndjson/ndjson.github.io/issues/1#issuecomment-109935996 https://github.com/ndjson/ndjson.github.io/issues/1#issuecom...
    - fulafel 6y ago

sradman 6y ago

Gigabytes per second has been a common metric used to compare column store performance (compressed bitmap indexes), a sweet spot for SIMD. Daniel Lemire is involved with both Roaring Bitmaps [1] and EWAH [2]. It is real-world in terms of exposing I/O bottlenecks, if they exist in the architecture (compared to GPU, for example). I suspect it is the default/natural metric these researchers use rather than an exercise in cherry picking results. The underlying question is a good one: when does it make sense to adopt a SIMD library for JSON parsing? [1] https://roaringbitmap.org/ https://roaringbitmap.org/ [2] https://github.com/lemire/javaewah https://github.com/lemire/javaewah

jonstewart 6y ago

Many “real-world” applications deal with massive json files.

jfkebwjsbx 6y ago
Yeah, although one can argue they are badly designed if they do so from the performance/energy perspective.
- jonstewart 6y ago
  Of course, but there are many situations in which one must deal with the input one is given.

cbsmith 6y ago

I'm not sure why you'd think that, but yes there is no special optimizations for files of size above about 512-bits IIRC.