4 ms·

Really? I just tried with a sample of one of my files, using python 3.6. With text in memory, 1M smallish strings decoding is 12s with ujson and 25s with the bu

by IanCal 7y ago

Really? I just tried with a sample of one of my files, using python 3.6. With text in memory, 1M smallish strings decoding is 12s with ujson and 25s with the built in json. orjson and rapidjson seemed about the same for this test.

umvi 7y ago
Hmm, I'm starting to think my definition of "small" is different from everybody else's. I consider "smallish" JSON to be at most 100KB (and on average 1-10 KB). 1 MB I consider to be "large"... basically anything that doesn't parse near instantaneously I consider "large".
- Uehreka 7y ago
  I think GP might've been saying 1M small JSON documents, rather than a single document containing 1M strings.
- ken 7y ago
  Isn't that tautological? Clearly, no JSON parser provides a significant gain on parse speed of small JSON documents, when you define large as those where parse time is significant.
  - umvi 7y ago
    Well, Python2 -> Python3 introduced a dramatic performance increase, even for small JSON documents.
  - IanCal 7y ago
    These were small though, I just made a small sample of a million of them and still saw a 2x improvement.
  - coldtea 7y ago
    >Isn't that tautological? No, it's no. >Clearly, no JSON parser provides a significant gain on parse speed of small JSON documents, when you define large as those where parse time is significant. A rather bizarre deduction. E.g. "define large as those where parse time is significant". So let's say a large file is one whose parse time is > 10 seconds. So, while looking at small, ie. < 10 second parse time files, if a JSON parser takes files that others load in 500ms and load them in 20ms, then that's "a significant gain on parse speed of small JSON documents"... We didn't have to change any of the definitions...
- nicoburns 7y ago
  If you use a language like Rust (which is really easy for these kind of jobs), then 1mb will parse almost instantly. I've been working with a 90gb JSON file recently. I'd probably start counting a file as large at around 100mb.
  - aisengard 7y ago
    I have to ask - why? I understand if you're dealing with an external resource that you can't change, but I'd still like to ask them the same question. JSON's strength, it seems to me, is that it's great for human consumption. It's a very readable data structure when serialized! But who is ever going to read 90GB of JSON?
    - nly 7y ago
      Not OP, but I deal with a variety of proprietary binary/encoded files in the 10s of MB to the 10s of GB range at work, and we have started decoding to JSON as the first step in our normalization, indexing and analytics pipeline (To be honest we essentially just throw everything in to MongoDB). Daily we're in the terabyte range. It's the perfect solution because you can use the same decoder + something like jq for debugging and manipulation on the command line.
      - e12e 7y ago
        A question - do know of any good resources for jq? I haven't quite been "angry enough" to use jq much beyond using it for formatting json - but that's also partly because I can't seem to quite grasp the syntax. It isn't quite "awk-ish", not quite cut/join/sort/uniq. But also not quite graphql? Which I suppose would be the "natural" language for a json-query tool? I feel like I'm missing a piece - and missing out...
        chucky_z 7y ago
        The manual is pretty dang good, tbh. https://stedolan.github.io/jq/manual/ https://stedolan.github.io/jq/manual/
- IanCal 7y ago
  Oh I think you misunderstood, they are all small, there's just a million of them. I think they were all about 1k.
acdha 7y ago
Note the qualifiers about document size — anything measured in millions probably isn't “smallish”, and unless your app does nothing other than decode those strings the total runtime change is unlikely to be anywhere near 2:1.
- IanCal 7y ago
  Each document was small, there was just a lot of them. They can be huge changes. With slightly larger docs we had a setup that had to return batches of results back. The bottleneck was absolutely the single thread part of the API encoding and decoding JSON. Even if the docs were just the size of my example that's 25s for 1m docs. With a few threads I can easily run 10k+/s of the useful part so that'd be 100s + 25s + encode time. With a few more threads hitting 50k/s that's 20s useful time + 25s + more. With larger docs it becomes far more of a problem. More importantly the fix is a few seconds to implement.
  - acdha 7y ago
    Again, note the qualifiers: nobody is arguing that faster JSON parsing isn't a good thing, only that anything measured in the millions is probably not considered “small”. A significant amount of code is never profiled since it's fast enough and if someone does find the stdlib json library to be using enough time to care about it is certainly easily replaced.
    - IanCal 7y ago
      The small qualifier seemed to be on the size of the documents, and on small documents I see a 2x or often higher improvement. At a scale of 10k docs that's still 100ms difference, and in my case can be a huge proportion of the total time spent (more with larger docs, and more again if you include encoding time).
    - deleted 7y ago
      [deleted]
- naniwaduni 7y ago
  Plenty of things measured in millions are small. A million bytes is a small amount of memory. There are plenty of domains where a working definition of "small" is roughly "can be slurped into memory on a single node without jumping through hoops for provisioning". A million iterations that total to order of ten seconds seems quite reasonable for a quick performance comparison. It's just underspecified for the rest of us to take away much beyond that.

by IanCal 7y ago

umvi 7y ago

Hmm, I'm starting to think my definition of "small" is different from everybody else's. I consider "smallish" JSON to be at most 100KB (and on average 1-10 KB). 1 MB I consider to be "large"... basically anything that doesn't parse near instantaneously I consider "large".

Uehreka 7y ago
I think GP might've been saying 1M small JSON documents, rather than a single document containing 1M strings.
ken 7y ago
Isn't that tautological? Clearly, no JSON parser provides a significant gain on parse speed of small JSON documents, when you define large as those where parse time is significant.
- umvi 7y ago
  Well, Python2 -> Python3 introduced a dramatic performance increase, even for small JSON documents.
- IanCal 7y ago
  These were small though, I just made a small sample of a million of them and still saw a 2x improvement.
- coldtea 7y ago
  >Isn't that tautological? No, it's no. >Clearly, no JSON parser provides a significant gain on parse speed of small JSON documents, when you define large as those where parse time is significant. A rather bizarre deduction. E.g. "define large as those where parse time is significant". So let's say a large file is one whose parse time is > 10 seconds. So, while looking at small, ie. < 10 second parse time files, if a JSON parser takes files that others load in 500ms and load them in 20ms, then that's "a significant gain on parse speed of small JSON documents"... We didn't have to change any of the definitions...
nicoburns 7y ago
If you use a language like Rust (which is really easy for these kind of jobs), then 1mb will parse almost instantly. I've been working with a 90gb JSON file recently. I'd probably start counting a file as large at around 100mb.
- aisengard 7y ago
  I have to ask - why? I understand if you're dealing with an external resource that you can't change, but I'd still like to ask them the same question. JSON's strength, it seems to me, is that it's great for human consumption. It's a very readable data structure when serialized! But who is ever going to read 90GB of JSON?
  - nly 7y ago
    Not OP, but I deal with a variety of proprietary binary/encoded files in the 10s of MB to the 10s of GB range at work, and we have started decoding to JSON as the first step in our normalization, indexing and analytics pipeline (To be honest we essentially just throw everything in to MongoDB). Daily we're in the terabyte range. It's the perfect solution because you can use the same decoder + something like jq for debugging and manipulation on the command line.
    - e12e 7y ago
      A question - do know of any good resources for jq? I haven't quite been "angry enough" to use jq much beyond using it for formatting json - but that's also partly because I can't seem to quite grasp the syntax. It isn't quite "awk-ish", not quite cut/join/sort/uniq. But also not quite graphql? Which I suppose would be the "natural" language for a json-query tool? I feel like I'm missing a piece - and missing out...
      - chucky_z 7y ago
        The manual is pretty dang good, tbh. https://stedolan.github.io/jq/manual/ https://stedolan.github.io/jq/manual/
IanCal 7y ago
Oh I think you misunderstood, they are all small, there's just a million of them. I think they were all about 1k.

acdha 7y ago

Note the qualifiers about document size — anything measured in millions probably isn't “smallish”, and unless your app does nothing other than decode those strings the total runtime change is unlikely to be anywhere near 2:1.

IanCal 7y ago
Each document was small, there was just a lot of them. They can be huge changes. With slightly larger docs we had a setup that had to return batches of results back. The bottleneck was absolutely the single thread part of the API encoding and decoding JSON. Even if the docs were just the size of my example that's 25s for 1m docs. With a few threads I can easily run 10k+/s of the useful part so that'd be 100s + 25s + encode time. With a few more threads hitting 50k/s that's 20s useful time + 25s + more. With larger docs it becomes far more of a problem. More importantly the fix is a few seconds to implement.
- acdha 7y ago
  Again, note the qualifiers: nobody is arguing that faster JSON parsing isn't a good thing, only that anything measured in the millions is probably not considered “small”. A significant amount of code is never profiled since it's fast enough and if someone does find the stdlib json library to be using enough time to care about it is certainly easily replaced.
  - IanCal 7y ago
    The small qualifier seemed to be on the size of the documents, and on small documents I see a 2x or often higher improvement. At a scale of 10k docs that's still 100ms difference, and in my case can be a huge proportion of the total time spent (more with larger docs, and more again if you include encoding time).
  - deleted 7y ago
    [deleted]
naniwaduni 7y ago
Plenty of things measured in millions are small. A million bytes is a small amount of memory. There are plenty of domains where a working definition of "small" is roughly "can be slurped into memory on a single node without jumping through hoops for provisioning". A million iterations that total to order of ten seconds seems quite reasonable for a quick performance comparison. It's just underspecified for the rest of us to take away much beyond that.