7 ms·

> In the last year or two I have worked on and off on making PyPy's JSON faster, particularly when parsing large JSON files I think this is the key takeaway. I

• bjoli 7y ago
  I have had pretty large speedups in the past when switching JSON implementations. About 5x. It doesn't matter much when the times are 0.2 Vs 1s, but when you are working with a couple of MB of JSON the difference will definitely be noticeable. Python's STDlib is nice, but rarely spectacular. An old naive CSV parser I wrote for Chicken scheme (using a simple state machine) beat python's C-based one.
  • ehsankia 7y ago
    You're basically agreeing with the above, and I have experienced the same. The only place where this makes a difference is when the JSON file is in MBs. For smaller file, it's not really worth the extra library.
    • IanCal 7y ago
      Depends on how many of the small docs you deal with, I see useful differences on small documents because I process reasonable amounts of them. A 10s change on a million docs for just decoding for a single library is easily worth it for me.
  • acdha 7y ago
    > Python's STDlib is nice, but rarely spectacular. An old naive CSV parser I wrote for Chicken scheme (using a simple state machine) beat python's C-based one. The stdlib is also conservative: I've run into quite a few cases where someone had reported huge gains until they realized they weren't properly handling escaping, Unicode, etc. and once updated the gap narrowed considerably.
• IanCal 7y ago
  Really? I just tried with a sample of one of my files, using python 3.6. With text in memory, 1M smallish strings decoding is 12s with ujson and 25s with the built in json. orjson and rapidjson seemed about the same for this test.
  • umvi 7y ago
    Hmm, I'm starting to think my definition of "small" is different from everybody else's. I consider "smallish" JSON to be at most 100KB (and on average 1-10 KB). 1 MB I consider to be "large"... basically anything that doesn't parse near instantaneously I consider "large".
    • Uehreka 7y ago
      I think GP might've been saying 1M small JSON documents, rather than a single document containing 1M strings.
    • ken 7y ago
      Isn't that tautological? Clearly, no JSON parser provides a significant gain on parse speed of small JSON documents, when you define large as those where parse time is significant.
      • umvi 7y ago
        Well, Python2 -> Python3 introduced a dramatic performance increase, even for small JSON documents.
      • IanCal 7y ago
        These were small though, I just made a small sample of a million of them and still saw a 2x improvement.
      • coldtea 7y ago
        >Isn't that tautological? No, it's no. >Clearly, no JSON parser provides a significant gain on parse speed of small JSON documents, when you define large as those where parse time is significant. A rather bizarre deduction. E.g. "define large as those where parse time is significant". So let's say a large file is one whose parse time is > 10 seconds. So, while looking at small, ie. < 10 second parse time files, if a JSON parser takes files that others load in 500ms and load them in 20ms, then that's "a significant gain on parse speed of small JSON documents"... We didn't have to change any of the definitions...
• tachyonbeam 7y ago
  Something else to note here is that the Python interpreter is slow compared to native code, so if you're parsing a lot of small JSON files, whatever difference in speed the JSON parser makes is masked somewhat by the interpreter overhead (assuming you're doing something with that JSON you just parsed). When you're using a JIT like PyPy, the overhead of the rest of your code becomes less, and so then the bottleneck could become something like JSON parsing, I/O, etc.
  • weberc2 7y ago
    > so if you're parsing a lot of small JSON files, whatever difference in speed the JSON parser makes is masked somewhat by the interpreter overhead Does this vary based on the size/quantity of the input JSON files? I would expect this to hold regardless of size/quantity of the input files? Further, isn't CPython's implementation already native code? If so, I would expect the bottleneck to be allocating the output objects or pressure on the GC or something?
    • AaronFriel 7y ago
      Of course, the smaller the JSON file, the shorter the amount of time spent in native code, and proportionally more time will be spent in the interpreter around whatever looping construct is used to feed data into json.loads (and presumably process that data). # interpreter time spent here looping for file_name in stuff: # interpreter: do stuff with file name, load file into string, etc. str_data = ... # interpreter: dispatch call to native function # native: actually perform json.loads data = json.loads(str_data) # interpreter: time spent processing the data process_data(data) # what happens as native time goes to zero (json parse time decreases)? # answer: interpreter time dominates
      • weberc2 7y ago
        So "interpreter time" is defined as "time not spent parsing JSON" and thus "the less time you spend parsing JSON the less time you spend parsing JSON"? Of course that's correct, but it's so obvious as to be unsubstantial. Surely the OP is making a more interesting point...
• VHRanger 7y ago
  I would think that's because the overhead tends to dominate when calling on small JSON, whether it's function call overhead or python's interpreter, etc.