4 ms·

> so if you're parsing a lot of small JSON files, whatever difference in speed the JSON parser makes is masked somewhat by the interpreter overhead Does this v

> so if you're parsing a lot of small JSON files, whatever difference in speed the JSON parser makes is masked somewhat by the interpreter overhead Does this vary based on the size/quantity of the input JSON files? I would expect this to hold regardless of size/quantity of the input files? Further, isn't CPython's implementation already native code? If so, I would expect the bottleneck to be allocating the output objects or pressure on the GC or something?

AaronFriel 7y ago
Of course, the smaller the JSON file, the shorter the amount of time spent in native code, and proportionally more time will be spent in the interpreter around whatever looping construct is used to feed data into json.loads (and presumably process that data). # interpreter time spent here looping for file_name in stuff: # interpreter: do stuff with file name, load file into string, etc. str_data = ... # interpreter: dispatch call to native function # native: actually perform json.loads data = json.loads(str_data) # interpreter: time spent processing the data process_data(data) # what happens as native time goes to zero (json parse time decreases)? # answer: interpreter time dominates
- weberc2 7y ago
  So "interpreter time" is defined as "time not spent parsing JSON" and thus "the less time you spend parsing JSON the less time you spend parsing JSON"? Of course that's correct, but it's so obvious as to be unsubstantial. Surely the OP is making a more interesting point...
  - AaronFriel 7y ago
    No, that's it - for most json documents, the time the interpreter spends in all the stuff surrounding (and executing) json.loads dominates the time spent in native code parsing. That's why replacing json.loads with a faster native implementation doesn't result in noticeable improvements - it's speeding up the fastest bit, and if it's already 0.1% of the wall clock time, then you can't get much more than about 0.1% faster. PyPy is a JITed language though, so the interpreted bits are much faster. OP is making the point that Python 3/CPython is so slow that speeding up json.loads is a premature (and unnecessary) optimization.

AaronFriel 7y ago

Of course, the smaller the JSON file, the shorter the amount of time spent in native code, and proportionally more time will be spent in the interpreter around whatever looping construct is used to feed data into json.loads (and presumably process that data). # interpreter time spent here looping for file_name in stuff: # interpreter: do stuff with file name, load file into string, etc. str_data = ... # interpreter: dispatch call to native function # native: actually perform json.loads data = json.loads(str_data) # interpreter: time spent processing the data process_data(data) # what happens as native time goes to zero (json parse time decreases)? # answer: interpreter time dominates

weberc2 7y ago
So "interpreter time" is defined as "time not spent parsing JSON" and thus "the less time you spend parsing JSON the less time you spend parsing JSON"? Of course that's correct, but it's so obvious as to be unsubstantial. Surely the OP is making a more interesting point...
- AaronFriel 7y ago
  No, that's it - for most json documents, the time the interpreter spends in all the stuff surrounding (and executing) json.loads dominates the time spent in native code parsing. That's why replacing json.loads with a faster native implementation doesn't result in noticeable improvements - it's speeding up the fastest bit, and if it's already 0.1% of the wall clock time, then you can't get much more than about 0.1% faster. PyPy is a JITed language though, so the interpreted bits are much faster. OP is making the point that Python 3/CPython is so slow that speeding up json.loads is a premature (and unnecessary) optimization.