4 ms·
(one of the original authors of simdjson here) I thought a bit about this - I think you could do a lot faster if you had a 'bulk' method to create lots of Pyth
by glangdale 7y ago
(one of the original authors of simdjson here)
I thought a bit about this - I think you could do a lot faster if you had a 'bulk' method to create lots of Python objects in one hit. I don't know it that's feasible, as I don't know much about the innards of Python, but there's no reason one couldn't keep goofing around with SIMD code and branch-free stuff all the way until you've created objects.
This may be ferociously unmaintainable (i.e. depend on things that Python implementations don't advertise as a part of the API) but I don't know. I doubt you're the only one to have this problem.
- TkTech 7y agoLike you said, it would unfortunately be unmaintainable, requiring a fork of CPython. Aggressive optimizations like this could go into pypy, but IMO CPython is still trying to be a "reasonably readable" reference implementation. I think the time is currently best spent trying to push more of the work back into native code, extending the simple filtering that's in there now to also handle mutation (similar to jq), reducing the amount of back-and-forth with the interpreter.