3 ms·
Given that schema is known, should be able to avoid general JSON parsing. Would be much faster.
by vjerancrnjak 2y ago
Given that schema is known, should be able to avoid general JSON parsing. Would be much faster.
- necubi 2y agoThe schema is only known at runtime, which prevents ahead-of-time code generation. JITing would be probably be an improvement though (at least in our use case, stream processing, where we're basically always willing to pay higher upfront costs for higher long-term performance). (Actually, the original version of Arroyo was purely based around ahead-of-time code generation, and used serde_json for deserialization. I wrote at length why we decided to move away from that approach here: https://www.arroyo.dev/blog/why-arrow-and-datafusion https://www.arroyo.dev/blog/why-arrow-and-datafusion).