I've dealt with hundreds of gigabytes of JSON data like that in the fintech world And yeah I just streamed the data thru the processor and avoid malloc at all

I've dealt with hundreds of gigabytes of JSON data like that in the fintech world And yeah I just streamed the data thru the processor and avoid malloc at all costs and it works in seconds I bet this method works even faster though somehow if they felt compelled to throw research money at all at it