5 ms·
I think it's because Go's encoding/json package doesn't support incremental parsing.
by icholy 3y ago
I think it's because Go's encoding/json package doesn't support incremental parsing.
- wwader 3y agoI think https://pkg.go.dev/encoding/json#Decoder https://pkg.go.dev/encoding/json#Decoder do support steaming at least. Here is gojq's stream mode https://github.com/itchyny/gojq/blob/main/cli/stream.go https://github.com/itchyny/gojq/blob/main/cli/stream.go
- Alifatisk 3y agoIs there an issue on this?
- zimpenfish 3y agoIt does - I patched that into my local fork of `gron` two years ago.
- zimpenfish 3y agoJust done a test with my 800MB stress test file. `jq`: 1m26s 21G resident `mygron -e --no-sort`: 18m14s 19M resident `gron --no-sort`: 1m51s OOM killed at 54G resident
- xiphias2 3y agoCan you try https://github.com/adamritter/fastgron https://github.com/adamritter/fastgron as a comparision?
- zimpenfish 3y ago`fastgron`: 8.5s 2.2G resident edit: Interestingly whilst doing this test, I piped the output into `fastgron -u` (39.5G resident) and `jq` rejected that. Will have to investigate further but it's a bit of a flaw if it can't rehydrate its own output into valid JSON.
- xiphias2 3y agoThanks, it's a clear bug. I created a new issue for it: https://github.com/adamritter/fastgron/issues/19 https://github.com/adamritter/fastgron/issues/19
- xiphias2 3y agoI released fastgron v0.7.5 which contains fixed in string escaping. Could you please take another look?
- zimpenfish 3y agoAlready commented on the github issue but will have another look, yep.
- xiphias2 3y agoSure, I saw it, thanks! I fixed the semicolon bug, but of course correctness is more important.
- zimpenfish 3y agoUpdate for anyone following - 0.7.6 recreates my 800MB input JSON correctly after trip through `fastgron | fastgron -u` which is good work.
- Alifatisk 3y ago> `gron --no-sort`: 1m51s OOM killed at 54G resident Oh dear
- zimpenfish 3y agoIf I remember correctly, it took a 128GB AWS EC2 to parse that file without OOMing. Go is not that efficient at deep multi-level size- and type-unknown data structures.
- icholy 3y agoThanks for the follow up. Is your fork public?
- zimpenfish 3y agoThere's a branch here - https://github.com/rjp/gron/tree/f/reduce-memory https://github.com/rjp/gron/tree/f/reduce-memory