4 ms·
The walkthrough is very nice, how to do this if you're going to do it. If you're going for pure performance in a production environment you might take a look a
by eatonphil 3y ago
The walkthrough is very nice, how to do this if you're going to do it.
If you're going for pure performance in a production environment you might take a look at Daniel Lemire's work: https://github.com/simdjson/simdjson https://github.com/simdjson/simdjson. Or the MinIO port of it to Go: https://github.com/minio/simdjson-go https://github.com/minio/simdjson-go.
- vjerancrnjak 3y agoIf your JSON always looks the same you can also do better than general JSON parsers.
- lylejantzi3rd 3y agoAndreas Fredriksson demonstrates exactly that in this video: https://vimeo.com/644068002 https://vimeo.com/644068002
- diarrhea 3y agoI wonder: can fast, special-case JSON parsers be dynamically autogenerated from JSON Schemas? Perhaps some macro-ridden Rust monstrosity that spits out specialised parsers at compile time, dynamically…
- minhazm 3y agoFor json schema specifically there are some tools like go-jsonschema[1] but I've never used them personally. But you can use something like ffjson[2] in go to generate a static serialize/deserialize function based on a struct definition. [1] https://github.com/omissis/go-jsonschema https://github.com/omissis/go-jsonschema [2] https://github.com/pquerna/ffjson https://github.com/pquerna/ffjson
- atombender 3y agoHey, go-jsonschema is my project. (Someone else just took over maintaining it, though.) It still relies on the standard Go parser; all it does it generate structs with the right types and tags.
- galangalalgol 3y agoDoesn't the serde crate's json support do precisely this? It generates structs that have optional in all the right places and with all the right types anyway. Seems like the llvm optimiser can probably do something useful with that even if the serde feature isn't using apriori knowledge out of the schema.
- dleeftink 3y agoSomewhat tangentially related, Fabian Iwand posted this regex prefix tree visualiser/generator last week [0], which may offer some inspiration for prototyping auto generated schemas.
- atombender 3y agoYou forgot to include the link?
- dleeftink 3y agoForgot to include the link: [0]: https://observablehq.com/@mootari/regex-from-strings https://observablehq.com/@mootari/regex-from-strings
- PartiallyTyped 3y ago
- loeg 3y agoYou might also move to something other than JSON if parsing it is a significant part of your workload.
- haswell 3y agoMost of the times I’ve had to deal with JSON performance issues, it involved a 3rd party API and JSON was the only option. If you’re building something net-new and know you’ll have these problems out the gate, something other than JSON might be feasible, but the moment some other system not in the closed loop needs to work with the data, you’re back to JSON and any associated perf issues.
- fooster 3y agoLast time I compared the performance of various json parsers the simd one turned out to be disappointingly slow.
- Thaxll 3y agoThe fastest json lib in Go is the one done by the company behind Tiktok.
- ken47 3y agoFastest at what?
- cannonpalms 3y ago> For all sizes of json and all scenarios of usage, Sonic performs best. The repository has benchmarks
- mananaysiempre 3y agoI’m not seeing simdjson in them though? I must be missing something because the Go port of it is explicitly mentioned in the motivation[1] (not the real thing, though). [1] https://github.com/bytedance/sonic/blob/main/docs/INTRODUCTION.md https://github.com/bytedance/sonic/blob/main/docs/INTRODUCTI...
- rockinghigh 3y agohttps://github.com/bytedance/sonic https://github.com/bytedance/sonic
- pizzafeelsright 3y agoExcellent treat vector.
- pizzafeelsright 3y agoExcellent treat vector.
- lionkor 3y agosimdjson has not been the fastest for a long long time
- jzwinck 3y agoWhat is faster? According to https://github.com/kostya/benchmarks#json https://github.com/kostya/benchmarks#json nothing is.