5 ms·

Show HN: A fast JSON library for Go

119 points by goccy 6y ago

goccy 6y ago
I develop a super fast JSON library ( https://github.com/goccy/go-json https://github.com/goccy/go-json ) . It is drop-in replacement of encoding/json. This library is available by simply replacing the import statement. It's more compatible and much faster than json-iterator/go. Please give me your feedback ! Thanks !
- krenel 6y ago
  Hi mate, very interesting library. Keep it up. Improvement suggestion: While reading the README, the JSON library comparison [1] it's confusing, at least to me. It uses symbols like: ○, and △ which are not obvious. I would rather use "yes/no" or the classic check/x (unicode chars, HN does not support them here) if you want to use symbols. I don't know what "△" even means, so I cannot suggest an alternative. Just my 2 cents. [1]: https://github.com/goccy/go-json#json-library-comparison https://github.com/goccy/go-json#json-library-comparison Edit: This 2021 I have the objective to contribute more to OS projects. Is it OKay if I submit a PR with the change? Thanks!
  - YesThatTom2 6y ago
    Another suggestion in the chart: it isn't obvious which line is about your project because it doesnt list the `goccy/` part. I would add that and move it to the top, out it and the standard in bold.
  - goccy 6y ago
    Yes, Of course !! Welcome !!
- 0x008 6y ago
  As a Non-Gopher, is the stdlib so slow?
  - goccy 6y ago
    I think it depends on the application. In many cases, encoding/json is fast enough and stable, but I think you can try to use a faster library if JSON encoding / decoding is performance critical on your application.
    - 0x008 6y ago
      Or you could try something like Apache Arrow or Flexbuffers?
      - jerf 6y ago
        If you write off the "JSON", sure, there's a bajillion options. But that's not always possible or desirable. None of those bajillion options have the combination of support and mindshare that JSON has obtained, for better or worse.
  - fredrikholm 6y ago
    We're talking differences of a thousandth of a millisecond (0.001ms) for most structs, but yes. encoding/json from stdlib makes a lot of allocations (~30) due its use of reflection, which is notoriously slow.
- krylon 6y ago
  How does it handle de-serializing to structs with interface members? With encoding/json and ffjson, I had to write some code to handle that myself and add a type-tag member to my struct, which wasn't a deal-breaker, but if I could avoid that, I would like that very much.
AYBABTME 6y ago
Very interesting read in the README. It's a bit scary, so I'm not sure I'd use this in a real program, but I learned some cool stuff reading the description!
- goccy 6y ago
  Thank you for your feedback! In fact, it's understandable to be scared because it uses a lot of uncommon techniques. By growing this project, I would like to be able to use advanced techniques with peace of mind.
mbertschler 6y ago
Great job. I starred this repository because of the interesting optimization techniques, could come in handy in my own projects.
- goccy 6y ago
  Thank you so much !
kryps 6y ago
How is the performance compared with simdjson (https://github.com/simdjson/simdjson https://github.com/simdjson/simdjson)?
- goccy 6y ago
  I tried to compare it with simdjson-go, but the interface of this library is very different from other libraries and it was difficult to compare. However, I think simdjson-go has the best decoder performance.
  - peterohler 6y ago
    Not according to this comparison https://github.com/ohler55/compare-go-json https://github.com/ohler55/compare-go-json.
chmike 6y ago
Could you please reference it on awesome go and publish a reference to it on go-nuts ? I would be interested to see feedback from go experts.
kevincox 6y ago
I find the circles, crosses and triangles on https://github.com/goccy/go-json#json-library-comparison https://github.com/goccy/go-json#json-library-comparison hard to comprehend. I guess that circle means "yes", cross is "no" and triangle is "partial" or something. However that is really just based on context. (I guessed the symbols from what I thought the data should be) I would suggest using words like "yes", "no" and "partial". They are short and clear. If you really want to do symbols I recommend using for yes, and still spelling out whatever triangle is supposed to mean.
- idobe 6y ago
  The author looks to be Japanese, so that's probably why. In Japan, grade school teachers often use the symbols like so: ○ = correct × = incorrect △ = partially incorrect I'd prefer the yes, no, partial though as well.
  - kevincox 6y ago
    Ah, thanks for explaining that. However yes, for an English README it probably makes sense to stick to words to clear any confusion.
    - goccy 6y ago
      I fixed it ! Thanks !
- mst 6y ago
  My first thought was "yes", "no" and "ish" since the last one is a similar length to the first two. My second thought was that that was probably just going to move the confusion to non-native english speakers. Given a math background, ~~ for 'ish' is also tempting ... but while writing this out I've circled back around to "I can't think of anything better than partial".
konart 6y ago
Great job! >json-iterator/go ... it hasn't been supported for a long time. I wouldn't say so. The latest release was last summer and the latest commit was last november.
cfors 6y ago
The README is a treasure trove of information on go internals, thank you for writing all of that out. Fascinating to read what looks like your your thought process as you built this library, I always sort of thought README's should sometimes be a "diary of design decisions" That being said, I'm not really at a point where json decoding is a bottleneck in any of the code I write, but will keep this in mind. Great job!
- goccy 6y ago
  Thank you for your nice feedback !
mkarlsch 6y ago
That looks great, especially some of the unique optimisations. I gave it a nonscientific test run with a set of 1k different 2-4kb JSON encoded messages that we saw in our day to day traffic using the default Go benchmark library. Compared to easyjson (generated parser) goccy/go-json is unfortunately 20-25% slower and allocated four times the number of bytes. Any chance that you will implement code generation like easyjson?
- jdknezek 6y ago
  Are you unmarshaling to interface{}? Have you tried commenting out the easyjson json.Un/Marshaler implementations? By default they wrap the easyjson code and add reflection overhead. I had the same results when testing with a project that is already using easyjson, but after commenting out easyjson's json.Un/Marshaler impls I am seeing much improved performance using goccy/go-json.
  - mkarlsch 6y ago
    Indeed. Thanks for the hint! While the allocations are still 2.5x, not by accident calling into easyjson makes goccy/go-json perform faster than easyjson with generated code by ~10%. That is some impressive result.
    - goccy 6y ago
      Thank you for your report. It's an interesting result. If you have code that can be reproduced, I will try to optimize it to be faster than easyjson.
peterohler 6y ago
Have you compare it to OjG (https://github.com/ohler55/ojg https://github.com/ohler55/ojg)? You could also add your parser to https://github.com/ohler55/compare-go-json https://github.com/ohler55/compare-go-json.

119 points by goccy 6y ago

I develop a super fast JSON library ( https://github.com/goccy/go-json https://github.com/goccy/go-json ) . It is drop-in replacement of encoding/json. This library is available by simply replacing the import statement. It's more compatible and much faster than json-iterator/go. Please give me your feedback ! Thanks !

krenel 6y ago
Hi mate, very interesting library. Keep it up. Improvement suggestion: While reading the README, the JSON library comparison [1] it's confusing, at least to me. It uses symbols like: ○, and △ which are not obvious. I would rather use "yes/no" or the classic check/x (unicode chars, HN does not support them here) if you want to use symbols. I don't know what "△" even means, so I cannot suggest an alternative. Just my 2 cents. [1]: https://github.com/goccy/go-json#json-library-comparison https://github.com/goccy/go-json#json-library-comparison Edit: This 2021 I have the objective to contribute more to OS projects. Is it OKay if I submit a PR with the change? Thanks!
- YesThatTom2 6y ago
  Another suggestion in the chart: it isn't obvious which line is about your project because it doesnt list the `goccy/` part. I would add that and move it to the top, out it and the standard in bold.
- goccy 6y ago
  Yes, Of course !! Welcome !!
0x008 6y ago
As a Non-Gopher, is the stdlib so slow?
- goccy 6y ago
  I think it depends on the application. In many cases, encoding/json is fast enough and stable, but I think you can try to use a faster library if JSON encoding / decoding is performance critical on your application.
  - 0x008 6y ago
    Or you could try something like Apache Arrow or Flexbuffers?
    - jerf 6y ago
      If you write off the "JSON", sure, there's a bajillion options. But that's not always possible or desirable. None of those bajillion options have the combination of support and mindshare that JSON has obtained, for better or worse.
- fredrikholm 6y ago
  We're talking differences of a thousandth of a millisecond (0.001ms) for most structs, but yes. encoding/json from stdlib makes a lot of allocations (~30) due its use of reflection, which is notoriously slow.
krylon 6y ago
How does it handle de-serializing to structs with interface members? With encoding/json and ffjson, I had to write some code to handle that myself and add a type-tag member to my struct, which wasn't a deal-breaker, but if I could avoid that, I would like that very much.

AYBABTME 6y ago

Very interesting read in the README. It's a bit scary, so I'm not sure I'd use this in a real program, but I learned some cool stuff reading the description!

goccy 6y ago
Thank you for your feedback! In fact, it's understandable to be scared because it uses a lot of uncommon techniques. By growing this project, I would like to be able to use advanced techniques with peace of mind.

mbertschler 6y ago

Great job. I starred this repository because of the interesting optimization techniques, could come in handy in my own projects.

goccy 6y ago
Thank you so much !

kryps 6y ago

How is the performance compared with simdjson (https://github.com/simdjson/simdjson https://github.com/simdjson/simdjson)?

goccy 6y ago
I tried to compare it with simdjson-go, but the interface of this library is very different from other libraries and it was difficult to compare. However, I think simdjson-go has the best decoder performance.
- peterohler 6y ago
  Not according to this comparison https://github.com/ohler55/compare-go-json https://github.com/ohler55/compare-go-json.

chmike 6y ago

Could you please reference it on awesome go and publish a reference to it on go-nuts ? I would be interested to see feedback from go experts.

kevincox 6y ago

I find the circles, crosses and triangles on https://github.com/goccy/go-json#json-library-comparison https://github.com/goccy/go-json#json-library-comparison hard to comprehend. I guess that circle means "yes", cross is "no" and triangle is "partial" or something. However that is really just based on context. (I guessed the symbols from what I thought the data should be) I would suggest using words like "yes", "no" and "partial". They are short and clear. If you really want to do symbols I recommend using for yes, and still spelling out whatever triangle is supposed to mean.

idobe 6y ago
The author looks to be Japanese, so that's probably why. In Japan, grade school teachers often use the symbols like so: ○ = correct × = incorrect △ = partially incorrect I'd prefer the yes, no, partial though as well.
- kevincox 6y ago
  Ah, thanks for explaining that. However yes, for an English README it probably makes sense to stick to words to clear any confusion.
  - goccy 6y ago
    I fixed it ! Thanks !
mst 6y ago
My first thought was "yes", "no" and "ish" since the last one is a similar length to the first two. My second thought was that that was probably just going to move the confusion to non-native english speakers. Given a math background, ~~ for 'ish' is also tempting ... but while writing this out I've circled back around to "I can't think of anything better than partial".

konart 6y ago

Great job! >json-iterator/go ... it hasn't been supported for a long time. I wouldn't say so. The latest release was last summer and the latest commit was last november.

cfors 6y ago

The README is a treasure trove of information on go internals, thank you for writing all of that out. Fascinating to read what looks like your your thought process as you built this library, I always sort of thought README's should sometimes be a "diary of design decisions" That being said, I'm not really at a point where json decoding is a bottleneck in any of the code I write, but will keep this in mind. Great job!

goccy 6y ago
Thank you for your nice feedback !

mkarlsch 6y ago

That looks great, especially some of the unique optimisations. I gave it a nonscientific test run with a set of 1k different 2-4kb JSON encoded messages that we saw in our day to day traffic using the default Go benchmark library. Compared to easyjson (generated parser) goccy/go-json is unfortunately 20-25% slower and allocated four times the number of bytes. Any chance that you will implement code generation like easyjson?

jdknezek 6y ago
Are you unmarshaling to interface{}? Have you tried commenting out the easyjson json.Un/Marshaler implementations? By default they wrap the easyjson code and add reflection overhead. I had the same results when testing with a project that is already using easyjson, but after commenting out easyjson's json.Un/Marshaler impls I am seeing much improved performance using goccy/go-json.
- mkarlsch 6y ago
  Indeed. Thanks for the hint! While the allocations are still 2.5x, not by accident calling into easyjson makes goccy/go-json perform faster than easyjson with generated code by ~10%. That is some impressive result.
  - goccy 6y ago
    Thank you for your report. It's an interesting result. If you have code that can be reproduced, I will try to optimize it to be faster than easyjson.

peterohler 6y ago

Have you compare it to OjG (https://github.com/ohler55/ojg https://github.com/ohler55/ojg)? You could also add your parser to https://github.com/ohler55/compare-go-json https://github.com/ohler55/compare-go-json.