3 ms·

Parsing JSON is probably an order of magnitude more inefficient than working with any binary message formats in terms of speed and energy.

garmaine 7y ago
...why? That's not at all obvious.
- StreamBright 7y ago
  Because accessing fields in a binary format is a seek() while in a text format it is much more complicated. Try to build a mental model what is required to parse text based formats. You can have a look to results of performance testing involving these binary message formats vs json. http://zderadicka.eu/comparison-of-json-like-serializations-json-vs-ubjson-vs-messagepack-vs-cbor/ http://zderadicka.eu/comparison-of-json-like-serializations-... https://github.com/ludocode/schemaless-benchmarks#speed---desktop-pc https://github.com/ludocode/schemaless-benchmarks#speed---de... https://eng.uber.com/trip-data-squeeze/ https://eng.uber.com/trip-data-squeeze/ http://ugorji.net/blog/benchmarking-serialization-in-go http://ugorji.net/blog/benchmarking-serialization-in-go
  - garmaine 7y ago
    Sorry, I read "order of magnitude more efficient" and was confused.
    - StreamBright 7y ago
      If you take into consideration the sum of energy it takes to process an average daily dataset, then yes I think it will be an order of magnitude more efficient (in terms of energy use).
signa11 7y ago
try parsing a tcp/udp header encoded in json vs as it ‘normally’ is. see if you can do it at line rate :o)
- StreamBright 7y ago
  Why would I want to do that?? :) I was using JSON and Avro in production and it was pretty obvious that Avro just beats the shit out of JSON libraries, even though there are state of the art libraries like Boon[1]. 1. https://github.com/boonproject/boon/wiki/Boon-JSON-in-five-minutes https://github.com/boonproject/boon/wiki/Boon-JSON-in-five-m... For me it does not really matter which binary message format you are using, I think there is not a huge difference between the different libraries. There are some feature differences for sure, library quality maybe. If you are going for the most performance that is possible to achiave you could look at SBE[2] developed by performance freaks. 2. https://github.com/real-logic/simple-binary-encoding https://github.com/real-logic/simple-binary-encoding
  - signa11 7y ago
    > Why would I want to do that?? :) because you _seem_ to imply that json parsing is faster/better than binary parsing...
    - StreamBright 7y ago
      >> Parsing JSON is probably an order of magnitude more ____inefficient___ than working with any binary message formats in terms of speed and energy. Do I?
      - signa11 7y ago
        oh dear lord ! how dumb of me :o) apologies...

garmaine 7y ago

...why? That's not at all obvious.

StreamBright 7y ago
Because accessing fields in a binary format is a seek() while in a text format it is much more complicated. Try to build a mental model what is required to parse text based formats. You can have a look to results of performance testing involving these binary message formats vs json. http://zderadicka.eu/comparison-of-json-like-serializations-json-vs-ubjson-vs-messagepack-vs-cbor/ http://zderadicka.eu/comparison-of-json-like-serializations-... https://github.com/ludocode/schemaless-benchmarks#speed---desktop-pc https://github.com/ludocode/schemaless-benchmarks#speed---de... https://eng.uber.com/trip-data-squeeze/ https://eng.uber.com/trip-data-squeeze/ http://ugorji.net/blog/benchmarking-serialization-in-go http://ugorji.net/blog/benchmarking-serialization-in-go
- garmaine 7y ago
  Sorry, I read "order of magnitude more efficient" and was confused.
  - StreamBright 7y ago
    If you take into consideration the sum of energy it takes to process an average daily dataset, then yes I think it will be an order of magnitude more efficient (in terms of energy use).

signa11 7y ago

try parsing a tcp/udp header encoded in json vs as it ‘normally’ is. see if you can do it at line rate :o)

StreamBright 7y ago
Why would I want to do that?? :) I was using JSON and Avro in production and it was pretty obvious that Avro just beats the shit out of JSON libraries, even though there are state of the art libraries like Boon[1]. 1. https://github.com/boonproject/boon/wiki/Boon-JSON-in-five-minutes https://github.com/boonproject/boon/wiki/Boon-JSON-in-five-m... For me it does not really matter which binary message format you are using, I think there is not a huge difference between the different libraries. There are some feature differences for sure, library quality maybe. If you are going for the most performance that is possible to achiave you could look at SBE[2] developed by performance freaks. 2. https://github.com/real-logic/simple-binary-encoding https://github.com/real-logic/simple-binary-encoding
- signa11 7y ago
  > Why would I want to do that?? :) because you _seem_ to imply that json parsing is faster/better than binary parsing...
  - StreamBright 7y ago
    >> Parsing JSON is probably an order of magnitude more ____inefficient___ than working with any binary message formats in terms of speed and energy. Do I?
    - signa11 7y ago
      oh dear lord ! how dumb of me :o) apologies...