5 ms·

People keep reinventing JSON trying to beat gzip, and it never beats the simplicity of gzipping JSON...

Beltiras 4y ago
This actually does. I used Python gzip to see how the minified would compress and came up with 157 bytes to JCOF's 134. Using the same mechanism to gzip the JCOF saves only 4 bytes. I'm actually rather impressed.
- tmp_anon_22 4y ago
  I think people would reject your samples because the data size is too small. Production JSON comes in all shapes and sizes and situations where you care about performance usually indicates much larger payloads.
  - Beltiras 4y ago
    If your egress is paramount, you should of course benchmark everything to shreds. Then pick what fits.
- zimpenfish 4y ago
  It might on small files but it doesn't on (some) bigger ones. e.g. I dumped a Minecraft region file into JSON giving 85M. `jcof` turns this into 66M. `gzip` on the original json gives 14M for `-1` and 11M for `-9`. `zstd` also gives 11M. But because the Minecraft JSON is highly redundant, I also tried it on 84M of fake social graph data. `jcof` gave 44M, `gzip -9` gave 19M and `zstd` gave 18M. In summary, neat idea, but you're maybe better off with `gzip` or `zstd` in the real world.
  - Beltiras 4y ago
    JCOF and gzip are not mutex.
    - masklinn 4y ago
      No, but when you get ~95% of the benefit (per other people's data) while keeping all the flexibility, compatibility with everything under the sun, and native-performance parsing everywhere, the supposed improvement looks a lot less like one.
      - Beltiras 4y ago
        A sibling has the data. For that case it's 10% of egress which can be a healthy chunk of change on your bill. For all such use cases you should of course benchmark a lot and then pick what fits. I think this tool is a nice one to have in the toolbox. The low-rate files seem to be close to optimal.
        zimpenfish 4y ago
        > A sibling has the data. And now has updated data showing that sorting your keys lets `lzma` beat `jcof|lzma` at every level. The egress bill has been reduced! All rejoice!
        zimpenfish 4y ago
        EDIT: Nope, I was wrong, I was doing `lzma` against `jcof|zstd`. `jcof|lzma` is still sneaking ~1M below `lzma` at all levels.
        KronisLV 4y ago
        > For that case it's 10% of egress which can be a healthy chunk of change on your bill. For all such use cases you should of course benchmark a lot and then pick what fits. Provided that you don't run into the issue of one end only talking in language X, which does not have JCOF support due to there not being any libraries for it yet. In the GitHub repo, it seems like there is only the JavaScript reference implementation for now.
        Beltiras 4y ago
        The design documents are really thorough. I don't think I would be more than a day making a serializer in Python for example. Lack of libraries just means an interested developer needs to tend to the matter and solve the issue.
xsmasher 4y ago
comets.json: JSON: 51949 bytes JCOF: 37480 bytes (0.721x) JSON ZIPED: 15178 bytes Zipped json wins again.
- conradev 4y ago
  Now I am curious – how does zipped JCOF turn out? I know that’s not the author’s intention, but I’m curious how it compresses
  - bearjaws 4y ago
    It's nearly irrelevant, given that I can gzip JSON and send it to literally anything built in the last 10+ years. It's so ubiquitous now that many web servers will just accept gzipped content without even exposing that fact to the back end servers. You can get all the benefits of highly compressed JSON and never have interacted with gzip. If it could somehow produce a significant reduction with fewer CPU cycles, there may be a really niche use case, but I don't see those often.
    - conradev 4y ago
      Oh, it is completely irrelevant! I just want a data point for my mental model of gzip performance. Specifically, I am curious if the optimizations (object tables and separator elimination) are made irrelevant by gzip's dictionary, or if they result in better compression
      - twawaaay 4y ago
        In my experience it should not make much or any difference. I had devs try to "optimise" the JSON our services were returning by doing things like abbreviating field names that were repeated very often or replacing long string constants with short integers or removing formatting, and so on. When I asked them to do actual test when the service returns gzipped output and they found out that even reducing the JSON size this way by half does not make any measurable difference on the compressed stream in most cases.
heresie-dabord 4y ago
People keep trying to re-invent a human-readable, parsable, efficient, robust format that can be validated. But "easy" wins so we keep returning to CSV and JSON. ^_^
nine_k 4y ago
Gzip may be standard and supported transparently by your http library. OTOH zstd [1] should be significantly more efficient, while being as fast or faster. [1]: https://en.wikipedia.org/wiki/Zstd https://en.wikipedia.org/wiki/Zstd

This actually does. I used Python gzip to see how the minified would compress and came up with 157 bytes to JCOF's 134. Using the same mechanism to gzip the JCOF saves only 4 bytes. I'm actually rather impressed.

tmp_anon_22 4y ago
I think people would reject your samples because the data size is too small. Production JSON comes in all shapes and sizes and situations where you care about performance usually indicates much larger payloads.
- Beltiras 4y ago
  If your egress is paramount, you should of course benchmark everything to shreds. Then pick what fits.
zimpenfish 4y ago
It might on small files but it doesn't on (some) bigger ones. e.g. I dumped a Minecraft region file into JSON giving 85M. `jcof` turns this into 66M. `gzip` on the original json gives 14M for `-1` and 11M for `-9`. `zstd` also gives 11M. But because the Minecraft JSON is highly redundant, I also tried it on 84M of fake social graph data. `jcof` gave 44M, `gzip -9` gave 19M and `zstd` gave 18M. In summary, neat idea, but you're maybe better off with `gzip` or `zstd` in the real world.
- Beltiras 4y ago
  JCOF and gzip are not mutex.
  - masklinn 4y ago
    No, but when you get ~95% of the benefit (per other people's data) while keeping all the flexibility, compatibility with everything under the sun, and native-performance parsing everywhere, the supposed improvement looks a lot less like one.
    - Beltiras 4y ago
      A sibling has the data. For that case it's 10% of egress which can be a healthy chunk of change on your bill. For all such use cases you should of course benchmark a lot and then pick what fits. I think this tool is a nice one to have in the toolbox. The low-rate files seem to be close to optimal.
      - zimpenfish 4y ago
        > A sibling has the data. And now has updated data showing that sorting your keys lets `lzma` beat `jcof|lzma` at every level. The egress bill has been reduced! All rejoice!
        zimpenfish 4y ago
        EDIT: Nope, I was wrong, I was doing `lzma` against `jcof|zstd`. `jcof|lzma` is still sneaking ~1M below `lzma` at all levels.
      - KronisLV 4y ago
        > For that case it's 10% of egress which can be a healthy chunk of change on your bill. For all such use cases you should of course benchmark a lot and then pick what fits. Provided that you don't run into the issue of one end only talking in language X, which does not have JCOF support due to there not being any libraries for it yet. In the GitHub repo, it seems like there is only the JavaScript reference implementation for now.
        Beltiras 4y ago
        The design documents are really thorough. I don't think I would be more than a day making a serializer in Python for example. Lack of libraries just means an interested developer needs to tend to the matter and solve the issue.

xsmasher 4y ago

comets.json: JSON: 51949 bytes JCOF: 37480 bytes (0.721x) JSON ZIPED: 15178 bytes Zipped json wins again.

conradev 4y ago
Now I am curious – how does zipped JCOF turn out? I know that’s not the author’s intention, but I’m curious how it compresses
- bearjaws 4y ago
  It's nearly irrelevant, given that I can gzip JSON and send it to literally anything built in the last 10+ years. It's so ubiquitous now that many web servers will just accept gzipped content without even exposing that fact to the back end servers. You can get all the benefits of highly compressed JSON and never have interacted with gzip. If it could somehow produce a significant reduction with fewer CPU cycles, there may be a really niche use case, but I don't see those often.
  - conradev 4y ago
    Oh, it is completely irrelevant! I just want a data point for my mental model of gzip performance. Specifically, I am curious if the optimizations (object tables and separator elimination) are made irrelevant by gzip's dictionary, or if they result in better compression
    - twawaaay 4y ago
      In my experience it should not make much or any difference. I had devs try to "optimise" the JSON our services were returning by doing things like abbreviating field names that were repeated very often or replacing long string constants with short integers or removing formatting, and so on. When I asked them to do actual test when the service returns gzipped output and they found out that even reducing the JSON size this way by half does not make any measurable difference on the compressed stream in most cases.

heresie-dabord 4y ago

People keep trying to re-invent a human-readable, parsable, efficient, robust format that can be validated. But "easy" wins so we keep returning to CSV and JSON. ^_^

nine_k 4y ago

Gzip may be standard and supported transparently by your http library. OTOH zstd [1] should be significantly more efficient, while being as fast or faster. [1]: https://en.wikipedia.org/wiki/Zstd https://en.wikipedia.org/wiki/Zstd