15 ms·
This actually does. I used Python gzip to see how the minified would compress and came up with 157 bytes to JCOF's 134. Using the same mechanism to gzip the JCO
by Beltiras 4y ago
This actually does. I used Python gzip to see how the minified would compress and came up with 157 bytes to JCOF's 134. Using the same mechanism to gzip the JCOF saves only 4 bytes. I'm actually rather impressed.
- tmp_anon_22 4y agoI think people would reject your samples because the data size is too small. Production JSON comes in all shapes and sizes and situations where you care about performance usually indicates much larger payloads.
- Beltiras 4y agoIf your egress is paramount, you should of course benchmark everything to shreds. Then pick what fits.
- zimpenfish 4y agoIt might on small files but it doesn't on (some) bigger ones. e.g. I dumped a Minecraft region file into JSON giving 85M. `jcof` turns this into 66M. `gzip` on the original json gives 14M for `-1` and 11M for `-9`. `zstd` also gives 11M. But because the Minecraft JSON is highly redundant, I also tried it on 84M of fake social graph data. `jcof` gave 44M, `gzip -9` gave 19M and `zstd` gave 18M. In summary, neat idea, but you're maybe better off with `gzip` or `zstd` in the real world.
- Beltiras 4y agoJCOF and gzip are not mutex.
- masklinn 4y agoNo, but when you get ~95% of the benefit (per other people's data) while keeping all the flexibility, compatibility with everything under the sun, and native-performance parsing everywhere, the supposed improvement looks a lot less like one.
- Beltiras 4y agoA sibling has the data. For that case it's 10% of egress which can be a healthy chunk of change on your bill. For all such use cases you should of course benchmark a lot and then pick what fits. I think this tool is a nice one to have in the toolbox. The low-rate files seem to be close to optimal.
- zimpenfish 4y ago> A sibling has the data. And now has updated data showing that sorting your keys lets `lzma` beat `jcof|lzma` at every level. The egress bill has been reduced! All rejoice!
- zimpenfish 4y agoEDIT: Nope, I was wrong, I was doing `lzma` against `jcof|zstd`. `jcof|lzma` is still sneaking ~1M below `lzma` at all levels.
- KronisLV 4y ago> For that case it's 10% of egress which can be a healthy chunk of change on your bill. For all such use cases you should of course benchmark a lot and then pick what fits. Provided that you don't run into the issue of one end only talking in language X, which does not have JCOF support due to there not being any libraries for it yet. In the GitHub repo, it seems like there is only the JavaScript reference implementation for now.
- Beltiras 4y agoThe design documents are really thorough. I don't think I would be more than a day making a serializer in Python for example. Lack of libraries just means an interested developer needs to tend to the matter and solve the issue.
- zimpenfish 4y agoSee https://news.ycombinator.com/item?id=32111239 https://news.ycombinator.com/item?id=32111239 where I tested gzip vs jcof vs jcof+gzip (tl;dr: you save 2M at the cost of an extra CPU intensive process)
- luhn 4y agoYou just blew my mind that “mutex” is a portmanteau of “mutually exclusive.” How have I been programming for 20 years and never realized this?
- Beltiras 4y agoPerhaps you haven't had to program concurrencies or thought much about bitwise operators. It's not very frequent anymore. Used to be.