5 ms·

Show HN: Homogenius – Packer/unpacker to reduce the size of JSON

58 points by afshinmeh 12y ago

akx 12y ago
I think this needs some sort of marker for the compressed form to make it clear it's not to be digested as is. Maybe `$HMGNS$` or something as the first key of the array?
beefsack 12y ago
Out of interest, do you have any size comparisons compared to gzipping the raw homogenous JSON text? Edit: I did a quick test by myself, using the first example but repeated for 10,000 objects: Raw: 550111 Homogenius: 150183 Raw gzipped: 1688 Homogenius gzipped: 419 Interesting to see a ~4x between raw JSON and Homogenius JSON both when compressed and not compressed.
- walrus 12y ago
  In most real-world cases, it'll look like this: gzip+Homogenius < gzip < Homogenius < original JSON gzip+Homogenius will typically be better than just gzip because Homogenius 'knows' about the structure of JSON. gzip will typically be better than Homogenius (without gzip) because it can compress the payload. Here's the size (in bytes) of the sample files given with the project: 69934 packed-homogenius.json.gz 79550 verbose.json.gz 213742 packed-homogenius.json 265177 verbose.json
  - beefsack 12y ago
    You had the same idea as me, I think I posted my edit almost exactly when you posted this! Interesting to see a similar correlation in your results in the ratio between the two non-gzipped and gzipped files.
  - _delirium 12y ago
    I get the same ordering with gzip, but it seems sensitive to compression algorithm. With bzip2 (default settings, v1.0.6), homogenius seems to actually worsen the compression ratio, at least on this one example: 52093 verbose.json.bz2 56298 packed-homogenius.json.bz2 68521 packed-homogenius.json.gz 79222 verbose.json.gz 213742 packed-homogenius.json 265177 verbose.json
    - Scaevolus 12y ago
      BWT-based compressors like bzip2 do best when their inputs have highly repetitive structure. In a JSON file with many repeated keys, the information that `"k` is usually followed by `ey_1": "` is compressed very effectively. For similar reasons, bzip2 tends to outperform gzip on executables-- it can more effectively model the conditional probability of opcode sequences. See http://mattmahoney.net/dc/dce.html#Section_55 http://mattmahoney.net/dc/dce.html#Section_55 for more discussion
- deleted 12y ago
  [deleted]
- afshinmeh 12y ago
  Actually I'm trying to create a new datatype for homogenous data types, which is prevents repeating keys that happened in JSON datatype.
- sauere 12y ago
  Semi-OT: i am amazed to see how many big sites do not have gzip/deflate compression enabled in their webservers. Especially on mobile connections this can actually make a difference.
bussiere 12y ago
You loose in human readability but you win in size. An interesting project and i think about a lot of uses for it (save bandwith, avoid timeout etc ...) thks to make it and i will use it (in go i think).
- sauere 12y ago
  Yep. Depending on the scenario, i don't think this the loss of readability is worth it. Using gzip on the webserver already cuts enough.
- grimlck 12y ago
  Except that if you don't care about human readability, you might as well use a binary format and decrease size even more
_ducky 12y ago
I think [Transit][1] by Cognitect addresses this issue as well. [1]: https://github.com/cognitect/transit-format#caching https://github.com/cognitect/transit-format#caching
- afshinmeh 12y ago
  Actually I'm trying to create a new datatype for homogenous data types, which is prevents repeating keys that happened in JSON datatype.
ah- 12y ago
So it's a schema plus interned values? Do you think it might be better on real-world data to use just one mapping of value id to value instead of one per column/key?
codehero 12y ago
I see two great things about this tool, and not exactly for what it was designed: 1) Fix bandwidth problems caused by API designer laziness. 2) Detect API designer laziness and programmer laziness. The API designer laziness merits description. Essentially, if you as an API host server are sending back tons of redundant data, you are doing a disservice to your user by not passing references to this data in the first place. The results in your API should either include an entities section or links to retrieve more data. (Note the scale of repeated data I am talking about here are subtrees, not just key/value) The programmer laziness is when an he just sends back JSON.stringify(someVeryLargeObject), instead of making sure only a minimal object is sent back. So we can detect this laziness by running this tool and looking for 4x compression gains!
- Ixiaus 12y ago
  Bandwidth issues can be solved well with msgpack too.
- Guillaume86 12y ago
  It's not always lazyness, several small connections can be considered worse than one bigger payload in some scenarii. There's no universal truth about API design, every case is different.
- AYBABTME 12y ago
  Sending too much data is often better than having to do many RPC calls. It's the same problem than normalizing/denormalizing DB schemas, except that DB engines are smart (denormalizing can hurt perf) while RPC clients aren't.
shocks 12y ago
Any plans to add this to npm?
- afshinmeh 12y ago
  Here you go https://www.npmjs.org/package/homogenius https://www.npmjs.org/package/homogenius
eli 12y ago
If size is a concern, wouldn't a binary message format make more sense?
- afshinmeh 12y ago
  Binary format will help as well in homogenius format. Please note that I'm defining a new datatype, not only a compressor.
lololololololz 12y ago
like Alan Turing lolol
stinky613 12y ago
Doing this with JSON is a neat idea. I recently learned that Excel workbooks do this using a sharedStrings.xml file in the .xlsx package.
stefan_kendall3 12y ago
No benchmarking or comparisons with gzip.
imaginenore 12y ago
Why not just use MessagePack (aka MsgPack) http://msgpack.org/ http://msgpack.org/
- dennyabraham 12y ago
  MsgPack is great, but pure JSON compaction methods like this or transit can be used directly in a browser.
  - imaginenore 12y ago
    So can MsgPack https://github.com/creationix/msgpack-js https://github.com/creationix/msgpack-js https://github.com/msgpack/msgpack-javascript https://github.com/msgpack/msgpack-javascript
placeybordeaux 12y ago
I really don't see too many use cases here. Both parties have to buy into it, then at that point you might as well use a binary protocol.
- jarin 12y ago
  For some apps, there are cases where you want to load the entire dataset and then operate over it (see pourover.js http://newsdev.github.io/pourover/ http://newsdev.github.io/pourover/ )

58 points by afshinmeh 12y ago

akx 12y ago

I think this needs some sort of marker for the compressed form to make it clear it's not to be digested as is. Maybe `$HMGNS$` or something as the first key of the array?

beefsack 12y ago

Out of interest, do you have any size comparisons compared to gzipping the raw homogenous JSON text? Edit: I did a quick test by myself, using the first example but repeated for 10,000 objects: Raw: 550111 Homogenius: 150183 Raw gzipped: 1688 Homogenius gzipped: 419 Interesting to see a ~4x between raw JSON and Homogenius JSON both when compressed and not compressed.

walrus 12y ago
In most real-world cases, it'll look like this: gzip+Homogenius < gzip < Homogenius < original JSON gzip+Homogenius will typically be better than just gzip because Homogenius 'knows' about the structure of JSON. gzip will typically be better than Homogenius (without gzip) because it can compress the payload. Here's the size (in bytes) of the sample files given with the project: 69934 packed-homogenius.json.gz 79550 verbose.json.gz 213742 packed-homogenius.json 265177 verbose.json
- beefsack 12y ago
  You had the same idea as me, I think I posted my edit almost exactly when you posted this! Interesting to see a similar correlation in your results in the ratio between the two non-gzipped and gzipped files.
- _delirium 12y ago
  I get the same ordering with gzip, but it seems sensitive to compression algorithm. With bzip2 (default settings, v1.0.6), homogenius seems to actually worsen the compression ratio, at least on this one example: 52093 verbose.json.bz2 56298 packed-homogenius.json.bz2 68521 packed-homogenius.json.gz 79222 verbose.json.gz 213742 packed-homogenius.json 265177 verbose.json
  - Scaevolus 12y ago
    BWT-based compressors like bzip2 do best when their inputs have highly repetitive structure. In a JSON file with many repeated keys, the information that `"k` is usually followed by `ey_1": "` is compressed very effectively. For similar reasons, bzip2 tends to outperform gzip on executables-- it can more effectively model the conditional probability of opcode sequences. See http://mattmahoney.net/dc/dce.html#Section_55 http://mattmahoney.net/dc/dce.html#Section_55 for more discussion
deleted 12y ago
[deleted]
afshinmeh 12y ago
Actually I'm trying to create a new datatype for homogenous data types, which is prevents repeating keys that happened in JSON datatype.
sauere 12y ago
Semi-OT: i am amazed to see how many big sites do not have gzip/deflate compression enabled in their webservers. Especially on mobile connections this can actually make a difference.

bussiere 12y ago

You loose in human readability but you win in size. An interesting project and i think about a lot of uses for it (save bandwith, avoid timeout etc ...) thks to make it and i will use it (in go i think).

sauere 12y ago
Yep. Depending on the scenario, i don't think this the loss of readability is worth it. Using gzip on the webserver already cuts enough.
grimlck 12y ago
Except that if you don't care about human readability, you might as well use a binary format and decrease size even more

_ducky 12y ago

I think [Transit][1] by Cognitect addresses this issue as well. [1]: https://github.com/cognitect/transit-format#caching https://github.com/cognitect/transit-format#caching

afshinmeh 12y ago
Actually I'm trying to create a new datatype for homogenous data types, which is prevents repeating keys that happened in JSON datatype.

ah- 12y ago

So it's a schema plus interned values? Do you think it might be better on real-world data to use just one mapping of value id to value instead of one per column/key?

codehero 12y ago

I see two great things about this tool, and not exactly for what it was designed: 1) Fix bandwidth problems caused by API designer laziness. 2) Detect API designer laziness and programmer laziness. The API designer laziness merits description. Essentially, if you as an API host server are sending back tons of redundant data, you are doing a disservice to your user by not passing references to this data in the first place. The results in your API should either include an entities section or links to retrieve more data. (Note the scale of repeated data I am talking about here are subtrees, not just key/value) The programmer laziness is when an he just sends back JSON.stringify(someVeryLargeObject), instead of making sure only a minimal object is sent back. So we can detect this laziness by running this tool and looking for 4x compression gains!

Ixiaus 12y ago
Bandwidth issues can be solved well with msgpack too.
Guillaume86 12y ago
It's not always lazyness, several small connections can be considered worse than one bigger payload in some scenarii. There's no universal truth about API design, every case is different.
AYBABTME 12y ago
Sending too much data is often better than having to do many RPC calls. It's the same problem than normalizing/denormalizing DB schemas, except that DB engines are smart (denormalizing can hurt perf) while RPC clients aren't.

shocks 12y ago

Any plans to add this to npm?

afshinmeh 12y ago
Here you go https://www.npmjs.org/package/homogenius https://www.npmjs.org/package/homogenius

eli 12y ago

If size is a concern, wouldn't a binary message format make more sense?

afshinmeh 12y ago
Binary format will help as well in homogenius format. Please note that I'm defining a new datatype, not only a compressor.

lololololololz 12y ago

like Alan Turing lolol

stinky613 12y ago

Doing this with JSON is a neat idea. I recently learned that Excel workbooks do this using a sharedStrings.xml file in the .xlsx package.

stefan_kendall3 12y ago

No benchmarking or comparisons with gzip.

imaginenore 12y ago

Why not just use MessagePack (aka MsgPack) http://msgpack.org/ http://msgpack.org/

dennyabraham 12y ago
MsgPack is great, but pure JSON compaction methods like this or transit can be used directly in a browser.
- imaginenore 12y ago
  So can MsgPack https://github.com/creationix/msgpack-js https://github.com/creationix/msgpack-js https://github.com/msgpack/msgpack-javascript https://github.com/msgpack/msgpack-javascript

placeybordeaux 12y ago

I really don't see too many use cases here. Both parties have to buy into it, then at that point you might as well use a binary protocol.

jarin 12y ago
For some apps, there are cases where you want to load the entire dataset and then operate over it (see pourover.js http://newsdev.github.io/pourover/ http://newsdev.github.io/pourover/ )