6 ms·
JCOF: JSON-like Compact Object Format
- faleidel 4y agoI would like to see gzip compression added to the benchmark
- mg 4y agocurl https://raw.githubusercontent.com/mortie/jcof/main/tests/corpus/meteorites.json | gzip -9 | wc Gives me 34569 So the comparison is: JSON: 244920 bytes JCOF: 87028 bytes GZIP: 34569 bytes
- eof 4y agoTo be fair you would gzip the JCOF encoding in this example too. Author mentions gzip doesn’t work for some use cases. For use case mentioned I’d expect sqlite to be similar, at least that is the default thing I’d reach for. If for some reason sqlite wasn’t sufficient probably a custom binary encoding controlled and updated via code instead of config would be next.
- jffry 4y agometeorites.json, re-encoded via both JSON.stringify() and jcof.stringify(): json jcof jcof/json plain 244975 87083 0.355 gzip -6 35829 33152 0.925 gzip -9 34384 32875 0.956 xz -9 27864 28696 1.030 And I imagine this is close to ideal for jcof. So unless that last few % really matters, gzipped JSON is probably much better in the general case.
- zimpenfish 4y ago> To be fair you would gzip the JCOF encoding in this example too. Just tested my 84M fake social media file - `jcof` gives 44M, `gzip` gives 19M, `jcof+gzip` gives 17M. In essence, you've gained 2M for two CPU intensive procedures instead of one. Doesn't seem all that worth it?
- Beltiras 4y ago10% is a lot of your egress.
- zimpenfish 4y agoA fair point. Prompted me to check if the higher zstd levels worked any better on my 84MB fake social graph - nope - and then if LZMA was any good - yes, `lzma` at 5 or higher on the raw JSON beats `jcof | lzma` by ~2M every time. `lzma -4` beats it by ~400k. If I sort my object keys (a la `jq -S`), `lzma` beats `jcof|lzma` at every level (`gzip` never gets close, `zstd` gets closer.)
- zimpenfish 4y agoEDIT: Nope, I was wrong, I was doing `lzma` against `jcof|zstd`. `jcof|lzma` is still sneaking ~1M below `lzma` at all levels.
- KerrAvon 4y agoSerialization formats that dedupe and perform other fancy processing trade increased CPU time for decreased memory usage. People who care about CPU time will want to measure the CPU time hit as well as the memory savings. This format is also not interoperable without a decoding library, which kind of invalidates the comparison to JSON. If you're going to do this, why not just go full binary and save even more memory?
- throwawaymaths 4y agoYou're decoding JSON in all cases. Usually network dominates over cpu. Sometimes it's useful to have human readable serializations, especially if you expect to operate in anger.
- lijogdfljk 4y agoDepends on your needs. When i'm really concerned about deserialization time, i try to skip it entirely. Partial|Total zero copy deserialization is the thing to reach for. Various libs exist, https://rkyv.org/ https://rkyv.org/ is what i've been toying with recently. For those unfamiliar, here is the interesting bit: > rkyv implements total zero-copy deserialization, which guarantees that no data is copied during deserialization and no work is done to deserialize data. It achieves this by structuring its encoded representation so that it is the same as the in-memory representation of the source type.
- conradev 4y agoThank you for linking to this! It is super cool that it serializes hash tables and b-trees because that is where serde’s zero copy parsing ends It seems perfect for my use case, which is putting stuff in a chunk of memory shared between processes
- linker3000 4y agoSlight awkwardness with the spoken pronunciation.
- taddevries 4y agoGlad I'm not the only one who reads it that way.
- jsiaajdsdaa 4y agoJ cough?
- HelloNurse 4y agoJack off, I'm afraid.
- jsiaajdsdaa 4y agoInteresting, I wonder which type of English speakers would see that as their initial hunch for pronunciation.
- tgv 4y agoWhile Oscar Wilde would undoubtedly have something witty to say about that, I guess it's a matter of stress: if you read it as a word and assume stress on the (invisible) first vowel, it's almost inevitable: juh-cof.
- jsiaajdsdaa 4y agoBut we dont say juh sahn for JSON? Hence why Jcough seemed intuitive.
- tgv 4y agoLanguage is not very deterministic. An unknown word can remind you of more than one other word, which can influence the way you pronounce it, certainly in English where pronunciation is so irregular. That’s not a fact of course, just a possible factor.
- sto_hristo 4y agoThing about JSON is the ease of reading and writing it, regardless the size of the document. Compression deals with the rest.
- Someone 4y agoIt may be easy to read and write, but few libraries implement it to the letter because the format doesn’t support extremely common use cases. For those who disagree, try finding a json library that can reliably round-trip the data in a json file (i.e. ignoring spacing, tabs, and extra line breaks) Common breaking points are discrimination between integers and doubles (e.g read 0.0 and write 0) and limits to the number of digits in a number. It’s futile to try and change it now, but I think json would have been better if it had disjunct sets of integers and floats, did 64-bit ints (1) and IEEE doubles (1) as opposed to arbitrary length integers and floats, standardized serialization of time stamps and durations. Requiring parsers to keep keys in file order also might have been a good thing. (1) I know that opens a rat’s nest of users wanting unsigned 64-bit integers, signed 16-bit, 128-bit quad floats, etc, but I don’t see how not addressing that problem at all is better than picking reasonable values, event though that gives up on supporting those alternatives. Now, we have libraries making choices there, hopefullly in compatible ways.
- HelloNurse 4y agoNeither fully human-editable nor really compact, but an interesting experiment nonetheless.
- andrew_ 4y agoNevermind readable
- mywittyname 4y agoI'm still not sure wtf the leading 0, on every object is supposed to represent.
- SahAssar 4y agoI think that's representing the keys, so if you'd have multiple types of objects in an array the key table would have multiple rows and the leading number index for each object would define what the keys for that object would be.
- bearjaws 4y agoPeople keep reinventing JSON trying to beat gzip, and it never beats the simplicity of gzipping JSON...
- Beltiras 4y agoThis actually does. I used Python gzip to see how the minified would compress and came up with 157 bytes to JCOF's 134. Using the same mechanism to gzip the JCOF saves only 4 bytes. I'm actually rather impressed.
- tmp_anon_22 4y agoI think people would reject your samples because the data size is too small. Production JSON comes in all shapes and sizes and situations where you care about performance usually indicates much larger payloads.
- Beltiras 4y agoIf your egress is paramount, you should of course benchmark everything to shreds. Then pick what fits.
- zimpenfish 4y agoIt might on small files but it doesn't on (some) bigger ones. e.g. I dumped a Minecraft region file into JSON giving 85M. `jcof` turns this into 66M. `gzip` on the original json gives 14M for `-1` and 11M for `-9`. `zstd` also gives 11M. But because the Minecraft JSON is highly redundant, I also tried it on 84M of fake social graph data. `jcof` gave 44M, `gzip -9` gave 19M and `zstd` gave 18M. In summary, neat idea, but you're maybe better off with `gzip` or `zstd` in the real world.
- Beltiras 4y agoJCOF and gzip are not mutex.
- 4y ago
- indymike 4y agoJCOF is not usable in many of the use cases where JSON is. Here is why: * JSON is really schemaless so I don't have to assume all objects are shaped the same or that they are even the same kind of object. This allows for streaming serialization, and does not require the data structure to be known or to introspect data to create the heading lines. * Nested objects look to be difficult, especially if the schema is not known. * JSON is very human friendly, JCOF is not. I do think the format is better than CSV, TSV and other delimited flat record formats. JCOF appears to be able to handle objects that are not tabular, which CSV just can't really do. A replacement for JSON this is not, and storage efficiency isn't really relevant because of the difference in purpose.
- nine_k 4y agoJCOF is just not radical enough. It should stop pretending to remain human-readable, go binary, and become a variant of protocol buffers, thrift, etc. BTW being schemaless is a boon during initial hacking things together, and an impediment when operating and developing the system further down the line. It's the data equivalent of dynamic typing vs static typing, interpreted vs compiled in code.
- indymike 4y ago> It should stop pretending to remain human-readable, go binary, and become a variant of protocol buffers, thrift, etc. There are lots of ways standard ways to do serialization to binary formats. https://en.wikipedia.org/wiki/Comparison_of_data-serialization_formats#Comparison_of_binary_formats https://en.wikipedia.org/wiki/Comparison_of_data-serializati... > BTW being schemaless is a boon during initial hacking things together, and an impediment when operating and developing the system further down the line. It's the data equivalent of dynamic typing vs static typing, interpreted vs compiled in code. Sounds like XML, which is a whole other discussion :-)
- mort96 4y agoHey, JCOF author here. I didn't notice this had made it to HN before now. * JCOF allows for streaming serialization if you want, you can just use the key-value object syntax and use inline strings everywhere. You won't get much size improvement over JSON when using a streaming approach, but you can even do a half-and-half approach; maybe you have a set of strings which you know you're using a lot, so you put those strings in the string table, then you do a streaming encode of the rest of your data, with string references to the string table. You could even do the same with object shapes, where you hard-code a couple of object shapes you know you use a lot, and use the JSON-style key-value syntax for other kinds of objects. Basically, you can choose where you want to be on the efficiency vs streaming curve. * Nested objects aren't difficult; an object can go in any place where a value can go, just like in JSON. This object can be either a "shaped" object where the keys are listed in the object shape table, or it can be a "keyed" object where you write out the keys inline (as either literal strings or references to the string table). * JCOF is human-writable; you can write a document which looks a whole lot like JSON. This is a valid document: `;;{"people":[{"name":"Bob",Aage":36},{"name":"Alice","age":40}]}`. But optimal JCOF is mostly unreadable. So it's less human friendly than JSON, more human friendly than binary formats; it's a trade-off. I think JCOF hits enough interesting, unexplored points on enough relevant curves that it has some value. It also ends up being smaller than any of the binary formats I've tried like MessagePack and CBOR (even with the string references extension), while being text-based. It's obviously not going to replace JSON, but it's useful in some cases.
- Scaevolus 4y ago> gzipping stuff from JavaScript in the browser is inconvenient until TextEncoderStream is supported in Firefox There are a pure-JS DEFLATE implementations with reasonable performance-- I recommend fflate.
- peanut_worm 4y agoJSON is supposed to be human readable this doesn’t even seem like it would address the same issues.
- m463 4y agoI'm always conflicted. They're improvements, they're experiments, they're an attempt to move forward. Somebody cares! but Randall Munroe said it pretty well: https://xkcd.com/927/ https://xkcd.com/927/
- talideon 4y agoJSON isn't mean to be efficient, but to be a good compromise between being human readable/writable, easily generated/parsed by a machine, and familiar. CBOR and BSON make more sense, to be frank, for any use case I could think of using this in.
- xsmasher 4y agoI don't think BSON de-dupes the keys in the manner that JCOF does? But it has other nice properties. In a game I worked on we used a BSON library that didn't really "parse" the data when you loaded a file; it just kept a pointer to the original binary data and let you perform queries against it. We used this to keep our memory footprint low; instead of loading BSON files we would memmap them (getting a pointer that can be used to read the data right off the disk) and passed that to the BSON library. Bang, we could map a 10MB file and it wouldn't use 10MB of system memory; it would just swap portions in and out as they were used. This has performance tradeoffs but for us (on mobile, where there is no swap space) the memory savings was worth it.
- dathinab 4y agono offense, but this seems pointless tbh. If I need a more compact no longer at all human readable format I can use a compact binary format. If not I can use JSON which has wide spread support. or just use a compression (after which the difference between JSON and JCOF should be negligible)
- mort96 4y agoI'm curious, which more compact format would you use? Because I have looked a bit and been unable to find one, both MessgePack and CBOR produce pretty large objects. CBOR with the string reference extension comes closer, but it's still bigger, and string references don't seem very well supported in the CBOR ecosystem. Do you mean you'd use a format with a schema?
- LAC-Tech 4y agoThe comments here are really negative. I think this is a great idea. JSON is incredibly wasteful and any improvement is welcome. And yes, you can gzip JSON. still wastes a lot of space when you actually need it in a format where you might want to read individual fields.
- smitty1e 4y agoIf a specific use-case can benefit from it, fine. I love me some jq, not moving much data as JSON, and so this amounts to an interesting exercise.