3 ms·
Serialization formats that dedupe and perform other fancy processing trade increased CPU time for decreased memory usage. People who care about CPU time will wa
by KerrAvon 4y ago
Serialization formats that dedupe and perform other fancy processing trade increased CPU time for decreased memory usage. People who care about CPU time will want to measure the CPU time hit as well as the memory savings.
This format is also not interoperable without a decoding library, which kind of invalidates the comparison to JSON. If you're going to do this, why not just go full binary and save even more memory?
- throwawaymaths 4y agoYou're decoding JSON in all cases. Usually network dominates over cpu. Sometimes it's useful to have human readable serializations, especially if you expect to operate in anger.
- lijogdfljk 4y agoDepends on your needs. When i'm really concerned about deserialization time, i try to skip it entirely. Partial|Total zero copy deserialization is the thing to reach for. Various libs exist, https://rkyv.org/ https://rkyv.org/ is what i've been toying with recently. For those unfamiliar, here is the interesting bit: > rkyv implements total zero-copy deserialization, which guarantees that no data is copied during deserialization and no work is done to deserialize data. It achieves this by structuring its encoded representation so that it is the same as the in-memory representation of the source type.
- conradev 4y agoThank you for linking to this! It is super cool that it serializes hash tables and b-trees because that is where serde’s zero copy parsing ends It seems perfect for my use case, which is putting stuff in a chunk of memory shared between processes