3 ms·

We have a great plethora of binary serialization libraries now, but I've noticed none of them offer the following: * Specification of the number of bits I want

by shdh 3y ago

We have a great plethora of binary serialization libraries now, but I've noticed none of them offer the following: * Specification of the number of bits I want to cap out a field at during serialization, ie: `int` that only uses 3 bits. * Delta encoding for serialization and deserialization, this would further decrease the size of each message if there is an older message that I can use as the initial message to delta encode/decode from.

jeffbee 3y ago
> `int` that only uses 3 bits. CBOR approximates this, since it has several different widths for integers. > an older message that I can use as the initial message to delta encode/decode from. General-purpose compression on the encoded stream would do something toward this goal, but some protocol buffers library implementations offer merge functions. The question is what semantics of "merge" you expect. For repeated fields do you want to append or clobber?
- Dylan16807 3y ago
  CBOR has different integer widths, yes, but 1/2/3/5/9 bytes is not particularly flexible. And if someone is asking for exact bit sizes and deltas, they probably don't want a format that embeds every key in every message.
no_circuit 3y ago
Take a look at FAST protocol [1]. It has been around for a while. Was created for market/trading data. There appears to be some open source implementations, but I don't think in general they'd be maintained well since trading is, well, secretive. [1] https://en.wikipedia.org/wiki/FAST_protocol https://en.wikipedia.org/wiki/FAST_protocol
IshKebab 3y ago
Most formats use varints, so you can't have a 3-bit int but they will store a 64-bit int in one byte if it fits. Going to smaller than a byte isn't worth the extra complexity and slowness. If you're that space sensitive you need to add proper compression. By delta compression you mean across messages? Yeah I've never seen that but it's hard to imagine a scenario where it would be useful and worth the insane complexity.
wichert 3y ago
zserio [1] has the former at least. It isn't intended for the same use cases as protobuf/capnproto/flatbutter though; in particular it has no backward or forwards compatibility. But it's great for situations where you know exactly what software is used on both ends and you need small data and fast en-/decoding. [1] http://zserio.org/doc/ZserioLanguageOverview.html#bit-field-types http://zserio.org/doc/ZserioLanguageOverview.html#bit-field-...
wiml 3y ago
One thing I liked about Ada, the small amount I used it, is it has actual subtypes: you could define a variable as an integer within a specific range, and the compiler would (presumably) choose an appropriate underlying storage type for it.

by shdh 3y ago

jeffbee 3y ago

> `int` that only uses 3 bits. CBOR approximates this, since it has several different widths for integers. > an older message that I can use as the initial message to delta encode/decode from. General-purpose compression on the encoded stream would do something toward this goal, but some protocol buffers library implementations offer merge functions. The question is what semantics of "merge" you expect. For repeated fields do you want to append or clobber?

Dylan16807 3y ago
CBOR has different integer widths, yes, but 1/2/3/5/9 bytes is not particularly flexible. And if someone is asking for exact bit sizes and deltas, they probably don't want a format that embeds every key in every message.

no_circuit 3y ago

Take a look at FAST protocol [1]. It has been around for a while. Was created for market/trading data. There appears to be some open source implementations, but I don't think in general they'd be maintained well since trading is, well, secretive. [1] https://en.wikipedia.org/wiki/FAST_protocol https://en.wikipedia.org/wiki/FAST_protocol

IshKebab 3y ago

Most formats use varints, so you can't have a 3-bit int but they will store a 64-bit int in one byte if it fits. Going to smaller than a byte isn't worth the extra complexity and slowness. If you're that space sensitive you need to add proper compression. By delta compression you mean across messages? Yeah I've never seen that but it's hard to imagine a scenario where it would be useful and worth the insane complexity.

wichert 3y ago

zserio [1] has the former at least. It isn't intended for the same use cases as protobuf/capnproto/flatbutter though; in particular it has no backward or forwards compatibility. But it's great for situations where you know exactly what software is used on both ends and you need small data and fast en-/decoding. [1] http://zserio.org/doc/ZserioLanguageOverview.html#bit-field-types http://zserio.org/doc/ZserioLanguageOverview.html#bit-field-...

wiml 3y ago

One thing I liked about Ada, the small amount I used it, is it has actual subtypes: you could define a variable as an integer within a specific range, and the compiler would (presumably) choose an appropriate underlying storage type for it.