3 ms·
In this case, a 64-bit little-endian unsigned integer Ought to be enough for anyone. [1] Using variable-length "micro-compression" schemes like these merges co
by xjay 3y ago
In this case, a 64-bit little-endian unsigned integer Ought to be enough for anyone. [1]
Using variable-length "micro-compression" schemes like these merges concerns [2], add processing overhead, and lose out on the simplicity and efficiency of working on the data directly in memory, maybe even reusing the same memory/cache line after checking identifiers, etc.
The 7-bit VLQ scheme [3] makes sense for MIDI sequences (1980s), as they were stored on MIDI hardware with limited memory, and had basic processing needs.
[1] https://quoteinvestigator.com/2011/09/08/640k-enough/ https://quoteinvestigator.com/2011/09/08/640k-enough/
[2] https://en.wikipedia.org/wiki/Separation_of_concerns https://en.wikipedia.org/wiki/Separation_of_concerns
[3] https://en.wikipedia.org/wiki/Variable-length_quantity https://en.wikipedia.org/wiki/Variable-length_quantity
- monocasa 3y agoEh, I don't think the .wasm file is intended to be an in memory format. For an on wire/storage format, variable length integers like protobuf uses is a perfectly valid choice.
- dataangel 3y agoIt's actually the protobuf design mistake that most adversely affects decoding performance. The original creator who went on to make Cap'N Proto makes the excellent point that it's better to leave the extra bytes and then just slap a compressor that only works on runs of 0 bytes on top. Faster for the same savings. There are also newer schemes from Daniel Lemire and others more amenable to SIMD decoding.
- monocasa 3y agoSure, but that decoding speed penalty doesn't stop protobuf (and similar encodings using leb128 like thrift) from being some of the most used binary encodings on the planet, even in relatively low latency spaces. Additionally, Cap'N Proto RLE encodes the bytestream, which is pretty similar to the overhead of LEB128 integers. And on top of all of that, .wasm isn't intended to be an IPC format, so the fact that it's choices are typical of that space (despite not being leading edge) reflect very well for the space it's intended to be in.