4 ms·

I implemented a streaming deserializer for MessagePack data in C for small microcontrollers. It is quite small and not complete. Then I tested it with test dat

I implemented a streaming deserializer for MessagePack data in C for small microcontrollers. It is quite small and not complete. Then I tested it with test data streams generated by the reference C++ implementation and Python implementation. It's actually kind of a pain, because the C++ serializer generates a variety of different data types depending on the values you are packing, not the type of the values you are packing. Let's say I encode a uint32_t field. The stream might get a uint8, a uint16, a uint32, or even an int32 (for reasons that completely elude me). Also, C++ strings come out as the 'ext' type while Python strings come out as the 'string' type, so I have to accommodate both, even though they are both basically byte strings. So, I want to tell the deserializer what kind of output field I'm expecting, for each struct member or array, and then look in the data stream to see if there is a data object there that came from the same type. But this is impossible, so the per-type decode functions have to be quite complicated to handle a variety of types. So - it works but I can't do much on the deserialization side to verify that the data I'm unpacking really matches what was encoded. I can only detect very broken cases, like when I'm unpacking a uint32_t and in the stream there is an int32_t with a negative value. I guess this is mostly done for optimization, but two things would make it a lot better: - if the spec actually specified how data types in different languages were allowed to be encoded - if the encoded data contained _two_ type fields, one indicating the original source data type and another indicating the type it was encoded into. Basically the spec is just way too "loose" to make it usable for the use case I'm trying to use it for, which is to easily generate data that is sent to a micro and stored in EEPROM, then deserialized out of EEPROM later. That's probably not very close to a use case the original designer had in mind. But I haven't found anything that works better (less decode logic).

ludocode 7y ago
The spec doesn't really make this clear, but reifying the packed encoding types is not, I think, how MessagePack was intended to be used. At least it's not how it's implemented in MessagePack libraries. As far as I know they pretty much all use the most efficient representation for all values, so the original data type is always lost. You generally shouldn't worry about the low-level type that a value was encoded into in the MessagePack stream. It's dynamically typed, so you should just care about values. When encoding you should allow the encoder to use the most efficient representation, and when decoding you should be able to tell your MessagePack parser the integer width you want instead of caring about the original type or how it was encoded. It should then accept any packed integer type as long as the value is in range. This is how my MessagePack implementation works as well. If you expect to receive an integer that fits in, say, `uint16_t`, you can call `mpack_expect_u16()` or `mpack_node_u16()`, and it will allow any integer representation as long as the value is in range. It sounds like this is where you were going with your implementation as well, so this may not be comforting because it's not what you want, but it is at least the correct way to understand the format. I've talked about this pretty extensively and wrote up a protocol clarifications document that explains a bit more about how and why MessagePack libraries discard integer width and signedness: https://github.com/ludocode/mpack/issues/35 https://github.com/ludocode/mpack/issues/35 https://github.com/ludocode/mpack/blob/develop/docs/protocol.md https://github.com/ludocode/mpack/blob/develop/docs/protocol... If you really want things like original integer width represented in the format, ultimately you're going to want to use a different format, probably one that is non-dynamic and uses schemas. As far as the string vs ext, you may have meant string vs bin; there was a format change a while back that separated string and bin types and not all MessagePack libraries have adapted to that. Many libraries (including mine) support a compatibility mode so they will use only compatible string representations.

ludocode 7y ago

The spec doesn't really make this clear, but reifying the packed encoding types is not, I think, how MessagePack was intended to be used. At least it's not how it's implemented in MessagePack libraries. As far as I know they pretty much all use the most efficient representation for all values, so the original data type is always lost. You generally shouldn't worry about the low-level type that a value was encoded into in the MessagePack stream. It's dynamically typed, so you should just care about values. When encoding you should allow the encoder to use the most efficient representation, and when decoding you should be able to tell your MessagePack parser the integer width you want instead of caring about the original type or how it was encoded. It should then accept any packed integer type as long as the value is in range. This is how my MessagePack implementation works as well. If you expect to receive an integer that fits in, say, `uint16_t`, you can call `mpack_expect_u16()` or `mpack_node_u16()`, and it will allow any integer representation as long as the value is in range. It sounds like this is where you were going with your implementation as well, so this may not be comforting because it's not what you want, but it is at least the correct way to understand the format. I've talked about this pretty extensively and wrote up a protocol clarifications document that explains a bit more about how and why MessagePack libraries discard integer width and signedness: https://github.com/ludocode/mpack/issues/35 https://github.com/ludocode/mpack/issues/35 https://github.com/ludocode/mpack/blob/develop/docs/protocol.md https://github.com/ludocode/mpack/blob/develop/docs/protocol... If you really want things like original integer width represented in the format, ultimately you're going to want to use a different format, probably one that is non-dynamic and uses schemas. As far as the string vs ext, you may have meant string vs bin; there was a format change a while back that separated string and bin types and not all MessagePack libraries have adapted to that. Many libraries (including mine) support a compatibility mode so they will use only compatible string representations.