6 ms·

MessagePack should be replaced with http://cbor.io http://cbor.io everywhere, as CBOR is an actual IETF RFC. Even if that kills the naming pun opportunity.

tptacek 10y ago
Nobody should really care whether something is or isn't an IETF standard. The cart has been dragging the horses with internet standardization for well over a decade now.
- dchest 10y ago
  Indeed. MessagePack should be replaced with CBOR in new protocols, as CBOR is a very nicely designed format.
  - tptacek 10y ago
    Isn't CBOR just IETF's bikeshedded MessagePack? I'm not trying to be snarky: I don't have strong opinions about encoding formats. "CBOR is better than MessagePack because it's better designed" is a good argument, I think --- but: you'd want it to be so much better that the difference is material. "CBOR is better than MessagePack because it's standardized" is not, I think, a persuasive argument.
    - dchest 10y ago
      Yeah, I skipped all the drama, read the spec and implemented an encoder/decoder. CBOR is just how MessagePack-like format should have been done from the beginning: it's technically superior in a sense that it's neat and simple, replacing many specialized rules with one generalization. I agree that IETF standardization isn't a good argument (well, for some it is), that's why I replied with a better argument :) But, seriously, I won't say that everyone should replace MessagePack everywhere with CBOR, both work fine (as long as you use the latest version of MessagePack, with binary/string distinction).
      - theamk 10y ago
        Looking at CBOR spec, I see that it is just more complex. Two ways to encode lists/strings -- indefinite-length and strings. 16-bit floats in the main spec. Separation of "null" and "undefined" values. And tags, with which define things like decimal fractions, bigints, and regexps. The tags are the worst, actually. Sure, spec says "decoders do not need to understand tags", but this is not really the case. For example, if someone has floating point numbers and worries about precision loss, they can store the value as _decimal fraction_ (per section 2.4.3). This means that your decoders have to support both tagging and your favorite bignum library just to make sense of the data. In comparison, in msgpack (or json or xml or anything else) you would just have to store a string representation -- trivial to convert to either regular floats if you do not care, or to pass to your favorite bignum library (and this will be simple, as all of them support constructor based on ascii strings). In general, I think optional tags in data-interchange protocols are a very bad idea. For example, there is a tag for "Standard date/time string" and for "Epoch-based date/time". Which means that either: - You schema says "date/time", and your decoder now must support both of them (and probably untagged strings, and integers too). So this is an extra complexity in your decoder. - Your schema says "date/time in 'Standard date/time string' format", and now every encoder user must make sure the emitted value is encoded and tagged appropriately. This means you cannot do `x = cbor_encode({"now": date})`, you have to read your encoder documentation to make sure the CBOR encoder you are using will generate the required encoding. So extra complexity in either case, and no real benefits. Better stick to msgpack, at least it has no extensions defined currently.
        dchest 10y ago
        Seriously, I just don't understand how can you described the benefits of CBOR and claim that they are its drawbacks. Yes, you have to read the documentation of your encoder/decoder to understand what tag values it maps to your programming language's objects, but if you need to encode or decode those same values with MessagePack you'll have to define your own format for them and document it. You just moved this problem up the stack, but with an ad-hoc format. Separation of "null" and "undefined" is for full JavaScript support. Before C99, C didn't have boolean type, but you wouldn't complain if serialization formats had them, would you? Same thing "undefined": while it's useless in most other languages, it's useful to have it for JavaScript. I don't like float16 too - they took 19 lines of decoder code to support (no need to support it in encoder) — but it's the same situation as with "undefined" — some people need it.
  - camgunz 10y ago
    CBOR is MessagePack. At least cbor-ruby started with the MessagePack sources. There's nothing at all complicated about MessagePack, there are a few types, they all have small prefixes, end of format. The story is that Carsten took MessagePack, wrote a standard and added some things he wanted, and called it something else. I personally think this is pretty bad faith, but I admit I haven't been involved in the community or discussions, so it's possible there are some things I don't know. Disclaimer: I wrote and maintain a MessagePack library.
    - dchest 10y ago
      I don't really care about communities around serialization formats — I use it to encode and decode things, not for entertainment. I'd probably use CBOR even if Hitler himself designed it. Since you wrote a MessagePack library, I think you'll be surprised how regular CBOR is compared to MessagePack: Hitler found a brilliant generalization by introducing a neat [type|length] encoding instead of just [type], which applies to all types – integers, floats, strings, maps, etc.
      - camgunz 10y ago
        > (CBOR introduced) a neat [type|length] encoding instead of just [type], which > applies to all types - integers, floats, strings, maps, etc. Not really. For example there's no 11-bit integer type. In fact, CBOR is more-or-less identical to MP in this regard: there are a set number of (we'll use CBOR parlance here) tags, each of which indicates the length of succeeding binary data. If you're just saying "length follows type", then this is in fact exactly how MP works. If you have 30 bytes of raw data, you use the bin 8 tag (0xc4), write 0x1e for the length, and then memcpy your data into your buffer. --- I'd like to discuss CBOR's design, which I think is very poorly thought out. --- CBOR is broken because of indefinite-length data types. A CBOR implementation has to accept an infinite stream of data. Besides being impossible, this is unacceptable for pretty much any application. It's also poorly specified; Section 3.1 reads: In a streaming application, a data stream may be composed of a sequence of CBOR data items concatenated back-to-back. In such an environment, the decoder immediately begins decoding a new data item if data is found after the end of a previous data item. "Immediately" is troubling here, in fact, the section goes on: Note that some applications and protocols will not want to use indefinite-length encoding. Using indefinite-length encoding allows an encoder to not need to marshal all the data for counting, but it requires a decoder to allocate increasing amounts of memory while waiting for the end of the item. This might be fine for some applications but not others. I really don't know what to do here. The data format requires your application to allocate indefinitely, and the spec even says "this might be bad for you". Come on. --- CBOR's tags are pretty much MP v5's Extension types, but the spec makes them more confusing. Here's an excerpt: Decoders do not need to understand tags, and thus tags may be of little value in applications where the implementation decoding that stream know the semantic meaning of each item in the data flow. Their primary purpose in this specification is to define common data types such as dates. So far, so good. CBOR then goes on to define more than a dozen tags: - Standard date/time string - Epoch-based date/time - Positive bignum - Negative bignum - Decimal fraction - Bigfloat - Expected conversion to base64url encoding - Expected conversion to base64 encoding - Expected conversion to base16 encoding - Encoded CBOR data item - URI - base64url - base64 - Regular expression - MIME message - Self-describe CBOR Besides being over-engineered ("Expected conversion to X"? "MIME message"?!), this puts pressure on implementations to support these extension types. The reason MP doesn't do this is that, for example, BigNum support isn't free and you don't want to tie your binary format to it. CBOR's spec says implementations are free to not convert these (at least, that's how I interpret "Decoders do not need to understand tags"), but it creates a schism: most dynamic languages include BigNum support (Ruby, Python, etc.), but at least C and C++ don't. This creates pressure on C/C++ implementations to pull in a lot of extra baggage, something the MP authors specifically and thoughtfully avoided. Even worse, tags are undefined behavior in CBOR. Section 3.5 reads: A decoder that comes across a tag (Section 2.4) that it does not recognize, such as a tag that was added to the IANA registry after the decoder was deployed or a tag that the decoder chose not to implement, might issue a warning, might stop processing altogether, might handle the error and present the unknown tag value together with the contained data item to the application (as is expected of generic decoders), might ignore the tag and simply present the contained data item only to the application, or take some other type of action. "some other type of action" is deeply worrying. Even more deeply worrying is that this applies to anything at all in CBOR: A decoder that comes across a simple value (Section 2.3) that it does not recognize, such as a value that was added to the IANA registry after the decoder was deployed or a value that the decoder chose not to implement, might issue a warning, might stop processing altogether, might handle the error by making the unknown value available to the application as such (as is expected of generic decoders), or take some other type of action. I choose not to implement `int`. I decide instead to fill up your home folder. I'm a compliant CBOR implementation. --- Canonical CBOR is a bad idea, as outlined by ludocode elsewhere. Generally speaking, canonicalization is application-specific and attempts to codify it end up being nonsensical, overwrought, or incomplete. And hey look, CBOR's is nonsensical AND incomplete: it punts on floats and tags. --- There is no technical justification for CBOR. I know some places require standards, and that could've been solved by working with Sadayuki. I'm legitimately at a loss. --- I know you're not into the "drama" part, but I think it's beneficial to get into it a little. Feel free to ignore; it's why I put it down here. I'm not gonna characterize things. I'll just list links. I feel like that's the most fair way to do things. https://github.com/msgpack/msgpack/issues/13 https://github.com/msgpack/msgpack/issues/13 https://github.com/msgpack/msgpack/issues/121 https://github.com/msgpack/msgpack/issues/121 https://github.com/msgpack/msgpack/issues/129 https://github.com/msgpack/msgpack/issues/129 https://tools.ietf.org/html/draft-bormann-apparea-bpack-01 https://tools.ietf.org/html/draft-bormann-apparea-bpack-01 https://tools.ietf.org/html/rfc7049 https://tools.ietf.org/html/rfc7049 http://www6.ietf.org/mail-archive/web/apps-discuss/current/msg08914.html http://www6.ietf.org/mail-archive/web/apps-discuss/current/m... http://www6.ietf.org/mail-archive/web/apps-discuss/current/msg08915.html http://www6.ietf.org/mail-archive/web/apps-discuss/current/m... http://www6.ietf.org/mail-archive/web/apps-discuss/current/msg08927.html http://www6.ietf.org/mail-archive/web/apps-discuss/current/m... http://www6.ietf.org/mail-archive/web/apps-discuss/current/msg08973.html http://www6.ietf.org/mail-archive/web/apps-discuss/current/m... http://www6.ietf.org/mail-archive/web/apps-discuss/current/msg08987.html http://www6.ietf.org/mail-archive/web/apps-discuss/current/m... http://www6.ietf.org/mail-archive/web/apps-discuss/current/msg08989.html http://www6.ietf.org/mail-archive/web/apps-discuss/current/m... It's also worth pointing out that CBOR's RFC is incorrect and unfair in its description of MP. Heres's Section E.2.: MessagePack has been essentially stable since it was first published around 2011; it has not yet had a transition. The evolution of MessagePack is impeded by an imperative to maintain complete backwards compatibility with existing stored data, while only few bytecodes are still available for extension. Repeated requests over the years from the MessagePack user community to separate out binary and text strings in the encoding recently have led to an extension proposal that would leave MessagePack's "raw" data ambiguous between its usages for binary and text data. The extension mechanism for MessagePack remains unclear. MP v5 specifies separate types for string data and raw binary data. MP v5 is backwards-compatible with v4; in fact the v5 standard says implementations should provide a v4 compatibility mode. Nothing about the separate types for string and raw data is ambiguous; they're as separate as any other variable-length type. MP v5 includes a clear and simple extension mechanism: MessagePack allows applications to define application-specific types using the Extension type. Extension type consists of an integer and byte array where the integer represents a kind of type and the byte array represents data. Applications can assign 0 - 127 to store application-specific type information. MessagePack reserves -1 - -128 for future extension to add predefined types which will be described in separated documents.
ludocode 10y ago
No, CBOR should not replace MessagePack. The CBOR spec was written without the participation of the MessagePack community or the consent of its creator. CBOR added a number of questionable and unpopular features that lacked community consensus such as variable-length arrays and maps, or prefix tags for an arbitrary list of semantic types (URI, mimetype, base64, etc.) It also proposes various levels of strictness and canonicalization such as key sorting of maps based on binary representation. All of this complicates encoding and parsing which runs counter to the goals of a simple binary format.
- dchest 10y ago
  Wait, since when do you have to have the consent of the creator of one serialization format to create another? Tagged values are useful! Because of the simplicity of CBOR (compared to MessagePack), it's about 3 lines of code to implement them in the encoder. I'm not sure I understand your complaint about canonical mode. First of all, it's optional, and secondly, it's actually good that it's defined in the spec, otherwise you'd have multiple incompatible formats like canonical JSON.
  - ludocode 10y ago
    Of course they don't need consent. The point is that standardization is meaningless without the support and participation of implementers and users. This is why CBOR has made no progress in replacing MessagePack. The point about strictness and canonical mode are that they are poorly thought out. For example, sorting based on binary representation requires multi-pass encoding which is slow, complex, error prone, and completely non-intuitive: [1,2,3] comes before 100000 which comes before [1,2,3,4].

tptacek 10y ago

Nobody should really care whether something is or isn't an IETF standard. The cart has been dragging the horses with internet standardization for well over a decade now.

dchest 10y ago
Indeed. MessagePack should be replaced with CBOR in new protocols, as CBOR is a very nicely designed format.
- tptacek 10y ago
  Isn't CBOR just IETF's bikeshedded MessagePack? I'm not trying to be snarky: I don't have strong opinions about encoding formats. "CBOR is better than MessagePack because it's better designed" is a good argument, I think --- but: you'd want it to be so much better that the difference is material. "CBOR is better than MessagePack because it's standardized" is not, I think, a persuasive argument.
  - dchest 10y ago
    Yeah, I skipped all the drama, read the spec and implemented an encoder/decoder. CBOR is just how MessagePack-like format should have been done from the beginning: it's technically superior in a sense that it's neat and simple, replacing many specialized rules with one generalization. I agree that IETF standardization isn't a good argument (well, for some it is), that's why I replied with a better argument :) But, seriously, I won't say that everyone should replace MessagePack everywhere with CBOR, both work fine (as long as you use the latest version of MessagePack, with binary/string distinction).
    - theamk 10y ago
      Looking at CBOR spec, I see that it is just more complex. Two ways to encode lists/strings -- indefinite-length and strings. 16-bit floats in the main spec. Separation of "null" and "undefined" values. And tags, with which define things like decimal fractions, bigints, and regexps. The tags are the worst, actually. Sure, spec says "decoders do not need to understand tags", but this is not really the case. For example, if someone has floating point numbers and worries about precision loss, they can store the value as _decimal fraction_ (per section 2.4.3). This means that your decoders have to support both tagging and your favorite bignum library just to make sense of the data. In comparison, in msgpack (or json or xml or anything else) you would just have to store a string representation -- trivial to convert to either regular floats if you do not care, or to pass to your favorite bignum library (and this will be simple, as all of them support constructor based on ascii strings). In general, I think optional tags in data-interchange protocols are a very bad idea. For example, there is a tag for "Standard date/time string" and for "Epoch-based date/time". Which means that either: - You schema says "date/time", and your decoder now must support both of them (and probably untagged strings, and integers too). So this is an extra complexity in your decoder. - Your schema says "date/time in 'Standard date/time string' format", and now every encoder user must make sure the emitted value is encoded and tagged appropriately. This means you cannot do `x = cbor_encode({"now": date})`, you have to read your encoder documentation to make sure the CBOR encoder you are using will generate the required encoding. So extra complexity in either case, and no real benefits. Better stick to msgpack, at least it has no extensions defined currently.
      - dchest 10y ago
        Seriously, I just don't understand how can you described the benefits of CBOR and claim that they are its drawbacks. Yes, you have to read the documentation of your encoder/decoder to understand what tag values it maps to your programming language's objects, but if you need to encode or decode those same values with MessagePack you'll have to define your own format for them and document it. You just moved this problem up the stack, but with an ad-hoc format. Separation of "null" and "undefined" is for full JavaScript support. Before C99, C didn't have boolean type, but you wouldn't complain if serialization formats had them, would you? Same thing "undefined": while it's useless in most other languages, it's useful to have it for JavaScript. I don't like float16 too - they took 19 lines of decoder code to support (no need to support it in encoder) — but it's the same situation as with "undefined" — some people need it.
- camgunz 10y ago
  CBOR is MessagePack. At least cbor-ruby started with the MessagePack sources. There's nothing at all complicated about MessagePack, there are a few types, they all have small prefixes, end of format. The story is that Carsten took MessagePack, wrote a standard and added some things he wanted, and called it something else. I personally think this is pretty bad faith, but I admit I haven't been involved in the community or discussions, so it's possible there are some things I don't know. Disclaimer: I wrote and maintain a MessagePack library.
  - dchest 10y ago
    I don't really care about communities around serialization formats — I use it to encode and decode things, not for entertainment. I'd probably use CBOR even if Hitler himself designed it. Since you wrote a MessagePack library, I think you'll be surprised how regular CBOR is compared to MessagePack: Hitler found a brilliant generalization by introducing a neat [type|length] encoding instead of just [type], which applies to all types – integers, floats, strings, maps, etc.
    - camgunz 10y ago
      > (CBOR introduced) a neat [type|length] encoding instead of just [type], which > applies to all types - integers, floats, strings, maps, etc. Not really. For example there's no 11-bit integer type. In fact, CBOR is more-or-less identical to MP in this regard: there are a set number of (we'll use CBOR parlance here) tags, each of which indicates the length of succeeding binary data. If you're just saying "length follows type", then this is in fact exactly how MP works. If you have 30 bytes of raw data, you use the bin 8 tag (0xc4), write 0x1e for the length, and then memcpy your data into your buffer. --- I'd like to discuss CBOR's design, which I think is very poorly thought out. --- CBOR is broken because of indefinite-length data types. A CBOR implementation has to accept an infinite stream of data. Besides being impossible, this is unacceptable for pretty much any application. It's also poorly specified; Section 3.1 reads: In a streaming application, a data stream may be composed of a sequence of CBOR data items concatenated back-to-back. In such an environment, the decoder immediately begins decoding a new data item if data is found after the end of a previous data item. "Immediately" is troubling here, in fact, the section goes on: Note that some applications and protocols will not want to use indefinite-length encoding. Using indefinite-length encoding allows an encoder to not need to marshal all the data for counting, but it requires a decoder to allocate increasing amounts of memory while waiting for the end of the item. This might be fine for some applications but not others. I really don't know what to do here. The data format requires your application to allocate indefinitely, and the spec even says "this might be bad for you". Come on. --- CBOR's tags are pretty much MP v5's Extension types, but the spec makes them more confusing. Here's an excerpt: Decoders do not need to understand tags, and thus tags may be of little value in applications where the implementation decoding that stream know the semantic meaning of each item in the data flow. Their primary purpose in this specification is to define common data types such as dates. So far, so good. CBOR then goes on to define more than a dozen tags: - Standard date/time string - Epoch-based date/time - Positive bignum - Negative bignum - Decimal fraction - Bigfloat - Expected conversion to base64url encoding - Expected conversion to base64 encoding - Expected conversion to base16 encoding - Encoded CBOR data item - URI - base64url - base64 - Regular expression - MIME message - Self-describe CBOR Besides being over-engineered ("Expected conversion to X"? "MIME message"?!), this puts pressure on implementations to support these extension types. The reason MP doesn't do this is that, for example, BigNum support isn't free and you don't want to tie your binary format to it. CBOR's spec says implementations are free to not convert these (at least, that's how I interpret "Decoders do not need to understand tags"), but it creates a schism: most dynamic languages include BigNum support (Ruby, Python, etc.), but at least C and C++ don't. This creates pressure on C/C++ implementations to pull in a lot of extra baggage, something the MP authors specifically and thoughtfully avoided. Even worse, tags are undefined behavior in CBOR. Section 3.5 reads: A decoder that comes across a tag (Section 2.4) that it does not recognize, such as a tag that was added to the IANA registry after the decoder was deployed or a tag that the decoder chose not to implement, might issue a warning, might stop processing altogether, might handle the error and present the unknown tag value together with the contained data item to the application (as is expected of generic decoders), might ignore the tag and simply present the contained data item only to the application, or take some other type of action. "some other type of action" is deeply worrying. Even more deeply worrying is that this applies to anything at all in CBOR: A decoder that comes across a simple value (Section 2.3) that it does not recognize, such as a value that was added to the IANA registry after the decoder was deployed or a value that the decoder chose not to implement, might issue a warning, might stop processing altogether, might handle the error by making the unknown value available to the application as such (as is expected of generic decoders), or take some other type of action. I choose not to implement `int`. I decide instead to fill up your home folder. I'm a compliant CBOR implementation. --- Canonical CBOR is a bad idea, as outlined by ludocode elsewhere. Generally speaking, canonicalization is application-specific and attempts to codify it end up being nonsensical, overwrought, or incomplete. And hey look, CBOR's is nonsensical AND incomplete: it punts on floats and tags. --- There is no technical justification for CBOR. I know some places require standards, and that could've been solved by working with Sadayuki. I'm legitimately at a loss. --- I know you're not into the "drama" part, but I think it's beneficial to get into it a little. Feel free to ignore; it's why I put it down here. I'm not gonna characterize things. I'll just list links. I feel like that's the most fair way to do things. https://github.com/msgpack/msgpack/issues/13 https://github.com/msgpack/msgpack/issues/13 https://github.com/msgpack/msgpack/issues/121 https://github.com/msgpack/msgpack/issues/121 https://github.com/msgpack/msgpack/issues/129 https://github.com/msgpack/msgpack/issues/129 https://tools.ietf.org/html/draft-bormann-apparea-bpack-01 https://tools.ietf.org/html/draft-bormann-apparea-bpack-01 https://tools.ietf.org/html/rfc7049 https://tools.ietf.org/html/rfc7049 http://www6.ietf.org/mail-archive/web/apps-discuss/current/msg08914.html http://www6.ietf.org/mail-archive/web/apps-discuss/current/m... http://www6.ietf.org/mail-archive/web/apps-discuss/current/msg08915.html http://www6.ietf.org/mail-archive/web/apps-discuss/current/m... http://www6.ietf.org/mail-archive/web/apps-discuss/current/msg08927.html http://www6.ietf.org/mail-archive/web/apps-discuss/current/m... http://www6.ietf.org/mail-archive/web/apps-discuss/current/msg08973.html http://www6.ietf.org/mail-archive/web/apps-discuss/current/m... http://www6.ietf.org/mail-archive/web/apps-discuss/current/msg08987.html http://www6.ietf.org/mail-archive/web/apps-discuss/current/m... http://www6.ietf.org/mail-archive/web/apps-discuss/current/msg08989.html http://www6.ietf.org/mail-archive/web/apps-discuss/current/m... It's also worth pointing out that CBOR's RFC is incorrect and unfair in its description of MP. Heres's Section E.2.: MessagePack has been essentially stable since it was first published around 2011; it has not yet had a transition. The evolution of MessagePack is impeded by an imperative to maintain complete backwards compatibility with existing stored data, while only few bytecodes are still available for extension. Repeated requests over the years from the MessagePack user community to separate out binary and text strings in the encoding recently have led to an extension proposal that would leave MessagePack's "raw" data ambiguous between its usages for binary and text data. The extension mechanism for MessagePack remains unclear. MP v5 specifies separate types for string data and raw binary data. MP v5 is backwards-compatible with v4; in fact the v5 standard says implementations should provide a v4 compatibility mode. Nothing about the separate types for string and raw data is ambiguous; they're as separate as any other variable-length type. MP v5 includes a clear and simple extension mechanism: MessagePack allows applications to define application-specific types using the Extension type. Extension type consists of an integer and byte array where the integer represents a kind of type and the byte array represents data. Applications can assign 0 - 127 to store application-specific type information. MessagePack reserves -1 - -128 for future extension to add predefined types which will be described in separated documents.

ludocode 10y ago

No, CBOR should not replace MessagePack. The CBOR spec was written without the participation of the MessagePack community or the consent of its creator. CBOR added a number of questionable and unpopular features that lacked community consensus such as variable-length arrays and maps, or prefix tags for an arbitrary list of semantic types (URI, mimetype, base64, etc.) It also proposes various levels of strictness and canonicalization such as key sorting of maps based on binary representation. All of this complicates encoding and parsing which runs counter to the goals of a simple binary format.

dchest 10y ago
Wait, since when do you have to have the consent of the creator of one serialization format to create another? Tagged values are useful! Because of the simplicity of CBOR (compared to MessagePack), it's about 3 lines of code to implement them in the encoder. I'm not sure I understand your complaint about canonical mode. First of all, it's optional, and secondly, it's actually good that it's defined in the spec, otherwise you'd have multiple incompatible formats like canonical JSON.
- ludocode 10y ago
  Of course they don't need consent. The point is that standardization is meaningless without the support and participation of implementers and users. This is why CBOR has made no progress in replacing MessagePack. The point about strictness and canonical mode are that they are poorly thought out. For example, sorting based on binary representation requires multi-pass encoding which is slow, complex, error prone, and completely non-intuitive: [1,2,3] comes before 100000 which comes before [1,2,3,4].