6 ms·

How does ASN.1 compare to something like binary protobuf? I have no data to back this up, but I was under the impression ASN.1 is one of those specifications t

by ahepp 5y ago

How does ASN.1 compare to something like binary protobuf? I have no data to back this up, but I was under the impression ASN.1 is one of those specifications that is so large and complicated it’s essentially impossible to implement correctly; and that ASN.1 parsers are a pretty infamous source of security bugs (although I mean, most parsers seem to be).

rixed 5y ago
ASN1 has a much richer type system than protobufs, and is more general in that it accept several encodings, some of which optimised for size, some for speed, some for simplicity. I believe we could prune all the unnecessary 90s fad from the specs and end up with something actually pretty nice. I wish Google had done this as ASN2 instead of protobuf.
- bawolff 5y ago
  > ASN1 has a much richer type system than protobufs, and is more general in that it accept several encodings, some of which optimised for size, some for speed, some for simplicity. You say that like its a good thing.
  - sandermvanvliet 5y ago
    Protobuf doesn’t have a date/date time/time type which means you get dates encoded as strings and all the fun that entails. Not saying that either is better but just sometimes you actually do want more types instead of less.
    - Matthias247 5y ago
      Well there is Timestamp defined as a well known type which is available to all implementations despite not being a primitive type [1]. Plus one is obviously able to define any other custom types if necessary- eg as seen in [2]. [1] https://developers.google.com/protocol-buffers/docs/reference/google.protobuf#google.protobuf.Timestamp https://developers.google.com/protocol-buffers/docs/referenc... [2] https://github.com/googleapis/googleapis/blob/master/google/type/datetime.proto https://github.com/googleapis/googleapis/blob/master/google/...
      - sandermvanvliet 5y ago
        Ok my bad, I was looking to quickly through the base types in the proto reference
      - ithkuil 5y ago
        Well known types strike an interesting balance between IDL and compiler simplicity vs ergonomics. For example maps were added in proto3. They use the same binary encoding as the proto2 pattern that you could use to implement mappings (repeated field of key value pair messages). Adding maps as first class citizen in the IDL dealt with the lack of type parameters that would have been needed to implement a Map well-known type.
    - stjohnswarts 5y ago
      Time doesn't have to be hard though? Just stick to an international standard and stick to it. Reject anything that doesn't comply with your chosen format. It's better to fail than to continue running after you receive garbage data. I admit we have a close system at work be we never have time issues.
  - rixed 5y ago
    How could a richer type system or more choices regarding the encoding be a bad thing? I don't understand your remark.
- floatboth 5y ago
  A big design goal of Protobuf was long-term evolvability of message definitions with backward and forward compatibility. How rich is ASN.1 in that aspect?
  - p_l 5y ago
    Protobuf has known issues with backward/forward compatibility that require special attention, ASN.1 makes it IMHO remarkably easier (also consider kerberos which uses ASN.1, defining each extensible piece of data with extra tag declaring whether it must be parsed or can be ignored and passed on)
    - ithkuil 5y ago
      Can you point us to which well known backward compatibility issues protobuf has?
  - rixed 5y ago
    This is indeed important for a serialization format to permit extensions, and ASN1 types can accordingly be extended in a backward compatible way (for instance, enumerations can be said to also accept other constructors to be defined in the future).
memling 5y ago
> How does ASN.1 compare to something like binary protobuf? Protobuf is probably easier to work with in many cases, but there are a few issues that I've run across that makes protobuf a greater challenge in production: 1. Backwards compatibility. proto2 vs. proto3 for example introduced changes that you can work around but could cause issues if you're upgrading. Among those changes was the handling of optional vs. required elements; this may have changed since, but IIRC proto3 makes all elements optional. ASN.1 also supports explicit extensibility and message versioning. 2. ASN.1 supports canonical encodings (CER and DER for sure, and I think there are canonical encodings for PER and XML), which means that the same data are encoded exactly the same way across the wire. Protobuf doesn't guarantee element ordering. When hashing messages, this is important. 3. ASN.1 supports length and value constraints, and compilers can take advantage of these in message validation and (in some cases) message compression. Protobuf, last I checked, did not. ASN.1 predates open source software, though, and that means that most (all?) of the viable solutions are commercial (Objective Systems, OSS Nokalva, Marben, etc), or else rolled in-house (e.g., erlang). My guess is that the older standards were designed in part by LISPers: I'm not sure how else you would produce a general solution for MACRO constructions in their absence.
- p_l 5y ago
  It's also much easier to construct a safe parser if you can use metaprogramming and effectively build a parser from generic blocks. Consider also how ASN.1 heavily uses specifications that are supposed to be read by compiler generators, something that also is replicated in Protobuf, but with less expressive power.

by ahepp 5y ago

rixed 5y ago

ASN1 has a much richer type system than protobufs, and is more general in that it accept several encodings, some of which optimised for size, some for speed, some for simplicity. I believe we could prune all the unnecessary 90s fad from the specs and end up with something actually pretty nice. I wish Google had done this as ASN2 instead of protobuf.

bawolff 5y ago
> ASN1 has a much richer type system than protobufs, and is more general in that it accept several encodings, some of which optimised for size, some for speed, some for simplicity. You say that like its a good thing.
- sandermvanvliet 5y ago
  Protobuf doesn’t have a date/date time/time type which means you get dates encoded as strings and all the fun that entails. Not saying that either is better but just sometimes you actually do want more types instead of less.
  - Matthias247 5y ago
    Well there is Timestamp defined as a well known type which is available to all implementations despite not being a primitive type [1]. Plus one is obviously able to define any other custom types if necessary- eg as seen in [2]. [1] https://developers.google.com/protocol-buffers/docs/reference/google.protobuf#google.protobuf.Timestamp https://developers.google.com/protocol-buffers/docs/referenc... [2] https://github.com/googleapis/googleapis/blob/master/google/type/datetime.proto https://github.com/googleapis/googleapis/blob/master/google/...
    - sandermvanvliet 5y ago
      Ok my bad, I was looking to quickly through the base types in the proto reference
    - ithkuil 5y ago
      Well known types strike an interesting balance between IDL and compiler simplicity vs ergonomics. For example maps were added in proto3. They use the same binary encoding as the proto2 pattern that you could use to implement mappings (repeated field of key value pair messages). Adding maps as first class citizen in the IDL dealt with the lack of type parameters that would have been needed to implement a Map well-known type.
  - stjohnswarts 5y ago
    Time doesn't have to be hard though? Just stick to an international standard and stick to it. Reject anything that doesn't comply with your chosen format. It's better to fail than to continue running after you receive garbage data. I admit we have a close system at work be we never have time issues.
- rixed 5y ago
  How could a richer type system or more choices regarding the encoding be a bad thing? I don't understand your remark.
floatboth 5y ago
A big design goal of Protobuf was long-term evolvability of message definitions with backward and forward compatibility. How rich is ASN.1 in that aspect?
- p_l 5y ago
  Protobuf has known issues with backward/forward compatibility that require special attention, ASN.1 makes it IMHO remarkably easier (also consider kerberos which uses ASN.1, defining each extensible piece of data with extra tag declaring whether it must be parsed or can be ignored and passed on)
  - ithkuil 5y ago
    Can you point us to which well known backward compatibility issues protobuf has?
- rixed 5y ago
  This is indeed important for a serialization format to permit extensions, and ASN1 types can accordingly be extended in a backward compatible way (for instance, enumerations can be said to also accept other constructors to be defined in the future).

memling 5y ago

> How does ASN.1 compare to something like binary protobuf? Protobuf is probably easier to work with in many cases, but there are a few issues that I've run across that makes protobuf a greater challenge in production: 1. Backwards compatibility. proto2 vs. proto3 for example introduced changes that you can work around but could cause issues if you're upgrading. Among those changes was the handling of optional vs. required elements; this may have changed since, but IIRC proto3 makes all elements optional. ASN.1 also supports explicit extensibility and message versioning. 2. ASN.1 supports canonical encodings (CER and DER for sure, and I think there are canonical encodings for PER and XML), which means that the same data are encoded exactly the same way across the wire. Protobuf doesn't guarantee element ordering. When hashing messages, this is important. 3. ASN.1 supports length and value constraints, and compilers can take advantage of these in message validation and (in some cases) message compression. Protobuf, last I checked, did not. ASN.1 predates open source software, though, and that means that most (all?) of the viable solutions are commercial (Objective Systems, OSS Nokalva, Marben, etc), or else rolled in-house (e.g., erlang). My guess is that the older standards were designed in part by LISPers: I'm not sure how else you would produce a general solution for MACRO constructions in their absence.

p_l 5y ago
It's also much easier to construct a safe parser if you can use metaprogramming and effectively build a parser from generic blocks. Consider also how ASN.1 heavily uses specifications that are supposed to be read by compiler generators, something that also is replicated in Protobuf, but with less expressive power.