5 ms·

The grandfather of protobuf. Lost in the tales of time.

by dbyte 2y ago

css 2y ago
iMessage uses a very strange amalgamation of typedstream (message content), keyed archives (app messages, sticker data), and protobufs (Digital Touch, handwriting) for different features. I wonder what motivated all of those design decisions.
- bengarney 2y ago
  This is stuff is such a PIA to parse. I assume it's just different teams doing different features over the years, and being alternately repulsed/seduced by each format. Probably features are implemented as libraries so there isn't a master oversight - they aren't trying to make iMessage's internal formats follow a consistent plan, just let all the libs coexist...
  - pixel_tracing 2y ago
    As someone who used to work on that team, it’s so interesting hearing thoughts from external public on the team.
    - ajtaylor 2y ago
      I would love to hear your thoughts as an insider.
  - meibo 2y ago
    Maybe they should be repulsed, considering all of the journalists that are getting persecuted and/or murdered because they are getting pwned through iMessage serialization bugs :)
- userbinator 2y ago
  "Those who don't understand ASN.1 are doomed to reinvent it, poorly." That said, it could be much worse --- JSON, or XML.
dim13 2y ago
Grandfather of Protobuf is ASN.1
- foobiekr 2y ago
  Very much so. Pretty much all of these protocols are simplifications of asn1 and in some cases (like protobuf) there are a handful of things that got lost because the wire formats didn’t have them as they didn’t need them. A schema indicator being the single biggest flaw in protobuf.
  - orf 2y ago
    Why is the lack of a schema indicator the biggest flaw of protobuf?
    - ptspts 2y ago
      If you parse a serialized protobuf byte array without having a .proto file, you have no way to dustinguish a byte string field from a nested message field. Thus you have no way to know how deep your parser should go.
      - css 2y ago
        Semi-related, one of the `imessage-exporter` contributors provided a great write-up on reverse engineering the handwritten and digital touch message protobufs [0]. The reconstructed proto files are [1] [2]. [0]: https://github.com/trymoose/handwriting2svg/blob/0eb56cf458207bb1c2ceea48cf4b6b6510fa7b13/DISCOVERY.md https://github.com/trymoose/handwriting2svg/blob/0eb56cf4582... [1]: https://github.com/ReagentX/imessage-exporter/blob/beeb853b211b52d71bf5f8e6e241810b89bd93cc/imessage-database/src/message_types/handwriting/handwriting.proto https://github.com/ReagentX/imessage-exporter/blob/beeb853b2... [2]: https://github.com/ReagentX/imessage-exporter/blob/beeb853b211b52d71bf5f8e6e241810b89bd93cc/imessage-database/src/message_types/digital_touch/digital_touch.proto https://github.com/ReagentX/imessage-exporter/blob/beeb853b2...
    - foobiekr 2y ago

The grandfather of protobuf. Lost in the tales of time.

by dbyte 2y ago

The grandfather of protobuf. Lost in the tales of time.

css 2y ago

iMessage uses a very strange amalgamation of typedstream (message content), keyed archives (app messages, sticker data), and protobufs (Digital Touch, handwriting) for different features. I wonder what motivated all of those design decisions.

bengarney 2y ago
This is stuff is such a PIA to parse. I assume it's just different teams doing different features over the years, and being alternately repulsed/seduced by each format. Probably features are implemented as libraries so there isn't a master oversight - they aren't trying to make iMessage's internal formats follow a consistent plan, just let all the libs coexist...
- pixel_tracing 2y ago
  As someone who used to work on that team, it’s so interesting hearing thoughts from external public on the team.
  - ajtaylor 2y ago
    I would love to hear your thoughts as an insider.
- meibo 2y ago
  Maybe they should be repulsed, considering all of the journalists that are getting persecuted and/or murdered because they are getting pwned through iMessage serialization bugs :)
userbinator 2y ago
"Those who don't understand ASN.1 are doomed to reinvent it, poorly." That said, it could be much worse --- JSON, or XML.

dim13 2y ago

Grandfather of Protobuf is ASN.1

foobiekr 2y ago
Very much so. Pretty much all of these protocols are simplifications of asn1 and in some cases (like protobuf) there are a handful of things that got lost because the wire formats didn’t have them as they didn’t need them. A schema indicator being the single biggest flaw in protobuf.
- orf 2y ago
  Why is the lack of a schema indicator the biggest flaw of protobuf?
  - ptspts 2y ago
    If you parse a serialized protobuf byte array without having a .proto file, you have no way to dustinguish a byte string field from a nested message field. Thus you have no way to know how deep your parser should go.
    - css 2y ago
      Semi-related, one of the `imessage-exporter` contributors provided a great write-up on reverse engineering the handwritten and digital touch message protobufs [0]. The reconstructed proto files are [1] [2]. [0]: https://github.com/trymoose/handwriting2svg/blob/0eb56cf458207bb1c2ceea48cf4b6b6510fa7b13/DISCOVERY.md https://github.com/trymoose/handwriting2svg/blob/0eb56cf4582... [1]: https://github.com/ReagentX/imessage-exporter/blob/beeb853b211b52d71bf5f8e6e241810b89bd93cc/imessage-database/src/message_types/handwriting/handwriting.proto https://github.com/ReagentX/imessage-exporter/blob/beeb853b2... [2]: https://github.com/ReagentX/imessage-exporter/blob/beeb853b211b52d71bf5f8e6e241810b89bd93cc/imessage-database/src/message_types/digital_touch/digital_touch.proto https://github.com/ReagentX/imessage-exporter/blob/beeb853b2...
  - foobiekr 2y ago