6 ms·

It's amazing how many people are trying to reinvent protocol buffers! Every time I see something like this I think the developer didn't do their research or may

by msoad 10y ago

It's amazing how many people are trying to reinvent protocol buffers! Every time I see something like this I think the developer didn't do their research or maybe they wanted to make a hobby project anyway. Stuff like this is dangerous to use in production. Even JSON as simple as it looks had a lot of bugs that are now. If you want typed data structure transfer, use protocol buffer.

bascule 10y ago
Did you even read the post? First paragraph: "Its primary intended use is in cryptographic authentication contexts, particularly ones where JSON is used as a human-friendly alternative representation of data in a system which otherwise works natively in a binary format." See also the "Content-Aware Hashing" section: The goal of this format is to enable content-aware hashing which produces the same digest for data encoded either as TJSON or a binary format such as Protobufs. I am using it in conjunction with Protobufs.
- mianos 10y ago
  So they are inventing ASN.1 again, for the third time. Next thing they will invent distinguished encoding rules so data can be hashed without decoding.
  - bascule 10y ago
    No, far from "inventing ASN.1 again", TJSON could potentially be a very useful format for representing equivalent structures to ASN.1, similar to: https://github.com/google/der-ascii https://github.com/google/der-ascii
dewitt 10y ago
I take it you don't know who either of the two authors are? - https://en.wikipedia.org/wiki/Ben_Laurie https://en.wikipedia.org/wiki/Ben_Laurie - https://github.com/tarcieri https://github.com/tarcieri They know about protocol buffers.
- Cyph0n 10y ago
  Pretty damn impressive. I never knew about them either to be honest.
chubot 10y ago
That's not correct -- Protocol buffers differ from both JSON and TJSON because the schema isn't part of the protocol. That is, the information on the wire doesn't contain enough information to interpret the data -- the schema has to be compiled into or otherwise included in the binary to know what the data means. That's not the case with JSON or TJSON.
- bascule 10y ago
  Yes, this is a very important point, (T)JSON is self-describing in ways protobufs or other similar on-the-wire formats are not. Making sense of a protobuf involves both knowing what type of protobuf you're looking at in advance (protobufs are NOT self-identifying) and loading the corresponding schema. Otherwise each member of a protobuf is identified by an integer, so good luck guessing what each field represents.
alfalfasprout 10y ago
Enough people have given reasons why Protobufs (and other schema-based interchange formats) are different from the unstructured JSON/BSON/TSON/whatever. But if you are using a schema there are now far better alternatives to Protobufs. If you're primarily sending data over the network I've found Google's Flatbuffers to be great. For writing to disk Capn'Proto is similar and equally good. Receiving market data where every nanosecond matters? Simple Binary Encoding (SBE). All of these formats employ some form of code generation to extract values from what are essentially cleverly packed structs. All data is sent little endian and follow machine word sizes. Not truly cross-language but Flatbuffers has native support for C/C++, Python, Java, Go, C# and 3rd party support for Rust.
- bascule 10y ago
  The main reason to prefer protobufs is GRPC: there is now a pseudo-standard HTTP/2-based RPC format with many robust language implementations. That's mostly to say: I think I've generally observed GRPC being embraced. My intended deployment profile is having a single HTTP(/2) server listening on a single TCP port which can speak JSON over HTTP as well as protos over GRPC, both of which can be authenticated by the same objecthash/signature. You can store whatever format you want on disk, but if you primarily intend to serve proto-consuming clients, you might as well store protos on disk so what you serve to the network is an opaque blob of bytes with no transcoding. Don't get me wrong, I really love capnp, particularly the CapTP-like features, but I feel like many of the novelties of its IDL/serialization format (possibly ones involving kentonv's original work before he left Google) have actually shipped in proto3. I really love capnp, but there's this handwavy "this is the way the wind is blowing" argument to be made for GRPC, I think.
mahyarm 10y ago
Protobuf have a binary size bloat issue compared to JSON. JSON wins for similar reasons why HTTP 1.1 won. It's a human readable, simple format and performant enough the majority of development cases. Human readable makes debugging easier. I hope with tjson it will help increase parser speeds with it's type hints.
Matthias247 10y ago
I wouldn't necessarily compare it to protobuf, since that in general requires a schema for exact parsing and forwarding. However there are other encodings which should already cover all the desired features. E.g. CBOR is a standardized JSON-like binary format which has the possibility to store exact types for binary data, dates, etc... Yeah, such a format isn't directly human readable on the wire, but you can just get in into whatever string-like representation you want in your program. And parsers are for sure not harder to write than for most text formats.
- bascule 10y ago
  I am curious if you simply missed the numerous places this post refers to CBOR? Such as this: There exists a binary analogue of JWT called CWT which is based on the Compact Binary Object Representation (CBOR) standard. Unfortunately you can’t convert JWTs to CWTs without the original issuer re-issuing them and re-signing them in the new format. Or this: This is a pervasive problem for anyone who would like to store authenticated/signed data natively in a binary format (e.g. Protobufs, Thrift, capnp, MessagePack, BSON, or CBOR), but also permit clients to work natively with a JSON API without necessarily being aware of a full-and-evolving schema. JSON is a human-meaningful serialization format, as opposed to all the binary formats named in the post, including CBOR. See also: https://github.com/tjson/tjson-spec/issues/26 https://github.com/tjson/tjson-spec/issues/26 TJSON could potentially provide non-lossy transcoding to/from the similarly tagged types in CBOR in ways JSON itself cannot.

by msoad 10y ago

bascule 10y ago

Did you even read the post? First paragraph: "Its primary intended use is in cryptographic authentication contexts, particularly ones where JSON is used as a human-friendly alternative representation of data in a system which otherwise works natively in a binary format." See also the "Content-Aware Hashing" section: The goal of this format is to enable content-aware hashing which produces the same digest for data encoded either as TJSON or a binary format such as Protobufs. I am using it in conjunction with Protobufs.

mianos 10y ago
So they are inventing ASN.1 again, for the third time. Next thing they will invent distinguished encoding rules so data can be hashed without decoding.
- bascule 10y ago
  No, far from "inventing ASN.1 again", TJSON could potentially be a very useful format for representing equivalent structures to ASN.1, similar to: https://github.com/google/der-ascii https://github.com/google/der-ascii

dewitt 10y ago

I take it you don't know who either of the two authors are? - https://en.wikipedia.org/wiki/Ben_Laurie https://en.wikipedia.org/wiki/Ben_Laurie - https://github.com/tarcieri https://github.com/tarcieri They know about protocol buffers.

Cyph0n 10y ago
Pretty damn impressive. I never knew about them either to be honest.

chubot 10y ago

That's not correct -- Protocol buffers differ from both JSON and TJSON because the schema isn't part of the protocol. That is, the information on the wire doesn't contain enough information to interpret the data -- the schema has to be compiled into or otherwise included in the binary to know what the data means. That's not the case with JSON or TJSON.

bascule 10y ago
Yes, this is a very important point, (T)JSON is self-describing in ways protobufs or other similar on-the-wire formats are not. Making sense of a protobuf involves both knowing what type of protobuf you're looking at in advance (protobufs are NOT self-identifying) and loading the corresponding schema. Otherwise each member of a protobuf is identified by an integer, so good luck guessing what each field represents.

alfalfasprout 10y ago

Enough people have given reasons why Protobufs (and other schema-based interchange formats) are different from the unstructured JSON/BSON/TSON/whatever. But if you are using a schema there are now far better alternatives to Protobufs. If you're primarily sending data over the network I've found Google's Flatbuffers to be great. For writing to disk Capn'Proto is similar and equally good. Receiving market data where every nanosecond matters? Simple Binary Encoding (SBE). All of these formats employ some form of code generation to extract values from what are essentially cleverly packed structs. All data is sent little endian and follow machine word sizes. Not truly cross-language but Flatbuffers has native support for C/C++, Python, Java, Go, C# and 3rd party support for Rust.

bascule 10y ago
The main reason to prefer protobufs is GRPC: there is now a pseudo-standard HTTP/2-based RPC format with many robust language implementations. That's mostly to say: I think I've generally observed GRPC being embraced. My intended deployment profile is having a single HTTP(/2) server listening on a single TCP port which can speak JSON over HTTP as well as protos over GRPC, both of which can be authenticated by the same objecthash/signature. You can store whatever format you want on disk, but if you primarily intend to serve proto-consuming clients, you might as well store protos on disk so what you serve to the network is an opaque blob of bytes with no transcoding. Don't get me wrong, I really love capnp, particularly the CapTP-like features, but I feel like many of the novelties of its IDL/serialization format (possibly ones involving kentonv's original work before he left Google) have actually shipped in proto3. I really love capnp, but there's this handwavy "this is the way the wind is blowing" argument to be made for GRPC, I think.

mahyarm 10y ago

Protobuf have a binary size bloat issue compared to JSON. JSON wins for similar reasons why HTTP 1.1 won. It's a human readable, simple format and performant enough the majority of development cases. Human readable makes debugging easier. I hope with tjson it will help increase parser speeds with it's type hints.

Matthias247 10y ago

I wouldn't necessarily compare it to protobuf, since that in general requires a schema for exact parsing and forwarding. However there are other encodings which should already cover all the desired features. E.g. CBOR is a standardized JSON-like binary format which has the possibility to store exact types for binary data, dates, etc... Yeah, such a format isn't directly human readable on the wire, but you can just get in into whatever string-like representation you want in your program. And parsers are for sure not harder to write than for most text formats.

bascule 10y ago
I am curious if you simply missed the numerous places this post refers to CBOR? Such as this: There exists a binary analogue of JWT called CWT which is based on the Compact Binary Object Representation (CBOR) standard. Unfortunately you can’t convert JWTs to CWTs without the original issuer re-issuing them and re-signing them in the new format. Or this: This is a pervasive problem for anyone who would like to store authenticated/signed data natively in a binary format (e.g. Protobufs, Thrift, capnp, MessagePack, BSON, or CBOR), but also permit clients to work natively with a JSON API without necessarily being aware of a full-and-evolving schema. JSON is a human-meaningful serialization format, as opposed to all the binary formats named in the post, including CBOR. See also: https://github.com/tjson/tjson-spec/issues/26 https://github.com/tjson/tjson-spec/issues/26 TJSON could potentially provide non-lossy transcoding to/from the similarly tagged types in CBOR in ways JSON itself cannot.