5 ms·

NoProto: Flexible, Fast and Compact Serialization with RPC

gravypod 5y ago
One of the big benefits of protos is that you have a language-agnostic schema that can evolve safely and backwards compatibly. I'd love for some team to join that with better performance as it seems like it's possible from what this project has done.
omegalulw 5y ago
How is serialization and de-serialization faster than compiled formats with just spend? If you update many if the fields some more than once, the overhead would be higher?
Groxx 5y ago
Every popular RPC format I'm aware of supports dynamic schemas, they're just generally not used unless strictly necessary. E.g. Cap'n Proto has SchemaLoader (dynamically loading compiled schemas) and SchemaParser (parsing schema strings into generic objects): https://capnproto.org/cxx.html#dynamic-reflection https://capnproto.org/cxx.html#dynamic-reflection, protobuf has "Dynamic Message": https://developers.google.com/protocol-buffers/docs/reference/cpp/google.protobuf.dynamic_message https://developers.google.com/protocol-buffers/docs/referenc... , etc. Plus, literally anything you can compile can be done dynamically, so it's not like this is an inherent quality of an encoding, only at best a specific tool. Broadly: I'm sorta curious about the details here, but I'll have to read more later. It seems to be claiming the world without a whole lot of evidence, and some incorrect claims, while being implemented as a fairly simple (often good!) serialized form which is also quite large (less good!). Also I have no idea why it's so obsessed with sort-ability. It's not like the encoded data as a whole can be meaningfully sorted, and per-value that sounds like "is big-endian"... E.g. this is the serialized form of a pre-defined struct with a single field, like `{"age":20}`: https://docs.rs/no_proto/0.9.60/no_proto/format/index.html https://docs.rs/no_proto/0.9.60/no_proto/format/index.html // [0, 0, 0, 0, 0, 6, 0, 0, 0, 26, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 20] // [ root ptr, vtable, data] which is pretty far from protocol buffers': https://developers.google.com/protocol-buffers/docs/encoding https://developers.google.com/protocol-buffers/docs/encoding 08 96 20 ^-- 01 in sample This sort of waste in favor of simplicity seems to be spread throughout the encoding. I'll happily admit that I think NoProto's format is the easier of the two to read and understand by hand, but that's not generally why we choose binary encodings. It's possible the simplicity gives it a performance edge, but I strongly suspect that, in practice, that's influenced far more by the various implementations' internal details than the encoding itself.
vlovich123 5y ago
Does anyone know what the `mutable` column means? I think the cap'n'proto section may not be correct as I think you can do read/modify/write. > all the buffers/data is trusted This part is also wrong I think. Cap'n'proto is fine to use with untrusted data AFAIK. > Thus, Cap’n Proto checks the structural integrity of the message just like any other serialization protocol would. And, just like any other protocol, it is up to the app to check the validity of the content. Likely the author is referring to this section of the home page: > As of this writing, Cap’n Proto has not undergone a security review, therefore we suggest caution when handling messages from untrusted sources Which seems more like a disclaimer than "you can't do this" (security is on by default as compared with FlatBuffers where it's opt-in).
- kentonv 5y ago
  Cap'n Proto is only sortof-mutable. You can update e.g. an integer in-place, but if you want to change anything that is variable-width, e.g. modifying a string, adding/removing inner objects, etc., then it can get awkward. New objects will be added to the end of the buffer, and removed objects will just be zero'd out, wasting the memory. You can "garbage-collect" by making a whole-tree copy, of course. But yeah all the stuff about untrusted data is BS. NoProto's readme lists "Safely handle untrusted data" as an advantage vs. every other format it compares against, including Protobuf and JSON. What? Every serious serialization format supports untrusted data.
  - kentonv 5y ago
    Note that Cap'n Proto's mutability limitations are inherent to being a zero-copy protocol -- it's necessary to allocate messages in contiguous memory, tightly packed, which makes it hard to be able to allocate memory dynamically. NoProto also claims to be zero-copy, which would seem to subject it to the same issues. I couldn't find anything in the docs explaining how they avoid it.
erik_seaberg 5y ago
Being able to quickly replace or append a field without re-serializing the entire struct can be useful, but the overhead for it is pretty substantial (an extra u32 for every field, plus 1/4 of another u32 for next vtable in the list). Every u32 always consumes four (unaligned?) bytes. I think “far more space efficient” than Avro is pretty unlikely. I hope he’s also looking at ASN.1 PER for more “don’t write anything the reader already knows” ideas in the vein of Avro. Without OIDs or UUIDs for versions of types, most of these formats seem equally prone to treating invalid messages as unpredictable gibberish or erroring out on an invalid size or offset.
tomberek 5y ago
At what point do you throw in the towel and just send in-memory representations of SQLite DBs around?
spenczar5 5y ago
This seems pretty confused. The "compiled vs dynamic" distinction is a property of the implementation, not of the protocol. For example, you can certainly compile Avro into Go source files [0]. You can even compile Avro loaded schemas _during runtime_ into Python bytecode, since Python is interpreted [1]. This even works if you have the _wrong schema document_ for the message (you'll just get the subset of fields which are accurately described), because of Avro's schema compatibility rules. Likewise, you can deserialize arbitrary protobuf messages during runtime without a compilation step, if you have a description for the message schema. The Python protobuf library has had a "ParseMessage" API forever, and protoreflect [2] exists for Go. (In case it's not obvious, I mostly work in Python and Go but I am completely certain analogues exist in other major languages). There is a very big and important difference between a protocol and the implementation of a protocol. I think this README's author is not clear on that difference, which shows up in other claims ("Deserialization is incrimental", for example) too. --- [0] https://github.com/actgardner/gogen-avro https://github.com/actgardner/gogen-avro [1] https://github.com/spenczar/avroc https://github.com/spenczar/avroc [2] https://pkg.go.dev/google.golang.org/protobuf/reflect/protoreflect https://pkg.go.dev/google.golang.org/protobuf/reflect/protor...
- mumblemumble 5y ago
  To add to that, protobuf even some facilities for self-describing messages. gRPC has a reflection API so that you can dynamically download the schema for messages. It's a lot like Swagger only it works better with less effort.
pmalynin 5y ago
Tbh it’s kind of funny folks keep reinventing the wheel when we have had portable, standardized serialization format for around 50 years: DER.
- nly 5y ago
  Tooling for ASN.1 formats f*** suck, that's why. The only truly fully featured, well-maintained open source implementation for C/C++ seems to be asn1c from Lev Walkin and the dev experience is a world away from what you get using something like Avro, CapnProto or Protobufs.
- baybal2 5y ago
  ASN1 is 50 years old? Really?

One of the big benefits of protos is that you have a language-agnostic schema that can evolve safely and backwards compatibly. I'd love for some team to join that with better performance as it seems like it's possible from what this project has done.

omegalulw 5y ago

How is serialization and de-serialization faster than compiled formats with just spend? If you update many if the fields some more than once, the overhead would be higher?

Groxx 5y ago

Every popular RPC format I'm aware of supports dynamic schemas, they're just generally not used unless strictly necessary. E.g. Cap'n Proto has SchemaLoader (dynamically loading compiled schemas) and SchemaParser (parsing schema strings into generic objects): https://capnproto.org/cxx.html#dynamic-reflection https://capnproto.org/cxx.html#dynamic-reflection, protobuf has "Dynamic Message": https://developers.google.com/protocol-buffers/docs/reference/cpp/google.protobuf.dynamic_message https://developers.google.com/protocol-buffers/docs/referenc... , etc. Plus, literally anything you can compile can be done dynamically, so it's not like this is an inherent quality of an encoding, only at best a specific tool. Broadly: I'm sorta curious about the details here, but I'll have to read more later. It seems to be claiming the world without a whole lot of evidence, and some incorrect claims, while being implemented as a fairly simple (often good!) serialized form which is also quite large (less good!). Also I have no idea why it's so obsessed with sort-ability. It's not like the encoded data as a whole can be meaningfully sorted, and per-value that sounds like "is big-endian"... E.g. this is the serialized form of a pre-defined struct with a single field, like `{"age":20}`: https://docs.rs/no_proto/0.9.60/no_proto/format/index.html https://docs.rs/no_proto/0.9.60/no_proto/format/index.html // [0, 0, 0, 0, 0, 6, 0, 0, 0, 26, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 20] // [ root ptr, vtable, data] which is pretty far from protocol buffers': https://developers.google.com/protocol-buffers/docs/encoding https://developers.google.com/protocol-buffers/docs/encoding 08 96 20 ^-- 01 in sample This sort of waste in favor of simplicity seems to be spread throughout the encoding. I'll happily admit that I think NoProto's format is the easier of the two to read and understand by hand, but that's not generally why we choose binary encodings. It's possible the simplicity gives it a performance edge, but I strongly suspect that, in practice, that's influenced far more by the various implementations' internal details than the encoding itself.

vlovich123 5y ago

Does anyone know what the `mutable` column means? I think the cap'n'proto section may not be correct as I think you can do read/modify/write. > all the buffers/data is trusted This part is also wrong I think. Cap'n'proto is fine to use with untrusted data AFAIK. > Thus, Cap’n Proto checks the structural integrity of the message just like any other serialization protocol would. And, just like any other protocol, it is up to the app to check the validity of the content. Likely the author is referring to this section of the home page: > As of this writing, Cap’n Proto has not undergone a security review, therefore we suggest caution when handling messages from untrusted sources Which seems more like a disclaimer than "you can't do this" (security is on by default as compared with FlatBuffers where it's opt-in).

kentonv 5y ago
Cap'n Proto is only sortof-mutable. You can update e.g. an integer in-place, but if you want to change anything that is variable-width, e.g. modifying a string, adding/removing inner objects, etc., then it can get awkward. New objects will be added to the end of the buffer, and removed objects will just be zero'd out, wasting the memory. You can "garbage-collect" by making a whole-tree copy, of course. But yeah all the stuff about untrusted data is BS. NoProto's readme lists "Safely handle untrusted data" as an advantage vs. every other format it compares against, including Protobuf and JSON. What? Every serious serialization format supports untrusted data.
- kentonv 5y ago
  Note that Cap'n Proto's mutability limitations are inherent to being a zero-copy protocol -- it's necessary to allocate messages in contiguous memory, tightly packed, which makes it hard to be able to allocate memory dynamically. NoProto also claims to be zero-copy, which would seem to subject it to the same issues. I couldn't find anything in the docs explaining how they avoid it.

erik_seaberg 5y ago

Being able to quickly replace or append a field without re-serializing the entire struct can be useful, but the overhead for it is pretty substantial (an extra u32 for every field, plus 1/4 of another u32 for next vtable in the list). Every u32 always consumes four (unaligned?) bytes. I think “far more space efficient” than Avro is pretty unlikely. I hope he’s also looking at ASN.1 PER for more “don’t write anything the reader already knows” ideas in the vein of Avro. Without OIDs or UUIDs for versions of types, most of these formats seem equally prone to treating invalid messages as unpredictable gibberish or erroring out on an invalid size or offset.

tomberek 5y ago

At what point do you throw in the towel and just send in-memory representations of SQLite DBs around?

spenczar5 5y ago

This seems pretty confused. The "compiled vs dynamic" distinction is a property of the implementation, not of the protocol. For example, you can certainly compile Avro into Go source files [0]. You can even compile Avro loaded schemas _during runtime_ into Python bytecode, since Python is interpreted [1]. This even works if you have the _wrong schema document_ for the message (you'll just get the subset of fields which are accurately described), because of Avro's schema compatibility rules. Likewise, you can deserialize arbitrary protobuf messages during runtime without a compilation step, if you have a description for the message schema. The Python protobuf library has had a "ParseMessage" API forever, and protoreflect [2] exists for Go. (In case it's not obvious, I mostly work in Python and Go but I am completely certain analogues exist in other major languages). There is a very big and important difference between a protocol and the implementation of a protocol. I think this README's author is not clear on that difference, which shows up in other claims ("Deserialization is incrimental", for example) too. --- [0] https://github.com/actgardner/gogen-avro https://github.com/actgardner/gogen-avro [1] https://github.com/spenczar/avroc https://github.com/spenczar/avroc [2] https://pkg.go.dev/google.golang.org/protobuf/reflect/protoreflect https://pkg.go.dev/google.golang.org/protobuf/reflect/protor...

mumblemumble 5y ago
To add to that, protobuf even some facilities for self-describing messages. gRPC has a reflection API so that you can dynamically download the schema for messages. It's a lot like Swagger only it works better with less effort.

pmalynin 5y ago

Tbh it’s kind of funny folks keep reinventing the wheel when we have had portable, standardized serialization format for around 50 years: DER.

nly 5y ago
Tooling for ASN.1 formats f*** suck, that's why. The only truly fully featured, well-maintained open source implementation for C/C++ seems to be asn1c from Lev Walkin and the dev experience is a world away from what you get using something like Avro, CapnProto or Protobufs.
baybal2 5y ago
ASN1 is 50 years old? Really?