3 ms·

Fast and dynamic encoding of Protocol Buffers in Go

50 points by g0xA52A2A 4y ago

jeffbee 4y ago
Very nice! Puts together many of my favorite protobuf techniques. I am glad that people do not feel that they are required to use generated code when they choose the protobuf format. One thing I will quibble about is the representation of the tag number 19, named SrcCountry. The author shows it as binary 10011010 but this is impossible because protobuf tags are themselves encoded as continuation bit variable-length numbers and cannot end in a byte with the most significant bit set. That's why fields 1-15 are the magic fast-path fields taking only 1 byte. Field 19 would need two bytes... 10000001_00011010 or in hex 81 1A
- vbernat 4y ago
  Author here. The tag is shown not encoded as a variable int (I admit this is not super clear). Its encoded version is 0x9a 0x01. I think this is correct. This the result of (19 << 3) + 2 encoded as a varint.
  - jeffbee 4y ago
    Oh I see, the thing on the left is logical and the thing on the right is on the wire. Thanks for helping me read it.
bamazizi 4y ago
This project does have potentials. We were using Gogo up until a few months ago until it was deprecated and replaced the pipeline with Buf.build. It took several days of head scratching but once we figured it out it was basically ZERO code change in 100s of proto and Go files. If vtprotobuf would be adopted as an optional compile module in Buf then I would love try it. I just don't think we would want to move away from Buf.
everybodyknows 4y ago
> There exist other alternatives, including Cap’n Proto, CBOR, and Avro. The names of the alternatives are links to references. All technical articles should start this way, well done.
irq-1 4y ago
> Some paragraphs were editorialized by ChatGPT, using “editorialize and keep it short” as a prompt. The result was proofread by a human for correctness. The main idea is that ChatGPT should be better at English than me. This should be at the TOP not at the end.
- sliken 4y ago
  Why? The author is putting effort into writing as clearly as they can, do the tools really matter? Do I need to announce that my sister is proof reading my writing? Or that I use the assistant inside Microsoft Word? Where do you draw the line?

50 points by g0xA52A2A 4y ago

Very nice! Puts together many of my favorite protobuf techniques. I am glad that people do not feel that they are required to use generated code when they choose the protobuf format. One thing I will quibble about is the representation of the tag number 19, named SrcCountry. The author shows it as binary 10011010 but this is impossible because protobuf tags are themselves encoded as continuation bit variable-length numbers and cannot end in a byte with the most significant bit set. That's why fields 1-15 are the magic fast-path fields taking only 1 byte. Field 19 would need two bytes... 10000001_00011010 or in hex 81 1A

vbernat 4y ago
Author here. The tag is shown not encoded as a variable int (I admit this is not super clear). Its encoded version is 0x9a 0x01. I think this is correct. This the result of (19 << 3) + 2 encoded as a varint.
- jeffbee 4y ago
  Oh I see, the thing on the left is logical and the thing on the right is on the wire. Thanks for helping me read it.

bamazizi 4y ago

This project does have potentials. We were using Gogo up until a few months ago until it was deprecated and replaced the pipeline with Buf.build. It took several days of head scratching but once we figured it out it was basically ZERO code change in 100s of proto and Go files. If vtprotobuf would be adopted as an optional compile module in Buf then I would love try it. I just don't think we would want to move away from Buf.

everybodyknows 4y ago

> There exist other alternatives, including Cap’n Proto, CBOR, and Avro. The names of the alternatives are links to references. All technical articles should start this way, well done.

irq-1 4y ago

> Some paragraphs were editorialized by ChatGPT, using “editorialize and keep it short” as a prompt. The result was proofread by a human for correctness. The main idea is that ChatGPT should be better at English than me. This should be at the TOP not at the end.

sliken 4y ago
Why? The author is putting effort into writing as clearly as they can, do the tools really matter? Do I need to announce that my sister is proof reading my writing? Or that I use the assistant inside Microsoft Word? Where do you draw the line?