3 ms·

Just for fun, how often do regular-sized companies that deal in regular-sized traffic need Protobuf to accomplish their goals in the first place, compared to JS

Just for fun, how often do regular-sized companies that deal in regular-sized traffic need Protobuf to accomplish their goals in the first place, compared to JSON or even XML with basic string marshalling?

tcfhgj 9mo ago
Well, protobuf allows to generate easy to use code for parsing defined data and service stubs for many languages and is one of the faster and less bandwidth wasting options
bluGill 9mo ago
In most languages protobuf is eaiser because it generates the boilerplate. And protobuf is cross language so even if you are working in javascript where json is native protobuf is still faster because the other side can be whatever and you are not spending their time parsing.
- t-writescode 9mo ago
  In most languages I’ve worked in, there is no boiler plate for json either, and barely any for XML. You make a data class of some sort and it “just works”. Not having that functionality is a weakness of a language or its support tools at this point, to me.
jonathanstrange 9mo ago
Protobuf is fantastic because it separates the definition from the language. When you make changes, you recompile your definitions to native code and you can be sure it will stay compatible with other languages and implementations.
- speed_spread 9mo ago
  You mean like WSDL, OpenAPI and every other schema definition format? Well I agree. Contract-first is great. You provide your clients with the specs and let them generate their own bindings. And as a client they're great too because I can also easily generate a mock server implementation that I can use in tests.
Chiron1991 9mo ago
It's not just about traffic. IoT devices (or any other low-powered devices for that matter) also like protobuf because of its comparatively high efficiency.
tuetuopay 9mo ago
Type safety. The contract is the law instead of a suggestion like JSON. Having a way to describe your whole API and generate bindings is a godsend. Yes, it can be done with JSON and OpenApi, yet it’s not mandatory.
- 9rx 9mo ago
  > Yes, it can be done with JSON and OpenApi, yet it’s not mandatory. It is not mandatory for Protobuf either. You can construct a protobuf message with an implied structure just as you can with JSON. It does not violate the spec. Protobuf ultimately gets the nod because it has better tooling (which isn't to be taken as praise towards Protobuf's tooling, but OpenAPI is worse).
vouwfietsman 9mo ago
Besides the other comments already here about code gen & contracts, a bigger one for me to step away from json/xml is binary serialization. It sounds weird, and its totally dependent on your use case, but binary serialization can make a giant difference. For me, I work with 3D data which is primarily (but not only) tightly packed arrays of floats & ints. I have a bunch of options available: 1. JSON/XML, readable, easy to work with, relatively bulky (but not as bad as people think if you compress) but no random access, and slow floating point parsing, great extensibility. 2. JSON/XML + base64, OK to work with, quite bulky, no random access, faster parsing, but no structure, extensible. 3. Manual binary serialization: hard to work with, OK size (esp compressed), random access if you put in the effort, optimal parsing, not extensible unless you put in a lot of effort. 4. Flatbuffers/protobuf/capn-proto/etc: easy to work with, great size (esp compressed), random access, close-to-optimal parsing, extensible. Basically if you care about performance, you would really like to just have control of the binary layout of your data, but you generally don't want to design extensibility and random access yourself, so you end up sacrificing explicit layout (and so some performance) by choosing a convenient lib. We are a very regularly sized company, but our 3D data spans hundreds of terabytes. (also, no, there is no general purpose 3D format available to do this work, gltf and friends are great but have a small range of usecases)
- physicsguy 9mo ago
  This was the norm many years ago, I worked on a simulation software which existed long before Protobuf was even an apple in it's authors eyes. The whole thing was on a server architecture with a Java (later ported to Qt) GUI and a C++ core. The solver periodically sent data in a custom binary format over TCP for vector fields and things.
- t-writescode 9mo ago
  This use case totally makes sense of course. I’m thinking about why people use Protobuf for their string, uuid and int powered CRUD app.
  - tucnak 9mo ago
    You're making assumptions about what kind of software people write. For a Hacker News degenerate, everything in the world revolves around bean-counting B2B SaaS CRUD crap, but it doesn't mean it's all there is to the world, right? You would be shocked how much networked computer software (not everything is a website) exists that is NOT a CRUD "app."
pjmlp 9mo ago
I never used it, coding since 1986.
izacus 9mo ago
I dunno, are you sure you can manually write correct de/serializaiton for JSON and XML so strings, floats and integer formats correctly get parsed between JavaScript, Java, Python, Go, Rust, C++ and any other languages? Do you want to maintain that and debug that? Do you want to do all of that without help of a compiler enforcing the schema and failing compiles/CI when someone accidentally changes the schema? Because you get all of that with protobuf if you use them appropriately. You can of course build all of this yourself... and maybe it'll even be as efficient, performant and supported. Maybe.
- nicman23 9mo ago
  i mean you can always go mono or duo language and then it is really not that of an issue
  - eklavya 9mo ago
    That would make sense if protobuf was complex, bloated, slow. But it's not, so the question should be why not use it, unless you are doing browser stuff.
    - 9rx 9mo ago
      If you are going to use it elsewhere, why not use it for browser stuff too?
      - eklavya 9mo ago
        I would advise against it. Too much friction, try it, maybe you will have a different experience than mine.
        9rx 9mo ago
        I am curious about what kind of friction you encountered. Were you generating ad-hoc protobuf messages? Assuming you were using Protobufs as they are usually used, meaning under generated code, I saw no difference between using it in Javascript and any other language in my experience. The wire format is beyond your concern. At least it is no more of your concern than it is in any other environment. There are a number of different generator implementations for Javascript/Typescript. Some of them have some peculiar design choices. Is that where you found issue? I would certainly agree with that, but others aren't so bad. That doesn't really have anything to do with the browser, though. You'd have the same problem using protobufs under Node.js.

tcfhgj 9mo ago

Well, protobuf allows to generate easy to use code for parsing defined data and service stubs for many languages and is one of the faster and less bandwidth wasting options

bluGill 9mo ago

In most languages protobuf is eaiser because it generates the boilerplate. And protobuf is cross language so even if you are working in javascript where json is native protobuf is still faster because the other side can be whatever and you are not spending their time parsing.

t-writescode 9mo ago
In most languages I’ve worked in, there is no boiler plate for json either, and barely any for XML. You make a data class of some sort and it “just works”. Not having that functionality is a weakness of a language or its support tools at this point, to me.

jonathanstrange 9mo ago

Protobuf is fantastic because it separates the definition from the language. When you make changes, you recompile your definitions to native code and you can be sure it will stay compatible with other languages and implementations.

speed_spread 9mo ago
You mean like WSDL, OpenAPI and every other schema definition format? Well I agree. Contract-first is great. You provide your clients with the specs and let them generate their own bindings. And as a client they're great too because I can also easily generate a mock server implementation that I can use in tests.

Chiron1991 9mo ago

It's not just about traffic. IoT devices (or any other low-powered devices for that matter) also like protobuf because of its comparatively high efficiency.

tuetuopay 9mo ago

Type safety. The contract is the law instead of a suggestion like JSON. Having a way to describe your whole API and generate bindings is a godsend. Yes, it can be done with JSON and OpenApi, yet it’s not mandatory.

9rx 9mo ago
> Yes, it can be done with JSON and OpenApi, yet it’s not mandatory. It is not mandatory for Protobuf either. You can construct a protobuf message with an implied structure just as you can with JSON. It does not violate the spec. Protobuf ultimately gets the nod because it has better tooling (which isn't to be taken as praise towards Protobuf's tooling, but OpenAPI is worse).

vouwfietsman 9mo ago

Besides the other comments already here about code gen & contracts, a bigger one for me to step away from json/xml is binary serialization. It sounds weird, and its totally dependent on your use case, but binary serialization can make a giant difference. For me, I work with 3D data which is primarily (but not only) tightly packed arrays of floats & ints. I have a bunch of options available: 1. JSON/XML, readable, easy to work with, relatively bulky (but not as bad as people think if you compress) but no random access, and slow floating point parsing, great extensibility. 2. JSON/XML + base64, OK to work with, quite bulky, no random access, faster parsing, but no structure, extensible. 3. Manual binary serialization: hard to work with, OK size (esp compressed), random access if you put in the effort, optimal parsing, not extensible unless you put in a lot of effort. 4. Flatbuffers/protobuf/capn-proto/etc: easy to work with, great size (esp compressed), random access, close-to-optimal parsing, extensible. Basically if you care about performance, you would really like to just have control of the binary layout of your data, but you generally don't want to design extensibility and random access yourself, so you end up sacrificing explicit layout (and so some performance) by choosing a convenient lib. We are a very regularly sized company, but our 3D data spans hundreds of terabytes. (also, no, there is no general purpose 3D format available to do this work, gltf and friends are great but have a small range of usecases)

physicsguy 9mo ago
This was the norm many years ago, I worked on a simulation software which existed long before Protobuf was even an apple in it's authors eyes. The whole thing was on a server architecture with a Java (later ported to Qt) GUI and a C++ core. The solver periodically sent data in a custom binary format over TCP for vector fields and things.
t-writescode 9mo ago
This use case totally makes sense of course. I’m thinking about why people use Protobuf for their string, uuid and int powered CRUD app.
- tucnak 9mo ago
  You're making assumptions about what kind of software people write. For a Hacker News degenerate, everything in the world revolves around bean-counting B2B SaaS CRUD crap, but it doesn't mean it's all there is to the world, right? You would be shocked how much networked computer software (not everything is a website) exists that is NOT a CRUD "app."

pjmlp 9mo ago

I never used it, coding since 1986.

izacus 9mo ago

I dunno, are you sure you can manually write correct de/serializaiton for JSON and XML so strings, floats and integer formats correctly get parsed between JavaScript, Java, Python, Go, Rust, C++ and any other languages? Do you want to maintain that and debug that? Do you want to do all of that without help of a compiler enforcing the schema and failing compiles/CI when someone accidentally changes the schema? Because you get all of that with protobuf if you use them appropriately. You can of course build all of this yourself... and maybe it'll even be as efficient, performant and supported. Maybe.

nicman23 9mo ago
i mean you can always go mono or duo language and then it is really not that of an issue
- eklavya 9mo ago
  That would make sense if protobuf was complex, bloated, slow. But it's not, so the question should be why not use it, unless you are doing browser stuff.
  - 9rx 9mo ago
    If you are going to use it elsewhere, why not use it for browser stuff too?
    - eklavya 9mo ago
      I would advise against it. Too much friction, try it, maybe you will have a different experience than mine.
      - 9rx 9mo ago
        I am curious about what kind of friction you encountered. Were you generating ad-hoc protobuf messages? Assuming you were using Protobufs as they are usually used, meaning under generated code, I saw no difference between using it in Javascript and any other language in my experience. The wire format is beyond your concern. At least it is no more of your concern than it is in any other environment. There are a number of different generator implementations for Javascript/Typescript. Some of them have some peculiar design choices. Is that where you found issue? I would certainly agree with that, but others aren't so bad. That doesn't really have anything to do with the browser, though. You'd have the same problem using protobufs under Node.js.