19 ms·

Why I stopped using JSON for my APIs

180 points by barremian 10mo ago

spagoop 10mo ago
Is it just me or is this article insanely confusing? With all due respect to the author, please be mindful of copy editing LLM-assisted writing. There is a really interesting discussion underneath of this as to the limitations of JSON along with potential alternatives, but I can't help but distrust this writing due to how much it sounds like an LLM.
- port11 10mo ago
  I don't think it's LLM-generated or even assisted. It's kinda like how I write when I don't want to really argue a point but rather get to the good bits. Seems like the author just wanted to talk about Protobuf without bothering too much about the issues with JSON (though some are mentioned).
- dkdcio 10mo ago
  do you have any evidence that the author used a LLM? focusing on the content, instead of the tooling used to write the content, leads to a lot more productive discussions I promise you cannot tell LLM-generated content from non-LLM generated content. what you think you’re detecting is poor quality, which is orthogonal to the tooling used
  - spagoop 10mo ago
    Fair point, to be constructive here, LLMs seem to love lists and emphasizing random words / phrases with bold. Those two are everywhere. Not a smoking gun but enough to tune out. I am not dismissing this as being slop and actually have no beef with using LLMs to write but yes, as you call out, I think it's just poorly written or perhaps I'm not the specific audience for this. Sorry if this is bad energy, I appreciate the write up regardless.
- pavel_lishin 10mo ago
  > Is it just me or is this article insanely confusing? I didn't find it confusing. I found it unconvincing, but the argument itself was pretty clear. I just disagreed with it.
- phyzome 10mo ago
  It wasn't confusing, but yeah, it smelled strongly of LLMs.
codewritero 10mo ago
I love to see people advocating for better protocols and standards but seeing the title I expected the author to present something which would be better in the sense of supporting the same or more use cases with better efficiency and/or ergonomics and I don't think that protobuf does that. Protobuf has advantages, but is missing support for a tons of use cases where JSON thrives due to the strict schema requirement. A much stronger argument could be made for CBOR as a replacement for JSON for most use cases. CBOR has the same schema flexibility as JSON but has a more concise encoding.
- port11 10mo ago
  I think the strict schema of Protobuf might be one of the major improvements, as most APIs don't publish a JSON schema? I've always had to use ajv or superstruct to make sure payloads match a schema, Protobuf doesn't need that (supposedly).
  - thayne 10mo ago
    One limitation of protobuf 3 schemas, is they doen't allow required fields. That makes it easier to remove the field in a later version in a backwards compatible way, but sometimes fields really are required, and the message doesn't make any sense without them. Ideally, IMO, if the message is missing those fields, it would fail to parse successfully. But with protobuf, you instead get a default value, which could potentially cause subtle bugs.
    - port11 10mo ago
      Okay, this is a definite issue, you're still stuck validating inputs/outputs.
- youngtaff 10mo ago
  We need browsers to support CBOR APIs… and it shouldn’t be that hard as they all have internal implementations now
  - 10mo ago
written-beyond 10mo ago
Idk I built a production system and ensured all data transfers, client to server and server to client were proto buf and it was a pain. Technically, it sounds really good but the actual act of managing it is hell. That or I need a lot of practice to use them, at that point shouldn't I just use JSON and get on with my life.
- Arainach 10mo ago
  What issues did you have? In my experience, most things that could be called painful with protobuf would be bigger pains with things like JSON. Making changes to messages in a backwards-compatible way can be annoying, but JSON allowing you to shoot yourself in the foot will take more time and effort to fix when it's corrupting data in prod than protobuf giving you a compile error would.
  - written-beyond 10mo ago
    Well at the bare minimum setting up proto files and knowing where they live across many projects. If they live in their own project, making a single project be buildable with a git clone gets progressively more complex. You now need sub modules to pull in your protobuf definitions. You now also need the protobuf tool chain to be available in your environment you just cloned to. If that environment has the wrong version the build fails, it starts to get frustrating pretty fast. Compare that to json, yes I don't get versioning and a bunch of other fancy features but... I get to finish my work, build and test pretty quickly.
- barremian 10mo ago
  I think the it-just-works nature and human readability for debugging JSON cannot be overstated. Most people and projects are content to just use JSON even if protos offer some advantages, if not only to save time and resources. Whether the team saves times in the longer when using protos is a question in its own.
  - IshKebab 10mo ago
    There are plenty of it-doesn't-just-work things about JSON though. Sending binary data or 64-bit integers is a huge pain. Or maps with non-string keys, or ordered maps. Plus JSON doesn't scale well with message size because it doesn't use TLV so parsing any of a message requires parsing all of it. It's not some perfect format. That said, I'm disappointed with Protobuf too. Especially the handling of optional/default fields. I believe they did eventually add an `optional` tag so you can at least distinguish missing vs default field values. The lack of required fields makes it very annoying to work with though. And no, there's no issue with required fields in general. The only reason it doesn't have them is because the implementation in Protobuf 2 caused issues and Google had to have a smooth transition from that. If you're starting a greenfield project it seems silly to opt in to Google's tech debt. If you're thinking "but schema evolution!!" well yeah all you need is a way to have versioned structs and then you can mark fields as being required for ranges of versions. So you can still remove required fields without breaking backwards compatibility.
esafak 10mo ago
It's premature and maybe presumptuous of him to be advertising protobufs when he hasn't heard of the alternatives yet. I'll engage the article after he discovers them...
- deleted 10mo ago
  [deleted]
pzmarzly 10mo ago
> With Protobuf, that’s impossible. Unless your servers and clients push at different time, thus are compiled with different versions of your specs, then many safety bets are off. There are ways to be mostly safe (never reuse IDs, use unknown-field-friendly copying methods, etc.), but distributed systems are distributed systems, and protobuf isn't a silver bullet that can solve all problems on author's list. On the upside, it seems like protobuf3 fixed a lot of stuff I used to hate about protobuf2. Issues like: > if the field is not a message, it has two states: > - ... > - the field is set to the default (zero) value. It will not be serialized to the wire. In fact, you cannot determine whether the default (zero) value was set or parsed from the wire or not provided at all are now gone if you stick to using protobuf3 + `message` keyword. That's really cool.
- brabel 10mo ago
  No type system survives going through a network.
  - dhussoe 10mo ago
    yes, but any sane JSON parsing library (Rust Serde, kotlinx-serialization, Swift, etc.) will raise an error when you have the wrong type or are missing a required field. and any JSON parsing callsite is very likely also an IO callsite so you need to handle errors there anyways, all IO can fail. then you log it or recover or whatever you do when IO fails in some other way in that situation. this seems like a problem only if you use JSON.parse or json.loads etc. and then just cross your fingers and hope that the types are correct, basically doing the silent equivalent of casting an "any" type to some structure that you assume is correct, rather than strictly parsing (parse, don't validate) into a typed structure before handing that off to other code.
    - koakuma-chan 10mo ago
      > strictly parsing (parse, don't validate) That's called validating? Zod is a validation library. But yeah, people really need to start strictly parsing/validating their data. One time I had an interview and I was told yOu DoN'T tRuSt YoUr BaCkeNd?!?!?!?
volemo 10mo ago
S-expressions exist since 1960, what more do you need? /s
Jemaclus 10mo ago
"Better than JSON" is a pretty bold claim, and even though the article makes some great cases, the author is making some trade-offs that I wouldn't make, based on my 20+ year career and experience. The author makes a statement at the beginning: "I find it surprising that JSON is so omnipresent when there are far more efficient alternatives." We might disagree on what "efficient" means. OP is focusing on computer efficiency, where as you'll see, I tend to optimize for human efficiency (and, let's be clear, JSON is efficient _enough_ for 99% of computer cases). I think the "human readable" part is often an overlooked pro by hardcore protobuf fans. One of my fundamental philosophies of engineering historically has been "clarity over cleverness." Perhaps the corollary to this is "...and simplicity over complexity." And I think protobuf, generally speaking, falls in the cleverness part, and certainly into the complexity part (with regards to dependencies). JSON, on the other hand, is ubiquitous, human readable (clear), and simple (little-to-no dependencies). I've found in my career that there's tremendous value in not needing to execute code to see what a payload contains. I've seen a lot of engineers (including myself, once upon a time!) take shortcuts like using bitwise values and protobufs and things like that to make things faster or to be clever or whatever. And then I've seen those same engineers, or perhaps their successors, find great difficulty in navigating years-old protobufs, when a JSON payload is immediately clear and understandable to any human, technical or not, upon a glance. I write MUDs for fun, and one of the things that older MUD codebases do is that they use bit flags to compress a lot of information into a tiny integer. To know what conditions a player has (hunger, thirst, cursed, etc), you do some bit manipulation and you wind up with something like 31 that represents the player being thirsty (1), hungry (2), cursed (4), with haste (8), and with shield (16). Which is great, if you're optimizing for integer compression, but it's really bad when you want a human to look at it. You have to do a bunch of math to sort of de-compress that integer into something meaningful for humans. Similarly with protobuf, I find that it usually optimizes for the wrong thing. To be clear, one of my other fundamental philosophies about engineering is that performance is king and that you should try to make things fast, but there are certainly diminishing returns, especially in codebases where humans interact frequently with the data. Protobufs make things fast at a cost, and that cost is typically clarity and human readability. Versioning also creates more friction. I've seen teams spend an inordinate amount of effort trying to ensure that both the producer and consumer are using the same versions. This is not to say that protobufs are useless. It's great for enforcing API contracts at the code level, and it provides those speed improvements OP mentions. There are certain high-throughput use-cases where this complexity and relative opaqueness is not only an acceptable trade off, but the right one to make. But I've found that it's not particularly common, and people reaching for protobufs are often optimizing for the wrong things. Again, clarity over cleverness and simplicity over complexity. I know one of the arguments is "it's better for situations where you control both sides," but if you're in any kind of team with more than a couple of engineers, this stops being true. Even if your internal API is controlled by "us," that "us" can sometimes span 100+ engineers, and you might as well consider it a public API. I'm not a protobuf hater, I just think that the vast majority of engineers would go through their careers without ever touching protobufs, never miss it, never need it, and never find themselves where eking out that extra performance is truly worth the hassle.
catchmeifyoucan 10mo ago
I wonder if we can write an API w/ JSON the usual way and change the final packaging to send it over protobuf.
- pstuart 10mo ago
  If you're using Go then this framework let's you work with protobufs and gives you a JSON rest-like service for free: https://github.com/grpc-ecosystem/grpc-gateway https://github.com/grpc-ecosystem/grpc-gateway
- bglusman 10mo ago
  Sure... https://protobuf.dev/programming-guides/json/ https://protobuf.dev/programming-guides/json/ I was pushing at one point for us to have some code in our protobuf parsers that would essentially allow reading messages in either JSON or binary format, though to be fair there's some overhead that way by doing some kind of try/catch, but, for some use cases I think it's worth it...
- lanigone 10mo ago
  envoy (the proxy) can transcode RESTful APIs to internal grpc services and vice versa. you can even map like url params etc. to proto fields. it works well, even server streaming.
brabel 10mo ago
Mandatory comment about ASN.1, a protocol from 1984, already did what Protobuf does, with more flexibility. Yes, it's a bit ugly but if you stick to the DER encoding it's really not worse than Protbuf at all. Check out the Wikipedia example: https://en.wikipedia.org/wiki/ASN.1#Example_encoded_in_DER https://en.wikipedia.org/wiki/ASN.1#Example_encoded_in_DER Protobuf is ok but if you actually look at how the serializers work, it's just too complex for what it achieves.
- bloppe 10mo ago
  What makes it too complex in your opinion?
  - kragen 10mo ago
    Experience with ASN.1.
    - kstrauser 10mo ago
      Same. My take on ASN.1 is that no one would pay me what I would ask to work on ASN.1. I’d only touch it if I had to parse files from an outside source, and a package already exists in my language of choice that’s capable of parsing those files.
- zzo38computer 10mo ago
  I also think ASN.1 DER is better (there are other formats, but in my opinion, DER is the only good one, because BER is too messy). I use it in some of my stuff, and when I can, my new designs also use ASN.1 DER rather than using JSON and Protobuf etc. (Some types are missing from standard ASN.1 but I made up a variant called "ASN.1X" which adds some additional types such as key/value list and some others. With the key/value list type added, it is now a superset of the data model of JSON, so you can convert JSON to ASN.1X DER.) (I wrote a implementation of DER encoding/decoding in C, which is public domain and FOSS.)
- morshu9001 10mo ago
  ASN.1 is way overengineered to the point of making it hard to support. You don't need inheritance for example.
wilg 10mo ago
One of the best parts of Protobuf is that there's a fully compatible JSON serialization and deserialization spec, so you can offer a parallel JSON API with minimal extra work.
- bglusman 10mo ago
  Yes! Came to comments to see if that was discussed/commented on above with link: https://protobuf.dev/programming-guides/json/ https://protobuf.dev/programming-guides/json/
pyrolistical 10mo ago
Compressed JSON is good enough and requires less human communication initially. Sure it will blow up in your face when a field goes missing or value changes type. People who advocate paying the higher cost ahead of time to perfectly type the entire data structure AND propose a process to do perform version updates to sync client/server are going to lose most of the time. The zero cost of starting with JSON is too compelling even if it has a higher total cost due to production bugs later on. When judging which alternative will succeed, lower perceived human cost beats lower machine cost every time. This is why JSON is never going away, until it gets replaced with something with even lower human communication cost.
- deleted 10mo ago
  [deleted]
- esafak 10mo ago
  It won't go away in the same way COBOL won't. That does not mean we should be using it everywhere for greenfield projects.
- DyslexicAtheist 10mo ago
  > People who advocate paying the higher cost ahead of time to perfectly type the entire data structure AND propose a process to do perform version updates to sync client/server are going to lose most of the time. that's true. But people also rather argue about security vulnerabilities than getting it right from the get-go. Why spend an extra 15 mins effort during design when you can spend 3 months revisiting the ensuing problem later.
  - jstanley 10mo ago
    Alternatively: why spend an extra 15 mins on protobuf every other day, when you can put off the 3-month JSON-revisiting project forever?
    - OccamsMirror 10mo ago
      I use ConnectRPC (proto). I definitely do not spend any extra time. In fact the generated types for my backend and frontend saves me time.
taco_emoji 10mo ago
i really hate this blog post format of posting the new thing you discovered as if it's objectively better than the previous thing. no it's not, you just like it better, which is FINE, just own it
morshu9001 10mo ago
Protos are great. Last time I did a small project in NodeJS, I set up a server that defines the entire API in a .proto and serves each endpoint as either proto or json, depending on the content header. Even if the clients want to use json, at least I can define the whole API in proto spec instead of something like Swagger. So my question is, why didn't Google just provide that as a library? The setup wasn't hard but wasn't trivial either, and had several "wrong" ways to set up the proto side. They also bait most people with gRPC, which is its own separate annoying thing that requires HTTP/2, which even Google's own cloud products don't support well (e.g App Engine). P.S. Text proto is also the best static config language. More readable than JSON, less error-prone than YAML, more structure than both.
- noctune 10mo ago
  You might be interested in https://connectrpc.com/ https://connectrpc.com/. It's basically what you describe, though it's not clear to me how well supported it is.
  - morshu9001 10mo ago
    Yeah that one looked good. I don't remember why I didn't use it that time, maybe just felt it was easy enough to DIY that I didn't feel like using another dep (given that I already knew express and proto in isolation). The thing is, Google themselves had to lead the way on this if they wanted protobuf to be mainstream like JSON.
- c-cube 10mo ago
  That's what Twirp (https://github.com/twitchtv/twirp https://github.com/twitchtv/twirp) is about. Protobuf or JSON, over any HTTP, with a simple URL schema. It's fairly simple.
  - rileymichael 10mo ago
    highly recommend twirp, even in current year. connectrpc seems to have stalled so there isn't a ton of languages w/server support, and because of the grpc interop on top of their own protocol it's a bit of an undertaking to roll your own. the twirp spec however is so simple you can throw together your own code generator in an afternoon for whatever language you want.
jtrn 10mo ago
I like Python-like indentation, but I usually read Python in an IDE or code blocks. JSON in a non-monospace environment might be problematic with some fonts. Hell, I pass JSON around in emails and word processors all the time.
wg0 10mo ago
Protos don't work out of the box in any browsers as far as I checked last time unless you're willing to deploy a proxy in front to do the translation and it requires extra dependency on the browser as well. Plus - tooling. JSON might not be simpler or strict but it gets the job done. If JSON's size or performance is causing you to go out of business, you surely have bigger problems than JSON.
- taftster 10mo ago
  Right, I find the use of protobuf lacking with direct support in the browser. Since JSON is a native data encoding format of the browser (effectively), it's just easier to have a JSON-based API. Yes, there are abstractions or other hacks that would bolt on protobuf support in the browser, but that's not ideal in my mind. Protobuf is an ideal exchange format when you're not dealing with the public as a whole. That is, a private or corporate API for your data processing pipelines, etc.
dhussoe 10mo ago
I like that JSON parsing libraries (Serde etc.) allow you to validate nullability constraints at parse-time. Protobuf's deliberate lack of support for required fields means that either you kick that down to every callsite, or you need to build another parsing layer on top of the generated protobuf code. Now, there is a serde_protobuf (I haven't used it) that I assume allows you to enforce nullability constraints but one of the article's points is that you can use the generated code directly and: > No manual validation. No JSON parsing. No risk of type errors. But this is not true—nullability errors are type errors. Manual validation is still required (except you should parse, not validate) to make sure that all of the fields your app expects are there in the response. And "manual" validation (again, parse don't validate) is not necessary with any good JSON parsing library, the library handles it.
- myvoiceismypass 10mo ago
  'Parse, don't validate' is such great reading: https://lexi-lambda.github.io/blog/2019/11/05/parse-don-t-validate/ https://lexi-lambda.github.io/blog/2019/11/05/parse-don-t-va...
recursivecaveat 10mo ago
My dream binary format is schema driven, as compact and efficient as Capt Proto or such, but just optionally embeds the entire schema into the message. Then we can write a vim plugin that just opens the file in human readable form without having to fish for the schema. Whenever I am using binary formats, it's because I have a list of millions of objects of the same types. Seems to me that you may as well tack 1KB of schema onto a 2GB message and make it self-describing so everyone's life is easier.
- barremian 10mo ago
  You could have a look at Avro (https://avro.apache.org/ https://avro.apache.org/) and Yardl (https://microsoft.github.io/yardl/ https://microsoft.github.io/yardl/).
- EdwardDiego 10mo ago
  As another user suggested, Avro is something to look into.
- HelloNurse 10mo ago
  For many web services it would be more often 200 KB of schema (many possible request and responses, some of them complex) tacked onto a less than 1 KB message (brief requests and acknowledgements without significant data inside).
- squirrellous 10mo ago
  It’s possible to build this around protobuf. Google has a rich internal protobuf ecosystem that does this and supports querying large amounts of protobuf data without specifying schemas. They are only selectively open sourced. Have a look at riegeli if you are interested. https://github.com/google/riegeli https://github.com/google/riegeli
coffeeaddict1 10mo ago
> protobuf As an aside, like all things Google, their C++ library is massive (14mb dll) and painful to build (takes nearly 10 minutes on my laptop).
- mb7733 10mo ago
  Yeah and (1) the codegen produces massive headers that slow compilation of anything that touches them (2) the generated classes are really awkward to use. Not a big fan of the experience of protobuffer generated code in a large C++ code base. It's lead to a huge layer of adapters and native c++ classes equivalent to the protobuffers classes to try and mitigate these issues.
keithnz 10mo ago
CBOR is a pretty good middle ground
teleforce 10mo ago
Kudos to the poster and the author of this article. I think this is by far the most insightful technical post I've read this year on HN. >If you develop or use an API, there’s a 99% chance it exchanges data encoded in JSON. Just wondering if the inherent defiencies of JSON can somewhat be improved by CUE lang since the former is very much pervasive and the latter understand JSON [1],[2]. [1] Configure Unify Execute (CUE): Validate, define, and use dynamic and text‑based data: https://cuelang.org/ https://cuelang.org/ [2] Cue – A language for defining, generating, and validating data: https://news.ycombinator.com/item?id=20847943 https://news.ycombinator.com/item?id=20847943
- Joel_Mckay 10mo ago
  BSON dealt with a lot of the JSON limitations, and ambiguous type issues. Batching with message pooling to a transaction payload size limit actually made it performant. =3
loph 10mo ago
How many times has this problem been "solved"? https://en.wikipedia.org/wiki/DCE/RPC https://en.wikipedia.org/wiki/DCE/RPC DCE/RPC worked in 1993, and still does today. Protocol buffers is just another IDL.
- PantaloonFlames 10mo ago
  DCE STILL WORKS? Where!??
JoshMock 10mo ago
Protobuf is a great format with a lot of benefits, but it's missing one that I wish it could support: zero-copy. The ability to transport data between processes, services and languages with effectively zero time spent on serialization and deserialization. It appears possible in some cases but it's not universally the case. Which means that similar binary transport formats that do support zero-copy, like Cap'n Proto, offer most or all of the perks described in this post, with the addition of ensuring that serialization and deserialization are not a bottleneck when passing data between processes.
- jonny_eh 10mo ago
  Is that a format/serialization issue, or library/implementation issue?
  - __s 10mo ago
    Format: https://news.ycombinator.com/item?id=23589117 https://news.ycombinator.com/item?id=23589117
  - ElectricalUnion 10mo ago
    Serialization issue. From the Introduction to Cap’n Proto: "Cap’n Proto is INFINITY TIMES faster than Protocol Buffers. (...) there is no encoding/decoding step. The Cap’n Proto encoding is appropriate both as a data interchange format and an in-memory representation, so once your structure is built, you can simply write the bytes straight out". I take it as a rationalization of what OLE Compound File Binary - internal Microsoft Office memory structures serialized "raw" as file format - would look like if they paid more attention to being backward and forward compatible and extensible.
    - TillE 10mo ago
      Google has a library/format for that too, with FlatBuffers. Different use cases and advantages really, not clearly better/worse.
      - 10mo ago
brunoluiz 10mo ago
I am curious why the author did not consider ConnectRPC (http://connectrpc.com/ http://connectrpc.com/), which could be a great middle ground since it is compatible with both Protobuf and JSON served APIs. It is developed by Buf, which has been a leader Protobuf tooling.
dfabulich 10mo ago
> With JSON, you often send ambiguous or non-guaranteed data. You may encounter a missing field, an incorrect type, a typo in a key, or simply an undocumented structure. With Protobuf, that’s impossible. Everything starts with a .proto file that defines the structure of messages precisely. This deeply misunderstands the philosophy of Protobuf. proto3 doesn't even support required fields. https://protobuf.dev/best-practices/dos-donts/ https://protobuf.dev/best-practices/dos-donts/ > Never add a required field, instead add `// required` to document the API contract. Required fields are considered harmful by so many they were removed from proto3 completely. Protobuf clients need to be written defensively, just like JSON API clients.
- klysm 10mo ago
  It’s also conflating the serialization format with contracts
  - shortrounddev2 10mo ago
    Most web frameworks do both at the same time to the point where having to write code which enforced a type contract after deserializing is a delabreaker for me. I eant to be able to define my DTOs in one place, once, and have it both deserialize and enforce types/format. Anything else is code smell
    - Seattle3503 10mo ago
      I'm in the same boat. I mostly write Rust and Python. Using serde_json and Pydantic, you get deserialization and validation at the same time. It allows you to de-serialize really "tight" types. Most of my APIs are internal APIs that accept breaking changes easily. My experience with protobufs is that it was created to solve problems in large systems with many teams and APIs, where backwards compatibility is important. There are certainly systems where you can't "just" push through a breaking API change, and in those cases protobufs make sense.
      - masklinn 10mo ago
_el1s7 10mo ago
Why I stopped caring about "Why I stopped [insert something widely used here]" click bait articles
_heimdall 10mo ago
I need to dig deeper into Protobuf. I've never quite understood the benefit of Protobug over XML.
PunchyHamster 10mo ago
"I've used this optimization technique to make app faster" The app 20req/sec The app after optimizations: 20req/sec (It waits for db query anyway)
- PantaloonFlames 10mo ago
  Yes. Proto makes sense when the request rate is much higher and the network is constrained. Otherwise, json is sufficient.
  - kragen 10mo ago
    If it's the network that's constrained and not the CPU, gzipped JSON will often beat protobufs.
bzmrgonz 10mo ago
What about TOON op? I understand that's the standard poised to takeover from Json. Your thoughts??
beders 10mo ago
Know the consumer of your API. If that is just your team, use whatever tech gets you there quick. However, if you need to provide some guarantees to a second or third party with your API, embrace standards like JSON, even better, use content negotiation.
rizky05 10mo ago
[dead]
socalgal2 10mo ago
https://github.com/ajv-validator/ajv https://github.com/ajv-validator/ajv 138 million downloads from npm in the last week. Yes, you can validate your JSON
cryptonector 10mo ago
If you're gonna switch from JSON to... PB, then you might as well switch to flatbuffers before you realize it's better than PB and save yourself a lot of trouble.
lenkite 10mo ago
There is no mention in this thread of how the author is using Dart and Shelf for his REST API's. Code is rather readable and elegant. This is not a combo I have every tried before. Does anyone have any experience of how it compares versus REST services written in Go/Rust/Python ?
lazy_afternoons 10mo ago
IIRC from the red wild boar book, JSONs biggest win is that it got the adoption. Getting everyone to agree on a standard was/is/will be the tougher part.
rock_artist 10mo ago
I feel few points weren’t addressed in the article. 1. Size, biggest problem with JSON can happen when things gets too big. So here other formats might be better. Yet, as a reminder JSON has the binary version named BSON. 2. Zero batteries. JSON is readable by humans but also almost self explanatory format. Most languages has built in or quick drop in for json. Still, it’s easy to implement a limited JSON parser from scratch when in need (eg. Pure on func in C on a tiny device). Working with Protobuf and MsgPack in the past, You have much more tooling involved especially if data passes between parts written in different languages. 3. Validation, JSON is simple. But there are solutions such as JSON Schema.
- paulddraper 10mo ago
  BSON is not simply a binary encoding of JSON. It is a JSON superset with binary encoding, created by MongoDB. (And there is even a JSON encoding of BSON, called extended JSON.) AFAIK there is no widely adopted binary pure adaptation of JSON. (There are application-specific storage formats, like PostgreSQL JSONB, or SQLite JSONB.) ——- Moreover, JSON is relatively compact. BSON or other self descriptive binary formats are often around the same size of JSON. MessagePack aggressively tries to be compact and is, depending on the data. BSON doesn’t try to be compact, rather it improves the parse speed.
  - WorldMaker 10mo ago
    CBOR [0] is probably the closest to a widely adopted "pure" adaptation of JSON. It is still technically a superset of JSON, but it tries to be closely matched and frequently cross-references the JSON specs directly, especially because it is also an IETF tracked standard like JSON, and in so far as widely adopted is concerned is included in web standards like WebAuthn today. (For instance, you can't handle Passkeys without some amount of CBOR. The presumed next steps to wider adoption now that all browsers have internal CBOR encoders/decoders would be to add a web platform JS API for it as well.) However, yes, JSON compresses extremely well even in ancient gzip, but especially in Brotli, and desiring compaction of your API responses alone isn't necessarily the best reason to prefer a binary encoding of JSON to just using JSON and letting compression do its thing. [0] https://www.rfc-editor.org/rfc/rfc8949.html https://www.rfc-editor.org/rfc/rfc8949.html
umvi 10mo ago
Protobufs are a pain to debug and maintain compared to json and modern browsers support zstd compression making json "efficient"
davedx 10mo ago
"Ultra-efficient" Searched the article, no mention of gzip, and how most of the time all that text data (html, js and css too!) you're sending over the wire will be automatically compressed to...... an efficient binary format! So really, the author should compare protobufs to gzipped JSON
- chillfox 10mo ago
  Last time I was evaluating different binary serialization formats for an API I was really hoping to get to use one of the cool ones, but gzipped JSON just beat everything and it wasn't even close.
  - theshrike79 10mo ago
    There are some compression formats that perform better than gzip, but it's very dependent on the data you're compressing and your timing requirements (is bandwidth or CPU more important to conserve). But in the end compressed JSON is pretty good. Not perfect, but good enough for many many things.
- MangoToupe 10mo ago
  I would think that serialization/deserialization time would be the largest drawback of json (at least for serving APIs). Pretty much all the other pain points can slowly be ironed out over time, albeit with deeply ugly solutions.
  - thayne 10mo ago
    It depends on what your data looks like. If your content is mostly UTF-8 text, with dynamic keys, then I wouldn't expect protobuf to have much of an advantage over JSON for parsing to an equivalent structure. On the other hand, if you have binary data that needs to be base64 encoded in JSON, then protobuf has a significant advantage.
- bloppe 10mo ago
  https://auth0.com/blog/beating-json-performance-with-protobuf/ https://auth0.com/blog/beating-json-performance-with-protobu...
sergiotapia 10mo ago
Personally, I looked into protobuf for our Elixir/React Native wombocombo but the second I realized we would have to deploy app updates when we added or removed a field from the response structure it became a non-starter. I can't imagine using protobuf when you're in the first 5 years of a product.
- xarope 10mo ago
  I'm pretty sure protobuf ignores new fields (if you add; assuming you add as an append, and not change the field ordering), and it recommends you not to remove a field to ensure backward compatibility.
gethly 10mo ago
Reading the first few paragraphs and immediately seeing PB made me instantly think of "Every master was once a beginner.". When you'll go through your own journey, and inevitably end back with json, do write another blog post :) ... we've all been there.
undefeated 10mo ago
Complaining about JSON and then proceeding to write your API in Dart is... interesting.
zabil 10mo ago
I have a slight dislike for JSON+REST for API's. The design overhead involved in determining the correct URL and HTTP method adds a layer of subjectivity to the design and bike shedding arguments. I’m not a huge fan of Protobuf/GRPC either, if there’s a better alternative I believe RPC is the right approach for exposing APIs.
globular-toast 10mo ago
> An API (Application Programming Interface) is a set of rules that allow two systems to communicate. In the web world, REST APIs ... are by far the most widespread. I too had this overly restrictive view of "APIs" for too long. One I started to think about it as the interface between any teo software components it really changed the way I did programming. In other words, a system itself is composed and that composition is done via APIs. There's no point treating "the API" as something special.
cheema33 10mo ago
I use GraphQL. It has a higher learning curve. But it addresses the shortcomings listed by the referenced blog article. It offers type safety, efficiency and modern tooling. And it is also human readable. If you use good tooling, you can have a mutation change a variable type in the database and that type change is automatically reflected in the middleware/backend and the typescript UI code. Not only that libraries like HotChocolate for asp.net come with built-in functions for filtering, pagination, streaming etc.
knallfrosch 10mo ago
I have never encountered the use case where the data sent by the backend over the normal CRUD operations was the bottleneck. But we have built protobuf into a web server that handles 2 requests per second. Why? We wanted to learn about it on the job. I think that's 99% of Protobuf usage.
mjmas 10mo ago
> The same message in Protobuf binary > → About 23 bytes. This appears to be a nice ai-generated result. There are already at least 23 bytes of data there (number 42 (1 byte) + string of 5 chars + string of 17 chars + 1 boolean), so that plus field overhead will be more.
liampulles 10mo ago
I know that OpenAPI code gen support is spotty, and that protobuf codegen (in my experience) is quite good, but all of this starts from the idea that the SaaS I'm consuming has actually documented their API properly. A sizable portion of the integrations I've built have had to be built by hand, because there are inevitable stupid quirks and/or failures I've had to work around. For these usecases, using JSON is preferable, because it is easy for me to see what I have actually been sent, not what the partially up to date spec says I should've been sent. This is consistent with the idea that communication over the internet should consist of (encrypted and compressed) plain text. It's because human beings are going to have to deal with human reality at the end of the day.
pjacotg 10mo ago
On the human readability concern, we use protobuf converted to text format. It looks JSON like so very readable and comes with all the other benefits of protobuf.
whatevaa 10mo ago
I can use text based API's (like with JSON) with nothing else but text editor and curl. No other tooling actually required. Meanwhile if binary protocol tooling in your stack sucks, then it just sucks. Had the joy of implementing calling SOAP service with client generated from wsdl in .net core 2/3 times. Tooling was shit poorly undocumented amd with crappy errors at that time. Run only with very specific versions in very specific way. And not much you can do about it, rolling your own SOAP client would be too expensive for our team. REST with JSON meanwhile is easy and we do it all the time, don't need any client, just give us request/response spec and any docs.
borplk 10mo ago
If you have a protobuf API, does it work in the js environment of browsers? Last time I checked (many years ago) the browser story wasn't good.
baquero 10mo ago
If you want to work with ProtoBuf as with other APIs, have a look at https://github.com/qaware/protocurl https://github.com/qaware/protocurl
protobuf-ai 10mo ago
I started working with protobuf on a project, and just went down the path of MCP. If anybody would like to try it out, it's here: https://www.protobuf.ai/ https://www.protobuf.ai/. Just a lightweight MCP server for schemas that plugs into a schema registry.
madFlasher 10mo ago
The author should check first the performance of protobuf serialization/deserialiazation in browsers Due to very native nature of JSON in browsers and node backend it usually also the fastest data format If for example you have C++ on backend and C++ on frontend - you'll definitely have some performance boost But for browsers usage the goal is not so obvious
asa400 10mo ago
Whenever people bring up binary formats I have to bring up Erlang External Term Format (ETF)[0]. It's the native format Erlang nodes use to serialize data to communicate with each other, but it's just a simple Type-Length-Value binary format, so anything can implement it. It's small enough that you can create a reasonably complete and fast implementation of it for your language in an afternoon. It's self-describing, so if you can read ETF you can read any ETF message, as there are no out of band schemas. I love it. [0] - https://www.erlang.org/doc/apps/erts/erl_ext_dist.html https://www.erlang.org/doc/apps/erts/erl_ext_dist.html
wallaconno 10mo ago
My criticism is that protobuf is a premature optimization for most projects. I want to love protobuf, but it usually slows down my development pace and it’s just not worth it for most web / small data projects. Distributing the contract with lock-step deploys between services is a bit much. JSON parsing time and size are not usually key factors in my projects. Protobuf doesn’t pose strict requirements for data validation, so I have to do that anyway. Losing data readability in transit is a huge problem. Protobuf seems like it would be amazing in situations where data is illegible and tightly coupled such as embedded CAN or radio.
stanfordkid 10mo ago
They didn't even mention MessagePack. Also there is a huge amount of developer over-head for using things like ProtoBuf. You can always validate your API responses with Zod or JSONSchema so that is a bit of a moot point!
firemelt 10mo ago
this article makes me keep json

180 points by barremian 10mo ago

Is it just me or is this article insanely confusing? With all due respect to the author, please be mindful of copy editing LLM-assisted writing. There is a really interesting discussion underneath of this as to the limitations of JSON along with potential alternatives, but I can't help but distrust this writing due to how much it sounds like an LLM.

port11 10mo ago
I don't think it's LLM-generated or even assisted. It's kinda like how I write when I don't want to really argue a point but rather get to the good bits. Seems like the author just wanted to talk about Protobuf without bothering too much about the issues with JSON (though some are mentioned).
dkdcio 10mo ago
do you have any evidence that the author used a LLM? focusing on the content, instead of the tooling used to write the content, leads to a lot more productive discussions I promise you cannot tell LLM-generated content from non-LLM generated content. what you think you’re detecting is poor quality, which is orthogonal to the tooling used
- spagoop 10mo ago
  Fair point, to be constructive here, LLMs seem to love lists and emphasizing random words / phrases with bold. Those two are everywhere. Not a smoking gun but enough to tune out. I am not dismissing this as being slop and actually have no beef with using LLMs to write but yes, as you call out, I think it's just poorly written or perhaps I'm not the specific audience for this. Sorry if this is bad energy, I appreciate the write up regardless.
pavel_lishin 10mo ago
> Is it just me or is this article insanely confusing? I didn't find it confusing. I found it unconvincing, but the argument itself was pretty clear. I just disagreed with it.
phyzome 10mo ago
It wasn't confusing, but yeah, it smelled strongly of LLMs.

codewritero 10mo ago

I love to see people advocating for better protocols and standards but seeing the title I expected the author to present something which would be better in the sense of supporting the same or more use cases with better efficiency and/or ergonomics and I don't think that protobuf does that. Protobuf has advantages, but is missing support for a tons of use cases where JSON thrives due to the strict schema requirement. A much stronger argument could be made for CBOR as a replacement for JSON for most use cases. CBOR has the same schema flexibility as JSON but has a more concise encoding.

port11 10mo ago
I think the strict schema of Protobuf might be one of the major improvements, as most APIs don't publish a JSON schema? I've always had to use ajv or superstruct to make sure payloads match a schema, Protobuf doesn't need that (supposedly).
- thayne 10mo ago
  One limitation of protobuf 3 schemas, is they doen't allow required fields. That makes it easier to remove the field in a later version in a backwards compatible way, but sometimes fields really are required, and the message doesn't make any sense without them. Ideally, IMO, if the message is missing those fields, it would fail to parse successfully. But with protobuf, you instead get a default value, which could potentially cause subtle bugs.
  - port11 10mo ago
    Okay, this is a definite issue, you're still stuck validating inputs/outputs.
youngtaff 10mo ago
We need browsers to support CBOR APIs… and it shouldn’t be that hard as they all have internal implementations now
- 10mo ago

written-beyond 10mo ago

Idk I built a production system and ensured all data transfers, client to server and server to client were proto buf and it was a pain. Technically, it sounds really good but the actual act of managing it is hell. That or I need a lot of practice to use them, at that point shouldn't I just use JSON and get on with my life.

Arainach 10mo ago
What issues did you have? In my experience, most things that could be called painful with protobuf would be bigger pains with things like JSON. Making changes to messages in a backwards-compatible way can be annoying, but JSON allowing you to shoot yourself in the foot will take more time and effort to fix when it's corrupting data in prod than protobuf giving you a compile error would.
- written-beyond 10mo ago
  Well at the bare minimum setting up proto files and knowing where they live across many projects. If they live in their own project, making a single project be buildable with a git clone gets progressively more complex. You now need sub modules to pull in your protobuf definitions. You now also need the protobuf tool chain to be available in your environment you just cloned to. If that environment has the wrong version the build fails, it starts to get frustrating pretty fast. Compare that to json, yes I don't get versioning and a bunch of other fancy features but... I get to finish my work, build and test pretty quickly.
barremian 10mo ago
I think the it-just-works nature and human readability for debugging JSON cannot be overstated. Most people and projects are content to just use JSON even if protos offer some advantages, if not only to save time and resources. Whether the team saves times in the longer when using protos is a question in its own.
- IshKebab 10mo ago
  There are plenty of it-doesn't-just-work things about JSON though. Sending binary data or 64-bit integers is a huge pain. Or maps with non-string keys, or ordered maps. Plus JSON doesn't scale well with message size because it doesn't use TLV so parsing any of a message requires parsing all of it. It's not some perfect format. That said, I'm disappointed with Protobuf too. Especially the handling of optional/default fields. I believe they did eventually add an `optional` tag so you can at least distinguish missing vs default field values. The lack of required fields makes it very annoying to work with though. And no, there's no issue with required fields in general. The only reason it doesn't have them is because the implementation in Protobuf 2 caused issues and Google had to have a smooth transition from that. If you're starting a greenfield project it seems silly to opt in to Google's tech debt. If you're thinking "but schema evolution!!" well yeah all you need is a way to have versioned structs and then you can mark fields as being required for ranges of versions. So you can still remove required fields without breaking backwards compatibility.

esafak 10mo ago

It's premature and maybe presumptuous of him to be advertising protobufs when he hasn't heard of the alternatives yet. I'll engage the article after he discovers them...

deleted 10mo ago
[deleted]

pzmarzly 10mo ago

> With Protobuf, that’s impossible. Unless your servers and clients push at different time, thus are compiled with different versions of your specs, then many safety bets are off. There are ways to be mostly safe (never reuse IDs, use unknown-field-friendly copying methods, etc.), but distributed systems are distributed systems, and protobuf isn't a silver bullet that can solve all problems on author's list. On the upside, it seems like protobuf3 fixed a lot of stuff I used to hate about protobuf2. Issues like: > if the field is not a message, it has two states: > - ... > - the field is set to the default (zero) value. It will not be serialized to the wire. In fact, you cannot determine whether the default (zero) value was set or parsed from the wire or not provided at all are now gone if you stick to using protobuf3 + `message` keyword. That's really cool.

brabel 10mo ago
No type system survives going through a network.
- dhussoe 10mo ago
  yes, but any sane JSON parsing library (Rust Serde, kotlinx-serialization, Swift, etc.) will raise an error when you have the wrong type or are missing a required field. and any JSON parsing callsite is very likely also an IO callsite so you need to handle errors there anyways, all IO can fail. then you log it or recover or whatever you do when IO fails in some other way in that situation. this seems like a problem only if you use JSON.parse or json.loads etc. and then just cross your fingers and hope that the types are correct, basically doing the silent equivalent of casting an "any" type to some structure that you assume is correct, rather than strictly parsing (parse, don't validate) into a typed structure before handing that off to other code.
  - koakuma-chan 10mo ago
    > strictly parsing (parse, don't validate) That's called validating? Zod is a validation library. But yeah, people really need to start strictly parsing/validating their data. One time I had an interview and I was told yOu DoN'T tRuSt YoUr BaCkeNd?!?!?!?

volemo 10mo ago

S-expressions exist since 1960, what more do you need? /s

Jemaclus 10mo ago

"Better than JSON" is a pretty bold claim, and even though the article makes some great cases, the author is making some trade-offs that I wouldn't make, based on my 20+ year career and experience. The author makes a statement at the beginning: "I find it surprising that JSON is so omnipresent when there are far more efficient alternatives." We might disagree on what "efficient" means. OP is focusing on computer efficiency, where as you'll see, I tend to optimize for human efficiency (and, let's be clear, JSON is efficient _enough_ for 99% of computer cases). I think the "human readable" part is often an overlooked pro by hardcore protobuf fans. One of my fundamental philosophies of engineering historically has been "clarity over cleverness." Perhaps the corollary to this is "...and simplicity over complexity." And I think protobuf, generally speaking, falls in the cleverness part, and certainly into the complexity part (with regards to dependencies). JSON, on the other hand, is ubiquitous, human readable (clear), and simple (little-to-no dependencies). I've found in my career that there's tremendous value in not needing to execute code to see what a payload contains. I've seen a lot of engineers (including myself, once upon a time!) take shortcuts like using bitwise values and protobufs and things like that to make things faster or to be clever or whatever. And then I've seen those same engineers, or perhaps their successors, find great difficulty in navigating years-old protobufs, when a JSON payload is immediately clear and understandable to any human, technical or not, upon a glance. I write MUDs for fun, and one of the things that older MUD codebases do is that they use bit flags to compress a lot of information into a tiny integer. To know what conditions a player has (hunger, thirst, cursed, etc), you do some bit manipulation and you wind up with something like 31 that represents the player being thirsty (1), hungry (2), cursed (4), with haste (8), and with shield (16). Which is great, if you're optimizing for integer compression, but it's really bad when you want a human to look at it. You have to do a bunch of math to sort of de-compress that integer into something meaningful for humans. Similarly with protobuf, I find that it usually optimizes for the wrong thing. To be clear, one of my other fundamental philosophies about engineering is that performance is king and that you should try to make things fast, but there are certainly diminishing returns, especially in codebases where humans interact frequently with the data. Protobufs make things fast at a cost, and that cost is typically clarity and human readability. Versioning also creates more friction. I've seen teams spend an inordinate amount of effort trying to ensure that both the producer and consumer are using the same versions. This is not to say that protobufs are useless. It's great for enforcing API contracts at the code level, and it provides those speed improvements OP mentions. There are certain high-throughput use-cases where this complexity and relative opaqueness is not only an acceptable trade off, but the right one to make. But I've found that it's not particularly common, and people reaching for protobufs are often optimizing for the wrong things. Again, clarity over cleverness and simplicity over complexity. I know one of the arguments is "it's better for situations where you control both sides," but if you're in any kind of team with more than a couple of engineers, this stops being true. Even if your internal API is controlled by "us," that "us" can sometimes span 100+ engineers, and you might as well consider it a public API. I'm not a protobuf hater, I just think that the vast majority of engineers would go through their careers without ever touching protobufs, never miss it, never need it, and never find themselves where eking out that extra performance is truly worth the hassle.

catchmeifyoucan 10mo ago

I wonder if we can write an API w/ JSON the usual way and change the final packaging to send it over protobuf.

pstuart 10mo ago
If you're using Go then this framework let's you work with protobufs and gives you a JSON rest-like service for free: https://github.com/grpc-ecosystem/grpc-gateway https://github.com/grpc-ecosystem/grpc-gateway
bglusman 10mo ago
Sure... https://protobuf.dev/programming-guides/json/ https://protobuf.dev/programming-guides/json/ I was pushing at one point for us to have some code in our protobuf parsers that would essentially allow reading messages in either JSON or binary format, though to be fair there's some overhead that way by doing some kind of try/catch, but, for some use cases I think it's worth it...
lanigone 10mo ago
envoy (the proxy) can transcode RESTful APIs to internal grpc services and vice versa. you can even map like url params etc. to proto fields. it works well, even server streaming.

brabel 10mo ago

Mandatory comment about ASN.1, a protocol from 1984, already did what Protobuf does, with more flexibility. Yes, it's a bit ugly but if you stick to the DER encoding it's really not worse than Protbuf at all. Check out the Wikipedia example: https://en.wikipedia.org/wiki/ASN.1#Example_encoded_in_DER https://en.wikipedia.org/wiki/ASN.1#Example_encoded_in_DER Protobuf is ok but if you actually look at how the serializers work, it's just too complex for what it achieves.

bloppe 10mo ago
What makes it too complex in your opinion?
- kragen 10mo ago
  Experience with ASN.1.
  - kstrauser 10mo ago
    Same. My take on ASN.1 is that no one would pay me what I would ask to work on ASN.1. I’d only touch it if I had to parse files from an outside source, and a package already exists in my language of choice that’s capable of parsing those files.
zzo38computer 10mo ago
I also think ASN.1 DER is better (there are other formats, but in my opinion, DER is the only good one, because BER is too messy). I use it in some of my stuff, and when I can, my new designs also use ASN.1 DER rather than using JSON and Protobuf etc. (Some types are missing from standard ASN.1 but I made up a variant called "ASN.1X" which adds some additional types such as key/value list and some others. With the key/value list type added, it is now a superset of the data model of JSON, so you can convert JSON to ASN.1X DER.) (I wrote a implementation of DER encoding/decoding in C, which is public domain and FOSS.)
morshu9001 10mo ago
ASN.1 is way overengineered to the point of making it hard to support. You don't need inheritance for example.

wilg 10mo ago

One of the best parts of Protobuf is that there's a fully compatible JSON serialization and deserialization spec, so you can offer a parallel JSON API with minimal extra work.

bglusman 10mo ago
Yes! Came to comments to see if that was discussed/commented on above with link: https://protobuf.dev/programming-guides/json/ https://protobuf.dev/programming-guides/json/

pyrolistical 10mo ago

Compressed JSON is good enough and requires less human communication initially. Sure it will blow up in your face when a field goes missing or value changes type. People who advocate paying the higher cost ahead of time to perfectly type the entire data structure AND propose a process to do perform version updates to sync client/server are going to lose most of the time. The zero cost of starting with JSON is too compelling even if it has a higher total cost due to production bugs later on. When judging which alternative will succeed, lower perceived human cost beats lower machine cost every time. This is why JSON is never going away, until it gets replaced with something with even lower human communication cost.

deleted 10mo ago
[deleted]
esafak 10mo ago
It won't go away in the same way COBOL won't. That does not mean we should be using it everywhere for greenfield projects.
DyslexicAtheist 10mo ago
> People who advocate paying the higher cost ahead of time to perfectly type the entire data structure AND propose a process to do perform version updates to sync client/server are going to lose most of the time. that's true. But people also rather argue about security vulnerabilities than getting it right from the get-go. Why spend an extra 15 mins effort during design when you can spend 3 months revisiting the ensuing problem later.
- jstanley 10mo ago
  Alternatively: why spend an extra 15 mins on protobuf every other day, when you can put off the 3-month JSON-revisiting project forever?
  - OccamsMirror 10mo ago
    I use ConnectRPC (proto). I definitely do not spend any extra time. In fact the generated types for my backend and frontend saves me time.

taco_emoji 10mo ago

i really hate this blog post format of posting the new thing you discovered as if it's objectively better than the previous thing. no it's not, you just like it better, which is FINE, just own it

morshu9001 10mo ago

Protos are great. Last time I did a small project in NodeJS, I set up a server that defines the entire API in a .proto and serves each endpoint as either proto or json, depending on the content header. Even if the clients want to use json, at least I can define the whole API in proto spec instead of something like Swagger. So my question is, why didn't Google just provide that as a library? The setup wasn't hard but wasn't trivial either, and had several "wrong" ways to set up the proto side. They also bait most people with gRPC, which is its own separate annoying thing that requires HTTP/2, which even Google's own cloud products don't support well (e.g App Engine). P.S. Text proto is also the best static config language. More readable than JSON, less error-prone than YAML, more structure than both.

noctune 10mo ago
You might be interested in https://connectrpc.com/ https://connectrpc.com/. It's basically what you describe, though it's not clear to me how well supported it is.
- morshu9001 10mo ago
  Yeah that one looked good. I don't remember why I didn't use it that time, maybe just felt it was easy enough to DIY that I didn't feel like using another dep (given that I already knew express and proto in isolation). The thing is, Google themselves had to lead the way on this if they wanted protobuf to be mainstream like JSON.
c-cube 10mo ago
That's what Twirp (https://github.com/twitchtv/twirp https://github.com/twitchtv/twirp) is about. Protobuf or JSON, over any HTTP, with a simple URL schema. It's fairly simple.
- rileymichael 10mo ago
  highly recommend twirp, even in current year. connectrpc seems to have stalled so there isn't a ton of languages w/server support, and because of the grpc interop on top of their own protocol it's a bit of an undertaking to roll your own. the twirp spec however is so simple you can throw together your own code generator in an afternoon for whatever language you want.

jtrn 10mo ago

I like Python-like indentation, but I usually read Python in an IDE or code blocks. JSON in a non-monospace environment might be problematic with some fonts. Hell, I pass JSON around in emails and word processors all the time.

wg0 10mo ago

Protos don't work out of the box in any browsers as far as I checked last time unless you're willing to deploy a proxy in front to do the translation and it requires extra dependency on the browser as well. Plus - tooling. JSON might not be simpler or strict but it gets the job done. If JSON's size or performance is causing you to go out of business, you surely have bigger problems than JSON.

taftster 10mo ago
Right, I find the use of protobuf lacking with direct support in the browser. Since JSON is a native data encoding format of the browser (effectively), it's just easier to have a JSON-based API. Yes, there are abstractions or other hacks that would bolt on protobuf support in the browser, but that's not ideal in my mind. Protobuf is an ideal exchange format when you're not dealing with the public as a whole. That is, a private or corporate API for your data processing pipelines, etc.

dhussoe 10mo ago

I like that JSON parsing libraries (Serde etc.) allow you to validate nullability constraints at parse-time. Protobuf's deliberate lack of support for required fields means that either you kick that down to every callsite, or you need to build another parsing layer on top of the generated protobuf code. Now, there is a serde_protobuf (I haven't used it) that I assume allows you to enforce nullability constraints but one of the article's points is that you can use the generated code directly and: > No manual validation. No JSON parsing. No risk of type errors. But this is not true—nullability errors are type errors. Manual validation is still required (except you should parse, not validate) to make sure that all of the fields your app expects are there in the response. And "manual" validation (again, parse don't validate) is not necessary with any good JSON parsing library, the library handles it.

myvoiceismypass 10mo ago
'Parse, don't validate' is such great reading: https://lexi-lambda.github.io/blog/2019/11/05/parse-don-t-validate/ https://lexi-lambda.github.io/blog/2019/11/05/parse-don-t-va...

recursivecaveat 10mo ago

My dream binary format is schema driven, as compact and efficient as Capt Proto or such, but just optionally embeds the entire schema into the message. Then we can write a vim plugin that just opens the file in human readable form without having to fish for the schema. Whenever I am using binary formats, it's because I have a list of millions of objects of the same types. Seems to me that you may as well tack 1KB of schema onto a 2GB message and make it self-describing so everyone's life is easier.

barremian 10mo ago
You could have a look at Avro (https://avro.apache.org/ https://avro.apache.org/) and Yardl (https://microsoft.github.io/yardl/ https://microsoft.github.io/yardl/).
EdwardDiego 10mo ago
As another user suggested, Avro is something to look into.
HelloNurse 10mo ago
For many web services it would be more often 200 KB of schema (many possible request and responses, some of them complex) tacked onto a less than 1 KB message (brief requests and acknowledgements without significant data inside).
squirrellous 10mo ago
It’s possible to build this around protobuf. Google has a rich internal protobuf ecosystem that does this and supports querying large amounts of protobuf data without specifying schemas. They are only selectively open sourced. Have a look at riegeli if you are interested. https://github.com/google/riegeli https://github.com/google/riegeli

coffeeaddict1 10mo ago

> protobuf As an aside, like all things Google, their C++ library is massive (14mb dll) and painful to build (takes nearly 10 minutes on my laptop).

mb7733 10mo ago
Yeah and (1) the codegen produces massive headers that slow compilation of anything that touches them (2) the generated classes are really awkward to use. Not a big fan of the experience of protobuffer generated code in a large C++ code base. It's lead to a huge layer of adapters and native c++ classes equivalent to the protobuffers classes to try and mitigate these issues.

keithnz 10mo ago

CBOR is a pretty good middle ground

teleforce 10mo ago

Kudos to the poster and the author of this article. I think this is by far the most insightful technical post I've read this year on HN. >If you develop or use an API, there’s a 99% chance it exchanges data encoded in JSON. Just wondering if the inherent defiencies of JSON can somewhat be improved by CUE lang since the former is very much pervasive and the latter understand JSON [1],[2]. [1] Configure Unify Execute (CUE): Validate, define, and use dynamic and text‑based data: https://cuelang.org/ https://cuelang.org/ [2] Cue – A language for defining, generating, and validating data: https://news.ycombinator.com/item?id=20847943 https://news.ycombinator.com/item?id=20847943

Joel_Mckay 10mo ago
BSON dealt with a lot of the JSON limitations, and ambiguous type issues. Batching with message pooling to a transaction payload size limit actually made it performant. =3

loph 10mo ago

How many times has this problem been "solved"? https://en.wikipedia.org/wiki/DCE/RPC https://en.wikipedia.org/wiki/DCE/RPC DCE/RPC worked in 1993, and still does today. Protocol buffers is just another IDL.

PantaloonFlames 10mo ago
DCE STILL WORKS? Where!??

JoshMock 10mo ago

Protobuf is a great format with a lot of benefits, but it's missing one that I wish it could support: zero-copy. The ability to transport data between processes, services and languages with effectively zero time spent on serialization and deserialization. It appears possible in some cases but it's not universally the case. Which means that similar binary transport formats that do support zero-copy, like Cap'n Proto, offer most or all of the perks described in this post, with the addition of ensuring that serialization and deserialization are not a bottleneck when passing data between processes.

jonny_eh 10mo ago
Is that a format/serialization issue, or library/implementation issue?
- __s 10mo ago
  Format: https://news.ycombinator.com/item?id=23589117 https://news.ycombinator.com/item?id=23589117
- ElectricalUnion 10mo ago
  Serialization issue. From the Introduction to Cap’n Proto: "Cap’n Proto is INFINITY TIMES faster than Protocol Buffers. (...) there is no encoding/decoding step. The Cap’n Proto encoding is appropriate both as a data interchange format and an in-memory representation, so once your structure is built, you can simply write the bytes straight out". I take it as a rationalization of what OLE Compound File Binary - internal Microsoft Office memory structures serialized "raw" as file format - would look like if they paid more attention to being backward and forward compatible and extensible.
  - TillE 10mo ago
    Google has a library/format for that too, with FlatBuffers. Different use cases and advantages really, not clearly better/worse.
    - 10mo ago

brunoluiz 10mo ago

I am curious why the author did not consider ConnectRPC (http://connectrpc.com/ http://connectrpc.com/), which could be a great middle ground since it is compatible with both Protobuf and JSON served APIs. It is developed by Buf, which has been a leader Protobuf tooling.

dfabulich 10mo ago

> With JSON, you often send ambiguous or non-guaranteed data. You may encounter a missing field, an incorrect type, a typo in a key, or simply an undocumented structure. With Protobuf, that’s impossible. Everything starts with a .proto file that defines the structure of messages precisely. This deeply misunderstands the philosophy of Protobuf. proto3 doesn't even support required fields. https://protobuf.dev/best-practices/dos-donts/ https://protobuf.dev/best-practices/dos-donts/ > Never add a required field, instead add `// required` to document the API contract. Required fields are considered harmful by so many they were removed from proto3 completely. Protobuf clients need to be written defensively, just like JSON API clients.

klysm 10mo ago
It’s also conflating the serialization format with contracts
- shortrounddev2 10mo ago
  Most web frameworks do both at the same time to the point where having to write code which enforced a type contract after deserializing is a delabreaker for me. I eant to be able to define my DTOs in one place, once, and have it both deserialize and enforce types/format. Anything else is code smell
  - Seattle3503 10mo ago
    I'm in the same boat. I mostly write Rust and Python. Using serde_json and Pydantic, you get deserialization and validation at the same time. It allows you to de-serialize really "tight" types. Most of my APIs are internal APIs that accept breaking changes easily. My experience with protobufs is that it was created to solve problems in large systems with many teams and APIs, where backwards compatibility is important. There are certainly systems where you can't "just" push through a breaking API change, and in those cases protobufs make sense.
    - masklinn 10mo ago

_el1s7 10mo ago

Why I stopped caring about "Why I stopped [insert something widely used here]" click bait articles

_heimdall 10mo ago

I need to dig deeper into Protobuf. I've never quite understood the benefit of Protobug over XML.

PunchyHamster 10mo ago

"I've used this optimization technique to make app faster" The app 20req/sec The app after optimizations: 20req/sec (It waits for db query anyway)

PantaloonFlames 10mo ago
Yes. Proto makes sense when the request rate is much higher and the network is constrained. Otherwise, json is sufficient.
- kragen 10mo ago
  If it's the network that's constrained and not the CPU, gzipped JSON will often beat protobufs.

bzmrgonz 10mo ago

What about TOON op? I understand that's the standard poised to takeover from Json. Your thoughts??

beders 10mo ago

Know the consumer of your API. If that is just your team, use whatever tech gets you there quick. However, if you need to provide some guarantees to a second or third party with your API, embrace standards like JSON, even better, use content negotiation.

rizky05 10mo ago

[dead]

socalgal2 10mo ago

https://github.com/ajv-validator/ajv https://github.com/ajv-validator/ajv 138 million downloads from npm in the last week. Yes, you can validate your JSON

cryptonector 10mo ago

If you're gonna switch from JSON to... PB, then you might as well switch to flatbuffers before you realize it's better than PB and save yourself a lot of trouble.

lenkite 10mo ago

There is no mention in this thread of how the author is using Dart and Shelf for his REST API's. Code is rather readable and elegant. This is not a combo I have every tried before. Does anyone have any experience of how it compares versus REST services written in Go/Rust/Python ?

lazy_afternoons 10mo ago

IIRC from the red wild boar book, JSONs biggest win is that it got the adoption. Getting everyone to agree on a standard was/is/will be the tougher part.

rock_artist 10mo ago

I feel few points weren’t addressed in the article. 1. Size, biggest problem with JSON can happen when things gets too big. So here other formats might be better. Yet, as a reminder JSON has the binary version named BSON. 2. Zero batteries. JSON is readable by humans but also almost self explanatory format. Most languages has built in or quick drop in for json. Still, it’s easy to implement a limited JSON parser from scratch when in need (eg. Pure on func in C on a tiny device). Working with Protobuf and MsgPack in the past, You have much more tooling involved especially if data passes between parts written in different languages. 3. Validation, JSON is simple. But there are solutions such as JSON Schema.

paulddraper 10mo ago
BSON is not simply a binary encoding of JSON. It is a JSON superset with binary encoding, created by MongoDB. (And there is even a JSON encoding of BSON, called extended JSON.) AFAIK there is no widely adopted binary pure adaptation of JSON. (There are application-specific storage formats, like PostgreSQL JSONB, or SQLite JSONB.) ——- Moreover, JSON is relatively compact. BSON or other self descriptive binary formats are often around the same size of JSON. MessagePack aggressively tries to be compact and is, depending on the data. BSON doesn’t try to be compact, rather it improves the parse speed.
- WorldMaker 10mo ago
  CBOR [0] is probably the closest to a widely adopted "pure" adaptation of JSON. It is still technically a superset of JSON, but it tries to be closely matched and frequently cross-references the JSON specs directly, especially because it is also an IETF tracked standard like JSON, and in so far as widely adopted is concerned is included in web standards like WebAuthn today. (For instance, you can't handle Passkeys without some amount of CBOR. The presumed next steps to wider adoption now that all browsers have internal CBOR encoders/decoders would be to add a web platform JS API for it as well.) However, yes, JSON compresses extremely well even in ancient gzip, but especially in Brotli, and desiring compaction of your API responses alone isn't necessarily the best reason to prefer a binary encoding of JSON to just using JSON and letting compression do its thing. [0] https://www.rfc-editor.org/rfc/rfc8949.html https://www.rfc-editor.org/rfc/rfc8949.html

umvi 10mo ago

Protobufs are a pain to debug and maintain compared to json and modern browsers support zstd compression making json "efficient"

davedx 10mo ago

"Ultra-efficient" Searched the article, no mention of gzip, and how most of the time all that text data (html, js and css too!) you're sending over the wire will be automatically compressed to...... an efficient binary format! So really, the author should compare protobufs to gzipped JSON

chillfox 10mo ago
Last time I was evaluating different binary serialization formats for an API I was really hoping to get to use one of the cool ones, but gzipped JSON just beat everything and it wasn't even close.
- theshrike79 10mo ago
  There are some compression formats that perform better than gzip, but it's very dependent on the data you're compressing and your timing requirements (is bandwidth or CPU more important to conserve). But in the end compressed JSON is pretty good. Not perfect, but good enough for many many things.
MangoToupe 10mo ago
I would think that serialization/deserialization time would be the largest drawback of json (at least for serving APIs). Pretty much all the other pain points can slowly be ironed out over time, albeit with deeply ugly solutions.
- thayne 10mo ago
  It depends on what your data looks like. If your content is mostly UTF-8 text, with dynamic keys, then I wouldn't expect protobuf to have much of an advantage over JSON for parsing to an equivalent structure. On the other hand, if you have binary data that needs to be base64 encoded in JSON, then protobuf has a significant advantage.
bloppe 10mo ago
https://auth0.com/blog/beating-json-performance-with-protobuf/ https://auth0.com/blog/beating-json-performance-with-protobu...

sergiotapia 10mo ago

Personally, I looked into protobuf for our Elixir/React Native wombocombo but the second I realized we would have to deploy app updates when we added or removed a field from the response structure it became a non-starter. I can't imagine using protobuf when you're in the first 5 years of a product.

xarope 10mo ago
I'm pretty sure protobuf ignores new fields (if you add; assuming you add as an append, and not change the field ordering), and it recommends you not to remove a field to ensure backward compatibility.

gethly 10mo ago

Reading the first few paragraphs and immediately seeing PB made me instantly think of "Every master was once a beginner.". When you'll go through your own journey, and inevitably end back with json, do write another blog post :) ... we've all been there.

undefeated 10mo ago

Complaining about JSON and then proceeding to write your API in Dart is... interesting.

zabil 10mo ago

I have a slight dislike for JSON+REST for API's. The design overhead involved in determining the correct URL and HTTP method adds a layer of subjectivity to the design and bike shedding arguments. I’m not a huge fan of Protobuf/GRPC either, if there’s a better alternative I believe RPC is the right approach for exposing APIs.

globular-toast 10mo ago

> An API (Application Programming Interface) is a set of rules that allow two systems to communicate. In the web world, REST APIs ... are by far the most widespread. I too had this overly restrictive view of "APIs" for too long. One I started to think about it as the interface between any teo software components it really changed the way I did programming. In other words, a system itself is composed and that composition is done via APIs. There's no point treating "the API" as something special.

cheema33 10mo ago

I use GraphQL. It has a higher learning curve. But it addresses the shortcomings listed by the referenced blog article. It offers type safety, efficiency and modern tooling. And it is also human readable. If you use good tooling, you can have a mutation change a variable type in the database and that type change is automatically reflected in the middleware/backend and the typescript UI code. Not only that libraries like HotChocolate for asp.net come with built-in functions for filtering, pagination, streaming etc.

knallfrosch 10mo ago

I have never encountered the use case where the data sent by the backend over the normal CRUD operations was the bottleneck. But we have built protobuf into a web server that handles 2 requests per second. Why? We wanted to learn about it on the job. I think that's 99% of Protobuf usage.

mjmas 10mo ago

> The same message in Protobuf binary > → About 23 bytes. This appears to be a nice ai-generated result. There are already at least 23 bytes of data there (number 42 (1 byte) + string of 5 chars + string of 17 chars + 1 boolean), so that plus field overhead will be more.

liampulles 10mo ago

I know that OpenAPI code gen support is spotty, and that protobuf codegen (in my experience) is quite good, but all of this starts from the idea that the SaaS I'm consuming has actually documented their API properly. A sizable portion of the integrations I've built have had to be built by hand, because there are inevitable stupid quirks and/or failures I've had to work around. For these usecases, using JSON is preferable, because it is easy for me to see what I have actually been sent, not what the partially up to date spec says I should've been sent. This is consistent with the idea that communication over the internet should consist of (encrypted and compressed) plain text. It's because human beings are going to have to deal with human reality at the end of the day.

pjacotg 10mo ago

On the human readability concern, we use protobuf converted to text format. It looks JSON like so very readable and comes with all the other benefits of protobuf.

whatevaa 10mo ago

I can use text based API's (like with JSON) with nothing else but text editor and curl. No other tooling actually required. Meanwhile if binary protocol tooling in your stack sucks, then it just sucks. Had the joy of implementing calling SOAP service with client generated from wsdl in .net core 2/3 times. Tooling was shit poorly undocumented amd with crappy errors at that time. Run only with very specific versions in very specific way. And not much you can do about it, rolling your own SOAP client would be too expensive for our team. REST with JSON meanwhile is easy and we do it all the time, don't need any client, just give us request/response spec and any docs.

borplk 10mo ago

If you have a protobuf API, does it work in the js environment of browsers? Last time I checked (many years ago) the browser story wasn't good.

baquero 10mo ago

If you want to work with ProtoBuf as with other APIs, have a look at https://github.com/qaware/protocurl https://github.com/qaware/protocurl

protobuf-ai 10mo ago

I started working with protobuf on a project, and just went down the path of MCP. If anybody would like to try it out, it's here: https://www.protobuf.ai/ https://www.protobuf.ai/. Just a lightweight MCP server for schemas that plugs into a schema registry.

madFlasher 10mo ago

The author should check first the performance of protobuf serialization/deserialiazation in browsers Due to very native nature of JSON in browsers and node backend it usually also the fastest data format If for example you have C++ on backend and C++ on frontend - you'll definitely have some performance boost But for browsers usage the goal is not so obvious

asa400 10mo ago

Whenever people bring up binary formats I have to bring up Erlang External Term Format (ETF)[0]. It's the native format Erlang nodes use to serialize data to communicate with each other, but it's just a simple Type-Length-Value binary format, so anything can implement it. It's small enough that you can create a reasonably complete and fast implementation of it for your language in an afternoon. It's self-describing, so if you can read ETF you can read any ETF message, as there are no out of band schemas. I love it. [0] - https://www.erlang.org/doc/apps/erts/erl_ext_dist.html https://www.erlang.org/doc/apps/erts/erl_ext_dist.html

wallaconno 10mo ago

My criticism is that protobuf is a premature optimization for most projects. I want to love protobuf, but it usually slows down my development pace and it’s just not worth it for most web / small data projects. Distributing the contract with lock-step deploys between services is a bit much. JSON parsing time and size are not usually key factors in my projects. Protobuf doesn’t pose strict requirements for data validation, so I have to do that anyway. Losing data readability in transit is a huge problem. Protobuf seems like it would be amazing in situations where data is illegible and tightly coupled such as embedded CAN or radio.

stanfordkid 10mo ago

They didn't even mention MessagePack. Also there is a huge amount of developer over-head for using things like ProtoBuf. You can always validate your API responses with Zod or JSONSchema so that is a bit of a moot point!

firemelt 10mo ago

this article makes me keep json