17 ms·
I gave up on protobufs years ago. The protobuf team has no idea how to write PHP and JS libraries. I got segfaults from using the PHP extension. The built-in to
by francislavoie 4y ago
I gave up on protobufs years ago. The protobuf team has no idea how to write PHP and JS libraries. I got segfaults from using the PHP extension. The built-in toJSON would return invalid JSON (missing braces for binary types). Ridiculous stuff.
I really just prefer to use JSON for everything. It's much easier to debug and observe traffic (browser Network tab). I like JSON-RPC, very simple spec (basically one page long). I don't like REST.
All that said, I'm really glad to see the community take things into their own hands.
- capableweb 4y agoAs always, each protocol/data format has it's place. You need to maximize the amount of data you send in each packet? Then protobuf is better than JSON. Need to support large amount of clients without any fuzz? Then JSON is better. Wanna pass around data you don't know the schema of? JSON again. Contexts matters, there is no silver bullets, everything has trade offs and so on, and so on.
- ninepoints 4y agoHonestly, gzipped json is likely much smaller than uncompressed protobuf. If you were going to use a binary protocol, why choose one that has no partial parsing/toc these days. There are much better alternatives IMO (flatbuffers being one of them)
- oll3 4y ago> Honestly, gzipped json is likely much smaller than uncompressed protobuf. Likely not. See here for a comparison: https://nilsmagnus.github.io/post/proto-json-sizes/ https://nilsmagnus.github.io/post/proto-json-sizes/ Btw, binary formats can also be compressed though it typically won't yield the same compression ratio as similar json would since there will be less repeation in the binary format.
- ninepoints 4y agoOr, we could have done a comparison with large strings and see the opposite result. Silly benchmark is silly (or should I say, specific)
- speedgoose 4y agoJSON messages in a compressed websocket stream are surprisingly tiny. Bigger than compressed protobuf packets but not by much, and much smaller than uncompressed protobuf packets.
- capableweb 4y agoYeah, which is probably fine in most cases but sometimes not (maybe the overhead is just 1.5x, but if you're doing thousands of messages per second (not the usual API<>browser communication for web users)) and then it matters. Again it's trade-offs and highly contextual.
- maccard 4y ago> Wanna pass around data you don't know the schema of? JSON again. This is a false flag. If you don't know the schema on the receiving (or sending, for that matter) side, then you can't do anything with the data, other than pass it on. If you _do_ know what it looks like, then it has an implicit schema whether you call it a schema or not.
- francislavoie 4y agoAt the time, we needed interop with C. So that's why we chose protobufs. But it was a nightmare to work with in other languages. Including C++ for cross platform desktop apps where cross compiling became a problem too. JSON in C is unfortunately way harder than in other modern languages (e.g. Go which makes it a breeze with struct tags and a great stdlib).
- depr 4y agoSurely the technical requirements of my specific use case are applicable to any use case.
- onion2k 4y agoIt's much easier to debug and observe traffic (browser Network tab). The DX for JSON things is much better. The UX for protobufs is much better (faster, less data over the wire, etc). Which you optimize for is up to you, but there isn't a straightforward "Use this tech because it's the best one."
- ZiiS 4y agoAssuming your developer time is contained improved DX often also leads to better UX (more features). So even if you are optimizing for UX you may well be better with JSON.
- halfmatthalfcat 4y agoSo does that make GraphQL the best then? JSON + faster/less data over the wire.
- counttheforks 4y agoNot when you count the DX of the backend developers. Good luck making a performant GraphQL backend that doesn't suffer the N+1 problem, and have fun whitelisting the GraphQL queries produced by your frontend, because attackers will be supplying their own queries with no regards to performance.
- halfmatthalfcat 4y ago
- fsaintjacques 4y agoJSON parsing is a minefield, especially in cross-platforms scenarios (language and/or library). You won't encounter those problems on toy project or simple CRUD applications. For example, as soon as you deal with (u)int64 where values are greater than 2^53, a simple round-trip to javascript can wreak silent havoc. See http://seriot.ch/projects/parsing_json.html http://seriot.ch/projects/parsing_json.html Protobuf support for google's first-class citizen languages is usually very good, i.e. C++, Java, Python and Go. For other languages, it depends on each implementation.
- arein3 4y agoNice article
- RedShift1 4y agoThough you're not wrong, in what common cases are integers larger than 2^53 required?
- Leherenn 4y agoTimestamps in nanoseconds is one.
- RedShift1 4y agoThat's not common, JS's built in Date doesn't even support nanoseconds.
- Leherenn 4y agoI guess it depends in which domain you work? In system programming, "clock_gettime" gives you nanoseconds. If you work with GPS timestamps, you have nanoseconds. Could it be that JS's Date doesn't support nanoseconds because it cannot represent them, which is the issue we are talking about here? Don't get me wrong, I understand this is not something that everyone uses every day, but to me it's a pretty straightforward example that can happen in a wide range of situations. It certainly happened to me/colleagues several times in several companies.
- fuzzy2 4y agoThe problem I see with JSON is its limited set of “native” types. I really wish it had specified support for proper numeric types (int, uint, various widths) and not just doubles. A timestamp type would be great as well. What I really like about Protocol Buffers is that you must write a schema to get started. No more JSON.stringify anything. Everything else sucks though.
- robertlagrant 4y agoI think we could remove about a quarter of all Javascript programming time if JSON had a native Date type.
- tough 4y agoWhat about tRPC?
- francislavoie 4y agoI would use tRPC if I used TypeScript in the backend. But I use PHP, so it's not viable.
- bitwize 4y agoI recommend Capnproto. Parsing time is zero, you can pretend you're a Microsoft programmer in the early 90s and just use the in-RAM struct as your wire format. Maybe it doesn't make sense for in-browser JS applications (though WASM is a different story) but for IPC and RPC in the general case, all parsing and unparsing does is generate waste heat. ALWAYS favor a binary format unless you have a really good reason otherwise.
- kccqzy 4y agoCapnproto is designed by Kenton, a former Google engineer who did a lot of work with protobufs at Google. I see Capnproto as the spiritual successor of protobuf, fixing many issues in protobufs. Also, Capnproto is quite extensively used in some Cloudflare products.
- artursapek 4y agoyour problem is that you're using PHP
- francislavoie 4y agoBad take. Modern PHP is great.
- artursapek 4y agolmao
- haberman 4y agoHi there, I am the primary maintainer of the PHP library as of the last few years. I have heard that there used to be a lot of crashes; the code was almost completely rewritten in 2020 and is in a much better state now. If you find a segfault and you have a repro, file a bug and we will fix it.
- asim 4y agoI think protobuf really works well on the backend and specifically with compiled languages like Go or C++ as per seen by the usage at Google and adoption of gRPC for Go based cloud tooling. Beyond that it's a huge failure. The generated code and usage for other languages is not idiomatic. In fact it's a hindrance and you can see that by the lack of adoption except by the largest orgs who are enforcing it using some sort of grpc-web bridge with types for the frontend. Ultimately you can just convert proto to OpenApi specs and do a much better job at custom client libs with that. I'm not a frontend dev. Most of my time was spent on the backend but what I'll say is I much prefer the fluidity and dynamic nature of JavaScript and the built in ability to deal with JSON that naturally become objects. All the type stuff is easy to do but with docs you can get away with not needing it. My feeling. Protobuf lives on for gRPC server side stuff but for everywhere else OpenApi is winning.
- bufbuild 4y agoIt's worth checking out our take on a lot of these problems: https://buf.build/blog/connect-web-protobuf-grpc-in-the-browser https://buf.build/blog/connect-web-protobuf-grpc-in-the-brow...
- asim 4y agoYea I'm aware of that. I wish you guys the best of luck. I tried a lot of this with Micro. I think it's the right direction especially if you can simplify the tooling. The hard part is just the adoption curve but I think you have a lot of funding to find your way through that.
- sa46 4y agoI like protobufs but I was also disappointed at the JS protobuf options. I disliked both the JS object representation and RPC transport. grpc-web in particular requires an Envoy proxy which seems absurdly heavyweight. I ended up using Twirp because Buf connect wasn't yet released or planned. I rolled my own JS representation. The major differences from Connect: - Avoid undefined if the message is not present on the wire and use an empty instance of the object instead. For recursive types, find the minimal set of fields to initialize as undefined instead of empty. - Transparently promote some protobuf types, like google.protobuf.Timestamp to a proper Instant type (from js-joda or similar library). This makes a surprisingly large difference on reducing the number of jumps from the UI to the API.