17 ms·

Protobuf-ES: Protocol Buffers TypeScript/JavaScript runtime

205 points by jacobwg 4y ago

aphexairlines 4y ago
Very good to have a smaller runtime. Some of the stated benefits of the new generator are silly and the comparison with ts-proto doesn't point out the actual improvement (generating d.ts files means your CI doesn't spend time type-checking and compiling ts sources), but nice to see this new alternative.
f_devd 4y ago
The same reason along with the fact that you had to generate code, as well as usually needing to convert it to a class afterward was the reason I wrote my own typescript-native binary serializer[0] (mostly based on C-FFI for compatibility) a few years ago. [0]: https://github.com/i404788/honeybuf https://github.com/i404788/honeybuf
francislavoie 4y ago
I gave up on protobufs years ago. The protobuf team has no idea how to write PHP and JS libraries. I got segfaults from using the PHP extension. The built-in toJSON would return invalid JSON (missing braces for binary types). Ridiculous stuff. I really just prefer to use JSON for everything. It's much easier to debug and observe traffic (browser Network tab). I like JSON-RPC, very simple spec (basically one page long). I don't like REST. All that said, I'm really glad to see the community take things into their own hands.
- capableweb 4y ago
  As always, each protocol/data format has it's place. You need to maximize the amount of data you send in each packet? Then protobuf is better than JSON. Need to support large amount of clients without any fuzz? Then JSON is better. Wanna pass around data you don't know the schema of? JSON again. Contexts matters, there is no silver bullets, everything has trade offs and so on, and so on.
  - ninepoints 4y ago
    Honestly, gzipped json is likely much smaller than uncompressed protobuf. If you were going to use a binary protocol, why choose one that has no partial parsing/toc these days. There are much better alternatives IMO (flatbuffers being one of them)
    - oll3 4y ago
      > Honestly, gzipped json is likely much smaller than uncompressed protobuf. Likely not. See here for a comparison: https://nilsmagnus.github.io/post/proto-json-sizes/ https://nilsmagnus.github.io/post/proto-json-sizes/ Btw, binary formats can also be compressed though it typically won't yield the same compression ratio as similar json would since there will be less repeation in the binary format.
      - ninepoints 4y ago
        Or, we could have done a comparison with large strings and see the opposite result. Silly benchmark is silly (or should I say, specific)
  - speedgoose 4y ago
    JSON messages in a compressed websocket stream are surprisingly tiny. Bigger than compressed protobuf packets but not by much, and much smaller than uncompressed protobuf packets.
    - capableweb 4y ago
      Yeah, which is probably fine in most cases but sometimes not (maybe the overhead is just 1.5x, but if you're doing thousands of messages per second (not the usual API<>browser communication for web users)) and then it matters. Again it's trade-offs and highly contextual.
  - maccard 4y ago
    > Wanna pass around data you don't know the schema of? JSON again. This is a false flag. If you don't know the schema on the receiving (or sending, for that matter) side, then you can't do anything with the data, other than pass it on. If you _do_ know what it looks like, then it has an implicit schema whether you call it a schema or not.
- onion2k 4y ago
  It's much easier to debug and observe traffic (browser Network tab). The DX for JSON things is much better. The UX for protobufs is much better (faster, less data over the wire, etc). Which you optimize for is up to you, but there isn't a straightforward "Use this tech because it's the best one."
  - ZiiS 4y ago
    Assuming your developer time is contained improved DX often also leads to better UX (more features). So even if you are optimizing for UX you may well be better with JSON.
    - onion2k 4y ago
      also leads to better UX (more features) More features is not a measure of better UX. In many cases (most cases!?) it's the opposite.
      - ZiiS 4y ago
        Sorry; I meant more polished features as much as more by count.
  - halfmatthalfcat 4y ago
    So does that make GraphQL the best then? JSON + faster/less data over the wire.
    - counttheforks 4y ago
      Not when you count the DX of the backend developers. Good luck making a performant GraphQL backend that doesn't suffer the N+1 problem, and have fun whitelisting the GraphQL queries produced by your frontend, because attackers will be supplying their own queries with no regards to performance.
      - halfmatthalfcat 4y ago
- fsaintjacques 4y ago
  JSON parsing is a minefield, especially in cross-platforms scenarios (language and/or library). You won't encounter those problems on toy project or simple CRUD applications. For example, as soon as you deal with (u)int64 where values are greater than 2^53, a simple round-trip to javascript can wreak silent havoc. See http://seriot.ch/projects/parsing_json.html http://seriot.ch/projects/parsing_json.html Protobuf support for google's first-class citizen languages is usually very good, i.e. C++, Java, Python and Go. For other languages, it depends on each implementation.
  - arein3 4y ago
    Nice article
  - RedShift1 4y ago
    Though you're not wrong, in what common cases are integers larger than 2^53 required?
    - Leherenn 4y ago
      Timestamps in nanoseconds is one.
      - RedShift1 4y ago
        That's not common, JS's built in Date doesn't even support nanoseconds.
        Leherenn 4y ago
        I guess it depends in which domain you work? In system programming, "clock_gettime" gives you nanoseconds. If you work with GPS timestamps, you have nanoseconds. Could it be that JS's Date doesn't support nanoseconds because it cannot represent them, which is the issue we are talking about here? Don't get me wrong, I understand this is not something that everyone uses every day, but to me it's a pretty straightforward example that can happen in a wide range of situations. It certainly happened to me/colleagues several times in several companies.
- fuzzy2 4y ago
  The problem I see with JSON is its limited set of “native” types. I really wish it had specified support for proper numeric types (int, uint, various widths) and not just doubles. A timestamp type would be great as well. What I really like about Protocol Buffers is that you must write a schema to get started. No more JSON.stringify anything. Everything else sucks though.
  - robertlagrant 4y ago
    I think we could remove about a quarter of all Javascript programming time if JSON had a native Date type.
- tough 4y ago
  What about tRPC?
  - francislavoie 4y ago
    I would use tRPC if I used TypeScript in the backend. But I use PHP, so it's not viable.
- bitwize 4y ago
  I recommend Capnproto. Parsing time is zero, you can pretend you're a Microsoft programmer in the early 90s and just use the in-RAM struct as your wire format. Maybe it doesn't make sense for in-browser JS applications (though WASM is a different story) but for IPC and RPC in the general case, all parsing and unparsing does is generate waste heat. ALWAYS favor a binary format unless you have a really good reason otherwise.
  - kccqzy 4y ago
    Capnproto is designed by Kenton, a former Google engineer who did a lot of work with protobufs at Google. I see Capnproto as the spiritual successor of protobuf, fixing many issues in protobufs. Also, Capnproto is quite extensively used in some Cloudflare products.
- artursapek 4y ago
  your problem is that you're using PHP
  - francislavoie 4y ago
    Bad take. Modern PHP is great.
    - artursapek 4y ago
      lmao
- haberman 4y ago
  Hi there, I am the primary maintainer of the PHP library as of the last few years. I have heard that there used to be a lot of crashes; the code was almost completely rewritten in 2020 and is in a much better state now. If you find a segfault and you have a repro, file a bug and we will fix it.
- asim 4y ago
  I think protobuf really works well on the backend and specifically with compiled languages like Go or C++ as per seen by the usage at Google and adoption of gRPC for Go based cloud tooling. Beyond that it's a huge failure. The generated code and usage for other languages is not idiomatic. In fact it's a hindrance and you can see that by the lack of adoption except by the largest orgs who are enforcing it using some sort of grpc-web bridge with types for the frontend. Ultimately you can just convert proto to OpenApi specs and do a much better job at custom client libs with that. I'm not a frontend dev. Most of my time was spent on the backend but what I'll say is I much prefer the fluidity and dynamic nature of JavaScript and the built in ability to deal with JSON that naturally become objects. All the type stuff is easy to do but with docs you can get away with not needing it. My feeling. Protobuf lives on for gRPC server side stuff but for everywhere else OpenApi is winning.
  - bufbuild 4y ago
    It's worth checking out our take on a lot of these problems: https://buf.build/blog/connect-web-protobuf-grpc-in-the-browser https://buf.build/blog/connect-web-protobuf-grpc-in-the-brow...
    - asim 4y ago
      Yea I'm aware of that. I wish you guys the best of luck. I tried a lot of this with Micro. I think it's the right direction especially if you can simplify the tooling. The hard part is just the adoption curve but I think you have a lot of funding to find your way through that.
- sa46 4y ago
  I like protobufs but I was also disappointed at the JS protobuf options. I disliked both the JS object representation and RPC transport. grpc-web in particular requires an Envoy proxy which seems absurdly heavyweight. I ended up using Twirp because Buf connect wasn't yet released or planned. I rolled my own JS representation. The major differences from Connect: - Avoid undefined if the message is not present on the wire and use an empty instance of the object instead. For recursive types, find the minimal set of fields to initialize as undefined instead of empty. - Transparently promote some protobuf types, like google.protobuf.Timestamp to a proper Instant type (from js-joda or similar library). This makes a surprisingly large difference on reducing the number of jumps from the UI to the API.
yawnxyz 4y ago
I keep trying to understand and use protobuf but every time I look at it and its API (this article included) I get more confused and have absolutely no idea how to implement it. I can't tell whether I'm just dumb or a really terrible developer, or if the docs or the thing itself is really hard to use?
- smaye81 4y ago
  I can empathize. I was the same way at first. What is it that you find confusing? Perhaps we can help clear it up or link you to helpful documentation (or improve our own docs).
- izacus 4y ago
  There are a few tricks to make them successful: 1. Your schema is the source of truth. 2. The protoc should generate code as part of your build (try not to check in generated proto code if at all possible). 3. Use generated code to output bytes/parse bytes (this depends on your HTTP/RPC library). The other trick is that you should use the exact same (!) schema file for your frontend and backend projects. This means that changing it should trigger regeneration of generated code for your clients and servers and then run CI on them. So if you accidentally introduce a breaking API change, the CI for broken client will fail before you deploy it.
  - quietbritishjim 4y ago
    > The other trick is that you should use the exact same (!) schema file for your frontend and backend projects. This means that changing it should trigger regeneration of generated code for your clients and servers and then run CI on them. You do not need to have the exact same schema file, in fact protobuf is carefully designed to avoid needing this. You need to follow some rules about what to do when fields are added or removed: * Generally, roll out the server side first then, once that is complete, start rolling out the client afterwards. * If a field is added (on the server side), make sure that it can be ignored on the client side, so old clients are not impacted. For example, don't add a "units" field that changes the meaning of existing "temperature" field (previously had to be fahrenheit, now can be celsius or fahrenheit). Instead add a separate field "temperature_celsius" and send both. (You can always remove the old one later on the server if new clients don't need it and you have 100% finished roll out of clients.) Note that receiving unexpected field data is not an error in protobuf, so the extra field won't cause any problems so long as it's not a problem at application level. * You can equally remove a field so long as the client isn't relying on it (in this case you may need to roll out client update first). More accurately (with proto3 syntax) it will appear as empty/zero so this needs to be OK. * You can't change a field's type e.g. from integer to double (or from one message type to another, but just adding a field to a message according to the above is OK). If you want to do that, go through a controlled process of adding a new field with the new type you want then removing the old field. * You are free to reorganise the order fields appear in the proto file but don't renumber the fields - the field number is what defines it in the binary encoding. In particular, if you remove field number 2 (for example) you should leave a gap (fields 1, 3, 4,... remaining) rather than renumbering the remaining ones to be contiguous. Depending on the application, it is often actually a good idea to have a completely separate copy of the proto file in the client and server applications, with the client proto typically lagging behind the server one.
arein3 4y ago
Why should usual developers use protobuf instead of json? You are just making your life harder If using compression the size is in the same ballpark (protobuf can be between 20% and 50% smaller). For 99% of users it should not make a difference. https://nilsmagnus.github.io/post/proto-json-sizes/#gzipped-json-and-gzipped-protobuf https://nilsmagnus.github.io/post/proto-json-sizes/#gzipped-...
- dboreham 4y ago
  To talk to a server that doesn't speak json.
  - arein3 4y ago
    For non-niche use cases that is a bad developer experience. If you are designing your own solution that uses protobuf instead of JSON say goodbye to a range of useful tools that the whole industry uses. From testing to automation it will be harder at every step, and you will have to find custom solutions instead of usual no-customization solution that works OOTB with JSON. It is a good way to frustrate your developers and generate sometimes brittle solutions related to testing/automation/infrastructure.
    - osigurdson 4y ago
      If PB becomes popular enough the tooling will materialize. The only difference is the schema needs to be discovered (somehow) with PB.
      - arein3 4y ago
        I would advise to be pragmatic and not risk. You are using it now, and my experience was worse with protobuf than with JSON.
        osigurdson 4y ago
        I would advise you not to assume you know what pragmatism means for the problems of others.
nathancahill 4y ago
About time. Looking forward to using this.
mootothemax 4y ago
While I enjoy protobufs, it's one of those things I feel I have to re-learn every time I write a new implementation. Never quite get to grips with them in the same way as, say, regex.
dimitropoulos 4y ago
I'm seeing a lot of comments here along the lines of "why even bother with protobuf (just use JSON, you fools!), and gRPC is a pain anyway!" There are two very very critical things to know if you find yourself thinking either of those two things: 1. To me the compression that protobuf offers, while cool, is not what I find most important about it (or, rather, is not the feature that I like the most). What is the most important (for me) is that the sterilization _forces your server and client to have a typed API contract_. This is a pretty big deal. Imagine you're in a situation where you spent months laboring over your OpenAPI config, just to later realize that > _oops! my OpenAPI config doesn't actually represent what my server is actually doing, and now my OpenAPI generated user documentation is wrong, and my TypeScript types I generated from the OpenAPI config are wrong too, which means I'll get runtime `TypeError`s on the frontend. This is a pretty common situation for people to find themselves in, and it's a tough spot to be in because... well... you can't ever escape. Because no one is really reliably generating actual running servers from OpenAPI configs, you hit this situation reallllly fast. Protobuf, on the other hand, is legitimately a spec-first workflow (by force -> which to me is a very good thing). Take a look at Postman's state of the API report, and you'll see that even their customers are only doing spec-first 10% of the time (and, this is supposed to be the cream of the crop in cutting-edge spec-first development). Why? Because it's really really really hard to do in practice. But protobuf makes it easy. That's a big deal. 2. gRPC and protobuf are not as deeply tied as you might think. You can use protobuf without gRPC at all, and in fact a lot of people are doing this. You can even do it over WebSockets, or anything else. Unfortunately, because the two technologies are often mentioned in the same breath we tend to deeply associate them. Years ago I had a project I needed to pick a wire compression thing for (it was just for fun) and since I was tired of always just using messagepack over and over again, I evaluated other options. I picked FlatBuffers in the end, but I wish I could go back in time and ask my past self why I didn't pick Protocol Buffers instead. I KNOW I remember looking into protobuf, but I think I just assumed that in order to do protobuf well you had to also use gRPC and since I wasn't up for learning gRPC, I gave up on protobuf. That's the beauty of what tools like Protobuf-ES are changing -> they're making protobuf accessible to everyone. ..and besides, there are other things that protobuf unlocks(namely the "reflection" API (very much like GraphQL's introspection API if you're familiar with that), but also things like documentation generation and enforcement of backwards-compatible changes) but the above are the two most appreciated by me.
fuzzy2 4y ago
I really wish someone would create a fork/variant of Protocol Buffers for all the folks that are not afraid of versioning message schemas. With actual guarantees as to what is in the message and what isn’t. I recently had to select a suitable data serialization format for a sort-of-distributed application. The goal was to force teams to declare/discuss contracts up-front, so schema-based. It was a very frustrating experience. All the formats with expressive schemas and type systems somehow don’t have a concept of required fields, or not anymore. How can you create a robust system like this?
- tantalor 4y ago
  Proto2 has required. > required: a well-formed message must have exactly one of this field. https://developers.google.com/protocol-buffers/docs/proto#specifying-rules https://developers.google.com/protocol-buffers/docs/proto#sp...
  - c-cube 4y ago
    Proto3 gained "required" back relatively recently. It's even in use in Opentelemetry proto files.
- IceWreck 4y ago
  Thrift (both fbthrift and apache thrift) has required fields
  - fuzzy2 4y ago
    You’re right! Dunno why I dismissed Thrift in my research, but I definitely missed this fact.
- bunderbunder 4y ago
  Protocol buffers used to have required fields. I worked at a company that kept using an old version specifically to keep having them. I’ve also talked to Googlers about this. My understanding is that having required fields created a robustness problem for them. My guess is that it’s the same situation as for basically everything else: the problems Google is trying to solve for, and the constraints on the solutions to those problems, are just different from what the rest of us are dealing with. That doesn’t make protocol buffers bad, but it maybe does make the level of mindshare Google gets in this corner of the technology world bad. That said, 100% agreed. Proto2 solves for that one use case (required fields), but not others such as "I want to talk to this API from the browser." I would love to have a solution that prioritized being easy to implement and support over being as heavily engineered - and, consequently, difficult to understand and use effectively - as FAANG-scale technologies tend to be.
kamilafsar 4y ago
Shameless plug to my project Phero [0]. It’s a bit like gRPC but specifically for full stack TypeScript projects. It has a minimal API, literally one function, with which you can expose your server’s functions. It will generate a Typesafe SDK for your frontend(s), packed with all models you’re using. It will also generate a server which will automatically validate input & output to your server. One thing I’ve seen no other similar solution do is the way we do error handling: throw an error on the server and catch it on the client as if it was a local error. As I said, it’s only meant for teams who have full stack TypeScript. For teams with polyglot stacks an intermediate like protobuf or GraphQL might make more sense. We generate a TS declaration file instead. [0] https://github.com/phero-hq/phero https://github.com/phero-hq/phero
- throwthere 4y ago
  tRPC is another similar library. https://trpc.io/docs/v10/quickstart https://trpc.io/docs/v10/quickstart
  - jasperpressplay 4y ago
    There’re some key differences though, one being you can use plain typescript types to define your models, instead of a validation lib like zod :)
    - simlevesque 4y ago
      Zod gives you a lot more of control on the schema.
      - jasperpressplay 4y ago
        That’s an interesting point, can you give an example of that?
        simlevesque 4y ago
        You can use any custom validation there is for any field.
tommiegannert 4y ago
Oh, nice! Another issue with the standard implementation is that the TypeScript declaration files are missing lots of stuff. E.g. deserialization functions actually take a jspb.ByteSource (which can be Uint8Array, base-64 string and others), but the declaration files use Uint8Array.
osigurdson 4y ago
Maybe what could be added is a debug header when using grpc. If it is present, the proto schema is sent with each request / response. Then the tooling can be enhanced to look for this. I suspect this would not be much heavier than json so it could be always left on for those who are ok with the overhead. Win win?
as_buggy_1 4y ago
It is unfair to compare ts-proto to this. ts-proto has a gRPC generator. > https://github.com/stephenh/ts-proto https://github.com/stephenh/ts-proto
- bufbuild 4y ago
  protobuf-es does as well! https://github.com/bufbuild/connect-web https://github.com/bufbuild/connect-web
  - ohnoesjmr 4y ago
    Its totally not obvious if this is usable for a non "connect" server side implementation (Grpc.net etc)?
deleted 4y ago
[deleted]
joshmarinacci 4y ago
Genuine question for someone who's used Protobufs in production. How do you avoid the sort of schema violation problems that CORBA and XML had back in their day? Are the schemas in each request?
- cgdub 4y ago
  We have rules around changing the schema. You can only add new fields and deprecate old fields. You can't rename or reuse fields. All new fields must be optional to make schema evolution possible. In practice, this is pretty similar to how you maintain compatibility with old clients in JSON. But with protobuf, you have the schema file to make code reviewing this kind of change easier.
- tommiegannert 4y ago
  I don't know what schema violations you refer to, but XML has the issue that everything is a string until it isn't. The wire encoding of Protobuf has more type information, so I'd say it's half-including the schema compared to XML. Similar to ASN.1 binary encodings, but while ASN.1 (DER, CER) allows you to override the actual wire type, Protobuf always says an integer is an integer on the wire. That means you can do way more parsing and validation in library code without knowing the schema. What's not in there is protection against violations such as sending multiple values for non-repeated fields (the latest value wins) or accidentally making non-backwards-compatible changes to the schema, e.g. reusing field numbers or changing types. The internal Google styleguide for Protobuf is a valuable resource for the collective avoidance of foot-firearms, and it's a bit sad it doesn't seem to be published externally. I don't know of any existing linter that could check for backwards compatibility , but I am interested in one. Might break out one I wrote to a separate library. There aren't too many situations that commonly causes breakages. One reason may be that developer names (text identifiers) are separate from on-wire/persistent names (integer tags). This leaves some wiggle-room for improving your development environment by incrementally clarifying names without breaking backwards-compatibility with already compiled code.
  - endtime 4y ago
    > I don't know of any existing linter that could check for backwards compatibility , but I am interested in one. Buf, from the OP, does this with `buf breaking`.
VLM 4y ago
I find this discussion fascinating as my only protobuf experience is with IoT devices on bandwidth-limited connections where every byte matters, so hearing JS web frontend people talk about the same protocol to transfer "large chunks of real time enduser data" is an interesting far opposite experience. Something "interesting" in the overall IoT protobuf experience is software licensing. I can write software under, GPL, or whatever, to access or otherwise interoperate with a REST API implemented using, BSD, or whatever, no problem as long as it follows reasonably close to generic REST standards. Just look at your swagger API docs or whatever system you use, it'll interoperate just fine. Ditto "most JSON stuff" just write your proprietary code to generate a JSON looking like some text documentation for a server under, perhaps, the MIT license, or whatever other license, and interoperability across licenses is easy and legal. Things get weird with protobuf .proto3 files where the server and ANY clients need to use the same .proto3 file to compile to whatever language they'd like to use. So what license do YOU apply to your .proto3 files to make sure other people can (or intentionally cannot) redistribute your .proto3 or compile software against it, etc? You can, intentionally or accidentally, implement all kinds of lockdown and restriction on APIs using protobufs via innocent or intentional copyright and license selection for a .proto3 file. For better or worse, protobufs are a tool invented primarily to enable the prevention of interoperability. If it was just about saving bytes on the wire we'd be doing IoT using zipped up JSONs or similar technology.
pcj-github 4y ago
The protobuf javascript generator and runtime really shines when your stack is compiled using the closure compiler; it shrinks down to very little code. Unfortunately, very few people outside google use this. The protobuf-gen-es plugin sounds nice, I'll definitely be checking it out. However, I don't understand the desire for people to write protoc plugins in anything but C++, and I'm not even a C++ developer (I much prefer js/ts for many things). Instead of a single c++ source file with a builtin cool templating library which compiles trivially to a tiny binary, you inherit a huge dependency chain that includes the typescipt compiler and nodejs runtime. Totally unnecessary IMHO.
- nr2x 4y ago
  Tech Island.
westurner 4y ago
Arrow Flight RPC (and Arrow Flight SQL, a faster alternative to ODBC/JDBC) are based on gRPC and protobufs: https://arrow.apache.org/docs/format/Flight.html https://arrow.apache.org/docs/format/Flight.html: > Arrow Flight is an RPC framework for high-performance data services based on Arrow data, and is built on top of gRPC and the IPC format. > Flight is organized around streams of Arrow record batches, being either downloaded from or uploaded to another service. A set of metadata methods offers discovery and introspection of streams, as well as the ability to implement application-specific methods. > Methods and message wire formats are defined by Protobuf, enabling interoperability with clients that may support gRPC and Arrow separately, but not Flight. However, Flight implementations include further optimizations to avoid overhead in usage of Protobuf (mostly around avoiding excessive memory copies "Powered By Apache Arrow in JS" https://arrow.apache.org/docs/js/index.html#powered-by-apache-arrow-in-js https://arrow.apache.org/docs/js/index.html#powered-by-apach...
- Kiro 4y ago
  What's with your slightly on-topic but mostly off-topic comments? Your post history almost looks like something written by GPT-3, following the same format and always linking to a lot of external resources that only briefly touch the subject.
  - kazinator 4y ago
    It looks like a bot which scans a comment, identifies some buzz phrases, and quotes those lines, replying with generic linked information (e.g. wikipedia) about those phrases.
    - westurner 4y ago
      No. Read: https://westurner.github.io/hnlog/ https://westurner.github.io/hnlog/ If you have a question about what I just took the time to share with you here, that would be great. Otherwise, I'm going to need you on Saturday.
      - deleted 4y ago
deleted 4y ago
[deleted]
mnutt 4y ago
I was curious how this performed server-side versus protobufjs. (what we're currently using) I hastily wired it up to protobufjs's benchmark suite. (https://github.com/protobufjs/protobuf.js/tree/master/bench https://github.com/protobufjs/protobuf.js/tree/master/bench) The suite is pretty ancient so getting buf-compiled esm added was a challenge. Granted, the benchmark is created for protobufjs and they probably optimize against it. Protobuf-ES was about 5.1x slower than protobufjs for encoding and 14.8x slower than protobufjs for decoding. This was run on my M1 with node 16.14; not particularly scientific, etc, etc.
- lowbloodsugar 4y ago
  Protobuf is really easy to write a working but slow version for.
- still_grokking 4y ago
  Any benchmarks against other PB implementations? Would love to see numbers against the JVM, and than against C++ or Rust. My gut feeling is anyway that PB in JS is likely a failure against just using JSON (which the JS runtime implements in efficient native code).
  - mnutt 4y ago
    No, but anecdotally the others are probably a lot faster. And you can do even better than that with zero-copy serializations. But protobuf has very wide support and is decent enough in js, at least server-to-server. Having a schema is very valuable, and there are substantial size wins over JSON, even with gzip.
SpaghettiX 4y ago
Protobufjs is good, but I can't use it because it's only a protobuf library, not a gRPC library. I end up having to use grpc-web, with all the problems it comes with. I was hoping Buf could solve that problem... Maybe in the future! :)
- conroy 4y ago
  They already have! Connect (https://github.com/bufbuild/connect-web https://github.com/bufbuild/connect-web) is what you're looking for, as it's grpc-web compatible.

205 points by jacobwg 4y ago

Very good to have a smaller runtime. Some of the stated benefits of the new generator are silly and the comparison with ts-proto doesn't point out the actual improvement (generating d.ts files means your CI doesn't spend time type-checking and compiling ts sources), but nice to see this new alternative.

f_devd 4y ago

The same reason along with the fact that you had to generate code, as well as usually needing to convert it to a class afterward was the reason I wrote my own typescript-native binary serializer[0] (mostly based on C-FFI for compatibility) a few years ago. [0]: https://github.com/i404788/honeybuf https://github.com/i404788/honeybuf

francislavoie 4y ago

I gave up on protobufs years ago. The protobuf team has no idea how to write PHP and JS libraries. I got segfaults from using the PHP extension. The built-in toJSON would return invalid JSON (missing braces for binary types). Ridiculous stuff. I really just prefer to use JSON for everything. It's much easier to debug and observe traffic (browser Network tab). I like JSON-RPC, very simple spec (basically one page long). I don't like REST. All that said, I'm really glad to see the community take things into their own hands.

capableweb 4y ago
As always, each protocol/data format has it's place. You need to maximize the amount of data you send in each packet? Then protobuf is better than JSON. Need to support large amount of clients without any fuzz? Then JSON is better. Wanna pass around data you don't know the schema of? JSON again. Contexts matters, there is no silver bullets, everything has trade offs and so on, and so on.
- ninepoints 4y ago
  Honestly, gzipped json is likely much smaller than uncompressed protobuf. If you were going to use a binary protocol, why choose one that has no partial parsing/toc these days. There are much better alternatives IMO (flatbuffers being one of them)
  - oll3 4y ago
    > Honestly, gzipped json is likely much smaller than uncompressed protobuf. Likely not. See here for a comparison: https://nilsmagnus.github.io/post/proto-json-sizes/ https://nilsmagnus.github.io/post/proto-json-sizes/ Btw, binary formats can also be compressed though it typically won't yield the same compression ratio as similar json would since there will be less repeation in the binary format.
    - ninepoints 4y ago
      Or, we could have done a comparison with large strings and see the opposite result. Silly benchmark is silly (or should I say, specific)
- speedgoose 4y ago
  JSON messages in a compressed websocket stream are surprisingly tiny. Bigger than compressed protobuf packets but not by much, and much smaller than uncompressed protobuf packets.
  - capableweb 4y ago
    Yeah, which is probably fine in most cases but sometimes not (maybe the overhead is just 1.5x, but if you're doing thousands of messages per second (not the usual API<>browser communication for web users)) and then it matters. Again it's trade-offs and highly contextual.
- maccard 4y ago
  > Wanna pass around data you don't know the schema of? JSON again. This is a false flag. If you don't know the schema on the receiving (or sending, for that matter) side, then you can't do anything with the data, other than pass it on. If you _do_ know what it looks like, then it has an implicit schema whether you call it a schema or not.
onion2k 4y ago
It's much easier to debug and observe traffic (browser Network tab). The DX for JSON things is much better. The UX for protobufs is much better (faster, less data over the wire, etc). Which you optimize for is up to you, but there isn't a straightforward "Use this tech because it's the best one."
- ZiiS 4y ago
  Assuming your developer time is contained improved DX often also leads to better UX (more features). So even if you are optimizing for UX you may well be better with JSON.
  - onion2k 4y ago
    also leads to better UX (more features) More features is not a measure of better UX. In many cases (most cases!?) it's the opposite.
    - ZiiS 4y ago
      Sorry; I meant more polished features as much as more by count.
- halfmatthalfcat 4y ago
  So does that make GraphQL the best then? JSON + faster/less data over the wire.
  - counttheforks 4y ago
    Not when you count the DX of the backend developers. Good luck making a performant GraphQL backend that doesn't suffer the N+1 problem, and have fun whitelisting the GraphQL queries produced by your frontend, because attackers will be supplying their own queries with no regards to performance.
    - halfmatthalfcat 4y ago
fsaintjacques 4y ago
JSON parsing is a minefield, especially in cross-platforms scenarios (language and/or library). You won't encounter those problems on toy project or simple CRUD applications. For example, as soon as you deal with (u)int64 where values are greater than 2^53, a simple round-trip to javascript can wreak silent havoc. See http://seriot.ch/projects/parsing_json.html http://seriot.ch/projects/parsing_json.html Protobuf support for google's first-class citizen languages is usually very good, i.e. C++, Java, Python and Go. For other languages, it depends on each implementation.
- arein3 4y ago
  Nice article
- RedShift1 4y ago
  Though you're not wrong, in what common cases are integers larger than 2^53 required?
  - Leherenn 4y ago
    Timestamps in nanoseconds is one.
    - RedShift1 4y ago
      That's not common, JS's built in Date doesn't even support nanoseconds.
      - Leherenn 4y ago
        I guess it depends in which domain you work? In system programming, "clock_gettime" gives you nanoseconds. If you work with GPS timestamps, you have nanoseconds. Could it be that JS's Date doesn't support nanoseconds because it cannot represent them, which is the issue we are talking about here? Don't get me wrong, I understand this is not something that everyone uses every day, but to me it's a pretty straightforward example that can happen in a wide range of situations. It certainly happened to me/colleagues several times in several companies.
fuzzy2 4y ago
The problem I see with JSON is its limited set of “native” types. I really wish it had specified support for proper numeric types (int, uint, various widths) and not just doubles. A timestamp type would be great as well. What I really like about Protocol Buffers is that you must write a schema to get started. No more JSON.stringify anything. Everything else sucks though.
- robertlagrant 4y ago
  I think we could remove about a quarter of all Javascript programming time if JSON had a native Date type.
tough 4y ago
What about tRPC?
- francislavoie 4y ago
  I would use tRPC if I used TypeScript in the backend. But I use PHP, so it's not viable.
bitwize 4y ago
I recommend Capnproto. Parsing time is zero, you can pretend you're a Microsoft programmer in the early 90s and just use the in-RAM struct as your wire format. Maybe it doesn't make sense for in-browser JS applications (though WASM is a different story) but for IPC and RPC in the general case, all parsing and unparsing does is generate waste heat. ALWAYS favor a binary format unless you have a really good reason otherwise.
- kccqzy 4y ago
  Capnproto is designed by Kenton, a former Google engineer who did a lot of work with protobufs at Google. I see Capnproto as the spiritual successor of protobuf, fixing many issues in protobufs. Also, Capnproto is quite extensively used in some Cloudflare products.
artursapek 4y ago
your problem is that you're using PHP
- francislavoie 4y ago
  Bad take. Modern PHP is great.
  - artursapek 4y ago
    lmao
haberman 4y ago
Hi there, I am the primary maintainer of the PHP library as of the last few years. I have heard that there used to be a lot of crashes; the code was almost completely rewritten in 2020 and is in a much better state now. If you find a segfault and you have a repro, file a bug and we will fix it.
asim 4y ago
I think protobuf really works well on the backend and specifically with compiled languages like Go or C++ as per seen by the usage at Google and adoption of gRPC for Go based cloud tooling. Beyond that it's a huge failure. The generated code and usage for other languages is not idiomatic. In fact it's a hindrance and you can see that by the lack of adoption except by the largest orgs who are enforcing it using some sort of grpc-web bridge with types for the frontend. Ultimately you can just convert proto to OpenApi specs and do a much better job at custom client libs with that. I'm not a frontend dev. Most of my time was spent on the backend but what I'll say is I much prefer the fluidity and dynamic nature of JavaScript and the built in ability to deal with JSON that naturally become objects. All the type stuff is easy to do but with docs you can get away with not needing it. My feeling. Protobuf lives on for gRPC server side stuff but for everywhere else OpenApi is winning.
- bufbuild 4y ago
  It's worth checking out our take on a lot of these problems: https://buf.build/blog/connect-web-protobuf-grpc-in-the-browser https://buf.build/blog/connect-web-protobuf-grpc-in-the-brow...
  - asim 4y ago
    Yea I'm aware of that. I wish you guys the best of luck. I tried a lot of this with Micro. I think it's the right direction especially if you can simplify the tooling. The hard part is just the adoption curve but I think you have a lot of funding to find your way through that.
sa46 4y ago
I like protobufs but I was also disappointed at the JS protobuf options. I disliked both the JS object representation and RPC transport. grpc-web in particular requires an Envoy proxy which seems absurdly heavyweight. I ended up using Twirp because Buf connect wasn't yet released or planned. I rolled my own JS representation. The major differences from Connect: - Avoid undefined if the message is not present on the wire and use an empty instance of the object instead. For recursive types, find the minimal set of fields to initialize as undefined instead of empty. - Transparently promote some protobuf types, like google.protobuf.Timestamp to a proper Instant type (from js-joda or similar library). This makes a surprisingly large difference on reducing the number of jumps from the UI to the API.

yawnxyz 4y ago

I keep trying to understand and use protobuf but every time I look at it and its API (this article included) I get more confused and have absolutely no idea how to implement it. I can't tell whether I'm just dumb or a really terrible developer, or if the docs or the thing itself is really hard to use?

smaye81 4y ago
I can empathize. I was the same way at first. What is it that you find confusing? Perhaps we can help clear it up or link you to helpful documentation (or improve our own docs).
izacus 4y ago
There are a few tricks to make them successful: 1. Your schema is the source of truth. 2. The protoc should generate code as part of your build (try not to check in generated proto code if at all possible). 3. Use generated code to output bytes/parse bytes (this depends on your HTTP/RPC library). The other trick is that you should use the exact same (!) schema file for your frontend and backend projects. This means that changing it should trigger regeneration of generated code for your clients and servers and then run CI on them. So if you accidentally introduce a breaking API change, the CI for broken client will fail before you deploy it.
- quietbritishjim 4y ago
  > The other trick is that you should use the exact same (!) schema file for your frontend and backend projects. This means that changing it should trigger regeneration of generated code for your clients and servers and then run CI on them. You do not need to have the exact same schema file, in fact protobuf is carefully designed to avoid needing this. You need to follow some rules about what to do when fields are added or removed: * Generally, roll out the server side first then, once that is complete, start rolling out the client afterwards. * If a field is added (on the server side), make sure that it can be ignored on the client side, so old clients are not impacted. For example, don't add a "units" field that changes the meaning of existing "temperature" field (previously had to be fahrenheit, now can be celsius or fahrenheit). Instead add a separate field "temperature_celsius" and send both. (You can always remove the old one later on the server if new clients don't need it and you have 100% finished roll out of clients.) Note that receiving unexpected field data is not an error in protobuf, so the extra field won't cause any problems so long as it's not a problem at application level. * You can equally remove a field so long as the client isn't relying on it (in this case you may need to roll out client update first). More accurately (with proto3 syntax) it will appear as empty/zero so this needs to be OK. * You can't change a field's type e.g. from integer to double (or from one message type to another, but just adding a field to a message according to the above is OK). If you want to do that, go through a controlled process of adding a new field with the new type you want then removing the old field. * You are free to reorganise the order fields appear in the proto file but don't renumber the fields - the field number is what defines it in the binary encoding. In particular, if you remove field number 2 (for example) you should leave a gap (fields 1, 3, 4,... remaining) rather than renumbering the remaining ones to be contiguous. Depending on the application, it is often actually a good idea to have a completely separate copy of the proto file in the client and server applications, with the client proto typically lagging behind the server one.

arein3 4y ago

Why should usual developers use protobuf instead of json? You are just making your life harder If using compression the size is in the same ballpark (protobuf can be between 20% and 50% smaller). For 99% of users it should not make a difference. https://nilsmagnus.github.io/post/proto-json-sizes/#gzipped-json-and-gzipped-protobuf https://nilsmagnus.github.io/post/proto-json-sizes/#gzipped-...

dboreham 4y ago
To talk to a server that doesn't speak json.
- arein3 4y ago
  For non-niche use cases that is a bad developer experience. If you are designing your own solution that uses protobuf instead of JSON say goodbye to a range of useful tools that the whole industry uses. From testing to automation it will be harder at every step, and you will have to find custom solutions instead of usual no-customization solution that works OOTB with JSON. It is a good way to frustrate your developers and generate sometimes brittle solutions related to testing/automation/infrastructure.
  - osigurdson 4y ago
    If PB becomes popular enough the tooling will materialize. The only difference is the schema needs to be discovered (somehow) with PB.
    - arein3 4y ago
      I would advise to be pragmatic and not risk. You are using it now, and my experience was worse with protobuf than with JSON.
      - osigurdson 4y ago
        I would advise you not to assume you know what pragmatism means for the problems of others.

nathancahill 4y ago

About time. Looking forward to using this.

mootothemax 4y ago

While I enjoy protobufs, it's one of those things I feel I have to re-learn every time I write a new implementation. Never quite get to grips with them in the same way as, say, regex.

dimitropoulos 4y ago

I'm seeing a lot of comments here along the lines of "why even bother with protobuf (just use JSON, you fools!), and gRPC is a pain anyway!" There are two very very critical things to know if you find yourself thinking either of those two things: 1. To me the compression that protobuf offers, while cool, is not what I find most important about it (or, rather, is not the feature that I like the most). What is the most important (for me) is that the sterilization _forces your server and client to have a typed API contract_. This is a pretty big deal. Imagine you're in a situation where you spent months laboring over your OpenAPI config, just to later realize that > _oops! my OpenAPI config doesn't actually represent what my server is actually doing, and now my OpenAPI generated user documentation is wrong, and my TypeScript types I generated from the OpenAPI config are wrong too, which means I'll get runtime `TypeError`s on the frontend. This is a pretty common situation for people to find themselves in, and it's a tough spot to be in because... well... you can't ever escape. Because no one is really reliably generating actual running servers from OpenAPI configs, you hit this situation reallllly fast. Protobuf, on the other hand, is legitimately a spec-first workflow (by force -> which to me is a very good thing). Take a look at Postman's state of the API report, and you'll see that even their customers are only doing spec-first 10% of the time (and, this is supposed to be the cream of the crop in cutting-edge spec-first development). Why? Because it's really really really hard to do in practice. But protobuf makes it easy. That's a big deal. 2. gRPC and protobuf are not as deeply tied as you might think. You can use protobuf without gRPC at all, and in fact a lot of people are doing this. You can even do it over WebSockets, or anything else. Unfortunately, because the two technologies are often mentioned in the same breath we tend to deeply associate them. Years ago I had a project I needed to pick a wire compression thing for (it was just for fun) and since I was tired of always just using messagepack over and over again, I evaluated other options. I picked FlatBuffers in the end, but I wish I could go back in time and ask my past self why I didn't pick Protocol Buffers instead. I KNOW I remember looking into protobuf, but I think I just assumed that in order to do protobuf well you had to also use gRPC and since I wasn't up for learning gRPC, I gave up on protobuf. That's the beauty of what tools like Protobuf-ES are changing -> they're making protobuf accessible to everyone. ..and besides, there are other things that protobuf unlocks(namely the "reflection" API (very much like GraphQL's introspection API if you're familiar with that), but also things like documentation generation and enforcement of backwards-compatible changes) but the above are the two most appreciated by me.

fuzzy2 4y ago

I really wish someone would create a fork/variant of Protocol Buffers for all the folks that are not afraid of versioning message schemas. With actual guarantees as to what is in the message and what isn’t. I recently had to select a suitable data serialization format for a sort-of-distributed application. The goal was to force teams to declare/discuss contracts up-front, so schema-based. It was a very frustrating experience. All the formats with expressive schemas and type systems somehow don’t have a concept of required fields, or not anymore. How can you create a robust system like this?

tantalor 4y ago
Proto2 has required. > required: a well-formed message must have exactly one of this field. https://developers.google.com/protocol-buffers/docs/proto#specifying-rules https://developers.google.com/protocol-buffers/docs/proto#sp...
- c-cube 4y ago
  Proto3 gained "required" back relatively recently. It's even in use in Opentelemetry proto files.
IceWreck 4y ago
Thrift (both fbthrift and apache thrift) has required fields
- fuzzy2 4y ago
  You’re right! Dunno why I dismissed Thrift in my research, but I definitely missed this fact.
bunderbunder 4y ago
Protocol buffers used to have required fields. I worked at a company that kept using an old version specifically to keep having them. I’ve also talked to Googlers about this. My understanding is that having required fields created a robustness problem for them. My guess is that it’s the same situation as for basically everything else: the problems Google is trying to solve for, and the constraints on the solutions to those problems, are just different from what the rest of us are dealing with. That doesn’t make protocol buffers bad, but it maybe does make the level of mindshare Google gets in this corner of the technology world bad. That said, 100% agreed. Proto2 solves for that one use case (required fields), but not others such as "I want to talk to this API from the browser." I would love to have a solution that prioritized being easy to implement and support over being as heavily engineered - and, consequently, difficult to understand and use effectively - as FAANG-scale technologies tend to be.

kamilafsar 4y ago

Shameless plug to my project Phero [0]. It’s a bit like gRPC but specifically for full stack TypeScript projects. It has a minimal API, literally one function, with which you can expose your server’s functions. It will generate a Typesafe SDK for your frontend(s), packed with all models you’re using. It will also generate a server which will automatically validate input & output to your server. One thing I’ve seen no other similar solution do is the way we do error handling: throw an error on the server and catch it on the client as if it was a local error. As I said, it’s only meant for teams who have full stack TypeScript. For teams with polyglot stacks an intermediate like protobuf or GraphQL might make more sense. We generate a TS declaration file instead. [0] https://github.com/phero-hq/phero https://github.com/phero-hq/phero

throwthere 4y ago
tRPC is another similar library. https://trpc.io/docs/v10/quickstart https://trpc.io/docs/v10/quickstart
- jasperpressplay 4y ago
  There’re some key differences though, one being you can use plain typescript types to define your models, instead of a validation lib like zod :)
  - simlevesque 4y ago
    Zod gives you a lot more of control on the schema.
    - jasperpressplay 4y ago
      That’s an interesting point, can you give an example of that?
      - simlevesque 4y ago
        You can use any custom validation there is for any field.

tommiegannert 4y ago

Oh, nice! Another issue with the standard implementation is that the TypeScript declaration files are missing lots of stuff. E.g. deserialization functions actually take a jspb.ByteSource (which can be Uint8Array, base-64 string and others), but the declaration files use Uint8Array.

osigurdson 4y ago

Maybe what could be added is a debug header when using grpc. If it is present, the proto schema is sent with each request / response. Then the tooling can be enhanced to look for this. I suspect this would not be much heavier than json so it could be always left on for those who are ok with the overhead. Win win?

as_buggy_1 4y ago

It is unfair to compare ts-proto to this. ts-proto has a gRPC generator. > https://github.com/stephenh/ts-proto https://github.com/stephenh/ts-proto

bufbuild 4y ago
protobuf-es does as well! https://github.com/bufbuild/connect-web https://github.com/bufbuild/connect-web
- ohnoesjmr 4y ago
  Its totally not obvious if this is usable for a non "connect" server side implementation (Grpc.net etc)?

deleted 4y ago

[deleted]

joshmarinacci 4y ago

Genuine question for someone who's used Protobufs in production. How do you avoid the sort of schema violation problems that CORBA and XML had back in their day? Are the schemas in each request?

cgdub 4y ago
We have rules around changing the schema. You can only add new fields and deprecate old fields. You can't rename or reuse fields. All new fields must be optional to make schema evolution possible. In practice, this is pretty similar to how you maintain compatibility with old clients in JSON. But with protobuf, you have the schema file to make code reviewing this kind of change easier.
tommiegannert 4y ago
I don't know what schema violations you refer to, but XML has the issue that everything is a string until it isn't. The wire encoding of Protobuf has more type information, so I'd say it's half-including the schema compared to XML. Similar to ASN.1 binary encodings, but while ASN.1 (DER, CER) allows you to override the actual wire type, Protobuf always says an integer is an integer on the wire. That means you can do way more parsing and validation in library code without knowing the schema. What's not in there is protection against violations such as sending multiple values for non-repeated fields (the latest value wins) or accidentally making non-backwards-compatible changes to the schema, e.g. reusing field numbers or changing types. The internal Google styleguide for Protobuf is a valuable resource for the collective avoidance of foot-firearms, and it's a bit sad it doesn't seem to be published externally. I don't know of any existing linter that could check for backwards compatibility , but I am interested in one. Might break out one I wrote to a separate library. There aren't too many situations that commonly causes breakages. One reason may be that developer names (text identifiers) are separate from on-wire/persistent names (integer tags). This leaves some wiggle-room for improving your development environment by incrementally clarifying names without breaking backwards-compatibility with already compiled code.
- endtime 4y ago
  > I don't know of any existing linter that could check for backwards compatibility , but I am interested in one. Buf, from the OP, does this with `buf breaking`.

VLM 4y ago

I find this discussion fascinating as my only protobuf experience is with IoT devices on bandwidth-limited connections where every byte matters, so hearing JS web frontend people talk about the same protocol to transfer "large chunks of real time enduser data" is an interesting far opposite experience. Something "interesting" in the overall IoT protobuf experience is software licensing. I can write software under, GPL, or whatever, to access or otherwise interoperate with a REST API implemented using, BSD, or whatever, no problem as long as it follows reasonably close to generic REST standards. Just look at your swagger API docs or whatever system you use, it'll interoperate just fine. Ditto "most JSON stuff" just write your proprietary code to generate a JSON looking like some text documentation for a server under, perhaps, the MIT license, or whatever other license, and interoperability across licenses is easy and legal. Things get weird with protobuf .proto3 files where the server and ANY clients need to use the same .proto3 file to compile to whatever language they'd like to use. So what license do YOU apply to your .proto3 files to make sure other people can (or intentionally cannot) redistribute your .proto3 or compile software against it, etc? You can, intentionally or accidentally, implement all kinds of lockdown and restriction on APIs using protobufs via innocent or intentional copyright and license selection for a .proto3 file. For better or worse, protobufs are a tool invented primarily to enable the prevention of interoperability. If it was just about saving bytes on the wire we'd be doing IoT using zipped up JSONs or similar technology.

pcj-github 4y ago

The protobuf javascript generator and runtime really shines when your stack is compiled using the closure compiler; it shrinks down to very little code. Unfortunately, very few people outside google use this. The protobuf-gen-es plugin sounds nice, I'll definitely be checking it out. However, I don't understand the desire for people to write protoc plugins in anything but C++, and I'm not even a C++ developer (I much prefer js/ts for many things). Instead of a single c++ source file with a builtin cool templating library which compiles trivially to a tiny binary, you inherit a huge dependency chain that includes the typescipt compiler and nodejs runtime. Totally unnecessary IMHO.

nr2x 4y ago
Tech Island.

westurner 4y ago

Arrow Flight RPC (and Arrow Flight SQL, a faster alternative to ODBC/JDBC) are based on gRPC and protobufs: https://arrow.apache.org/docs/format/Flight.html https://arrow.apache.org/docs/format/Flight.html: > Arrow Flight is an RPC framework for high-performance data services based on Arrow data, and is built on top of gRPC and the IPC format. > Flight is organized around streams of Arrow record batches, being either downloaded from or uploaded to another service. A set of metadata methods offers discovery and introspection of streams, as well as the ability to implement application-specific methods. > Methods and message wire formats are defined by Protobuf, enabling interoperability with clients that may support gRPC and Arrow separately, but not Flight. However, Flight implementations include further optimizations to avoid overhead in usage of Protobuf (mostly around avoiding excessive memory copies "Powered By Apache Arrow in JS" https://arrow.apache.org/docs/js/index.html#powered-by-apache-arrow-in-js https://arrow.apache.org/docs/js/index.html#powered-by-apach...

Kiro 4y ago
What's with your slightly on-topic but mostly off-topic comments? Your post history almost looks like something written by GPT-3, following the same format and always linking to a lot of external resources that only briefly touch the subject.
- kazinator 4y ago
  It looks like a bot which scans a comment, identifies some buzz phrases, and quotes those lines, replying with generic linked information (e.g. wikipedia) about those phrases.
  - westurner 4y ago
    No. Read: https://westurner.github.io/hnlog/ https://westurner.github.io/hnlog/ If you have a question about what I just took the time to share with you here, that would be great. Otherwise, I'm going to need you on Saturday.
    - deleted 4y ago

deleted 4y ago

[deleted]

mnutt 4y ago

I was curious how this performed server-side versus protobufjs. (what we're currently using) I hastily wired it up to protobufjs's benchmark suite. (https://github.com/protobufjs/protobuf.js/tree/master/bench https://github.com/protobufjs/protobuf.js/tree/master/bench) The suite is pretty ancient so getting buf-compiled esm added was a challenge. Granted, the benchmark is created for protobufjs and they probably optimize against it. Protobuf-ES was about 5.1x slower than protobufjs for encoding and 14.8x slower than protobufjs for decoding. This was run on my M1 with node 16.14; not particularly scientific, etc, etc.

lowbloodsugar 4y ago
Protobuf is really easy to write a working but slow version for.
still_grokking 4y ago
Any benchmarks against other PB implementations? Would love to see numbers against the JVM, and than against C++ or Rust. My gut feeling is anyway that PB in JS is likely a failure against just using JSON (which the JS runtime implements in efficient native code).
- mnutt 4y ago
  No, but anecdotally the others are probably a lot faster. And you can do even better than that with zero-copy serializations. But protobuf has very wide support and is decent enough in js, at least server-to-server. Having a schema is very valuable, and there are substantial size wins over JSON, even with gzip.

SpaghettiX 4y ago

Protobufjs is good, but I can't use it because it's only a protobuf library, not a gRPC library. I end up having to use grpc-web, with all the problems it comes with. I was hoping Buf could solve that problem... Maybe in the future! :)

conroy 4y ago
They already have! Connect (https://github.com/bufbuild/connect-web https://github.com/bufbuild/connect-web) is what you're looking for, as it's grpc-web compatible.