5 ms·

Interesting. I wonder to what extent it's found use at Google over this past decade. There are advantage to being language-specific, but a lot of disadvantages

Interesting. I wonder to what extent it's found use at Google over this past decade. There are advantage to being language-specific, but a lot of disadvantages, as well (speaking as someone who recently had to write some Elixir code to unmarshal a Ruby object...). It seems hard to introduce this since you're forcing all communicating services to be Go-based, which is kind of contrary to the independence that microservices usually affords you. Some of the benefits are simply design goals (e.g., top level arrays) which could also be done in a language-independent protocol. And even performance questions probably could. Like, Cap'n Proto I think is designed so that users of the protocol don't have to serialize/deserialize the data, right? They just pass it around and work with it directly. I can see Rob Pike being frustrated with Protocol Buffers at Google, and I don't begrudge anyone for taking a big shot like this, but I wonder if he's found any success with it.

lifthrasiir 3y ago
Yeah, after years of dealing with language-specific serialization formats---and inadvertently learning internals of them (including Go gob, Python pickle and PHP serialize), I'm over. And gob is not even a schematic serialization format (i.e. not only you don't need to define a schema beforehand, you can't). There is some interesting idea, but that's all. Use a well-known schemaless serialization format with some extensibility [1] if you really need. [1] Maybe there was no suitable one when Go was first created. Nowadays I believe CBOR is the best format for this job.
- packetlost 3y ago
  I'm in the same boat. Not to mention security concerns that often crop up in (interpreted) language specific deserialization (I'm looking at you, pickle, thinly veiled `eval()`). I agree that CBOR should generally be the serialization tool of choice for self-describing data (ie. in places where you might otherwise choose JSON). And if your language of choice doesn't have a CBOR lib, CBOR is fairly easy to implement and writing a encoder/decoder is very fun! I recently completed my implementation for the Gerbil Scheme language last week [0]. [0]: https://github.com/chiefnoah/gerbil-cbor https://github.com/chiefnoah/gerbil-cbor
- twotwotwo 3y ago
  Yep, I very much agree with this. It's probably inevitable that languages with reflection grow some kind of language-specific serialization format because a) they can and b) there's often some use case where it looks handy to have a format adapted to the quirks of the language. Plus, when some of these bespoke formats were created, the world hadn't converged as much as it has now around a few common text and binary formats. But now the interoperable serialization formats have a lot more energy spent on tools and such, and the formats are better-specified, to the point that you probably want to use them even where interoperability with other stuff doesn't force it.
danpalmer 3y ago
As an engineer at Google, my opinion on Protocol Buffers changed massively. Pre-Google I found them awkward, the language bindings in Python sucked, and I didn't really see the point. I knew a schema for services was a good idea, but protobufs didn't seem like the best option. The thing is that at Google protobufs are used everywhere. Like, absolutely everywhere. Think of all the places they could be, and it's way more. All the tooling understands them, code search and go-to-reference works on them everywhere, they are truly transformational on how many different services with many different implementations interact. Are they perfect? Far from it. But a Go-only implementation misses almost all the value of protobufs. If I was inventing them for Python I could do better (pickle? maybe not) but the whole point is that they aren't language/ecosystem/use-case specific. If this was Rob Pike's frustration then I can't help but feel he missed the point, or this post is a little disingenuous as to the benefits. I've not seen Gobs used in Google, but I'd imagine an engineer would need to make a very strong case against using protos between services, regardless of if both services are in Go.
- knorker 3y ago
  In my opinion this is pretty on-brand for Pike. Back when he did his best work, it was possible for one person to "just write the new thing", without making it fit with anything else. There was nothing else to fit it with. You could invent everything from scratch, and not only was it not a waste of time, if you were good enough it had a chance of being the best fit for purpose. You could take shortcuts. You could have every part of your system be "odd", because nothing was "even". That's not true anymore. And the way I see it Pike has not moved on. Science in general had this switch at one point, too. There was a point where one could know all of science. But it's long gone.
  - danpalmer 3y ago
    Interesting hypothesis. I can't comment on Pike's history here, but at Google there's certainly a noticeable difference between old special cases that were built pre-2015 ish, and the modern world where everything is very cohesive. I get the impression that there was a big push to achieve that, led from around that time by various products. It's hit some areas more than others, but there do seem to be almost no new products building in the way you've described now.
kunley 3y ago
Nice recap. By the way, just curious what is your opinion on MessagePack?
deleted 3y ago
[deleted]

lifthrasiir 3y ago

Yeah, after years of dealing with language-specific serialization formats---and inadvertently learning internals of them (including Go gob, Python pickle and PHP serialize), I'm over. And gob is not even a schematic serialization format (i.e. not only you don't need to define a schema beforehand, you can't). There is some interesting idea, but that's all. Use a well-known schemaless serialization format with some extensibility [1] if you really need. [1] Maybe there was no suitable one when Go was first created. Nowadays I believe CBOR is the best format for this job.

packetlost 3y ago
I'm in the same boat. Not to mention security concerns that often crop up in (interpreted) language specific deserialization (I'm looking at you, pickle, thinly veiled `eval()`). I agree that CBOR should generally be the serialization tool of choice for self-describing data (ie. in places where you might otherwise choose JSON). And if your language of choice doesn't have a CBOR lib, CBOR is fairly easy to implement and writing a encoder/decoder is very fun! I recently completed my implementation for the Gerbil Scheme language last week [0]. [0]: https://github.com/chiefnoah/gerbil-cbor https://github.com/chiefnoah/gerbil-cbor
twotwotwo 3y ago
Yep, I very much agree with this. It's probably inevitable that languages with reflection grow some kind of language-specific serialization format because a) they can and b) there's often some use case where it looks handy to have a format adapted to the quirks of the language. Plus, when some of these bespoke formats were created, the world hadn't converged as much as it has now around a few common text and binary formats. But now the interoperable serialization formats have a lot more energy spent on tools and such, and the formats are better-specified, to the point that you probably want to use them even where interoperability with other stuff doesn't force it.

danpalmer 3y ago

As an engineer at Google, my opinion on Protocol Buffers changed massively. Pre-Google I found them awkward, the language bindings in Python sucked, and I didn't really see the point. I knew a schema for services was a good idea, but protobufs didn't seem like the best option. The thing is that at Google protobufs are used everywhere. Like, absolutely everywhere. Think of all the places they could be, and it's way more. All the tooling understands them, code search and go-to-reference works on them everywhere, they are truly transformational on how many different services with many different implementations interact. Are they perfect? Far from it. But a Go-only implementation misses almost all the value of protobufs. If I was inventing them for Python I could do better (pickle? maybe not) but the whole point is that they aren't language/ecosystem/use-case specific. If this was Rob Pike's frustration then I can't help but feel he missed the point, or this post is a little disingenuous as to the benefits. I've not seen Gobs used in Google, but I'd imagine an engineer would need to make a very strong case against using protos between services, regardless of if both services are in Go.

knorker 3y ago
In my opinion this is pretty on-brand for Pike. Back when he did his best work, it was possible for one person to "just write the new thing", without making it fit with anything else. There was nothing else to fit it with. You could invent everything from scratch, and not only was it not a waste of time, if you were good enough it had a chance of being the best fit for purpose. You could take shortcuts. You could have every part of your system be "odd", because nothing was "even". That's not true anymore. And the way I see it Pike has not moved on. Science in general had this switch at one point, too. There was a point where one could know all of science. But it's long gone.
- danpalmer 3y ago
  Interesting hypothesis. I can't comment on Pike's history here, but at Google there's certainly a noticeable difference between old special cases that were built pre-2015 ish, and the modern world where everything is very cohesive. I get the impression that there was a big push to achieve that, led from around that time by various products. It's hit some areas more than others, but there do seem to be almost no new products building in the way you've described now.

kunley 3y ago

Nice recap. By the way, just curious what is your opinion on MessagePack?

deleted 3y ago

[deleted]