3 ms·

> The specific design decisions in Protobuf were not carefully tested or weighed against other possibilities. And just to be clear, I don't think this is bad.

> The specific design decisions in Protobuf were not carefully tested or weighed against other possibilities. And just to be clear, I don't think this is bad. On the contrary, I think Protobuf won because it did a wide variety of things "pretty well" while moving quickly and solving real problems. This is how the best technologies are usually made, not by academically trying to perfect everything, but by banging out something that works and running with it to solve real problems. If you try to carefully design everything perfectly upfront, you'll spend a huge amount of time on decisions that don't really matter.

bigiain 6y ago
“Rough consensus and running code.” Works pretty well most of the time.
0xbadcafebee 6y ago
If you're designing a critical component that's going to impact your business indefinitely, you can take some time for due diligence. This is basic Shift Left mentality and it has lots of benefits.
- kentonv 6y ago
  This was 2001-ish. No one at the time had any idea how widely the thing would end up being used, nor how big the company would grow. And spending too much time dwelling on each part of the tech stack then could easily have given a competitor the opportunity to pull ahead, in which case Google wouldn't be what it is today.
  - 0xbadcafebee 6y ago
    It's easy to sit in hindsight and say that all one's actions are justified, because otherwise the present wouldn't have happened. But it's a mistake to justify past actions judging solely on individual outcome. If it had all gone badly and Google had been sunk, these would've been listed as reasons why they should have done things differently. The individual action may be right or wrong in spite of any merit it may garnish.
haberman 6y ago
> This is how the best technologies are usually made, not by academically trying to perfect everything, but by banging out something that works and running with it to solve real problems. I think this is a really important point. I don't think I've ever told you this before, but I am really impressed by how quickly you turned out proto2. While there are some things here and there that we wish we could change, a lot of it holds up really well. A lot of decisions I made in upb early on, where I thought I was improving on proto2, actually turned out to be bad ideas and the proto2 design was better.
- kentonv 6y ago
  Thanks! Though in retrospect, I don't know if proto2 was a net win, given all the migration pain it caused. If I were doing it again I would take a more incremental improvement approach on proto1. It would have taken a lot longer but with less pain, I think. That said... who knows if that would have resulted in a better or worse outcome. Open sourcing would have taken a lot longer. Had I realized at the time that I was taking on a project with no good solutions... heh.
  - haberman 6y ago
    Yes, protobufs are a very interesting and fulfilling technology to work on, but also frustrating because there is so much API exposure that changing any existing API is like trying to run through molasses. A clean break a la proto1->proto2 opens up lots of possibilities on a much shorter time scale, but also creates a heavy migration burden. There are lot of improvements we can make without touching API, but whenever the API itself is a barrier to further improvements, there are just few good options for managing that. Your point about incremental changes reminds me of the Linus rant about "bundling", where he argues that incremental changes lead to a better result than a big bang rewrite: https://yarchive.net/comp/linux/bundling.html https://yarchive.net/comp/linux/bundling.html I very much agree with that approach when possible, but Linux has the benefit of a much narrower API offered to its users. The protobuf API is not only the API of the core library, but of every generated class. The massive surface area just makes any kind of change to generated APIs an enormous challenge (for example, returning string_view from accessors instead of std::string).

bigiain 6y ago

“Rough consensus and running code.” Works pretty well most of the time.

0xbadcafebee 6y ago

If you're designing a critical component that's going to impact your business indefinitely, you can take some time for due diligence. This is basic Shift Left mentality and it has lots of benefits.

kentonv 6y ago
This was 2001-ish. No one at the time had any idea how widely the thing would end up being used, nor how big the company would grow. And spending too much time dwelling on each part of the tech stack then could easily have given a competitor the opportunity to pull ahead, in which case Google wouldn't be what it is today.
- 0xbadcafebee 6y ago
  It's easy to sit in hindsight and say that all one's actions are justified, because otherwise the present wouldn't have happened. But it's a mistake to justify past actions judging solely on individual outcome. If it had all gone badly and Google had been sunk, these would've been listed as reasons why they should have done things differently. The individual action may be right or wrong in spite of any merit it may garnish.

haberman 6y ago

> This is how the best technologies are usually made, not by academically trying to perfect everything, but by banging out something that works and running with it to solve real problems. I think this is a really important point. I don't think I've ever told you this before, but I am really impressed by how quickly you turned out proto2. While there are some things here and there that we wish we could change, a lot of it holds up really well. A lot of decisions I made in upb early on, where I thought I was improving on proto2, actually turned out to be bad ideas and the proto2 design was better.

kentonv 6y ago
Thanks! Though in retrospect, I don't know if proto2 was a net win, given all the migration pain it caused. If I were doing it again I would take a more incremental improvement approach on proto1. It would have taken a lot longer but with less pain, I think. That said... who knows if that would have resulted in a better or worse outcome. Open sourcing would have taken a lot longer. Had I realized at the time that I was taking on a project with no good solutions... heh.
- haberman 6y ago
  Yes, protobufs are a very interesting and fulfilling technology to work on, but also frustrating because there is so much API exposure that changing any existing API is like trying to run through molasses. A clean break a la proto1->proto2 opens up lots of possibilities on a much shorter time scale, but also creates a heavy migration burden. There are lot of improvements we can make without touching API, but whenever the API itself is a barrier to further improvements, there are just few good options for managing that. Your point about incremental changes reminds me of the Linus rant about "bundling", where he argues that incremental changes lead to a better result than a big bang rewrite: https://yarchive.net/comp/linux/bundling.html https://yarchive.net/comp/linux/bundling.html I very much agree with that approach when possible, but Linux has the benefit of a much narrower API offered to its users. The protobuf API is not only the API of the core library, but of every generated class. The massive surface area just makes any kind of change to generated APIs an enormous challenge (for example, returning string_view from accessors instead of std::string).