4 ms·

Slightly off-topic, but related.. I have read that a common practice for managing proto files (or any schema definitions really) is to put them in a separate re

by bruth 9y ago

Slightly off-topic, but related.. I have read that a common practice for managing proto files (or any schema definitions really) is to put them in a separate repo/package to share. It seems pretty straightforward in my head and provides several advantages. However I still ask about any trade-offs when doing this in practice?

doh 9y ago
It has similar challenges than a monorepo. First challenge is, that you have to keep some kind of reference on what proto are you using within your project. What Google (and some others) do, is that they a) put all the proto files in a separate repo [0] and then generate them for each language separately (python [1], ...). This way you can use whichever proto file you need within your project, however you have to load more libraries than you need to. To be honest, it only makes slight difference when deploying, so not too bad. The second challenge is, that you have to generate the result files every time you make some kind of change. If you have a lot of proto files, then it may take some time to generate them and there are very little tools available to help you. Google open-sourced Artman [2] although it's more focused on APIs than managing shared protos. The massive advantage is that, because proto files are self-explanatory and if you put enough information in them can function as direct documentation of your API's interface, you don't need to fish out the requirements in the project or in the documentation but rather just directly read the proto file itself. But this does depend on the developers, to make it as consistent as possible, which is not always the case [3]. [0] https://github.com/googleapis/googleapis https://github.com/googleapis/googleapis [1] https://pypi.python.org/pypi/googleapis-common-protos https://pypi.python.org/pypi/googleapis-common-protos [2] https://github.com/googleapis/artman https://github.com/googleapis/artman [3] https://news.ycombinator.com/item?id=16166153 https://news.ycombinator.com/item?id=16166153
- marshallbrekka 9y ago
  The mono-repo of protos has been our approach, and has worked very well for us so far. Only difference is we publish the resulting code for all languages into a single artifact repo, which other projects use as a dependency.
  - doh 9y ago
    Hw many languages do you use? Do you generate separate folders for each language or what does the structure looks like?
- bruth 9y ago
  Thanks for the response! These are very good references. I like the breakdown of message and/or service definitions into separate files so it is easy to browse through the repository. I am assuming the version bump (v1 vs. v2) occurs when you introducing breaking changes to the API? I am aware that protobuf does a good job at allowing forwards and backwards compatibility at the message level..
  - doh 9y ago
    Correct on the versioning. We internally treat protos as an immutable descriptions of our API interfaces. That means whenever we need to change anything (outside of bugs), be it adding a new field, or changing the order, renaming fields, changing types, ... we start a new version. We also use a lot of inheritance of non-default types (timestamps, errors, ...) so it's important to make sure that we don't break anything for others.
    - bruth 9y ago
      Interesting so adding new fields even.. I suppose there is quite a bit of planning and iterating on the interface before making it public to reduce churn? I understand doing that for backwards incompatible changes.
      - doh 9y ago
        We add new fields (or redesign existing protos) very sporadically. We also don't have fully automated process to deal with a change in a proto within all projects so by using new versions we can signal to the developers that they should eventually adopt it. We are actively using 5 languages that have to deal with the changes, so we found it easier overall to do it this way.

by bruth 9y ago

doh 9y ago

It has similar challenges than a monorepo. First challenge is, that you have to keep some kind of reference on what proto are you using within your project. What Google (and some others) do, is that they a) put all the proto files in a separate repo [0] and then generate them for each language separately (python [1], ...). This way you can use whichever proto file you need within your project, however you have to load more libraries than you need to. To be honest, it only makes slight difference when deploying, so not too bad. The second challenge is, that you have to generate the result files every time you make some kind of change. If you have a lot of proto files, then it may take some time to generate them and there are very little tools available to help you. Google open-sourced Artman [2] although it's more focused on APIs than managing shared protos. The massive advantage is that, because proto files are self-explanatory and if you put enough information in them can function as direct documentation of your API's interface, you don't need to fish out the requirements in the project or in the documentation but rather just directly read the proto file itself. But this does depend on the developers, to make it as consistent as possible, which is not always the case [3]. [0] https://github.com/googleapis/googleapis https://github.com/googleapis/googleapis [1] https://pypi.python.org/pypi/googleapis-common-protos https://pypi.python.org/pypi/googleapis-common-protos [2] https://github.com/googleapis/artman https://github.com/googleapis/artman [3] https://news.ycombinator.com/item?id=16166153 https://news.ycombinator.com/item?id=16166153

marshallbrekka 9y ago
The mono-repo of protos has been our approach, and has worked very well for us so far. Only difference is we publish the resulting code for all languages into a single artifact repo, which other projects use as a dependency.
- doh 9y ago
  Hw many languages do you use? Do you generate separate folders for each language or what does the structure looks like?
bruth 9y ago
Thanks for the response! These are very good references. I like the breakdown of message and/or service definitions into separate files so it is easy to browse through the repository. I am assuming the version bump (v1 vs. v2) occurs when you introducing breaking changes to the API? I am aware that protobuf does a good job at allowing forwards and backwards compatibility at the message level..
- doh 9y ago
  Correct on the versioning. We internally treat protos as an immutable descriptions of our API interfaces. That means whenever we need to change anything (outside of bugs), be it adding a new field, or changing the order, renaming fields, changing types, ... we start a new version. We also use a lot of inheritance of non-default types (timestamps, errors, ...) so it's important to make sure that we don't break anything for others.
  - bruth 9y ago
    Interesting so adding new fields even.. I suppose there is quite a bit of planning and iterating on the interface before making it public to reduce churn? I understand doing that for backwards incompatible changes.
    - doh 9y ago
      We add new fields (or redesign existing protos) very sporadically. We also don't have fully automated process to deal with a change in a proto within all projects so by using new versions we can signal to the developers that they should eventually adopt it. We are actively using 5 languages that have to deal with the changes, so we found it easier overall to do it this way.