3 ms·

OP here. There’s a lot more to say about this stack, particularly learnings of grpc and envoy over 6 months. But overall I think the service framework and servi

OP here. There’s a lot more to say about this stack, particularly learnings of grpc and envoy over 6 months. But overall I think the service framework and service proxy pattern is really powerful.

ArtWomb 8y ago
Agreed. Lightweight, distributed tracing in real-time creates a feedback loop that can in turn inform design of your application. Even if you are only hosting a dozen microservice nodes, instead of 1000s. And if you have familiarity with Golang channel patterns such as timeouts, retrys, etc. It is sort of like second nature. Expect more on the state of the art as KubeCon vids come online ;) https://www.youtube.com/channel/UCvqbFHwN-nwalWPjPUKpvTA/videos https://www.youtube.com/channel/UCvqbFHwN-nwalWPjPUKpvTA/vid...
VectorLock 8y ago
Sounds like a great followup article ready to be written.
nindalf 8y ago
Looking forward to that article. I've read a lot about the features that envoy and grpc provide but it'd be really interesting to know how they mesh together and what problems they actually help to solve.
securingsincity 8y ago
I would love to hear how you are managing protofiles across repositories. Do you have a central repository for protofiles? It's something we're trying to figure out with our GRPC implementation.
- jpdb 8y ago
  This is something I've pondered myself, but haven't got a chance to play around with. My idea was simply to integrate it with make (or whatever build tool/script you prefer) where it pulls the .proto file from master in git unless you specify a branch. The downside is that you aren't versioning the protoc file with the client. It's probably best to separate the proto file for server and client and treat it as a dependency. If you're using a mono-repo the solution is possibly a little simpler.
  - be_erik 8y ago
    I recently built something like this: https://github.com/reverbdotcom/protopkg https://github.com/reverbdotcom/protopkg
- asdkhadsj 8y ago
  We opted to store proto files in a singular location. In my view, this has three benefits: 1. Protobufs importing each other becomes effortless. You don't need to download multiple repositories, if you have the proto repo, you have all the dependencies. 2. Decoupling "ownership" allows for a bit more freedom when creating proto files that describe shared data structures. 3. You get a singular location to browse infra. It doesn't describe relationships of course, but it's handy to be able to see what services you have and what their API looks like. Disclaimer: I don't use gRPC, but I do use Protobufs & an RPC, Twirp.
- nzoschke 8y ago
  OP here. We keep them all in one repo. Right now the entire API is a monorepo so it’s easy. But I plan to split it up soon... All the protos in one repo. Ideally public so Segment users can use it as a reference too! Then all the generated Go clients, servers, mocks, typescript, swagger, etc in another repo. This is what programs can import. Then we can build gRPC services in their own repos if we want to. And it’s easy to produce and consume messages for all the other services.

OP here. There’s a lot more to say about this stack, particularly learnings of grpc and envoy over 6 months. But overall I think the service framework and service proxy pattern is really powerful.

ArtWomb 8y ago

Agreed. Lightweight, distributed tracing in real-time creates a feedback loop that can in turn inform design of your application. Even if you are only hosting a dozen microservice nodes, instead of 1000s. And if you have familiarity with Golang channel patterns such as timeouts, retrys, etc. It is sort of like second nature. Expect more on the state of the art as KubeCon vids come online ;) https://www.youtube.com/channel/UCvqbFHwN-nwalWPjPUKpvTA/videos https://www.youtube.com/channel/UCvqbFHwN-nwalWPjPUKpvTA/vid...

VectorLock 8y ago

Sounds like a great followup article ready to be written.

nindalf 8y ago

Looking forward to that article. I've read a lot about the features that envoy and grpc provide but it'd be really interesting to know how they mesh together and what problems they actually help to solve.

securingsincity 8y ago

I would love to hear how you are managing protofiles across repositories. Do you have a central repository for protofiles? It's something we're trying to figure out with our GRPC implementation.

jpdb 8y ago
This is something I've pondered myself, but haven't got a chance to play around with. My idea was simply to integrate it with make (or whatever build tool/script you prefer) where it pulls the .proto file from master in git unless you specify a branch. The downside is that you aren't versioning the protoc file with the client. It's probably best to separate the proto file for server and client and treat it as a dependency. If you're using a mono-repo the solution is possibly a little simpler.
- be_erik 8y ago
  I recently built something like this: https://github.com/reverbdotcom/protopkg https://github.com/reverbdotcom/protopkg
asdkhadsj 8y ago
We opted to store proto files in a singular location. In my view, this has three benefits: 1. Protobufs importing each other becomes effortless. You don't need to download multiple repositories, if you have the proto repo, you have all the dependencies. 2. Decoupling "ownership" allows for a bit more freedom when creating proto files that describe shared data structures. 3. You get a singular location to browse infra. It doesn't describe relationships of course, but it's handy to be able to see what services you have and what their API looks like. Disclaimer: I don't use gRPC, but I do use Protobufs & an RPC, Twirp.
nzoschke 8y ago
OP here. We keep them all in one repo. Right now the entire API is a monorepo so it’s easy. But I plan to split it up soon... All the protos in one repo. Ideally public so Segment users can use it as a reference too! Then all the generated Go clients, servers, mocks, typescript, swagger, etc in another repo. This is what programs can import. Then we can build gRPC services in their own repos if we want to. And it’s easy to produce and consume messages for all the other services.