7 ms·

Go Micro: a standard library for distributed systems development

141 points by Kinrany 4y ago

esprehn 4y ago
Many similar projects are listed here: https://github.com/go-kit/kit#related-projects https://github.com/go-kit/kit#related-projects go-micro seems like it does a bit too much, like service discovery and balancing within the framework when that's likely better handled by an Envoy/Istio.
- jrockway 4y ago
  I think not having to run a sidecar is pretty valuable. The entire "service mesh" industry is predicated on not modifying existing code, which I totally get (ops teams can rarely get the dev team to understand their constraints), but if you are actually "devops" and the developers can handle mTLS, tracing, authn/authz, etc. inside their app in a standard way, the software stack is a lot easier to understand and maintain. Sidecar proxies also have the fatal flaw that they can't propagate a distributed trace, so if incoming RPC call triggers outgoing calls (common in microservices, for better or for worse), then you lose the link between them. To do that, your application has to remember the trace ID and pass it along from the server to the client. For reasons like that, I'm kind of bearish on premade service meshes. But, they are OK for mTLS. gRPC has added a lot of code in recent years to support xDS, which lets clients discover endpoints, load assignments, locality, etc. and lets your app do the right thing without an intermediate proxy. It can just connect directly to the control plane and make the same routing decisions a proxy would. That's the ideal situation to me.
  - esprehn 4y ago
    My experience is that tools like this don't scale in most organizations of even moderate size. Soon folks will want to write services in deno or ruby and the go first approach won't work anymore. Now you have "misbehaving" services connected to the system that may not implement load balancing or back off properly. I've also seen it cause a lot of pain during upgrades since upgrading the service framework is implicitly upgrading so many other components and the framework is probably not being diligent about only improving one component (ex. Auth) per release. Sometimes that leads to basically never upgrading to avoid the risk. It's true you need to do propagation for tracing but open telemetry has pretty broad support these days. That cost seems worth the stability and control.
    - hintymad 4y ago
      Netflix was using client-side discovery only. They did run into the issue of supporting multiple languages and invented their own sidecar, but it was a huge productivity boost for them as the decision and customization on service discovery and traffic routing was distributed to individual teams.
      - deleted 4y ago
        [deleted]
      - TheSwordsman 4y ago
        If you're talking about the sidecar process I think you're talking about, that thing sorta sucked. It was a great idea, but execution lacked and it caused all sorts of issues as a result. Also ended up becoming sort of abandoned, while people still relied on it for critical stuff. Netflix eventually moved away from HTTP+JSON to gRPC for backend services, and also started to expose internal DNS-based service discovery. This removed the need for that sidecar from _most_ places, since you could just generate code from protobufs, but it created other problems. Specifically, it was hard to apply the same resiliency patterns across multiple languages when calling those APIs, and so that would then contribute to weird incidents. I'm not there anymore, but last context I have is that things are moving to an Envoy-based model now. The idea is that everything will use the sidecar, and resiliency patterns will be baked in at that layer. I suspect that'll be a better architecture in the long run.
        hintymad 4y ago
        Thanks for the insight. > also started to expose internal DNS-based service discovery. Then this is similar to Consul-based service discovery. In that case, do you know how Netflix continued to support their client-defined routing rules, like selecting a specific set of nodes of a given service to route traffic to? I'd imagine that the number of IP addresses in an ARecord is limited too, so a service won't retrieve the entire list of IPs from a DNS record?
        4y ago
svnpenn 4y ago
Note, this module is Micro in name only: https://github.com/go-micro/go-micro/blob/master/go.mod https://github.com/go-micro/go-micro/blob/master/go.mod
marcus_holmes 4y ago
First thing I do on seeing a new go lib is check the go.mod for dependencies. ~25 direct dependencies is too many Firstly it's a security problem: not only have to vet this code, but also all the dependencies, and all their dependencies. It's going to be be quicker to just write the bits of this lib that I need (especially if I can copy/paste the tricky bits from here). Secondly, it shows that the maintainer wasn't being careful about their deps. That when they see a problem, they hunt for a solution that looks like it fits rather than really trying to dig in to the problem and grok it and work out the best solution. I'm all for not re-inventing the wheel, but you have to understand your load, your road surface, your motive force and your steering mechanism before deciding on a wheel design. Adding a cartwheel to a Formula 1 race car is not going to bring the happy faces. Too many dependencies is a sign of lazy thinking
- the_arun 4y ago
  Just an observation - Among contributors, I see only one active contributor. But there are ton of stars & forks.
- ericpauley 4y ago
  Including a package that solely provides a deque "specifically optimized to perform when used by Microservices": https://github.com/ef-ds/deque https://github.com/ef-ds/deque Sounds like a big red flag of unnecessary depedency bloat.
asim 4y ago
This is my library, unfortunately the website is filled with horrible popup ads. See https://github.com/go-micro/go-micro https://github.com/go-micro/go-micro instead.
- omgmajk 4y ago
  Unfortunate name since the text editor Micro is also written in Go. Might cross pollinate Google searches. But cool stuff mate, well done!
- aniforprez 4y ago
  This website seems like some mirror for gaming SEO and getting clicks to make money off ads
- janpieterz 4y ago
  It also links to https://github.com/Pixelmatic/go-micro?ref=golangexample.com https://github.com/Pixelmatic/go-micro?ref=golangexample.com instead, which hasn't been updated in a while.
- Cthulhu_ 4y ago
  I've flagged it; the github page is better, as are previous discussions: https://hn.algolia.com/?dateRange=all&page=0&prefix=true&query=go-micro&sort=byPopularity&type=story https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que...
- Kinrany 4y ago
  It looked okay with uBlock, sorry.
- bheadmaster 4y ago
  Wait, I've seen this repository on HN a few days ago: https://github.com/micro/micro https://github.com/micro/micro Are you affiliated with this repository? How is it related to yours?
  - asim 4y ago
    Yea, I wrote both. Go Micro was the first thing I open sourced in 2015. Eventually I moved on to building Micro, for a number of years both projects were in the same github org and worked together. After about 6 years I could see a split in the user base but also my own usage. People adopting Go Micro were those who wanted to plugin their own infrastructure and tools, whereas I wanted a more complete platform experience with Micro.
Kinrany 4y ago
@dang could you replace the link with https://github.com/go-micro/go-micro https://github.com/go-micro/go-micro ?
svnpenn 4y ago
Can someone explain whats going on? Just rewriting the URL and hiding the comments doesn't really make this the same post anymore. Might as well have just deleted it.
- asim 4y ago
  The original link posted was https://golangexample.com/go-micro-a-standard-library-for-distributed-systems-development/ https://golangexample.com/go-micro-a-standard-library-for-di... I'm not really sure what that is beyond a lot of popup ads. The HN admin may have replaced and reposted based on the request of the original poster.
maerF0x0 4y ago
see also: https://github.com/go-kit/kit https://github.com/go-kit/kit Or see also: https://medium.com/code-zen/why-i-don-t-use-go-web-frameworks-1087e1facfa4 https://medium.com/code-zen/why-i-don-t-use-go-web-framework... (or any of the dozens of blogs of people indicating why you dont need a framework for go)
- asim 4y ago
  I'd be interested to see whether people said the same about other languages with dominant frameworks now. I've personally learned that at very large people scale, frameworks end up emerging aka the common libraries grouped together into a package that everyone then adopts as the starter. Frameworks leads to standardisation which leads to reuse which leads to generally better software development as can be seen by things like Rails and Spring.
  - preseinger 4y ago
    In addition to the points raised in the other replies, I would observe that Go lacks most of the meta-programming features that frameworks rely upon to deliver value. You can't monkey-patch, you can't annotate functions, you can't really introspect types or do typical dynamic programming stuff. You can often use package reflect to hack up something approximating a solution to this class of problems, true. But, like package unsafe, package reflect sidesteps a bunch of language assumptions, conventions, idioms, etc. which means any code that uses reflection is going to be subtle, brittle. It's a tool of last resort, really, an escape hatch. Not something that you should consider part of your tool set. This is all by design, I think. Go tries to ensure that a straight-line reading of the code accurately represents the execution flow. So you can't really implement the inversion of control patterns that frameworks usually rely on without fighting the language.
  - geodel 4y ago
    I could understand Rails as it is for an already dynamic language. But Spring existence which very heavily rely on reflection always perplex me even though I am forced to use at work. It throws runtime exceptions which would otherwise been compiler error in Plain Java. Further it turns some straight forward business logic in to some cultish bulshittery of endless packages, classes, annotations and so on. But hey, there is no xml config now, so it all is super modern stuff. Just yesterday, I had to look into perf issue of Spring based project. It has probably 500 line of core business logic. But it is littered in tens of packages and dozens of java source files. So it is standardization for sure but it does not seem to be leading to better results. It is however giving a great cover for bad stuff because now code follows all best practices as per Spring framework.
hintymad 4y ago
How does the official grpc framework integrate with micro? It looks if I use Go grpc, I’d have to codegen the server code that depends on the grpc framework.
- asim 4y ago
  Go Micro has gRPC plugins for its client/server packages. You basically import these and it's immediately using gRPC under the covers. Go Micro has it's own code generation that looks similar to gRPC and based on protobuf.
golondon 4y ago
Seriously, what's the difference from the last time this is shared at HN?
- Kinrany 4y ago
  In retrospect that previous post might have inspired me to post this link indirectly. I found it after googling "distributed systems library".
- hu3 4y ago
  I get the sentiment and I share it sometimes, but reposting is not frowned upon in HN. Last post of that URL was 8 months ago according to https://hn.algolia.com/?dateRange=all&page=0&prefix=true&query=https%3A%2F%2Fgithub.com%2Fgo-micro%2Fgo-micro&sort=byDate&type=story https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que... Seems reasonable.
  - numlock86 4y ago
    I guess he was referring to this post two days ago: https://news.ycombinator.com/item?id=32994937 https://news.ycombinator.com/item?id=32994937
    - kitd 4y ago
      That's a different project
      - yencabulator 4y ago
        https://news.ycombinator.com/item?id=33034867 https://news.ycombinator.com/item?id=33034867

141 points by Kinrany 4y ago

Many similar projects are listed here: https://github.com/go-kit/kit#related-projects https://github.com/go-kit/kit#related-projects go-micro seems like it does a bit too much, like service discovery and balancing within the framework when that's likely better handled by an Envoy/Istio.

jrockway 4y ago
I think not having to run a sidecar is pretty valuable. The entire "service mesh" industry is predicated on not modifying existing code, which I totally get (ops teams can rarely get the dev team to understand their constraints), but if you are actually "devops" and the developers can handle mTLS, tracing, authn/authz, etc. inside their app in a standard way, the software stack is a lot easier to understand and maintain. Sidecar proxies also have the fatal flaw that they can't propagate a distributed trace, so if incoming RPC call triggers outgoing calls (common in microservices, for better or for worse), then you lose the link between them. To do that, your application has to remember the trace ID and pass it along from the server to the client. For reasons like that, I'm kind of bearish on premade service meshes. But, they are OK for mTLS. gRPC has added a lot of code in recent years to support xDS, which lets clients discover endpoints, load assignments, locality, etc. and lets your app do the right thing without an intermediate proxy. It can just connect directly to the control plane and make the same routing decisions a proxy would. That's the ideal situation to me.
- esprehn 4y ago
  My experience is that tools like this don't scale in most organizations of even moderate size. Soon folks will want to write services in deno or ruby and the go first approach won't work anymore. Now you have "misbehaving" services connected to the system that may not implement load balancing or back off properly. I've also seen it cause a lot of pain during upgrades since upgrading the service framework is implicitly upgrading so many other components and the framework is probably not being diligent about only improving one component (ex. Auth) per release. Sometimes that leads to basically never upgrading to avoid the risk. It's true you need to do propagation for tracing but open telemetry has pretty broad support these days. That cost seems worth the stability and control.
  - hintymad 4y ago
    Netflix was using client-side discovery only. They did run into the issue of supporting multiple languages and invented their own sidecar, but it was a huge productivity boost for them as the decision and customization on service discovery and traffic routing was distributed to individual teams.
    - deleted 4y ago
      [deleted]
    - TheSwordsman 4y ago
      If you're talking about the sidecar process I think you're talking about, that thing sorta sucked. It was a great idea, but execution lacked and it caused all sorts of issues as a result. Also ended up becoming sort of abandoned, while people still relied on it for critical stuff. Netflix eventually moved away from HTTP+JSON to gRPC for backend services, and also started to expose internal DNS-based service discovery. This removed the need for that sidecar from _most_ places, since you could just generate code from protobufs, but it created other problems. Specifically, it was hard to apply the same resiliency patterns across multiple languages when calling those APIs, and so that would then contribute to weird incidents. I'm not there anymore, but last context I have is that things are moving to an Envoy-based model now. The idea is that everything will use the sidecar, and resiliency patterns will be baked in at that layer. I suspect that'll be a better architecture in the long run.
      - hintymad 4y ago
        Thanks for the insight. > also started to expose internal DNS-based service discovery. Then this is similar to Consul-based service discovery. In that case, do you know how Netflix continued to support their client-defined routing rules, like selecting a specific set of nodes of a given service to route traffic to? I'd imagine that the number of IP addresses in an ARecord is limited too, so a service won't retrieve the entire list of IPs from a DNS record?
        4y ago

svnpenn 4y ago

Note, this module is Micro in name only: https://github.com/go-micro/go-micro/blob/master/go.mod https://github.com/go-micro/go-micro/blob/master/go.mod

marcus_holmes 4y ago

First thing I do on seeing a new go lib is check the go.mod for dependencies. ~25 direct dependencies is too many Firstly it's a security problem: not only have to vet this code, but also all the dependencies, and all their dependencies. It's going to be be quicker to just write the bits of this lib that I need (especially if I can copy/paste the tricky bits from here). Secondly, it shows that the maintainer wasn't being careful about their deps. That when they see a problem, they hunt for a solution that looks like it fits rather than really trying to dig in to the problem and grok it and work out the best solution. I'm all for not re-inventing the wheel, but you have to understand your load, your road surface, your motive force and your steering mechanism before deciding on a wheel design. Adding a cartwheel to a Formula 1 race car is not going to bring the happy faces. Too many dependencies is a sign of lazy thinking

the_arun 4y ago
Just an observation - Among contributors, I see only one active contributor. But there are ton of stars & forks.
ericpauley 4y ago
Including a package that solely provides a deque "specifically optimized to perform when used by Microservices": https://github.com/ef-ds/deque https://github.com/ef-ds/deque Sounds like a big red flag of unnecessary depedency bloat.

asim 4y ago

This is my library, unfortunately the website is filled with horrible popup ads. See https://github.com/go-micro/go-micro https://github.com/go-micro/go-micro instead.

omgmajk 4y ago
Unfortunate name since the text editor Micro is also written in Go. Might cross pollinate Google searches. But cool stuff mate, well done!
aniforprez 4y ago
This website seems like some mirror for gaming SEO and getting clicks to make money off ads
janpieterz 4y ago
It also links to https://github.com/Pixelmatic/go-micro?ref=golangexample.com https://github.com/Pixelmatic/go-micro?ref=golangexample.com instead, which hasn't been updated in a while.
Cthulhu_ 4y ago
I've flagged it; the github page is better, as are previous discussions: https://hn.algolia.com/?dateRange=all&page=0&prefix=true&query=go-micro&sort=byPopularity&type=story https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que...
Kinrany 4y ago
It looked okay with uBlock, sorry.
bheadmaster 4y ago
Wait, I've seen this repository on HN a few days ago: https://github.com/micro/micro https://github.com/micro/micro Are you affiliated with this repository? How is it related to yours?
- asim 4y ago
  Yea, I wrote both. Go Micro was the first thing I open sourced in 2015. Eventually I moved on to building Micro, for a number of years both projects were in the same github org and worked together. After about 6 years I could see a split in the user base but also my own usage. People adopting Go Micro were those who wanted to plugin their own infrastructure and tools, whereas I wanted a more complete platform experience with Micro.

Kinrany 4y ago

@dang could you replace the link with https://github.com/go-micro/go-micro https://github.com/go-micro/go-micro ?

svnpenn 4y ago

Can someone explain whats going on? Just rewriting the URL and hiding the comments doesn't really make this the same post anymore. Might as well have just deleted it.

asim 4y ago
The original link posted was https://golangexample.com/go-micro-a-standard-library-for-distributed-systems-development/ https://golangexample.com/go-micro-a-standard-library-for-di... I'm not really sure what that is beyond a lot of popup ads. The HN admin may have replaced and reposted based on the request of the original poster.

maerF0x0 4y ago

see also: https://github.com/go-kit/kit https://github.com/go-kit/kit Or see also: https://medium.com/code-zen/why-i-don-t-use-go-web-frameworks-1087e1facfa4 https://medium.com/code-zen/why-i-don-t-use-go-web-framework... (or any of the dozens of blogs of people indicating why you dont need a framework for go)

asim 4y ago
I'd be interested to see whether people said the same about other languages with dominant frameworks now. I've personally learned that at very large people scale, frameworks end up emerging aka the common libraries grouped together into a package that everyone then adopts as the starter. Frameworks leads to standardisation which leads to reuse which leads to generally better software development as can be seen by things like Rails and Spring.
- preseinger 4y ago
  In addition to the points raised in the other replies, I would observe that Go lacks most of the meta-programming features that frameworks rely upon to deliver value. You can't monkey-patch, you can't annotate functions, you can't really introspect types or do typical dynamic programming stuff. You can often use package reflect to hack up something approximating a solution to this class of problems, true. But, like package unsafe, package reflect sidesteps a bunch of language assumptions, conventions, idioms, etc. which means any code that uses reflection is going to be subtle, brittle. It's a tool of last resort, really, an escape hatch. Not something that you should consider part of your tool set. This is all by design, I think. Go tries to ensure that a straight-line reading of the code accurately represents the execution flow. So you can't really implement the inversion of control patterns that frameworks usually rely on without fighting the language.
- geodel 4y ago
  I could understand Rails as it is for an already dynamic language. But Spring existence which very heavily rely on reflection always perplex me even though I am forced to use at work. It throws runtime exceptions which would otherwise been compiler error in Plain Java. Further it turns some straight forward business logic in to some cultish bulshittery of endless packages, classes, annotations and so on. But hey, there is no xml config now, so it all is super modern stuff. Just yesterday, I had to look into perf issue of Spring based project. It has probably 500 line of core business logic. But it is littered in tens of packages and dozens of java source files. So it is standardization for sure but it does not seem to be leading to better results. It is however giving a great cover for bad stuff because now code follows all best practices as per Spring framework.

hintymad 4y ago

How does the official grpc framework integrate with micro? It looks if I use Go grpc, I’d have to codegen the server code that depends on the grpc framework.

asim 4y ago
Go Micro has gRPC plugins for its client/server packages. You basically import these and it's immediately using gRPC under the covers. Go Micro has it's own code generation that looks similar to gRPC and based on protobuf.

golondon 4y ago

Seriously, what's the difference from the last time this is shared at HN?

Kinrany 4y ago
In retrospect that previous post might have inspired me to post this link indirectly. I found it after googling "distributed systems library".
hu3 4y ago
I get the sentiment and I share it sometimes, but reposting is not frowned upon in HN. Last post of that URL was 8 months ago according to https://hn.algolia.com/?dateRange=all&page=0&prefix=true&query=https%3A%2F%2Fgithub.com%2Fgo-micro%2Fgo-micro&sort=byDate&type=story https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que... Seems reasonable.
- numlock86 4y ago
  I guess he was referring to this post two days ago: https://news.ycombinator.com/item?id=32994937 https://news.ycombinator.com/item?id=32994937
  - kitd 4y ago
    That's a different project
    - yencabulator 4y ago
      https://news.ycombinator.com/item?id=33034867 https://news.ycombinator.com/item?id=33034867