6 ms·

Show HN: Micro – a microservice toolkit

101 points by chuhnk 11y ago

swah 11y ago
> By now you may have heard of this new phenomenon, microservices. This reads like a joke... but reading on I don't understand if you meant it...
- trymas 11y ago
  My exact thoughts, as well.
- chuhnk 11y ago
  It's sattire for sure. Everything has a phase of being a buzzword. Buzzwords annoy the crap out of me but I've also been using the word microservices before it was a thing, back when just Netflix was blogging about it. I could say SOA or distributed systems but I say microservices because it encapsulates what I'm trying to say, "looseley coupled service oriented architecture with a bounded context", Adrian Cockroft's words.
brightball 11y ago
Although having another microservice toolkit for Go won't generate many headlines, the Sidecar approach they've described here does look pretty sharp. The combination RPC / Pubsub / Reverse Proxy + registry is pretty sharp.
webscalist 11y ago
github.com/micro/micro is this canonical import name? Serious Go projects usually have serious canonical import name. Examples, - golang.org/x/net/html/charset - google.golang.org/cloud - camlistore.org/pkg/app - willnorris.com/go/imageproxy
zongitsrinzler 11y ago
Kind of off topic but does anyone know which blog platform / theme this is? Looks like Ghost?
- zongitsrinzler 11y ago
  Snooping around in code shows that it is this one: https://github.com/TryGhost/Casper https://github.com/TryGhost/Casper
agentgt 11y ago
More and more I see toolkits basically turning microservice into a message queue system. More and more intelligent behavior is needed at the endpoints that its starting to leak into the pipes. Often times the behavior of the endpoints (e.g. discovery, reconnecting, timeout, etc) is so critical that you need that in all clients which totally defeats the "decoupling" of microservices [1] (ie any language or technology can participate). That is extremely complicated endpoint logic requires shared components (see referenced article). You could get all of the behavior by having a semi smart broker like RabbitMQ with out having an extremly complicated endpoints... but thats not microservice.. its evil old enterprise single point of failure. Perhaps I'm biased, old school or just so jaded of hearing "microservice" but I really think many companies particularly young startups could get more work done with an intelligent pipe ie message broker over fairly complicated to setup microservies and duplicated intelligent endpoint logic. TL;DR semi-smart pipes are looking better and better these days. [1]: http://www.microservices.com/ben-christensen-do-not-build-a-distributed-monolith http://www.microservices.com/ben-christensen-do-not-build-a-...
- lifty 11y ago
  Your observation about clients requiring more and more logic (e.g. discovery, reconnecting, timeout, etc) is spot on. It is a pretty big overhead and teams that are building an architecture based on micro-services should be prepared to pay that cost. I am not sure why people choose such an architecture, but for me the biggest reason to do this is to be able to run it on platforms like Mesos, Kubernetes, Nomad etc. Once you've aligned your system with such a platform you get plenty of benefits(improved workflow, reliability, fault tolerance, cloud mobility). But the price is pretty big. The good part is that good tooling is becoming more widespread and open source projects like this make it easier for people to start up.
  - spudfkc 11y ago
    I want to second the fact that a microservice architecture is great because of the ease of deployments (but only after the initial overhead!). I can attest first-hand to the large overhead that comes with creating something from scratch with micro-services - we are still battling with Mesos/Marathon, and recently found Mantl (https://mantl.io/ https://mantl.io/) which has been a huge help.
chuhnk 11y ago
Hi, author here. I built Micro based on my experience at Google and Hailo. At Hailo we built a global microservices platform with over 200 bespoke services in production by the time I left. I realised that more and more people were tackling the problem of scale both technically and as an organisation. I felt like the tools were lacking and most companies end up building their own from scratch. Micro was a way of creating a foundation for writing and running distributed systems. It's a pluggable architecture so that the underlying systems can be swapped out based on preference. Beyond Micro, I'm working on go-platform https://github.com/micro/go-platform https://github.com/micro/go-platform which addresses the higher level needs for microservices. Auth, monitoring, metrics, distributed tracing, etc, etc. Again pluggable like Micro. Still in it's infancy. Cheers Asim
- spriggan3 11y ago
  Is it fault tolerant ? does it support hot swapping ?
  - chuhnk 11y ago
    Fault tolerant in what way? And hot swapping in what way? Go micro uses client side selection and load balancing so you can set number of retries and the timeout period so that it will iterate through nodes for a given request. There's also implementations for rate limiting and circuit breakers as client side wrappers/middleware in the go-plugins repo https://github.com/micro/go-plugins/tree/master/wrapper https://github.com/micro/go-plugins/tree/master/wrapper. Plugins themselves can either be imported and added to a map within the go-micro/cmd package which can be set via flags or env vars or you can set it up yourself and pass it into the client/server. Go does not support DSOs yet so bit difficult to do "hot swapping" in that way if thats what you mean.
- tonyhb 11y ago
  Why did you decide to create micro and go-platform instead of contributing to go-kit (https://github.com/go-kit/kit https://github.com/go-kit/kit)?
  - 11y ago
saintfiends 11y ago
It's great that more and more toolkits are being released. But my biggest gripe with Microservice architectures and toolkits are that they ignore transactions across services.
- chuhnk 11y ago
  It's not that they're ignoring transactions across services, it's that it's a difficult problem that shouldn't actually be solved by a toolkit. It's something that has to be addressed on a per use case basis and most likely modelled differently based on your architectural and database choices. At Hailo we used Cassandra which was replicated globally. An eventually consistent database. We then used Zookeeper at a regional level to do locking for parts of the system that required consistency and serialisation. We made tradeoffs that allows to scale a global system. Data modelling and transactions were no easy thing but the benefits were clear. I did not envy the payments team and have a lot of respect for what they accomplished with said architecture.
crazy_geek 11y ago
The sidecar part of this looks like what is essentially SmartStack[1] but requiring the user be aware of it's existence (due to Micro's proto3 api) whereby users of SS can more or less be ignorant of it's existence. Actually, to fully to what the sidecar does, you'd want something like Kafka[2] or some other pubsub system too. [1] http://nerds.airbnb.com/smartstack-service-discovery-cloud/ http://nerds.airbnb.com/smartstack-service-discovery-cloud/ [2] http://kafka.apache.org/ http://kafka.apache.org/
awinter-py 11y ago
This reads like a litany of the services I forget how to set up between projects. Would be awesome to truly boot them all at once. Anything that can do most of the work of haproxy and be easier to administer gets my vote.
jorgecurio 11y ago
microservice is a horrible horrible idea to uproot your existing architecture or build on top of.
hire_charts 11y ago
Edit: As I posted below, my reasoning for this comment is perhaps better summarized by Zach Holman's discussion on the topic of github "streaks" and how he found himself burned out after a long series of streaks. Worth reading if you haven't already: https://zachholman.com/posts/streaks/ https://zachholman.com/posts/streaks/ ------------- Honest feedback--and this is a bit out of left field--but I think that the author of this project needs to SLOW DOWN. Here is his Github contribution chart: http://i.imgur.com/d1IA1if.png http://i.imgur.com/d1IA1if.png This... does not look like a sustainable level of work. 100 straight days of coding since last December? Yikes. This sounds like an interesting project, founded on good ideas, but sustaining this level of work will do no good if you burn out before there are other contributors to pick up the slack. I know that a lot of people can keep working this hard for much longer, but quality will inevitably start to suffer, well before you actually burn out. And, at the very least, you need to take the time to step back and validate your ideas. This is something you'll never do if you're always heads-down coding. I'm not saying that taking more breaks (weekends off, etc) works for everyone, but I just thought something needed to be said. I've seen friends and former colleagues fall into the trap of working on a project so deeply that it becomes a crazy obsession that they cannot escape. It's all they talk about and it can kind of scare people away. Not saying that this is such an extreme case, but it's something that can happen if you're not careful.
- timothycrosley 11y ago
  I couldn't disagree with this more. Just because you have 100 days of "coding" straight, doesn't mean it's just coding. It could be docs or anything else that's required. And most of those days look light. This is more like getting a habbit formed, then actually working 7 days a week. I mean look at my commit log: https://github.com/timothycrosley/ https://github.com/timothycrosley/, I plan on doing this the rest of my life. In reality, I've been coding every single day since I was 8. I'm now 26.
  - hire_charts 11y ago
    That's great -- and I'm glad you disagree. It's awesome to find a sustainable level of work that can be repeated like that, indefinitely. I felt the need to speak up because I've seen contribution graphs EXACTLY like the author's and yours, except where the end result was not so happy. Take Zach Holman's for example: https://zachholman.com/posts/streaks/ https://zachholman.com/posts/streaks/ (chart is 2/3 down the post) Zach's streak was great for him. Until suddenly it wasn't, and far too late he realized he was burnt out. He's been amazingly public about the whole experience (his subsequent sabbatical, then his being fired from Github, then his year of soul-searching), and for every public story like this I suspect there are many, many private examples that go unheard. I think it's great when you can reach such a peak of productivity, like in your case. But it's important to recognize that you're kind of an outlier. The endless contribution "streak" doesn't work for everyone, at least not always and indefinitely. I don't want to generalize about what is always a good level of work and what isn't. What I DO want to communicate is that it's important to be introspective and take periodic steps back to evaluate the sustainability. After all, a level of work that seems unsustainable might turn out to be quite manageable, and vice versa.

101 points by chuhnk 11y ago

swah 11y ago

> By now you may have heard of this new phenomenon, microservices. This reads like a joke... but reading on I don't understand if you meant it...

trymas 11y ago
My exact thoughts, as well.
chuhnk 11y ago
It's sattire for sure. Everything has a phase of being a buzzword. Buzzwords annoy the crap out of me but I've also been using the word microservices before it was a thing, back when just Netflix was blogging about it. I could say SOA or distributed systems but I say microservices because it encapsulates what I'm trying to say, "looseley coupled service oriented architecture with a bounded context", Adrian Cockroft's words.

brightball 11y ago

Although having another microservice toolkit for Go won't generate many headlines, the Sidecar approach they've described here does look pretty sharp. The combination RPC / Pubsub / Reverse Proxy + registry is pretty sharp.

webscalist 11y ago

github.com/micro/micro is this canonical import name? Serious Go projects usually have serious canonical import name. Examples, - golang.org/x/net/html/charset - google.golang.org/cloud - camlistore.org/pkg/app - willnorris.com/go/imageproxy

zongitsrinzler 11y ago

Kind of off topic but does anyone know which blog platform / theme this is? Looks like Ghost?

zongitsrinzler 11y ago
Snooping around in code shows that it is this one: https://github.com/TryGhost/Casper https://github.com/TryGhost/Casper

agentgt 11y ago

More and more I see toolkits basically turning microservice into a message queue system. More and more intelligent behavior is needed at the endpoints that its starting to leak into the pipes. Often times the behavior of the endpoints (e.g. discovery, reconnecting, timeout, etc) is so critical that you need that in all clients which totally defeats the "decoupling" of microservices [1] (ie any language or technology can participate). That is extremely complicated endpoint logic requires shared components (see referenced article). You could get all of the behavior by having a semi smart broker like RabbitMQ with out having an extremly complicated endpoints... but thats not microservice.. its evil old enterprise single point of failure. Perhaps I'm biased, old school or just so jaded of hearing "microservice" but I really think many companies particularly young startups could get more work done with an intelligent pipe ie message broker over fairly complicated to setup microservies and duplicated intelligent endpoint logic. TL;DR semi-smart pipes are looking better and better these days. [1]: http://www.microservices.com/ben-christensen-do-not-build-a-distributed-monolith http://www.microservices.com/ben-christensen-do-not-build-a-...

lifty 11y ago
Your observation about clients requiring more and more logic (e.g. discovery, reconnecting, timeout, etc) is spot on. It is a pretty big overhead and teams that are building an architecture based on micro-services should be prepared to pay that cost. I am not sure why people choose such an architecture, but for me the biggest reason to do this is to be able to run it on platforms like Mesos, Kubernetes, Nomad etc. Once you've aligned your system with such a platform you get plenty of benefits(improved workflow, reliability, fault tolerance, cloud mobility). But the price is pretty big. The good part is that good tooling is becoming more widespread and open source projects like this make it easier for people to start up.
- spudfkc 11y ago
  I want to second the fact that a microservice architecture is great because of the ease of deployments (but only after the initial overhead!). I can attest first-hand to the large overhead that comes with creating something from scratch with micro-services - we are still battling with Mesos/Marathon, and recently found Mantl (https://mantl.io/ https://mantl.io/) which has been a huge help.

chuhnk 11y ago

Hi, author here. I built Micro based on my experience at Google and Hailo. At Hailo we built a global microservices platform with over 200 bespoke services in production by the time I left. I realised that more and more people were tackling the problem of scale both technically and as an organisation. I felt like the tools were lacking and most companies end up building their own from scratch. Micro was a way of creating a foundation for writing and running distributed systems. It's a pluggable architecture so that the underlying systems can be swapped out based on preference. Beyond Micro, I'm working on go-platform https://github.com/micro/go-platform https://github.com/micro/go-platform which addresses the higher level needs for microservices. Auth, monitoring, metrics, distributed tracing, etc, etc. Again pluggable like Micro. Still in it's infancy. Cheers Asim

spriggan3 11y ago
Is it fault tolerant ? does it support hot swapping ?
- chuhnk 11y ago
  Fault tolerant in what way? And hot swapping in what way? Go micro uses client side selection and load balancing so you can set number of retries and the timeout period so that it will iterate through nodes for a given request. There's also implementations for rate limiting and circuit breakers as client side wrappers/middleware in the go-plugins repo https://github.com/micro/go-plugins/tree/master/wrapper https://github.com/micro/go-plugins/tree/master/wrapper. Plugins themselves can either be imported and added to a map within the go-micro/cmd package which can be set via flags or env vars or you can set it up yourself and pass it into the client/server. Go does not support DSOs yet so bit difficult to do "hot swapping" in that way if thats what you mean.
tonyhb 11y ago
Why did you decide to create micro and go-platform instead of contributing to go-kit (https://github.com/go-kit/kit https://github.com/go-kit/kit)?
- 11y ago

saintfiends 11y ago

It's great that more and more toolkits are being released. But my biggest gripe with Microservice architectures and toolkits are that they ignore transactions across services.

chuhnk 11y ago
It's not that they're ignoring transactions across services, it's that it's a difficult problem that shouldn't actually be solved by a toolkit. It's something that has to be addressed on a per use case basis and most likely modelled differently based on your architectural and database choices. At Hailo we used Cassandra which was replicated globally. An eventually consistent database. We then used Zookeeper at a regional level to do locking for parts of the system that required consistency and serialisation. We made tradeoffs that allows to scale a global system. Data modelling and transactions were no easy thing but the benefits were clear. I did not envy the payments team and have a lot of respect for what they accomplished with said architecture.

crazy_geek 11y ago

The sidecar part of this looks like what is essentially SmartStack[1] but requiring the user be aware of it's existence (due to Micro's proto3 api) whereby users of SS can more or less be ignorant of it's existence. Actually, to fully to what the sidecar does, you'd want something like Kafka[2] or some other pubsub system too. [1] http://nerds.airbnb.com/smartstack-service-discovery-cloud/ http://nerds.airbnb.com/smartstack-service-discovery-cloud/ [2] http://kafka.apache.org/ http://kafka.apache.org/

awinter-py 11y ago

This reads like a litany of the services I forget how to set up between projects. Would be awesome to truly boot them all at once. Anything that can do most of the work of haproxy and be easier to administer gets my vote.

jorgecurio 11y ago

microservice is a horrible horrible idea to uproot your existing architecture or build on top of.

hire_charts 11y ago

Edit: As I posted below, my reasoning for this comment is perhaps better summarized by Zach Holman's discussion on the topic of github "streaks" and how he found himself burned out after a long series of streaks. Worth reading if you haven't already: https://zachholman.com/posts/streaks/ https://zachholman.com/posts/streaks/ ------------- Honest feedback--and this is a bit out of left field--but I think that the author of this project needs to SLOW DOWN. Here is his Github contribution chart: http://i.imgur.com/d1IA1if.png http://i.imgur.com/d1IA1if.png This... does not look like a sustainable level of work. 100 straight days of coding since last December? Yikes. This sounds like an interesting project, founded on good ideas, but sustaining this level of work will do no good if you burn out before there are other contributors to pick up the slack. I know that a lot of people can keep working this hard for much longer, but quality will inevitably start to suffer, well before you actually burn out. And, at the very least, you need to take the time to step back and validate your ideas. This is something you'll never do if you're always heads-down coding. I'm not saying that taking more breaks (weekends off, etc) works for everyone, but I just thought something needed to be said. I've seen friends and former colleagues fall into the trap of working on a project so deeply that it becomes a crazy obsession that they cannot escape. It's all they talk about and it can kind of scare people away. Not saying that this is such an extreme case, but it's something that can happen if you're not careful.

timothycrosley 11y ago
I couldn't disagree with this more. Just because you have 100 days of "coding" straight, doesn't mean it's just coding. It could be docs or anything else that's required. And most of those days look light. This is more like getting a habbit formed, then actually working 7 days a week. I mean look at my commit log: https://github.com/timothycrosley/ https://github.com/timothycrosley/, I plan on doing this the rest of my life. In reality, I've been coding every single day since I was 8. I'm now 26.
- hire_charts 11y ago
  That's great -- and I'm glad you disagree. It's awesome to find a sustainable level of work that can be repeated like that, indefinitely. I felt the need to speak up because I've seen contribution graphs EXACTLY like the author's and yours, except where the end result was not so happy. Take Zach Holman's for example: https://zachholman.com/posts/streaks/ https://zachholman.com/posts/streaks/ (chart is 2/3 down the post) Zach's streak was great for him. Until suddenly it wasn't, and far too late he realized he was burnt out. He's been amazingly public about the whole experience (his subsequent sabbatical, then his being fired from Github, then his year of soul-searching), and for every public story like this I suspect there are many, many private examples that go unheard. I think it's great when you can reach such a peak of productivity, like in your case. But it's important to recognize that you're kind of an outlier. The endless contribution "streak" doesn't work for everyone, at least not always and indefinitely. I don't want to generalize about what is always a good level of work and what isn't. What I DO want to communicate is that it's important to be introspective and take periodic steps back to evaluate the sustainability. After all, a level of work that seems unsustainable might turn out to be quite manageable, and vice versa.