4 ms·
My current project consists of ~7 microservices, which has been wonderful for many reasons but exposed us to substantial complexity on this score. At present ~
by patio11 11y ago
My current project consists of ~7 microservices, which has been wonderful for many reasons but exposed us to substantial complexity on this score. At present ~2 of them have to receive communications internally or externally over HTTP, so we have those fronted by a proxy to make service discovery easier. (It is, regrettably, not an off-the-shelf proxy, since one of them required application-specific routing to a sticky host(s) depending on the contents of the HTTP request).
The thing which has made our lives MUCH easier from an orchestration, discovery, and routing perspective is NSQ, an OSS message bus. It's an event model rather than a request model -- producers of events say "Hey world, by the way, EventType just happened." and consumers can register to say "Apropos of nothing, please tell me about any EventType" without producers or consumers having to be aware of each other or aware of what a consumer intends to actually do with regards to the event.
Events go into topics ("a type of event") and each topic can have an arbitrary number of channels ("one thing which should happen for each event in this topic"). If you're clever about where you raise events, you can stitch together arbitrarily complex systems by just adding new consumers on new channels, without having to make the producers aware of the new functionality at all.
Concrete example: We wanted a log of all orders made to the stock exchange. We have a little utility, nsq-to-s3, which takes all the events on a topic/channel, persists them in memory/disk for a while, and periodically flushes that log to S3 for long-term storage. This trivialized the logging feature. (I was going to write our own nsq-to-s3 but an OSS project of that same name existed and was adequate to our needs, so we didn't have to go to 8 microservices... yet.)
This was much easier than making a generic S3 logging service and then making edits to seven microservice codebases to make a HTTP request out to the logging service, with e.g. retry/failure recovery/monitoring/etc, everywhere in those codebases we wanted something logged to S3.
- ma2rten 11y agoThis sounds similar to kafka, how does it compare?
- patio11 11y agoI have heard from several folks that they use Kafka where we'd use NSQ, but have insufficient experience with Kafka to give an informed opinion on the tradeoffs. I will say that few technologies I've used in my career were as easy to adopt as NSQ was.
- kasey_junk 11y agoKafka makes a central promise that NSQ doesn't, that of delivery order. This is a complex promise to maintain in a distributed system, and this complexity is apparent in operating a Kafka based system.
- buro9 11y agoThis is how Hailo https://sudo.hailoapp.com/services/2015/03/09/journey-into-a-microservice-world-part-2/ https://sudo.hailoapp.com/services/2015/03/09/journey-into-a... , Mondo https://getmondo.co.uk/ https://getmondo.co.uk/ and a few others have approached the orchestration. Hailo uses Rabbit, Mondo uses NSQ. It really starts to shine when you add controls to the consumers to enable soak testing of new services by either redirecting some small % of traffic to the new version dynamically.
- sciurus 11y ago> It is, regrettably, not an off-the-shelf proxy, since one of them required application-specific routing to a sticky host(s) depending on the contents of the HTTP request. Did haproxy not support the necessary ACLs for you to accomplish this? It might be possible in 1.6, which lets you process the HTTP request body in addition to the headers. http://blog.haproxy.com/2015/10/14/whats-new-in-haproxy-1-6/ http://blog.haproxy.com/2015/10/14/whats-new-in-haproxy-1-6/
- Simucal 11y agoDo you have a centralized nsqd instance, or do you have a 1:1 relationship between nsqd instances and producers?
- patio11 11y agoOne nsqlookupd, one nsqd per machine which hosts 1+ producers. We'll eventually have a small number of nsqlookupd.
- Erwin 11y agoTangentially, but is there a queueing service with good and fast retrying of messages with no destination? Basically I'd like the incoming router to send a message of a specific type and subtype to an existing listener, but if one does not exist I'd like to take back the message, spawn a new listener which will get on the queue and resend the message. I wonder if I could use a messaging service for keeping track of this, rather than have to manage my own map of what application backends are ready to process which type of messages ( spawning that backend takes a lot of resources so I want them to be born and die based on demand).
- arshmand 11y agoCurious to know how you handle dependencies between services. For example, say you have a User service, I could imagine creating a client library for interacting with this service (the library being responsible for knowing where the http end point is, serializing/unserializing JSON into domain objects, etc.). Then any service wanting to interact with the User service just adds this library as a dependency. Along those same lines how do you deal with fetching of domain objects and associated relationships from these services? For example, let's say in one instance you want a user and their manager and in another instance you want a user and their manager's manager and so forth for any relationship that a user might have (that the User service is responsible for).