5 ms·

Docker is a promising building block for solving logging and service management and that's something I've been playing with for a while. You stream your logs to

Docker is a promising building block for solving logging and service management and that's something I've been playing with for a while. You stream your logs to your container's stdout/stderr and can pick them up from the Docker REST API. You can also hit the Docker REST APIs across your hosts to discover the IP addresses and ports of the containers running various images. None of this is "solved" yet but it's getting there. Serialization... depends, but you can get pretty far with JSON over HTTP. So I guess you do need your microservices to be uniform in some respects, but the consistency of "every microservice is a Docker container exposing a REST API that speaks JSON and streams logs to stdout" would seem to be enough. It's easy to build that in several languages... if Rails makes sense, use Rails. If Sinatra makes sense, use Sinatra. If Java makes sense, use Java. You could run into trouble if you start using languages that only one or two of your staff members know, but it does allow for some flexibility (i.e. just because certain parts of the application are enterprisey doesn't mean everything has to be.)

AdrianRossouw 12y ago
Docker is currently switching to their own microservices library called libchan - https://github.com/docker/libchan https://github.com/docker/libchan It is msgpack over SPDY+TLS. Every language that will want to talk to docker will need a libchan implementation eventually. disclaimer: I'm the co-founder of the project to build the node.js version: https://github.com/graftjs/jschan https://github.com/graftjs/jschan My actual goal is to pull this into the browser though, and figure out what we can do with that : https://github.com/GraftJS/graft https://github.com/GraftJS/graft Until that is ready though, we have already built out our own microservices 'toolset' that does the [1] docker based deployment, and a small library that handles the [2] REST API, and that handles the [3] logging to logstash/kibana 1. http://longshoreman.io http://longshoreman.io 2. https://github.com/wayfin/waif https://github.com/wayfin/waif 3. https://github.com/wayfin/dockstash https://github.com/wayfin/dockstash
- PaulHoule 12y ago
  Hell yeah, latency matters and msgpack + SPDY beats the heck out of JSON/http.
bcoates 12y ago
Why not SNMP? It seems to be the protocol everyone loves to hate but there's a ton of ecosystem around it and it directly solves the status/discovery case
PaulHoule 12y ago
I definitely like the direction that Docker is going. JSON over HTTP sucks. I mean, it's OK if user experience doesn't matter, but if user experience does matter, then latency matters, and if latency matters you find pretty quickly that serialization/deserialization overhead is a big deal -- particularly in a microservices environment where an external request is going to mean a large number of microservice requests happen, which means data could be serialized/deserialized many times. If you're happy being in the middle of the pack, go on with JSON, but if you are playing to win you need something faster (preferably with a framework that means development as easy or easier than JSON) With multiple languages you don't just have the problem of training but you have other complexity-multiplying problems. For instance, one of the reasons why languages like Haskell are always a bridesmaid and never a bride is that they lack a complete "standard library". You need a urlencode() and decode function if you write web apps. If you write your own you may be more concerned with making it tail recursive than making it correct. I remember how the urldecode in Perl was busted for a decade and never got fixed, and that's a mainstream language. As flawed as PHP is, it took off because it had "good enough" implementations of everything web developers need in the stdlib. In a microservices system there tend to be functions similar to urlencode that need to be replicated across the system, and if these are implemented over and over again in different languages it is inevitable that some or all implementations will be buggy. If you can centralize these things in one library everybody depends upon, however, life is easy and happy.
- kalkal 12y ago
  > With multiple languages you don't just have the problem of training but you have other complexity-multiplying problems. > For instance, one of the reasons why languages like Haskell are always a bridesmaid and never a bride is that they lack a complete "standard library". You need a urlencode() and decode function if you write web apps. If you write your own you may be more concerned with making it tail recursive than making it correct. I remember how the urldecode in Perl was busted for a decade and never got fixed, and that's a mainstream language. As flawed as PHP is, it took off because it had "good enough" implementations of everything web developers need in the std lib. I don't understand what you're saying here - are you arguing that we shouldn't use languages that don't have web functionality built into the std lib? What about web frameworks, e.g. Java + Spring?
jburwell 12y ago
There are few issues with this approach. First, many security regulations require deterministic capture of audit-oriented log events (e.g. login failure/success, invalid access attempts, etc). Typically, you are required to provide a local spool to handle circumstances where a remote logging service is unavailable/overloaded. Simply logging to stdout is insufficient and hoping it gets picked by a process over HTTP is woefully inadequate in these circumstances. Second, as mentioned previously, HTTP headers will be larger than most logging events. Therefore, this approach likely doubles/triples (admittedly back of the napkin estimates) the data being transferred. rsyslog and its ilk learned this lesson long ago -- using a lightweight TCP protocol to minimize transfer overhead. Personally, I favor a locally spooling rsyslog instance in each container configured to push to a central logging service. Besides the issues of implementation, log data needs to be analyzed. Standardizing on a logging library and associated configuration provides important consistency to use a common set of log analytics in alerting systems not to mention keeping system admins performing forensics operations sane. Finally, in addition to logging, there is service instrumentation which requires consistency to yield useful visualization and alerting for operations teams. While Docker may allow permit a vast polyglot infrastructure, carefully choosing stacks deployed will have a tremendous impact on operational robustness. The greater the consistency across containers reduces the effort required to deploy and operate new services.
- superuser2 12y ago
  What you're attacking is not how Docker logging works. The Docker daemon collects stdout/stderr from the beginning of each container's life, always, and stores it locally. You can then ask the Docker API for all the log events from a particular container ID at any time. You would have one system that polls the Docker APIs on all your hosts and dumps the logs into the management system of your choice at some interval. Log events are not going to HTTP live, you are not making an HTTP request per entry, and the event will be captured whether or not the network is available at the moment.

AdrianRossouw 12y ago

Docker is currently switching to their own microservices library called libchan - https://github.com/docker/libchan https://github.com/docker/libchan It is msgpack over SPDY+TLS. Every language that will want to talk to docker will need a libchan implementation eventually. disclaimer: I'm the co-founder of the project to build the node.js version: https://github.com/graftjs/jschan https://github.com/graftjs/jschan My actual goal is to pull this into the browser though, and figure out what we can do with that : https://github.com/GraftJS/graft https://github.com/GraftJS/graft Until that is ready though, we have already built out our own microservices 'toolset' that does the [1] docker based deployment, and a small library that handles the [2] REST API, and that handles the [3] logging to logstash/kibana 1. http://longshoreman.io http://longshoreman.io 2. https://github.com/wayfin/waif https://github.com/wayfin/waif 3. https://github.com/wayfin/dockstash https://github.com/wayfin/dockstash

PaulHoule 12y ago
Hell yeah, latency matters and msgpack + SPDY beats the heck out of JSON/http.

bcoates 12y ago

Why not SNMP? It seems to be the protocol everyone loves to hate but there's a ton of ecosystem around it and it directly solves the status/discovery case

PaulHoule 12y ago

I definitely like the direction that Docker is going. JSON over HTTP sucks. I mean, it's OK if user experience doesn't matter, but if user experience does matter, then latency matters, and if latency matters you find pretty quickly that serialization/deserialization overhead is a big deal -- particularly in a microservices environment where an external request is going to mean a large number of microservice requests happen, which means data could be serialized/deserialized many times. If you're happy being in the middle of the pack, go on with JSON, but if you are playing to win you need something faster (preferably with a framework that means development as easy or easier than JSON) With multiple languages you don't just have the problem of training but you have other complexity-multiplying problems. For instance, one of the reasons why languages like Haskell are always a bridesmaid and never a bride is that they lack a complete "standard library". You need a urlencode() and decode function if you write web apps. If you write your own you may be more concerned with making it tail recursive than making it correct. I remember how the urldecode in Perl was busted for a decade and never got fixed, and that's a mainstream language. As flawed as PHP is, it took off because it had "good enough" implementations of everything web developers need in the stdlib. In a microservices system there tend to be functions similar to urlencode that need to be replicated across the system, and if these are implemented over and over again in different languages it is inevitable that some or all implementations will be buggy. If you can centralize these things in one library everybody depends upon, however, life is easy and happy.

kalkal 12y ago
> With multiple languages you don't just have the problem of training but you have other complexity-multiplying problems. > For instance, one of the reasons why languages like Haskell are always a bridesmaid and never a bride is that they lack a complete "standard library". You need a urlencode() and decode function if you write web apps. If you write your own you may be more concerned with making it tail recursive than making it correct. I remember how the urldecode in Perl was busted for a decade and never got fixed, and that's a mainstream language. As flawed as PHP is, it took off because it had "good enough" implementations of everything web developers need in the std lib. I don't understand what you're saying here - are you arguing that we shouldn't use languages that don't have web functionality built into the std lib? What about web frameworks, e.g. Java + Spring?

jburwell 12y ago

There are few issues with this approach. First, many security regulations require deterministic capture of audit-oriented log events (e.g. login failure/success, invalid access attempts, etc). Typically, you are required to provide a local spool to handle circumstances where a remote logging service is unavailable/overloaded. Simply logging to stdout is insufficient and hoping it gets picked by a process over HTTP is woefully inadequate in these circumstances. Second, as mentioned previously, HTTP headers will be larger than most logging events. Therefore, this approach likely doubles/triples (admittedly back of the napkin estimates) the data being transferred. rsyslog and its ilk learned this lesson long ago -- using a lightweight TCP protocol to minimize transfer overhead. Personally, I favor a locally spooling rsyslog instance in each container configured to push to a central logging service. Besides the issues of implementation, log data needs to be analyzed. Standardizing on a logging library and associated configuration provides important consistency to use a common set of log analytics in alerting systems not to mention keeping system admins performing forensics operations sane. Finally, in addition to logging, there is service instrumentation which requires consistency to yield useful visualization and alerting for operations teams. While Docker may allow permit a vast polyglot infrastructure, carefully choosing stacks deployed will have a tremendous impact on operational robustness. The greater the consistency across containers reduces the effort required to deploy and operate new services.

superuser2 12y ago
What you're attacking is not how Docker logging works. The Docker daemon collects stdout/stderr from the beginning of each container's life, always, and stores it locally. You can then ask the Docker API for all the log events from a particular container ID at any time. You would have one system that polls the Docker APIs on all your hosts and dumps the logs into the management system of your choice at some interval. Log events are not going to HTTP live, you are not making an HTTP request per entry, and the event will be captured whether or not the network is available at the moment.