4 ms·
While the article does not touch on this explicitly, Docker's Log Driver for Fluentd supports tags, which can be used to pass on container name/ID and other met
by ktamura 10y ago
While the article does not touch on this explicitly, Docker's Log Driver for Fluentd supports tags, which can be used to pass on container name/ID and other metadata to wherever logs are sent. Since Docker Machine itself allows these metadata to be passed on via API ({{.Name}}, etc.) one can create a fairly robust system to track log lineage.
- meddlepal 10y agoThat approach requires a lot of post processing to make it useful. Further it's pretty hard (near impossible) to do causal ordering.
- ktamura 10y agoI agree on the near-impossibility of causal ordering at scale. I don't see the need for a lot of "post processing". Adding filters in Fluentd is fairly simple: http://docs.fluentd.org/articles/filter-plugin-overview http://docs.fluentd.org/articles/filter-plugin-overview
- rdli 10y agoHow would passing on the container ID help in mapping multiple log messages to the same request? If a request is going through multiple containers, you'll get multiple container IDs. To solve this problem, you need a unique trace ID and vector clock, and this actually needs to propagated through your logging system, not through container tags.
- ktamura 10y agoAh, I completely misunderstood. Yes, you are right: Threading data together across multiple containers belonging to a single logical request is hard (and AFAIK, none of the container orchestration tools deal with it well).
- bogomipz 10y agoHow is the unique trace ID implemented? Vector clocks would be in use by your distributed data store for conflict resolution such as Riak does. Could you elaborate on how you've implemented such a system?
- rdli 10y agoIt's not about conflict resolution, it's about ordering. Imagine you have 3 services. A calls B calls C. A request goes into A. The request fails. Since there are 3 services involved, you want to be able to see the log statements of all 3 services, in order. To do this, you need a UUID that is passed from A to B to C and a vector clock that is incremented from A to B to C so you get perfect ordering. Passing the data around is not super hard (you can use HTTP headers, for example). The tricky bit is having a library that does the right thing in A and B and C. And if A is written in Java and B is written in NodeJS, that library better work on both!
- bogomipz 10y agoYes, this makes total sense. I think I completely misunderstood you post earlier. Thanks for the clarification.