4 ms·
The author does make a valid point in that logs are harder to collect the more processes you add. For example, I run gunicorn inside of a container with worker
by conradev 11y ago
The author does make a valid point in that logs are harder to collect the more processes you add.
For example, I run gunicorn inside of a container with worker processes, and I had to do some trickery (writing to /proc/1/fd/1) to redirect the logs of the worker processes to the stdout of the master process so that they would show up in `docker logs`.
- stephenr 11y agoIt's almost like the idea of relying on stdout from processes in a container to collect logs is fucking stupid. With all the hoops people jump through so they can say "we use Docker, we're cool" I think I shall refer to Docker fan-boys as Olympic Gymnasts from now on.
- XorNot 11y agoIf you drill through the docker source code, it becomes apparent that their are some rather serious issues to doing something other then this. I really really want a LOG directive for docker where I can just setup some pipes which the daemon will collect and tag into it's log files (would make dealing with old Java processes a lot easier). But actually finding somewhere to hook this logic in - huge task. A lot harder then grabbing stdout/stderr of the init process in the container.
- stephenr 11y agoThe whole point of getting logs out of containers is to centralise and store them somehow. This exact job is what Syslog (over a network) does.
- Gigablah 11y agoYou can use different logging drivers: https://docs.docker.com/engine/admin/logging/overview/ https://docs.docker.com/engine/admin/logging/overview/
- ibotty 11y agoThat does not answer the question on how to capture _different_ application logs.
- elbear 11y agoWhy not have the application log to a file on a data container?
- DanielDent 11y agoWell, now you've just created a new problem to deal with...
- lisivka 11y agoWho will manage these volumes, directories and log files? Why not just use existing infrastructure? I use containers for 10 years (they are called VServer), and I newer had a problem with standard setup: some less important logs are stored directly in container and rotated with logrotate, some more important logs (and error messages from less important logs) are forwarded to central server for analyzing, some are just disabled or discarded.
- DanielDent 11y agoI'm a heavy docker (ab)user, but I haven't gone through the source code - i.e. perhaps there is some complexity I'm overlooking. But couldn't Docker provide additional file handles for extra logging streams? Programs which understand they live in a Docker world could hook into an special system call directly. Non-Docker-Logging-Enabled apps could have a helper program (conceptually similar to 'ip netns exec') remap STDOUT & STDERR.
- Gigablah 11y agoOr maybe we can refrain from calling each other perjorative names? Wouldn't that be grand?
- aprdm 11y agoWhy is it stupid ? Is the same approach that heroku advocates. I have loads of containers logging to stdout and another container which collects all stdouts and send to a logging-as-a-service. Why is it a bad idea ?
- stephenr 11y ago> Why is it stupid Let's compare logging via stdout, to say syslog, with a simple test. How do you identify the difference between a warning and an error or an emergency, in a string sent to stdout? What happens if the message contains a newline. How do you know if what follows is a continuation of the log entry, or is a new log entry? If your application dares to have two processes in the same container (i know, crazy concept) how do you separate them. The "literally one process per container" idea is what both causes the issue, and causes the obvious solution to the issue to be even harder to solve for the cases where software doesn't natively support the syslog protocol.
- aprdm 11y agoWere you the one who downvotted me because you disagree? o.O Anyways, this is dealt with in the logging-as-a-service (LAAS) that receives the logs... say you are using the python logger, you have different types of logging (DEBUG, INFO, WARN, ERR). If you just start the line with one of those statuses, your LAAS will know it should email you or something like that. The print, or file, or whatever, it's just how your data is getting somewhere for the proper analysis. You know if it's a new line or not because every log message starts with the same format, something like {APP, TIME, SEVERITY} and then the message...
- stephenr 11y agoSo instead of having actually structured information in defined fields, you're relying on a specific service's format. So presumably you then have to try to get all the services you use (e.g. Apache / NGinx / HAproxy / Varnish / (my|postgre)sql / redis / memcache etc) to format logs in the same way, and do it to stdout... > Were you the one who downvotted me because you disagree? o.O Down voting on HN is broken[1], so I don't use it, ever[2]. 1. In the sense of, it does the wrong thing, not in the sense of it doesn't do anything. 2. Except when i hit the wrong thing because of the complete lack of usability on a mobile device.
- derFunk 11y agoNormally, if you run Docker you won't have a single container anyway, normally you want to scale out. Hence you'd be better off using a centralized logging infrastructure like ELK or one of the many commercial solutions like Splunk etc.
- LoSboccacc 11y agoI don't follow. If you go with one service one machine model, not using a container is more or less the same than just scripting a repeatable install. I'd say they are for isolation before they're for scaling out
- renke1 11y agoTrue, but it's not uncommon to only route the stdout/stderr of each container to fluentd, logstash and the like such that the containers do not have to care about where the logs go (other than just writing to stdout/stderr). And now that we have logging driver I definitely would want to avoid custom in-container log routing.
- derFunk 11y agoYes, it's good practice to have a separated logging container (eg logstash forwarder, filebeat etc), which accesses the shared log folder of the other container you want to actually forward the logs from. I'm recommending using file forwarder because this way you can simply categorize your logs by filename. Pushing all to stdout is a pain.
- fatherlinnux 11y agoYeah, the problem is getting them from each process in the container, then get them to the centralized logging. It can be done either way though. Especially with the syslog driver in Docker...
- deleted 11y ago[deleted]