5 ms·

it seems that none of the container frameworks are generally ready. Take for example k8s - I just started exploring it as something we could move to. https://g

it seems that none of the container frameworks are generally ready. Take for example k8s - I just started exploring it as something we could move to. https://github.com/kubernetes/kubernetes/issues/24677 https://github.com/kubernetes/kubernetes/issues/24677 - logging of application logs is an unsolved problem. And most of the proposals talk about creating yet another logger...rather than patching journald or whatever exists out there. For those of you who are running k8s in production - how are you doing logging ? does everyone roll their own ?

felixgallo 10y ago
logging of application logs generally in the world is an unsolved problem, but I feel like k8s does better than most; fluentd is the default, and then logs are collected by whatever end system you like the best that's appropriate for your needs (e.g. GKE -> fluentd -> Cloud Logging, and I've heard of K8S -> fluentd -> ELK).
- sandGorgon 10y ago
  since most people use k8s in the context of GKE.. there seems to be some magic that Google has put in. But if you read the bug, this doesnt work quite well in the wild. In fact, an entire spec is being written on it and right now the debate is whether to forward to journald and have it proxy further or reimplement a new logging format.
  - felixgallo 10y ago
    well, there's definitely systemd people pushing for journald, that's for sure. The first comment talks about how it doesn't fit, though. I wonder what in particular about the solution doesn't work for those people.
lobster_johnson 10y ago
Logging is definitely not an unsolved problem with K8s. It's trivial to set up Fluentd to snarf all container logs into some destination, such as Graylog, ElasticSearch or a simple centralized file system location. The Github issue talks a lot about "out of the box" setup via kube-up. If you're not using kube-up (and I wouldn't recommend using it for production setups), the problem is rather simple. The logging story could be better, but it's not "unsolved".
- sandGorgon 10y ago
  Thanks for that clarification. We are trying to get our feet wet by building a small 2 node setup. We don't want to stream our logs or setup fluent, etc. I just want to make sure my logs are captured and periodically rotated. Now, the newer docker allows me to use journald as logger (which means all logs are sent to journald on the HOST machine)... But I can't seem to figure out how to do this in k8s. Also as an aside, for production deployment on aws.. What would you suggest. I was thinking that kube-up is what i should use.
  - lobster_johnson 10y ago
    Yes, journald would definitely be superior, conceptually, to just writing to plain files. I can't help you there, though, as I've never tried setting it up. (As an aside, I don't understand why more Unix software doesn't consistently use syslog() — we already have a good, standard, abstract logging interface which can be implemented however you want on the host. The syslog protocol is awful, but the syslog() call should be reliable.) As for kube-up: It's a nice way to bootstrap, but it's an opaque setup that doesn't give you full control over your boxes, or of upgrades. I believe it's idempotent, but I wouldn't trust it to my production environment. Personally, I set up K8s from scratch on AWS via Salt + Debian/Ubuntu packages from Kismatic, and I recommend this approach. I have a Github repo that I'm planning to make semi-generic and public. Email me if you're interested.
  - robszumski 10y ago
    Check out kube-aws, a tool from CoreOS that will get you a proper CloudFormation/VPC set up in just a few minutes: https://coreos.com/kubernetes/docs/latest/kubernetes-on-aws.html https://coreos.com/kubernetes/docs/latest/kubernetes-on-aws.... If you're interested to see more about what it does behind the scenes, it is very similar to the full manual guide for setting up Kubernetes: https://coreos.com/kubernetes/docs/latest/getting-started.html https://coreos.com/kubernetes/docs/latest/getting-started.ht...
  - Goopplesoft 10y ago
    Whats the k8's limitation here? You can do it however you like. e.g. make a fluentd daemonset, deploy on both nodes have them slurp up to s3. There are tons of ways to handle the logs and k8s can either help or stay out of the way. https://github.com/fabric8io/fluent-plugin-kubernetes_metadata_filter https://github.com/fabric8io/fluent-plugin-kubernetes_metada...
Annatar 10y ago
it seems that none of the container frameworks are generally ready. None of the container frameworks on Linux are ready, but that doesn't mean that's the case in general.

felixgallo 10y ago

logging of application logs generally in the world is an unsolved problem, but I feel like k8s does better than most; fluentd is the default, and then logs are collected by whatever end system you like the best that's appropriate for your needs (e.g. GKE -> fluentd -> Cloud Logging, and I've heard of K8S -> fluentd -> ELK).

sandGorgon 10y ago
since most people use k8s in the context of GKE.. there seems to be some magic that Google has put in. But if you read the bug, this doesnt work quite well in the wild. In fact, an entire spec is being written on it and right now the debate is whether to forward to journald and have it proxy further or reimplement a new logging format.
- felixgallo 10y ago
  well, there's definitely systemd people pushing for journald, that's for sure. The first comment talks about how it doesn't fit, though. I wonder what in particular about the solution doesn't work for those people.

lobster_johnson 10y ago

Logging is definitely not an unsolved problem with K8s. It's trivial to set up Fluentd to snarf all container logs into some destination, such as Graylog, ElasticSearch or a simple centralized file system location. The Github issue talks a lot about "out of the box" setup via kube-up. If you're not using kube-up (and I wouldn't recommend using it for production setups), the problem is rather simple. The logging story could be better, but it's not "unsolved".

sandGorgon 10y ago
Thanks for that clarification. We are trying to get our feet wet by building a small 2 node setup. We don't want to stream our logs or setup fluent, etc. I just want to make sure my logs are captured and periodically rotated. Now, the newer docker allows me to use journald as logger (which means all logs are sent to journald on the HOST machine)... But I can't seem to figure out how to do this in k8s. Also as an aside, for production deployment on aws.. What would you suggest. I was thinking that kube-up is what i should use.
- lobster_johnson 10y ago
  Yes, journald would definitely be superior, conceptually, to just writing to plain files. I can't help you there, though, as I've never tried setting it up. (As an aside, I don't understand why more Unix software doesn't consistently use syslog() — we already have a good, standard, abstract logging interface which can be implemented however you want on the host. The syslog protocol is awful, but the syslog() call should be reliable.) As for kube-up: It's a nice way to bootstrap, but it's an opaque setup that doesn't give you full control over your boxes, or of upgrades. I believe it's idempotent, but I wouldn't trust it to my production environment. Personally, I set up K8s from scratch on AWS via Salt + Debian/Ubuntu packages from Kismatic, and I recommend this approach. I have a Github repo that I'm planning to make semi-generic and public. Email me if you're interested.
- robszumski 10y ago
  Check out kube-aws, a tool from CoreOS that will get you a proper CloudFormation/VPC set up in just a few minutes: https://coreos.com/kubernetes/docs/latest/kubernetes-on-aws.html https://coreos.com/kubernetes/docs/latest/kubernetes-on-aws.... If you're interested to see more about what it does behind the scenes, it is very similar to the full manual guide for setting up Kubernetes: https://coreos.com/kubernetes/docs/latest/getting-started.html https://coreos.com/kubernetes/docs/latest/getting-started.ht...
- Goopplesoft 10y ago
  Whats the k8's limitation here? You can do it however you like. e.g. make a fluentd daemonset, deploy on both nodes have them slurp up to s3. There are tons of ways to handle the logs and k8s can either help or stay out of the way. https://github.com/fabric8io/fluent-plugin-kubernetes_metadata_filter https://github.com/fabric8io/fluent-plugin-kubernetes_metada...

Annatar 10y ago

it seems that none of the container frameworks are generally ready. None of the container frameworks on Linux are ready, but that doesn't mean that's the case in general.