5 ms·

I've used K8S before, and those things are smooth once installation happen. We have a project which is looking at using Docker Swarm on a two node set up. The i

by hosh 10y ago

I've used K8S before, and those things are smooth once installation happen. We have a project which is looking at using Docker Swarm on a two node set up. The idea is that this will give us a baseline, and since it is looking more and more we'll move to GKE, this sets us up for a later move into GKE. I thought Docker Swarm would be mature enough to handle two nodes, for a short time (maybe half a year) until we make the leap to GKE. But based on the reports coming in from the wild, that's looking less and less likely. We hired a contractor to do this part. My team is so resource constrained that I don't have time for this, so we farmed it out. But now I'm thinking, the risk of project failure is much higher than I thought, made worse that the contractor is also showing signs of having poor communication skills. (I would rather be updated on things going wrong than to have someone try to be the hero or cowboy and figure it all out).

sandGorgon 10y ago
we are exactly in your boat. However, we are signing up for kubernetes since we want to keep the GKE option open. The only reason why we didnt go ahead is because of broken logging in k8s - https://github.com/kubernetes/kubernetes/issues/24677 https://github.com/kubernetes/kubernetes/issues/24677
- kozikow 10y ago
  I went through a few iterations for logging, but now I settled on using built-in GKE logging. Stdout logs from my containers are picked up by kubernetes and forwarded to stackdriver. Since it's just stdout I do not create too much lock-in. I use stackdriver dashboard for investigating recent logs and BigQuery exporter for complex analysis. My stdout logs are jsons, so I can export extra metadata without relying on regexps for analysis - I use https://pypi.python.org/pypi/python-json-logger https://pypi.python.org/pypi/python-json-logger.
  - sandGorgon 10y ago
    since we are not on GKE already, I want to be able to use k8s to forward to my host machine journald and thats broken. I think Google is doing a lot of handholding to get it working with stackdriver. This is the blocker for me. I cant switch to GKE already because I use AWS postgresql. But I want to use k8s :(
    - hosh 10y ago
      Thr K8S project I was involved in last year used AWS postgresql too. At that time, figuring out how to have persistant data was too much. Further, AWS EBS driver for K8S storage wasn't there. And, PetSets have not come out. With PetSets out, I think figuring out how to do a datadtore on K8S or GKE will be easier. (I had mentioned to my CTO, I don't know how to do persistant store on GKE; he pointed out the gcloud data store; I told him, that was not what I meant ;-)
      - sandGorgon 10y ago
        this is interesting. Were you on GKE or AWS. GKE is obvious, but if you were hosting on AWS.. what was your logging setup ? If you were not using AWS EBS .. what were you doing ?
        hosh 10y ago
        The owner of the company ran out of money before I could add logging, but my plan was to get it out to something like papertrail. I was on AWS. I sidestepped the issue by using AWS RDS (postgresql). I had tried to get the nascent EBS stuff working, but when I realized that I'd have to get a script to check if an EBS volume was formatted with a filesystem before mounting it in K8S, I stopped. This might have been improved by now.
        justinsb 10y ago
        I probably wrote that support (or at least maintain it), and you shouldn't ever have needed to add a script to format your disk: you declare the filesystem you want and it comes up formatted. If you didn't open an issue before please do so and I'll make double-sure it is now fixed (or point me to the issue if you already opened it!) On the logging front, kube-up comes up with automatic logging via fluentd to an ElasticSearch cluster hosted in k8s itself. You can relatively easily replace that ES cluster with an AWS ES cluster (using a proxy to do the AWS authentication), or you can reconfigure fluentd to run to AWS ES. Or you can set pretty easily set up something yourself using daemonsets if you'd rather use something like splunk, but I don't know if anyone has shared a config for this! A big shortcoming of the current fluentd/ES setup is that it also predates PetSets, and so it still doesn't use persistent storage in kube-up. I'm trying to fix this in time for 1.4 though! If you don't know about it, the sig-aws channel on the kubernetes slack is where the AWS folk tend to hang out and work through these snafus together - come join us :-)
  - tazjin 10y ago
    Just as a hint in case you're not aware of this: If you log errors in the format[1] expected by Stackdriver Error Reporting, your errors will automatically be picked up and grouped in that service as well. 1: https://cloud.google.com/error-reporting/docs/formatting-error-messages https://cloud.google.com/error-reporting/docs/formatting-err...

by hosh 10y ago

sandGorgon 10y ago

we are exactly in your boat. However, we are signing up for kubernetes since we want to keep the GKE option open. The only reason why we didnt go ahead is because of broken logging in k8s - https://github.com/kubernetes/kubernetes/issues/24677 https://github.com/kubernetes/kubernetes/issues/24677

kozikow 10y ago
I went through a few iterations for logging, but now I settled on using built-in GKE logging. Stdout logs from my containers are picked up by kubernetes and forwarded to stackdriver. Since it's just stdout I do not create too much lock-in. I use stackdriver dashboard for investigating recent logs and BigQuery exporter for complex analysis. My stdout logs are jsons, so I can export extra metadata without relying on regexps for analysis - I use https://pypi.python.org/pypi/python-json-logger https://pypi.python.org/pypi/python-json-logger.
- sandGorgon 10y ago
  since we are not on GKE already, I want to be able to use k8s to forward to my host machine journald and thats broken. I think Google is doing a lot of handholding to get it working with stackdriver. This is the blocker for me. I cant switch to GKE already because I use AWS postgresql. But I want to use k8s :(
  - hosh 10y ago
    Thr K8S project I was involved in last year used AWS postgresql too. At that time, figuring out how to have persistant data was too much. Further, AWS EBS driver for K8S storage wasn't there. And, PetSets have not come out. With PetSets out, I think figuring out how to do a datadtore on K8S or GKE will be easier. (I had mentioned to my CTO, I don't know how to do persistant store on GKE; he pointed out the gcloud data store; I told him, that was not what I meant ;-)
    - sandGorgon 10y ago
      this is interesting. Were you on GKE or AWS. GKE is obvious, but if you were hosting on AWS.. what was your logging setup ? If you were not using AWS EBS .. what were you doing ?
      - hosh 10y ago
        The owner of the company ran out of money before I could add logging, but my plan was to get it out to something like papertrail. I was on AWS. I sidestepped the issue by using AWS RDS (postgresql). I had tried to get the nascent EBS stuff working, but when I realized that I'd have to get a script to check if an EBS volume was formatted with a filesystem before mounting it in K8S, I stopped. This might have been improved by now.
        justinsb 10y ago
        I probably wrote that support (or at least maintain it), and you shouldn't ever have needed to add a script to format your disk: you declare the filesystem you want and it comes up formatted. If you didn't open an issue before please do so and I'll make double-sure it is now fixed (or point me to the issue if you already opened it!) On the logging front, kube-up comes up with automatic logging via fluentd to an ElasticSearch cluster hosted in k8s itself. You can relatively easily replace that ES cluster with an AWS ES cluster (using a proxy to do the AWS authentication), or you can reconfigure fluentd to run to AWS ES. Or you can set pretty easily set up something yourself using daemonsets if you'd rather use something like splunk, but I don't know if anyone has shared a config for this! A big shortcoming of the current fluentd/ES setup is that it also predates PetSets, and so it still doesn't use persistent storage in kube-up. I'm trying to fix this in time for 1.4 though! If you don't know about it, the sig-aws channel on the kubernetes slack is where the AWS folk tend to hang out and work through these snafus together - come join us :-)
- tazjin 10y ago
  Just as a hint in case you're not aware of this: If you log errors in the format[1] expected by Stackdriver Error Reporting, your errors will automatically be picked up and grouped in that service as well. 1: https://cloud.google.com/error-reporting/docs/formatting-error-messages https://cloud.google.com/error-reporting/docs/formatting-err...