7 ms·

I am using kubernetes instead of docker swarm to orchestrate docker images and all points mentioned in this article do not apply. My cluster is small - I have <

by kozikow 10y ago

I am using kubernetes instead of docker swarm to orchestrate docker images and all points mentioned in this article do not apply. My cluster is small - I have <100 machines at peak, but so far it feels ready for prime time. There are parts of docker that are relatively stable, have many other companies involved, and have been around for a while. There are also "got VC money, gotta monetise" parts that damage the reputation of stable parts.

hosh 10y ago
I've used K8S before, and those things are smooth once installation happen. We have a project which is looking at using Docker Swarm on a two node set up. The idea is that this will give us a baseline, and since it is looking more and more we'll move to GKE, this sets us up for a later move into GKE. I thought Docker Swarm would be mature enough to handle two nodes, for a short time (maybe half a year) until we make the leap to GKE. But based on the reports coming in from the wild, that's looking less and less likely. We hired a contractor to do this part. My team is so resource constrained that I don't have time for this, so we farmed it out. But now I'm thinking, the risk of project failure is much higher than I thought, made worse that the contractor is also showing signs of having poor communication skills. (I would rather be updated on things going wrong than to have someone try to be the hero or cowboy and figure it all out).
- sandGorgon 10y ago
  we are exactly in your boat. However, we are signing up for kubernetes since we want to keep the GKE option open. The only reason why we didnt go ahead is because of broken logging in k8s - https://github.com/kubernetes/kubernetes/issues/24677 https://github.com/kubernetes/kubernetes/issues/24677
  - kozikow 10y ago
    I went through a few iterations for logging, but now I settled on using built-in GKE logging. Stdout logs from my containers are picked up by kubernetes and forwarded to stackdriver. Since it's just stdout I do not create too much lock-in. I use stackdriver dashboard for investigating recent logs and BigQuery exporter for complex analysis. My stdout logs are jsons, so I can export extra metadata without relying on regexps for analysis - I use https://pypi.python.org/pypi/python-json-logger https://pypi.python.org/pypi/python-json-logger.
    - sandGorgon 10y ago
      since we are not on GKE already, I want to be able to use k8s to forward to my host machine journald and thats broken. I think Google is doing a lot of handholding to get it working with stackdriver. This is the blocker for me. I cant switch to GKE already because I use AWS postgresql. But I want to use k8s :(
      - hosh 10y ago
        Thr K8S project I was involved in last year used AWS postgresql too. At that time, figuring out how to have persistant data was too much. Further, AWS EBS driver for K8S storage wasn't there. And, PetSets have not come out. With PetSets out, I think figuring out how to do a datadtore on K8S or GKE will be easier. (I had mentioned to my CTO, I don't know how to do persistant store on GKE; he pointed out the gcloud data store; I told him, that was not what I meant ;-)
        sandGorgon 10y ago
        this is interesting. Were you on GKE or AWS. GKE is obvious, but if you were hosting on AWS.. what was your logging setup ? If you were not using AWS EBS .. what were you doing ?
        hosh 10y ago
        The owner of the company ran out of money before I could add logging, but my plan was to get it out to something like papertrail. I was on AWS. I sidestepped the issue by using AWS RDS (postgresql). I had tried to get the nascent EBS stuff working, but when I realized that I'd have to get a script to check if an EBS volume was formatted with a filesystem before mounting it in K8S, I stopped. This might have been improved by now.
        justinsb 10y ago
        I probably wrote that support (or at least maintain it), and you shouldn't ever have needed to add a script to format your disk: you declare the filesystem you want and it comes up formatted. If you didn't open an issue before please do so and I'll make double-sure it is now fixed (or point me to the issue if you already opened it!) On the logging front, kube-up comes up with automatic logging via fluentd to an ElasticSearch cluster hosted in k8s itself. You can relatively easily replace that ES cluster with an AWS ES cluster (using a proxy to do the AWS authentication), or you can reconfigure fluentd to run to AWS ES. Or you can set pretty easily set up something yourself using daemonsets if you'd rather use something like splunk, but I don't know if anyone has shared a config for this! A big shortcoming of the current fluentd/ES setup is that it also predates PetSets, and so it still doesn't use persistent storage in kube-up. I'm trying to fix this in time for 1.4 though! If you don't know about it, the sig-aws channel on the kubernetes slack is where the AWS folk tend to hang out and work through these snafus together - come join us :-)
brianwawok 10y ago
Heard this elsewhere. Why do people use Docker swarm with Kubernetes around? Does it require less setup or something?
- kozikow 10y ago
  I see two reasons for this: - If you don't use google container engine (hosted kubernetes, also known as GKE), kubernetes have a reputation of being hard to set up. I am running on GKE, so I can't comment too much. - It's hard to see how all of the orchestration/docker/etc. things play together when entering the area. I expect many people hear "docker" and want to just try "docker", not being aware that there exist alternatives for some parts, some parts are more reliable that others, etc. E.g. the article we are discussing seem to be doing this.
  - tazjin 10y ago
    Kubernetes definitely runs best on GKE, but projects like kops (https://github.com/kubernetes/kops https://github.com/kubernetes/kops) are making setups on AWS etc. a lot easier. You still have to deal with the other drawbacks of those platforms (slow networks, disks etc.) but that's not really a k8s issue.
  - rodgerd 10y ago
    > kubernetes have a reputation of being hard to set up Yeah, but there's no shortage of things wrapping Kuber (e.g. OpenShift).
- jjn2009 10y ago
  You can have a swarm with only a handful of commands now, with TLS (for docker api). Eventually stable docker host to docker host container networking will have encryption out of the box too. The number of steps for a production docker setup keeps getting shorter and shorter, kubernetes seems to be now focused on this aspect, they might catch up. This is all of course ignoring the IaaS solutions and obviously for the reasons stated in the article, swarm doesn't see those benefits with 1.12 because major features do not work yet which are supposedly stable.
chx 10y ago
> My cluster is small - I have <100 machines at peak Let me ask: WTF are people are doing that <100 machines is a "small cluster"? I ran a Top 100 (as measured by Quantcast) website with 23 machines and that included kit and caboodle -- Dev, Staging and Production environments. And quite some of that were just for HA purposes not because we needed that much... Stackexchange also runs about two dozen servers. Yes, yes, Google, Facebook runs datacenters but there's a power law kind of distribution here and the iron needs are falling very, very fast as you move from say Top 1 website to Top 30.
- kozikow 10y ago
  I have a processing pipeline that does crawling, indexing and some analytics on the crawled data - I am building a vertical-focused search engine focused on catering for some specific kinds of companies. Usually my cluster is very small, unless the pipeline is in progress. At this very moment I am not doing any processing, so I only have 3 machines up.
- brown9-2 10y ago
  Not all computing use cases are running a website.
- necubi 10y ago
  The number of machines you need to run a service is not really a linear function of your traffic. If you have a mostly static website that can be heavily cached/cdn'd, you can easily scale to thousands of requests a second with a small server footprint. I expect that's true of many of the top 100 sites as measured by visitors (like Quantcast does). But if you need to store a lot of data, or need to look up data with very low latency, or do CPU-intensive work for every request, you will end up with a lot more servers. (The other thing to consider is that SaSS companies can easily deal with more traffic than even the largest web sites, because they tend to aggregate traffic from many websites; Quantcast, for example, where I used to work, got hundreds of thousands of requests per second to its measurement endpoint.)
  - chx 10y ago
    Note: the site I mentioned did hit the database quite a few times for each page. It was a nice challenge.

by kozikow 10y ago

hosh 10y ago

I've used K8S before, and those things are smooth once installation happen. We have a project which is looking at using Docker Swarm on a two node set up. The idea is that this will give us a baseline, and since it is looking more and more we'll move to GKE, this sets us up for a later move into GKE. I thought Docker Swarm would be mature enough to handle two nodes, for a short time (maybe half a year) until we make the leap to GKE. But based on the reports coming in from the wild, that's looking less and less likely. We hired a contractor to do this part. My team is so resource constrained that I don't have time for this, so we farmed it out. But now I'm thinking, the risk of project failure is much higher than I thought, made worse that the contractor is also showing signs of having poor communication skills. (I would rather be updated on things going wrong than to have someone try to be the hero or cowboy and figure it all out).

sandGorgon 10y ago
we are exactly in your boat. However, we are signing up for kubernetes since we want to keep the GKE option open. The only reason why we didnt go ahead is because of broken logging in k8s - https://github.com/kubernetes/kubernetes/issues/24677 https://github.com/kubernetes/kubernetes/issues/24677
- kozikow 10y ago
  I went through a few iterations for logging, but now I settled on using built-in GKE logging. Stdout logs from my containers are picked up by kubernetes and forwarded to stackdriver. Since it's just stdout I do not create too much lock-in. I use stackdriver dashboard for investigating recent logs and BigQuery exporter for complex analysis. My stdout logs are jsons, so I can export extra metadata without relying on regexps for analysis - I use https://pypi.python.org/pypi/python-json-logger https://pypi.python.org/pypi/python-json-logger.
  - sandGorgon 10y ago
    since we are not on GKE already, I want to be able to use k8s to forward to my host machine journald and thats broken. I think Google is doing a lot of handholding to get it working with stackdriver. This is the blocker for me. I cant switch to GKE already because I use AWS postgresql. But I want to use k8s :(
    - hosh 10y ago
      Thr K8S project I was involved in last year used AWS postgresql too. At that time, figuring out how to have persistant data was too much. Further, AWS EBS driver for K8S storage wasn't there. And, PetSets have not come out. With PetSets out, I think figuring out how to do a datadtore on K8S or GKE will be easier. (I had mentioned to my CTO, I don't know how to do persistant store on GKE; he pointed out the gcloud data store; I told him, that was not what I meant ;-)
      - sandGorgon 10y ago
        this is interesting. Were you on GKE or AWS. GKE is obvious, but if you were hosting on AWS.. what was your logging setup ? If you were not using AWS EBS .. what were you doing ?
        hosh 10y ago
        The owner of the company ran out of money before I could add logging, but my plan was to get it out to something like papertrail. I was on AWS. I sidestepped the issue by using AWS RDS (postgresql). I had tried to get the nascent EBS stuff working, but when I realized that I'd have to get a script to check if an EBS volume was formatted with a filesystem before mounting it in K8S, I stopped. This might have been improved by now.
        justinsb 10y ago
        I probably wrote that support (or at least maintain it), and you shouldn't ever have needed to add a script to format your disk: you declare the filesystem you want and it comes up formatted. If you didn't open an issue before please do so and I'll make double-sure it is now fixed (or point me to the issue if you already opened it!) On the logging front, kube-up comes up with automatic logging via fluentd to an ElasticSearch cluster hosted in k8s itself. You can relatively easily replace that ES cluster with an AWS ES cluster (using a proxy to do the AWS authentication), or you can reconfigure fluentd to run to AWS ES. Or you can set pretty easily set up something yourself using daemonsets if you'd rather use something like splunk, but I don't know if anyone has shared a config for this! A big shortcoming of the current fluentd/ES setup is that it also predates PetSets, and so it still doesn't use persistent storage in kube-up. I'm trying to fix this in time for 1.4 though! If you don't know about it, the sig-aws channel on the kubernetes slack is where the AWS folk tend to hang out and work through these snafus together - come join us :-)

brianwawok 10y ago

Heard this elsewhere. Why do people use Docker swarm with Kubernetes around? Does it require less setup or something?

kozikow 10y ago
I see two reasons for this: - If you don't use google container engine (hosted kubernetes, also known as GKE), kubernetes have a reputation of being hard to set up. I am running on GKE, so I can't comment too much. - It's hard to see how all of the orchestration/docker/etc. things play together when entering the area. I expect many people hear "docker" and want to just try "docker", not being aware that there exist alternatives for some parts, some parts are more reliable that others, etc. E.g. the article we are discussing seem to be doing this.
- tazjin 10y ago
  Kubernetes definitely runs best on GKE, but projects like kops (https://github.com/kubernetes/kops https://github.com/kubernetes/kops) are making setups on AWS etc. a lot easier. You still have to deal with the other drawbacks of those platforms (slow networks, disks etc.) but that's not really a k8s issue.
- rodgerd 10y ago
  > kubernetes have a reputation of being hard to set up Yeah, but there's no shortage of things wrapping Kuber (e.g. OpenShift).
jjn2009 10y ago
You can have a swarm with only a handful of commands now, with TLS (for docker api). Eventually stable docker host to docker host container networking will have encryption out of the box too. The number of steps for a production docker setup keeps getting shorter and shorter, kubernetes seems to be now focused on this aspect, they might catch up. This is all of course ignoring the IaaS solutions and obviously for the reasons stated in the article, swarm doesn't see those benefits with 1.12 because major features do not work yet which are supposedly stable.

chx 10y ago

> My cluster is small - I have <100 machines at peak Let me ask: WTF are people are doing that <100 machines is a "small cluster"? I ran a Top 100 (as measured by Quantcast) website with 23 machines and that included kit and caboodle -- Dev, Staging and Production environments. And quite some of that were just for HA purposes not because we needed that much... Stackexchange also runs about two dozen servers. Yes, yes, Google, Facebook runs datacenters but there's a power law kind of distribution here and the iron needs are falling very, very fast as you move from say Top 1 website to Top 30.

kozikow 10y ago
I have a processing pipeline that does crawling, indexing and some analytics on the crawled data - I am building a vertical-focused search engine focused on catering for some specific kinds of companies. Usually my cluster is very small, unless the pipeline is in progress. At this very moment I am not doing any processing, so I only have 3 machines up.
brown9-2 10y ago
Not all computing use cases are running a website.
necubi 10y ago
The number of machines you need to run a service is not really a linear function of your traffic. If you have a mostly static website that can be heavily cached/cdn'd, you can easily scale to thousands of requests a second with a small server footprint. I expect that's true of many of the top 100 sites as measured by visitors (like Quantcast does). But if you need to store a lot of data, or need to look up data with very low latency, or do CPU-intensive work for every request, you will end up with a lot more servers. (The other thing to consider is that SaSS companies can easily deal with more traffic than even the largest web sites, because they tend to aggregate traffic from many websites; Quantcast, for example, where I used to work, got hundreds of thousands of requests per second to its measurement endpoint.)
- chx 10y ago
  Note: the site I mentioned did hit the database quite a few times for each page. It was a nice challenge.