4 ms·

I haven't played enough with Docker Swarm to run into any of these issues, but Rancher (www.rancher.com) does all of this, and more, very well. I am not affilia

I haven't played enough with Docker Swarm to run into any of these issues, but Rancher (www.rancher.com) does all of this, and more, very well. I am not affiliated with them - just a happy user and contributor of github issues.

sschueller 10y ago
Rancher is very cool but I wish that rancher OS was easier to lockdown. If you run it outside of AWS or Exoscale you don't get a firewall so you have to do that on the hosts.
jondubois 10y ago
I agree, Rancher just ties everything together really nicely and it works awesome with Kubernetes. I'm building a hosted Rancher cluster manager for my OSS project http://baasil.io/ http://baasil.io/.
drdaeman 10y ago
Went with them last week, as it seemed to be one of the most sane options out there (I've tried a lot). Still, has its share of issues. Few ones I've encountered so far: - Their built-in load balancing (HAProxy-based) is nearly impossible to debug. Literally no logging there. - No locality awareness. DNS queries always return all addresses they know about (including those that don't even work - https://github.com/rancher/rancher/issues/5792 https://github.com/rancher/rancher/issues/5792) and I haven't yet found any good way to prioritize containers co-running on the local host to the more distant ones (https://github.com/rancher/rancher/issues/5798 https://github.com/rancher/rancher/issues/5798 - if someone been to this situation and has some ideas, would appreciate any suggestions!). - Storage management was advertised, but can't find anything besides NFS (which is SPOF) and Amazon EFS (which I don't use). There was GlusterFS support, but it seems it was too broken so they had removed it or something like that. If one wants persistent storage, they'd better pin containers to hosts.
- InTheArena 10y ago
  GLuster is a disaster.
  - drdaeman 10y ago
    Haven't used it in non-toy environments, so won't argue with that. My actual issue is, there's effectively no distributed storage support in Rancher/Cattle at this moment, be it GlusterFS or anything else (for all I know, MooseFS worked quite well for us on one project). Just pointing it out, because for some reason I got quite a different impression from the website/docs. Every point of advertisement statements like "Rancher provides a full set of infrastructure services for containers, including networking, storage services, host management, load balancing and more." is to be taken with a huge bag of salt. (And that's by no means unique to Rancher.)
    - InTheArena 10y ago
      I agree on the general comments on storage. The Docker volume system seems to to be the ideal place to do this.
- bboreham 10y ago
  > I haven't yet found any good way to prioritize containers co-running on the local host You may find that RFC3484 helps; it prefers the address with the longest prefix in common with your own address so will tend to pick your own address. And you are probably getting this behaviour already.
  - drdaeman 10y ago
    Thanks for the pointers! Unfortunately, I don't think this applies. I'm certainly not getting this behavior - I wouldn't have even thought of it if I haven't observed higher latencies resulting from (sub)requests chain jumping from node to node back-and-forth, instead of staying within the node's boundaries. The problem is, the network space there is flat, not hierarchical - while I haven't looked at the actual implementation code, I believe container addresses are just randomly chosen from a single big 10.42/16 subnet and I'm unaware if there's a way that I can assign hosts, say, a /20 out of that space (yes, this would've solved things nicely).
    - bboreham 10y ago
      Oh. Right. I happen to work on a different Docker network, which 'chunks' the address space so containers on the same machine are very likely to have contiguous addresses. Hadn't occurred to me theirs doesn't do that.
      - drdaeman 10y ago
        Just curious - what networking/clustering solution you're using? (I'm asking, so the next time I'll have to make a choice between stacks I would be more aware about the finer details.) Thanks!
        bboreham 10y ago
        I work on Weave Net http://github.com/weaveworks/weave http://github.com/weaveworks/weave.

sschueller 10y ago

Rancher is very cool but I wish that rancher OS was easier to lockdown. If you run it outside of AWS or Exoscale you don't get a firewall so you have to do that on the hosts.

jondubois 10y ago

I agree, Rancher just ties everything together really nicely and it works awesome with Kubernetes. I'm building a hosted Rancher cluster manager for my OSS project http://baasil.io/ http://baasil.io/.

drdaeman 10y ago

Went with them last week, as it seemed to be one of the most sane options out there (I've tried a lot). Still, has its share of issues. Few ones I've encountered so far: - Their built-in load balancing (HAProxy-based) is nearly impossible to debug. Literally no logging there. - No locality awareness. DNS queries always return all addresses they know about (including those that don't even work - https://github.com/rancher/rancher/issues/5792 https://github.com/rancher/rancher/issues/5792) and I haven't yet found any good way to prioritize containers co-running on the local host to the more distant ones (https://github.com/rancher/rancher/issues/5798 https://github.com/rancher/rancher/issues/5798 - if someone been to this situation and has some ideas, would appreciate any suggestions!). - Storage management was advertised, but can't find anything besides NFS (which is SPOF) and Amazon EFS (which I don't use). There was GlusterFS support, but it seems it was too broken so they had removed it or something like that. If one wants persistent storage, they'd better pin containers to hosts.

InTheArena 10y ago
GLuster is a disaster.
- drdaeman 10y ago
  Haven't used it in non-toy environments, so won't argue with that. My actual issue is, there's effectively no distributed storage support in Rancher/Cattle at this moment, be it GlusterFS or anything else (for all I know, MooseFS worked quite well for us on one project). Just pointing it out, because for some reason I got quite a different impression from the website/docs. Every point of advertisement statements like "Rancher provides a full set of infrastructure services for containers, including networking, storage services, host management, load balancing and more." is to be taken with a huge bag of salt. (And that's by no means unique to Rancher.)
  - InTheArena 10y ago
    I agree on the general comments on storage. The Docker volume system seems to to be the ideal place to do this.
bboreham 10y ago
> I haven't yet found any good way to prioritize containers co-running on the local host You may find that RFC3484 helps; it prefers the address with the longest prefix in common with your own address so will tend to pick your own address. And you are probably getting this behaviour already.
- drdaeman 10y ago
  Thanks for the pointers! Unfortunately, I don't think this applies. I'm certainly not getting this behavior - I wouldn't have even thought of it if I haven't observed higher latencies resulting from (sub)requests chain jumping from node to node back-and-forth, instead of staying within the node's boundaries. The problem is, the network space there is flat, not hierarchical - while I haven't looked at the actual implementation code, I believe container addresses are just randomly chosen from a single big 10.42/16 subnet and I'm unaware if there's a way that I can assign hosts, say, a /20 out of that space (yes, this would've solved things nicely).
  - bboreham 10y ago
    Oh. Right. I happen to work on a different Docker network, which 'chunks' the address space so containers on the same machine are very likely to have contiguous addresses. Hadn't occurred to me theirs doesn't do that.
    - drdaeman 10y ago
      Just curious - what networking/clustering solution you're using? (I'm asking, so the next time I'll have to make a choice between stacks I would be more aware about the finer details.) Thanks!
      - bboreham 10y ago
        I work on Weave Net http://github.com/weaveworks/weave http://github.com/weaveworks/weave.