5 ms·

I will try and answer networking question. At scale a single host can be running may be 20 containers and port collision becomes a real problem. So imagine if

by gnufied 10y ago

I will try and answer networking question. At scale a single host can be running may be 20 containers and port collision becomes a real problem. So imagine if a container opened a port directly on host -we have to be careful that they don't step on each other toes. Even if all containers used some sort of contract about which port they are going to use - there are all sort of corner cases waiting to happen such as ephemeral ports(the port you bind to when you connect externally) taking over a port taken by real server app. I have seen two approaches being used to solve this problem: 1. Using Smartstack (http://nerds.airbnb.com/smartstack-service-discovery-cloud/ http://nerds.airbnb.com/smartstack-service-discovery-cloud/) the applications running inside container can run on any port but the port on which they are externally available is decided by orchestration service. Typically, no one talks to application inside container directly but they go through the haproxy configured on localhost. The advantage is - smartstack can remove a service if it is failing healthcheck etc. 2. The kubernetes/openshift approach of Software defined networking(https://github.com/coreos/flannel https://github.com/coreos/flannel). Although they also integrate with load balancers, so that is not the only way. I know if someone is just getting started with containers, it seems bit overwhelming to digest all this. But having worked in some large companies which are using containers at scale, it kinda makes sense.

justincormack 10y ago
The default setup of Docker does not make any assumptions about the host setup, so it assumes it might only have one IP address, so there is only one set of ports. It is perfectly ok if you have lots of IPs to put routed IPs on the `docker0` bridge, and never use port publishing at all, or to use some of the other optional setups, such as the new macvlan and ipvlan setups https://github.com/docker/docker/blob/master/experimental/vlan-networks.md https://github.com/docker/docker/blob/master/experimental/vl... which are the kind of production setups you may want if you run your own networking. But Docker cannot assume anything about the network setup in the default configuration, hence the use of published ports, which is kind of inconvenient but always works in any environment.
philsnow 10y ago
> Using smartstack the applications running inside container can run on any port but the port on which they are externally available is decided by orchestration service AFAIK this is not the case: with smartstack you configure your application to listen on some port, and you configure nerve to register <this machine's ip>:<the app's port> under <service name> in zk/etcd/whatever. You have to do both of these things; smartstack itself doesn't configure your app for you or make sure that nerve and your app agree on the port. I'm not aware of any such efforts to make smartstack aware of orchestration yet, but I haven't gone looking recently. It'd be entirely possible to make the app and nerve aware of network orchestration: have the orchestration layer 1. pick a port, 2. tell the app about it, and 3. configure nerve accordingly. In smartstack, each instance of <service name> can have different port numbers, so they could all be arbitrary high-numbered ports. edit: You could also just have your orchestration layer inform the service discovery layer itself (in the case of smartstack, by writing the app's ip:port to zk/etcd/whatever). nerve contains some local healthchecking (sanity checking that the app is up before registering it) and maybe a few other things, however, so I think if you're already using / considering smartstack it would make sense to keep using nerve instead of having orchestration do it.

by gnufied 10y ago

justincormack 10y ago

The default setup of Docker does not make any assumptions about the host setup, so it assumes it might only have one IP address, so there is only one set of ports. It is perfectly ok if you have lots of IPs to put routed IPs on the `docker0` bridge, and never use port publishing at all, or to use some of the other optional setups, such as the new macvlan and ipvlan setups https://github.com/docker/docker/blob/master/experimental/vlan-networks.md https://github.com/docker/docker/blob/master/experimental/vl... which are the kind of production setups you may want if you run your own networking. But Docker cannot assume anything about the network setup in the default configuration, hence the use of published ports, which is kind of inconvenient but always works in any environment.

philsnow 10y ago

> Using smartstack the applications running inside container can run on any port but the port on which they are externally available is decided by orchestration service AFAIK this is not the case: with smartstack you configure your application to listen on some port, and you configure nerve to register <this machine's ip>:<the app's port> under <service name> in zk/etcd/whatever. You have to do both of these things; smartstack itself doesn't configure your app for you or make sure that nerve and your app agree on the port. I'm not aware of any such efforts to make smartstack aware of orchestration yet, but I haven't gone looking recently. It'd be entirely possible to make the app and nerve aware of network orchestration: have the orchestration layer 1. pick a port, 2. tell the app about it, and 3. configure nerve accordingly. In smartstack, each instance of <service name> can have different port numbers, so they could all be arbitrary high-numbered ports. edit: You could also just have your orchestration layer inform the service discovery layer itself (in the case of smartstack, by writing the app's ip:port to zk/etcd/whatever). nerve contains some local healthchecking (sanity checking that the app is up before registering it) and maybe a few other things, however, so I think if you're already using / considering smartstack it would make sense to keep using nerve instead of having orchestration do it.