4 ms·

I am a broken record lately on here... Nomad is really awesome for avoiding configuration hell and having to manage multiple services for container orchestratio

I am a broken record lately on here... Nomad is really awesome for avoiding configuration hell and having to manage multiple services for container orchestration. It's a single binary and very very easy to setup and run. I actually prefer it to Swarm but YMMV. It uses Consul under the hood and has so far been bulletproof. (They all have their drawbacks / idiosyncrasies). https://www.nomadproject.io/ https://www.nomadproject.io/

gkze 10y ago
just curious - does it provide rolling updates to the jobs that you're running currently?
- leetrout 10y ago
  Nomad can do it but I've not used that feature.
andrenth 10y ago
The one feature that to me seems to be essential but appears to be missing from all these container orchestrators is the ability to tie a remote volume (Ceph/Gluster/Lustre/etc) to a container so that if a container is scheduled to run on a certain node, the volume will automatically be mounted on the same node. It seems from the mailing list that at least Nomad will have that at some point, but I have not seen much talk about it from Kubernetes or Docker Swarm.
- gkze 10y ago
  we tried using EFS for shared storage but quickly depleted our I/O bursting credits and our throughput dropped to a grinding halt, because our app's worklaod is both disk write and read intensive. No solutions yet
  - leetrout 10y ago
    Unrelated, just some color commentary, I put my first request that took over 24 hours to be approved... 150k+ provisioned iops. Seems they don't like doing that.
  - objectivefs 10y ago
    If you need more performance than EFS for your shared filesystem storage, you could give our ObjectiveFS (https://objectivefs.com https://objectivefs.com) a try. We see significantly higher read/write performance, especially for smaller files.
- tristanz 10y ago
  Kubernetes supports this via PersistentVolumes. Supported types include: GCEPersistentDisk AWSElasticBlockStore AzureFile FC (Fibre Channel) NFS iSCSI RBD (Ceph Block Device) CephFS Cinder (OpenStack block storage) Glusterfs VsphereVolume HostPath (single node testing only – local storage is not supported in any way and WILL NOT WORK in a multi-node cluster) http://kubernetes.io/docs/user-guide/persistent-volumes/ http://kubernetes.io/docs/user-guide/persistent-volumes/
  - andrenth 10y ago
    Interesting, thanks! To be honest I haven't looked too much into Kubernetes so far because of the emphasis in cloud deployment, while my interest is in setting it up in bare metal. This feature might make me give it a try though!
    - colemickens 10y ago
      There are plenty of people running Kubernetes on bare-metal. There are a number of resources out there for PXE-booting a Kubernetes cluster on bare-metal. Might be time to take a closer look ;).
      - andrenth 10y ago
        Yes, but the instructions that I've found to do so are based on installation via shell scripts, which I'm not comfortable with. I'd like to understand it well before running it even in a test setup.
        colemickens 10y ago
        The top result for "PXE Boot Kubernetes" is an extremely comprehensive step-by-step guide for it from CoreOS. I didn't see any magic shell scripts. The second top link is Ansible with links the constituent Ansible playbook. If you want insight into how the pieces fit together, the learnings from Kelsey Hightower's Kubernetes the Hard Way will certainly still map to bare-metal environments as well.
- leetrout 10y ago
  True. I didn't know that was even being discussed. I'd never really thought about the container host dynamically pulling in a volume as a dependency for a container. The best I've been able to do is use AWS EFS on my container hosts so that my ECS tasks with volume mounts find the same stuff everywhere.
- nzoschke 10y ago
  EFS on AWS is an interesting approach. It's a NFS mount that every instance and every container can share. I wouldn't run a database off it but it's been great for simple file synchronization across containers and container restarts.
gkze 10y ago
The point I'm trying to make is that these clustering capabilities are directly inside docker, so you don't have to run anything else but Docker. That is a huge win for us

gkze 10y ago

just curious - does it provide rolling updates to the jobs that you're running currently?

leetrout 10y ago
Nomad can do it but I've not used that feature.

andrenth 10y ago

The one feature that to me seems to be essential but appears to be missing from all these container orchestrators is the ability to tie a remote volume (Ceph/Gluster/Lustre/etc) to a container so that if a container is scheduled to run on a certain node, the volume will automatically be mounted on the same node. It seems from the mailing list that at least Nomad will have that at some point, but I have not seen much talk about it from Kubernetes or Docker Swarm.

gkze 10y ago
we tried using EFS for shared storage but quickly depleted our I/O bursting credits and our throughput dropped to a grinding halt, because our app's worklaod is both disk write and read intensive. No solutions yet
- leetrout 10y ago
  Unrelated, just some color commentary, I put my first request that took over 24 hours to be approved... 150k+ provisioned iops. Seems they don't like doing that.
- objectivefs 10y ago
  If you need more performance than EFS for your shared filesystem storage, you could give our ObjectiveFS (https://objectivefs.com https://objectivefs.com) a try. We see significantly higher read/write performance, especially for smaller files.
tristanz 10y ago
Kubernetes supports this via PersistentVolumes. Supported types include: GCEPersistentDisk AWSElasticBlockStore AzureFile FC (Fibre Channel) NFS iSCSI RBD (Ceph Block Device) CephFS Cinder (OpenStack block storage) Glusterfs VsphereVolume HostPath (single node testing only – local storage is not supported in any way and WILL NOT WORK in a multi-node cluster) http://kubernetes.io/docs/user-guide/persistent-volumes/ http://kubernetes.io/docs/user-guide/persistent-volumes/
- andrenth 10y ago
  Interesting, thanks! To be honest I haven't looked too much into Kubernetes so far because of the emphasis in cloud deployment, while my interest is in setting it up in bare metal. This feature might make me give it a try though!
  - colemickens 10y ago
    There are plenty of people running Kubernetes on bare-metal. There are a number of resources out there for PXE-booting a Kubernetes cluster on bare-metal. Might be time to take a closer look ;).
    - andrenth 10y ago
      Yes, but the instructions that I've found to do so are based on installation via shell scripts, which I'm not comfortable with. I'd like to understand it well before running it even in a test setup.
      - colemickens 10y ago
        The top result for "PXE Boot Kubernetes" is an extremely comprehensive step-by-step guide for it from CoreOS. I didn't see any magic shell scripts. The second top link is Ansible with links the constituent Ansible playbook. If you want insight into how the pieces fit together, the learnings from Kelsey Hightower's Kubernetes the Hard Way will certainly still map to bare-metal environments as well.
leetrout 10y ago
True. I didn't know that was even being discussed. I'd never really thought about the container host dynamically pulling in a volume as a dependency for a container. The best I've been able to do is use AWS EFS on my container hosts so that my ECS tasks with volume mounts find the same stuff everywhere.
nzoschke 10y ago
EFS on AWS is an interesting approach. It's a NFS mount that every instance and every container can share. I wouldn't run a database off it but it's been great for simple file synchronization across containers and container restarts.

gkze 10y ago

The point I'm trying to make is that these clustering capabilities are directly inside docker, so you don't have to run anything else but Docker. That is a huge win for us