5 ms·

I am looking into this now. Basically it’s either this or the super hard Kubernetes right? I find it hard to believe that we don’t have a middle ground?

by kfk 6y ago

dblooman 6y ago
There is AWS ECS as well
scrappyjoe 6y ago
K3s is pretty approachable.
tyingq 6y ago
Mesos and K3S are two other choices in that space.
sandGorgon 6y ago
There is the k3s distro of kubernetes
auspex 6y ago
Kubernetes really isn't as difficult as you might think. Early on it was "hard" because most people were learning containers as well as Kubernetes. Also how that we have managed services like EKS, GKE, AKS it's really straight forward to learn. I would suggest start with a GKE cluster and deploy nginx.
- kfk 6y ago
  There are many managed services, which is great, but how does this work when your storage is somewhere else? For instance I might decide to put the storage (postgres and minio) on a cheap Hetzner dedicated server, how would this work with EKS? I am especially trying to understand doing analytics and ML with dask.distributed. I think running compute on managed Kubernetes with storage inside another server could (hybrid?) should be ok, but I haven't figured this part yet.
  - auspex 6y ago
    You can use a persistent volume claim in multiple containers. I would suggest using a Google Disk or EBS volume (Depending on if you are using GKE or EKS. They can be mounted to multiple containers at a time (your consumers). Portable and durable. Keep this volume updated with your data. Maybe a sync process from your other server.
  - gas9S9zw3P9c 6y ago
    Yeah, hybrid cloud and multi-cluster is still not a first class citizen. Kubernetes wasn't originally meant for this, and while I think there exist some solutions, they are quite early stage. Of course nothing prevents you to put your data somewhere else and load it, you just won't be able to manage Kubernetes resources across clusters. But you wouldn't be able to do this with an approach like Docker Swarm either. It has nothing to do with managed services, it's just that Kubernetes clusters, managed or not, are not meant to span regions.
  - dilyevsky 6y ago
    You’ll get charged an arm and a leg for egress (from aws) so have to pretty much commit to aws built in storage (either ebs or efs mounted volumes) or local machine storage (eg nvme drives)
  - justRafi 6y ago
    You can use headless services[1] to incorporate external endpoints. There's also various CoreDNS plugins[2] that do wonders. [1]: https://kubernetes.io/docs/concepts/services-networking/service/#headless-services https://kubernetes.io/docs/concepts/services-networking/serv... [2]: https://coredns.io/plugins/ https://coredns.io/plugins/
jordanbeiber 6y ago
Nomad + Consul. Used it for years in prod - just awesome. I’ve also used k8s extensively and nomad/consul is more ”open” and less opinionated. If you also manage some form of legacy infrastructure and VMs, consul is hard to ignore. Two go-binaries - you’re up in minutes for a test drive.
- kfk 6y ago
  I looked into Nomad, did not understand much. I am pretty OK with docker and docker-compose, me and my team use docker to develop data workflows in python locally. So out of Docker I am much less comfortable. For instance, would JupyterHub for Kubernetes run on Nomad? https://zero-to-jupyterhub.readthedocs.io/en/latest/customizing/user-storage.html https://zero-to-jupyterhub.readthedocs.io/en/latest/customiz...
  - jordanbeiber 6y ago
    It’s not k8s api compatible - so no, not straight off. You can use nomad to schedule pretty much any workload you’d like, including docker containers. You could probably schedule a raw command that is “docker-compose”.
- heipei 6y ago
  sigh I've almost given up nudging folks on here to give Nomad a spin. It's so easy to deploy and play with, heck you can run a client and server with a single go binary on your local machine. If you compare that to all the hundreds of different (and soon-to-be-deprecated) installation instructions for k8s, k8s feels like a joke at best, or an evil plot at worst to get everyone to use hosted k8s. Honestly, people use hosted k8s because you need a full-time person/team to run it, so in the end, even though k8s is Open Source, Google (and AWS) still win... Nevermind that Nomad is fast, easy to grasp, very concise and can run non-Docker workloads. Oh well...
  - jordanbeiber 6y ago
    The “evil plot” is what I gather. K8s have it’s bright spots, but it can’t touch the flexibility of nomad & consul. Now that nomad/consul have autoscaling and csi as well as native envoy integration the gap, with regards to container workloads, have shrunk quite a bit!
gas9S9zw3P9c 6y ago
I don't understand why people think Kubernetes is "hard" - I find it quite straightforward and the documentation is excellent. I've been running my side projects on a single k8s clusters and rarely have issues. Yes, there is a learning curve. It's not a Heroku where you enter one command and it just works. You'll need to spend a few days to become familiar with the basics concepts. And it may take weeks grasp the advanced concepts. But it's less complexity than learning something like a new programming language. You'll probably get back the time you invested. Things like deploying production-ready application with a single command using e.g. Helm is quite powerful and can save you money over using managed services.
- ernststavrob 6y ago
  Kubernetes is "hard" when you're not running on the cloud. The solutions for persistent storage and networking are a mess when you're running on self-hosted infrastructure.
  - gas9S9zw3P9c 6y ago
    Fair enough, I was referring to "using" Kubernetes as opposed to managing your own infrastructure. I can imagine how running it on self-hosted servers outside of a popular cloud provider can be hard.
  - elsonrodriguez 6y ago
    This is a trap. The best way to do storage on K8s is not to do it. Especially on-premises. The networking used to be simple, you could just set up a few static routes and manually assign blocks of CIDRs to nodes and be done. I'm sure there there are some newer networking API components that obfuscated the whole thing in the name of "simplicity" because nobody understands networking anymore.
woudsma 6y ago
This was also my question/frustration. Kubernetes is not so easy to learn, especially if your focus is on other programming projects and you just want a easy and scalable personal server setup. I've created a tool called 'Swarmlet' over the past few weeks, which tries to mimic Dokku, so it combines git and Docker Swarm mode for easy app deployments. With some services included like Traefik for routing and automatic SSL using Let's Encrypt, with Consul as a distributed secrets store. https://github.com/swarmlet/swarmlet https://github.com/swarmlet/swarmlet Definitely not production-ready, and quite some things to do, but it's a nice POC which actually works for me.

I am looking into this now. Basically it’s either this or the super hard Kubernetes right? I find it hard to believe that we don’t have a middle ground?

by kfk 6y ago

I am looking into this now. Basically it’s either this or the super hard Kubernetes right? I find it hard to believe that we don’t have a middle ground?

dblooman 6y ago

There is AWS ECS as well

scrappyjoe 6y ago

K3s is pretty approachable.

tyingq 6y ago

Mesos and K3S are two other choices in that space.

sandGorgon 6y ago

There is the k3s distro of kubernetes

auspex 6y ago

Kubernetes really isn't as difficult as you might think. Early on it was "hard" because most people were learning containers as well as Kubernetes. Also how that we have managed services like EKS, GKE, AKS it's really straight forward to learn. I would suggest start with a GKE cluster and deploy nginx.

kfk 6y ago
There are many managed services, which is great, but how does this work when your storage is somewhere else? For instance I might decide to put the storage (postgres and minio) on a cheap Hetzner dedicated server, how would this work with EKS? I am especially trying to understand doing analytics and ML with dask.distributed. I think running compute on managed Kubernetes with storage inside another server could (hybrid?) should be ok, but I haven't figured this part yet.
- auspex 6y ago
  You can use a persistent volume claim in multiple containers. I would suggest using a Google Disk or EBS volume (Depending on if you are using GKE or EKS. They can be mounted to multiple containers at a time (your consumers). Portable and durable. Keep this volume updated with your data. Maybe a sync process from your other server.
- gas9S9zw3P9c 6y ago
  Yeah, hybrid cloud and multi-cluster is still not a first class citizen. Kubernetes wasn't originally meant for this, and while I think there exist some solutions, they are quite early stage. Of course nothing prevents you to put your data somewhere else and load it, you just won't be able to manage Kubernetes resources across clusters. But you wouldn't be able to do this with an approach like Docker Swarm either. It has nothing to do with managed services, it's just that Kubernetes clusters, managed or not, are not meant to span regions.
- dilyevsky 6y ago
  You’ll get charged an arm and a leg for egress (from aws) so have to pretty much commit to aws built in storage (either ebs or efs mounted volumes) or local machine storage (eg nvme drives)
- justRafi 6y ago
  You can use headless services[1] to incorporate external endpoints. There's also various CoreDNS plugins[2] that do wonders. [1]: https://kubernetes.io/docs/concepts/services-networking/service/#headless-services https://kubernetes.io/docs/concepts/services-networking/serv... [2]: https://coredns.io/plugins/ https://coredns.io/plugins/

jordanbeiber 6y ago

Nomad + Consul. Used it for years in prod - just awesome. I’ve also used k8s extensively and nomad/consul is more ”open” and less opinionated. If you also manage some form of legacy infrastructure and VMs, consul is hard to ignore. Two go-binaries - you’re up in minutes for a test drive.

kfk 6y ago
I looked into Nomad, did not understand much. I am pretty OK with docker and docker-compose, me and my team use docker to develop data workflows in python locally. So out of Docker I am much less comfortable. For instance, would JupyterHub for Kubernetes run on Nomad? https://zero-to-jupyterhub.readthedocs.io/en/latest/customizing/user-storage.html https://zero-to-jupyterhub.readthedocs.io/en/latest/customiz...
- jordanbeiber 6y ago
  It’s not k8s api compatible - so no, not straight off. You can use nomad to schedule pretty much any workload you’d like, including docker containers. You could probably schedule a raw command that is “docker-compose”.
heipei 6y ago
sigh I've almost given up nudging folks on here to give Nomad a spin. It's so easy to deploy and play with, heck you can run a client and server with a single go binary on your local machine. If you compare that to all the hundreds of different (and soon-to-be-deprecated) installation instructions for k8s, k8s feels like a joke at best, or an evil plot at worst to get everyone to use hosted k8s. Honestly, people use hosted k8s because you need a full-time person/team to run it, so in the end, even though k8s is Open Source, Google (and AWS) still win... Nevermind that Nomad is fast, easy to grasp, very concise and can run non-Docker workloads. Oh well...
- jordanbeiber 6y ago
  The “evil plot” is what I gather. K8s have it’s bright spots, but it can’t touch the flexibility of nomad & consul. Now that nomad/consul have autoscaling and csi as well as native envoy integration the gap, with regards to container workloads, have shrunk quite a bit!

gas9S9zw3P9c 6y ago

I don't understand why people think Kubernetes is "hard" - I find it quite straightforward and the documentation is excellent. I've been running my side projects on a single k8s clusters and rarely have issues. Yes, there is a learning curve. It's not a Heroku where you enter one command and it just works. You'll need to spend a few days to become familiar with the basics concepts. And it may take weeks grasp the advanced concepts. But it's less complexity than learning something like a new programming language. You'll probably get back the time you invested. Things like deploying production-ready application with a single command using e.g. Helm is quite powerful and can save you money over using managed services.

ernststavrob 6y ago
Kubernetes is "hard" when you're not running on the cloud. The solutions for persistent storage and networking are a mess when you're running on self-hosted infrastructure.
- gas9S9zw3P9c 6y ago
  Fair enough, I was referring to "using" Kubernetes as opposed to managing your own infrastructure. I can imagine how running it on self-hosted servers outside of a popular cloud provider can be hard.
- elsonrodriguez 6y ago
  This is a trap. The best way to do storage on K8s is not to do it. Especially on-premises. The networking used to be simple, you could just set up a few static routes and manually assign blocks of CIDRs to nodes and be done. I'm sure there there are some newer networking API components that obfuscated the whole thing in the name of "simplicity" because nobody understands networking anymore.

woudsma 6y ago

This was also my question/frustration. Kubernetes is not so easy to learn, especially if your focus is on other programming projects and you just want a easy and scalable personal server setup. I've created a tool called 'Swarmlet' over the past few weeks, which tries to mimic Dokku, so it combines git and Docker Swarm mode for easy app deployments. With some services included like Traefik for routing and automatic SSL using Let's Encrypt, with Consul as a distributed secrets store. https://github.com/swarmlet/swarmlet https://github.com/swarmlet/swarmlet Definitely not production-ready, and quite some things to do, but it's a nice POC which actually works for me.