4 ms·
"Lightweight Kubernetes" and then a graph involving 2 different nodes with 10+ services running on them. Nomad seems to be a truly "lightweight Kubernetes" and
by diggan 3y ago
"Lightweight Kubernetes" and then a graph involving 2 different nodes with 10+ services running on them.
Nomad seems to be a truly "lightweight Kubernetes" and for the small amount of time I've been using it, it seems to do it's job really well and it's easy to understand all the moving pieces without spending countless of hours reading docs and source code.
Although, it's hard to recommend Nomad for future uses as it sadly stopped being FOSS :/
- q3k 3y agoI mean, 2 nodes and 10 services is very light for what Kubernetes is designed to scale to.
- worksonmine 3y agoThe amount of services running is not an indication of how lightweight the underlying tech is. K3S is for orchestration across nodes, and by design you can't run services on the master for security reasons unless you manually change it. Are you trying to say that 10 services and 2 nodes would be less on Nomad?
- tarruda 3y ago> K3S is for orchestration across nodes, and by design you can't run services on the master for security reasons unless you manually change it. That has not been my experience with k3s. Unlike normal distributions, k3s servers do allow workload to run on them unless you manually disable. From https://docs.k3s.io/datastore/ha-embedded https://docs.k3s.io/datastore/ha-embedded : " An HA K3s cluster with embedded etcd is composed of: - Three or more server nodes that will serve the Kubernetes API and run other control plane services, as well as host the embedded etcd datastore. - Optional: Zero or more agent nodes that are designated to run your apps and services " So you can have a full HA k3s cluster with only 3 server nodes and 0 agent(worker) nodes.
- sleepybrett 3y agoThis is not true of k8s in general, maybe certain on prem 'distributions' enforce this, i'm not sure. Sometimes kubernetes admins/operators who are running their own control planes and etcd will run etcd on non k8s controlled nodes, or they will run it on the same nodes that run their control plane services (scheduler, apiserver, controller manager, etc). Often those nodes will be tainted in such a way that they don't get overloaded by a noisy neighbor 'customer pod', but that's in no way 'by design'.. perhaps by convention or best practice.
- deleted 3y ago[deleted]
- worksonmine 3y agoYeah, design was probably a poor choice of words, s/design/default/.
- worksonmine 3y agoYes, you can ignore the warnings and change the defaults, like I said? > unless you manually change it I ran into this myself just the other day.
- proxysna 3y agoNomad is an alternative to k8s, but it is not really a "lightweight k8s". Nomad and k8s are way to different to call them versions of eachother. And the whole license thing. Nothing changed for end users.
- _w1tm 3y ago> Nomad seems to be a truly "lightweight Kubernetes" and for the small amount of time I've been using it, it seems to do it's job really well and it's easy to understand all the moving pieces without spending countless of hours reading docs and source code. Does Nomad expose an API that you can extend with controllers running inside the cluster? Because Kubernetes without operators is not Kubernetes.
- proxysna 3y agoyes, you can do that. Here's an example. https://andydote.co.uk/2021/11/22/nomad-operator-pattern/ https://andydote.co.uk/2021/11/22/nomad-operator-pattern/
- moondev 3y ago> While Nomad doesn’t support the idea of Custom Resource Definitions, we can achieve an operator by utilising a regular Nomad job and the nomad HTTP API. Kubernetes operator is a combination of a CRD reconciled by a controller. Without the CRD it's simply a controller. I do like the linked post, it's well written and a valiant effort, but not close to comparable.
- illamint 3y ago> Because Kubernetes without operators is not Kubernetes. Alright, so, what is it, then? I've been unfortunate to work at firms that generally have a minimal level of competency with Kubernetes, but across several billion dollars worth of firms, not a one has used operators in any capacity, but they leverage Kubernetes substantially. Help me understand the gap, would you?
- figmert 3y agoNomad also doesn't have nearly half of the features that Kubernetes does. Need service discovery? Set up a Consul cluster. Need secret management? Install vault. Need vault enterprise? Install a separate Consul cluster! This was a few years ago, maybe it's changed? I dunno. Anyway, lightweight here means that whole bunch of external dependencies have been ripped out. E.g. AWS/GCP/Azure and I believe other things too.
- proxysna 3y agoYou can have sd and secrets with just nomad now, it is perfect for small projects in that way. K8s also can do secrets! base64 encoded secrets! But in case you need a more secure solution. Please pull in a Vault helm chart or mess around with sops or whatever. Want to scale out? See how long it will take you until you will have to move your etcd to dedicated hw/vm's. And let me tell you - setting up external etcd in prod sucks. Also maintaining mission control, because there is a bunch of ways to do things and devs love using things that they are not supposed to touch. Nomad approach is modular this is why it works. Need proper multi dc SD? please run a consul cluster (single binary + single config file)x3. Need mesh networking? same. Need secrets? Vault (single binary + single config file)x3. Pki? same. Or maybe you don't need it? Then you have an option to just not use it. One thing that i really don't like about it that there is not a lot of docs describing all the ways these tools can interact with each other.
- ownagefool 3y agoK8s supports encryption at rest, ACLs and audit logs. What it actually lacks is versioning, and a dynamic secrets engine, though you could build that with an operator, and things like the postgres-operator do. You also don't get the whole sealing thing, but I'd argue that's more annoying than useful.
- sleepybrett 3y ago> You also don't get the whole sealing thing, but I'd argue that's more annoying than useful. A-fucking-men.
- mfer 3y agoYou can run k3s with a single node. In that case it uses sqlite instead of etcd which is great for a smaller resource footprint. If you're comparing k8s distros, you'll be hard pressed to find a setup that uses fewer system resources.
- nullify88 3y agoIs it actually smaller though? For those non etcd data stores, Kine is often the component doing an etcd to database translation. Im my experience using it generated a load that was greater than a one node etcd instance.
- WJW 3y agoNomad is a lightweight container orchestrator, not a lightweight Kubernetes. The whole point is to keep syntax, terminology and config options etc as similar as possible between k8s and k3s.
- doctorpangloss 3y ago> "Lightweight Kubernetes" and then a graph involving 2 different nodes with 10+ services running on them. Among its many strengths, core services for running Kubernetes running inside Kubernetes itself is one of its greatest. > it's easy to understand all the moving pieces It's legit to leverage your pre-existing knowledge. Nomad + Linux Pets + Other Hashichorp Pets works well. There are many ways to run an application. Kubernetes + Machine Cattle, in my experience having started from zero, is superior to any other application deployment paradigm I've ever used.
- chrisweekly 3y ago"Machine Cattle"?
- sleepybrett 3y agoTreat your machines like cattle, not pets. When I used to run kubernetes in the pre-'kube as a service' days. We had two autoscaling groups, one which had 3 or 5 nodes in it to run the control plane nodes and an open eneded one for workers (which were managed by the cluster autoscaler). Each ASG was set up in such a way that the nodes would come up and join the cluster fully automated and they never needed to be shelled into for anything but troubleshooting the weirdest failures (it was early days, the kubelet and docker had some extra fun bugs).
- chrisweekly 3y agoOk sure, I'm very familiar w the cattle vs pets analogy / meme -- but the (capitalized!) "Machine Cattle" made it seem like a reference to a particular software tool or product.
- quickthrower2 3y agoProbably a reference to the pets v cattle analogy http://cloudscaling.com/blog/cloud-computing/the-history-of-pets-vs-cattle/ http://cloudscaling.com/blog/cloud-computing/the-history-of-...
- OhSoHumble 3y agoI've commented on this before in a different k8s thread (one about a k8s outage) but something that bears repeating is that the entire job market is Kubernetes. My personal experience is that it is very, very hard to find a job right now if your professional experience is primarily non-k8s orchestration systems. Most job positions out there require deep Kubernetes knowledge as well as hands-on experience with different supporting products like ArgoCD and Flux. I chose Nomad for a large initiative at my current employer and it is honestly pretty amazing given the use case but I regret choosing it because I feel like I'm unhirable now given that every devops/SRE/platform engineering position open on the market (each one with hundreds of applicants) is heavily Kubernetes focused.
- qudat 3y agoPlease forgive the slight tangent but Aptible (the most successful PaaS you never heard of) is hiring and we don’t use Kubernetes https://www.aptible.com/culture-hub/careers https://www.aptible.com/culture-hub/careers
- umvi 3y agoA lot of smaller SaaS companies avoid k8s like the plague and only start using it once they scale to a certain size. K8s might be ubiquitous past a certain scale, but there are many jobs operating below the threshold for which the complexity of your system with k8s outweighs the complexity of your system without k8s.
- hobofan 3y agoIf you find yourself on the Google cloud for some reason, GKE Autopilot is such a great hands-off solution that most alternative solutions will probably be more complex. I only recently had to use AWS EKS, and am pretty sure that many people dislike k8s because that's the main incarnation of it they experience (how is there no pre-installed ingress, and even the setup for their own load balancer is a 10+ step manual process?).
- hhh 3y ago
- themgt 3y agoNomad seems to be a truly "lightweight Kubernetes" k3s requirements: 1 node, 512MB RAM, 1 CPU core Nomad requirements: "Nomad servers may need to be run on large machine instances. We suggest having between 4-8+ cores, 16-32 GB+ of memory, 40-80 GB+ of fast disk and significant network bandwidth." https://docs.k3s.io/installation/requirements https://docs.k3s.io/installation/requirements https://developer.hashicorp.com/nomad/docs/install/production/requirements https://developer.hashicorp.com/nomad/docs/install/productio...
- rickette 3y agoHashicorp is playing safe here, Nomad can do with far far less resources.
- alanwreath 3y agoThat can be said of many systems, does hashicorp document it though? I’m guessing they must, otherwise that would be the first thing you hear on the other side of a support call.
- sleepybrett 3y agoalso with hashicorps new licensing regime...
- pojzon 3y agoThat does not impact 99% of ppl.. The licence change impacts only companies like Digger or Spacelift that build their platform abusing TF oss offering.
- emptysongglass 3y agoThat's a bad take and you know it. HashiCorp couldn't make money because their enterprise offering was slim on features and overpriced. Rather than compete, they're playing protect-the-castle with some aggressive plays to keep their company afloat. They're bad stewards of open source (always have been) and are now getting their just desserts.
- alanwreath 3y agoI think the tagline can be a bit misleading, you can run k3s on devices traditionally considered IoT (like the raspberry pi) but it will run on big heavy x86 servers too.
- cortesoft 3y agoK3s uses the exact same API as Kubernetes. Nomad does not. The idea is to more easily get going with Kubernetes, not to provide an alternative. Nomad and K3s serve a completely different use case.
- nonameiguess 3y agoIf Nomad provides workload scheduling, a runtime, workload networking, service networking, and some sort of overall orchestrator/controller plus an API, then it has these same services. Running them in one process instead of ten doesn't make it more lightweight. You're free to run either on one node, but other than for toy purposes like demos and learning, why use a cluster compute system when you're not actually running a cluster?