30 ms·

K3s – Lightweight Kubernetes

458 points by kadel 8y ago

pictur 8y ago
Do you think kubernetes are necessary for small businesses?
- redmorphium 8y ago
  No, for the same reason that Martin Fowler argues you should start with a monolith first before moving to a microservices architecture.
  - rhizome31 8y ago
    What is currently a good alternative for a small business who needs to distribute containers on a handful of nodes? Docker swarm mode?
    - MediumD 8y ago
      If you need to distribute containers on a handful of nodes, kubernetes is great (and far better than Docker swarm imho). However, most small businesses probably don't need to have a microservice setup and can simply deploy a simple monolith while they focus on getting their business off the ground.
      - rhizome31 8y ago
        Thank you. I was thinking that maybe swarm mode would be simpler to set up and maintain. Just to give more context, our use case is spawning Jupyter notebook in containers using JupyterHub and DockerSpawner so that each data scientist gets a personal ready-to-use environment to work with. With 4 data scientists we're beginning to reach the limits of the largest instance provided by our cloud provider and we're expecting to have more users soon. Actually our current cloud provider has just released a managed k8s service so that might be the way to go.
        MediumD 8y ago
        Gotcha. Yeah, I would much rather use a managed k8s service over deploying and maintaining my own. GCP's GKE has been great for us so far and very reasonably priced.
    - scarejunba 8y ago
      Kube is fine for this, but don't roll your own, use GKE. Perhaps you'd be better served using more managed services, though?
      - rhizome31 8y ago
        What do you mean by more managed services?
        icebraining 8y ago
        I'm guessing Heroku, Google App Engine, AWS Beanstalk and such.
    - djsumdog 8y ago
      A single baremetal server with just docker containers. It's much easier to migrate from just containers running on nodes to an orchestration system (marathon, nomad OR k3s/k8s; there are other options) than to start with the orchestration system.
      - rhizome31 8y ago
        This is the situation we're in. We're beginning to reach the limits of what can be handled on a single instance, mostly regarding memory.
      - noir_lord 8y ago
        In my case it's a single bare metal server with KVM guests and ansible. Docker is on my long term list (maybe) but the current system is fast, stable and requires little intervention so docker would have to offer a massive change not a step change to make it worth it. I always default to things I know until the benefits are clear and it's worked out well so far.
    - peterwwillis 8y ago
      Docker swarm, Consul+Nomad, AWS Fargate, Azure ACS, Google GCP, etc. Even DC/OS+Marathon is probably simpler. If you really think you need K8s, use a managed service provider like GKE, AKS, or a platform manager like Rancher. You need to pay a specialized company to build and operate it for you, or you aren't going to have a fun time. I highly recommend leveraging AWS, Google, Azure, etc services. Especially if you're a small business, spending a little for off the shelf tools that someone else operates and supports is going to help you big time.
- danaur 8y ago
  It wasn't before it existed and isn't now it's convenient though
- whazor 8y ago
  I think maybe eventually with cheaper ARM CPUs.
- sliken 8y ago
  Necessary, no. But with the hardware almost free, a 3 x Raspberry Pi cluster, switch, access pointer, and a small UPS would be a few $100 which allows for some interesting possibilities. Pretty much any business will want inventory tracking, tracking staff hours, monitoring cameras, sensors for detecting flooding/doors left open/freezers dying, running point of sales terminals, keeping the supply chain running smoothly, handling customer loyalty cards, handling returns, etc. An on site cluster can help with much of that, even if there's a hardware failure or the network is out. Question is how will these business find the apps they need that run turn key on some small k8s cluster?
  - vinceguidry 8y ago
    Hardware was never the bottleneck there. The systems could cost a hundred times that and it would still be a rounding error on the capex of even the smallest "real" business. The obstacle to using custom hardware and software in local biz is and has always been, the fact that by relying on it, the one guy who knows how to administer it all becomes the single point of failure for the whole business. No amount of tooling can save them from that. So local biz will always be beholden to the Oracles of the world, because smart individuals just can't be relied on. Businesses need to rely on other businesses. A business with a customer service departments and spare stock on hand. K3s or k8s might be used in the technology product offered by such a business, but this by itself won't change the economics of that market.
    - sliken 8y ago
      Dunno. I've know several business owners that didn't take credit cards until recently, and even then only did so because the newer companies have significantly lowered the investment cost. Maybe you don't consider small family owned restaurants "real". Seems like there's an opportunity for a cheap 3 node cluster, sold as a value add to businesses and supported a platform that application writers could target. They might well come with a support contract, some basic functionality, and then users could pay for inventory, staff scheduling, point of sales terminal support, integration with food delivery services, etc. Much like how some home/small business NASs these days have support for 100s of integrations to various online services.
- outime 8y ago
  Most likely not, as it adds quite a bit of complexity. It might be easier to start with simpler abstractions such as e.g. AWS ECS or similar alternatives (cloud or on-prem) if you want to use containers.
- sigi45 8y ago
  Nope but it solves problems you have to fix yourself otherwise anyway: - Rolling update? Build it yourself or get it with k8s - Infrastructure as code? Do it somehow yourself with ansible, awx or other setup OR just use k8s - Running different workloads on a bunch of hardware with resource scheduler -> some trickery or x different kind of vms I like it very much and i hope and guess that the k8s way will take over a lot. And since i'm working with k8s i really wanna have it at home. k3s sounds like a very good way forward. If you have a good working little setup, you don't need to replace it with k8s.
- mrweasel 8y ago
  Kubernetes isn't even necessary for medium and large businesses. It's more a question of what problem you're trying to solve. Kubernetes is only relevant if you have the need to scale your service quickly and/or often. If you run it on your own hardware the value is even more tricky to work out, it can still be a benefit. Whether or not it help you save on hardware cost is extremely dependent on your usage.
  - base698 8y ago
    I see it as decoupling the system administrator tasks from the software tasks. Sys admins can setup the OS, volumes, kubernetes, and monitoring processes. Application developers can do their own thing and deploy apps without having to care about the details of which mount their app needs for disks or the intricacies of whichever process runner they use.
- peterwwillis 8y ago
  No, in capital letters, with exclamation marks. (We're not allowed to be visibly dramatic on HN)
erulabs 8y ago
This is extremely welcome. Kube needs to glom on to all available architectures to fulfill its destiny, to some extent, and the memory/cpu usage of kubeapi et al. can be prohibitive for small setups. Having to revert to ansible/systemd in "some cases" really weakens the story of a universal datacenter O/S. My hope is for more competition for development k8s (or k3s!) - minikube and docker-for-desktop is plagued with high CPU usage problems, and OSX-wielding developers are still doubtful. It will require a lot of work and elbow grease to convince them. I've built a hosted platform that tries to address this, but it's a complex education problem, which I am not well suited to solve :(
- geerlingguy 8y ago
  Agreed. Kubernetes (installed via kubeadm) on 1 GB of RAM is a shaky proposition at best; I've been managing a cluster on Raspberry Pis and the master often hits swap managing three Nodes with a dozen or so Pods.
  - vhost- 8y ago
    K8s doesn’t support swap, you have to disable swap to even run kubelet.
    - phire 8y ago
      There is an override flag. It works, but it kind of invalidates all assumptions k8s makes about free memory on a node
  - moondev 8y ago
    What workloads are you running? With only 1gb, have you looked into setting resource limits for your pods?
- xur17 8y ago
  I really want to do this as well - we've just started looking at minikube for our group, and I'd love to use k3s instead, but does anyone know if it works on OSX? I was able to get a cluster running on my laptop in under a minute, which is pretty amazing.
detaro 8y ago
Looks quite interesting for small setups, curious to read more about the limitations (E.g. "Added sqlite3 as the default storage mechanism. etcd3 is still available, but not the default." - what availability promises can this make?)
- omeid2 8y ago
  In terms of etcd3 vs sqlite3, it is as reliable as most airplane systems that depend on it. https://www.sqlite.org/famous.html https://www.sqlite.org/famous.html I think the "high availability by redundancy" story is oversold.
  - philips 8y ago
    Alternatively the K3s authors could have embedded a single node etcd process into Kubernetes using the embed package instead of introducing sqlite. https://godoc.org/github.com/etcd-io/etcd/embed https://godoc.org/github.com/etcd-io/etcd/embed This is something the Kubernetes community might consider as well.
    - smarterclayton 8y ago
      Yeah, OpenShift did this from the very first version. It worked pretty well. Memory use was very reasonable from etcd 3.0 on.
  - rickycook 8y ago
    pretty sure critical airline systems have redundancy
  - tyingq 8y ago
    If someone wanted to do it, the lxc/lxd team has a distributed sqlite, so you could adapt that to k3s. Though I suppose that's not a good match for the stated purpose of k3s. https://github.com/lxc/lxd/blob/master/doc/database.md https://github.com/lxc/lxd/blob/master/doc/database.md https://github.com/CanonicalLtd/dqlite https://github.com/CanonicalLtd/dqlite
  - 8y ago
lprd 8y ago
A bit off topic, but has anyone used K8s along with Proxmox?
- Nux 8y ago
  Not a k8s user, but no reason it shouldn't work from what I read about it. Spawn a few CoreOS or RancherOS VMs and off you go. Some people even used it with LXC, though it seems a bit of a work. e.g. https://medium.com/@kvaps/run-kubernetes-in-lxc-container-f04aa94b6c9c https://medium.com/@kvaps/run-kubernetes-in-lxc-container-f0...
  - lprd 8y ago
    Gonna check this out, thank you! I tried installing CoreOS a couple years ago and I recall encountering some issues using LXC, but maybe it'll work now :)
- erinnh 8y ago
  Yes. I run my k8s cluster on a Proxmox host. Any specific questions you have?
  - lprd 8y ago
    I'm mostly curious how you got it all set up? Did you install something like CoreOS as the host?
    - erinnh 8y ago
      Sorry for the late answer. Host (inside the VM) is a plain Ubuntu 18.04 + Docker 18.06 and Kubernetes 1.13. Currently I have 1 Master and 2 Workers. I tried to use LXC (and it worked), but for some reason the same setup resulted in 50% (yes, 50%) IOWAIT with LXC in comparison to 4-5% in the 3 KVMs. I didnt really care to look too far into the reason. The server I run it on only has 2 7200rpm SATA drives in Raid 1, so its obviously not the best thing youd want for massive load or an actual production setup. But its basically my playpen + some private "production" servers, so with 1 User the performance doesnt really matter.
comboy 8y ago
Not trying to be snarky, just a sincere question - why would you use kubernetes for something like IoT?
- outime 8y ago
  I attended KubeCon China last year and there was a very good example of a K8s setup for IoT that scales pretty well. Check these links if you’re curious: - https://schd.ws/hosted_files/kccncchina2018english/18/ShengLiang-En.pdf https://schd.ws/hosted_files/kccncchina2018english/18/ShengL... - https://thenewstack.io/rancher-takes-kubernetes-management-to-china/ https://thenewstack.io/rancher-takes-kubernetes-management-t... Disclaimer: not affiliated with any of the companies above. Edit: formatting.
  - deleted 8y ago
    [deleted]
  - perone 8y ago
    Is there any link with the talk ?
- sliken 8y ago
  Because I want my doors, windows, garage door, smoke alarms, security system, and any other smart thing in my house even when the IoT controller dies. I also want as much functionality as possible if my ISP (Comcast) decides on another "maintenance" window. So that basically means local smarts and fall over in the case of a hardware failure.
  - comboy 8y ago
    Same. What does it have to do with kubernetes?
    - juiced 8y ago
      I like your username.
    - 8y ago
lostmsu 8y ago
Would it work on Android? That would be interesting.
- smw355 8y ago
  Darren mentioned on the CNCF webinar that he has plans to test it on chromeOS, which is similar to android, but that it hasn’t been done yet.
dankohn1 8y ago
Here's the recording of the webinar announcement: https://www.youtube.com/watch?v=5-5t672vFi4 https://www.youtube.com/watch?v=5-5t672vFi4
captn3m0 8y ago
@mods: This should be merged with https://news.ycombinator.com/item?id=19257675 https://news.ycombinator.com/item?id=19257675
zoobab 8y ago
I used k3s a while ago on my laptop to do some CI/CD with gitlab and arduino-cli to flash an ESP8266 device: https://fosdem.org/2019/schedule/event/hw_gitlab_ci_arduino/ https://fosdem.org/2019/schedule/event/hw_gitlab_ci_arduino/
zoobab 8y ago
Welcome to 2019 where an "embedded" system has 1GB RAM. The problem of GO is that all those unused libraries are loaded in RAM instead of staying on disk, and loaded when needed. An helloworld in GO (around 1.6MB) is still too large for my openwrt router.
- ithkuil 8y ago
  Like every process, the text segment is demand paged so it doesn't consume ram unless it's actually needed. What you probably mean is that multiple copies of the same library cannot share the same physical memory pages.
  - 05 8y ago
    Nope, they mean there's no flash memory available on the resource constrained device to store megabytes of useless runtime. Routers/other IOT Linux devices regularly use SPI ROM chips sized 4MB..16MB, you can't just install a 100MB Ubuntu/Arm rootfs on those.
    - londons_explore 8y ago
      I think the days of the small SPI flash chip are numbered. A a few hundreds of megabytes of flash will soon cost less than the plastic and manufacturing costs of the chip itself. At that point, manufacturers will be putting in 128megabyte flash chips simply because 16 megabyte flash chips cost more. The only place that argument won't apply is in microcontrollers where the flash is on the same die as the CPU.
      - 05 8y ago
        SPI chips are unreasonably expensive per MB compared to NAND, some reasons on why: https://electronics.stackexchange.com/questions/32200/why-is-spi-flash-memory-so-limited-in-max-size-and-cost-way-more-per-mb-than https://electronics.stackexchange.com/questions/32200/why-is...
      - mgoetzke 8y ago
sandGorgon 8y ago
Oh nice - this is by the Rancher guys. Would love to compare this not with k8s, but with Docker Swarm. Docker Swarm on Raspberry Pi is a very common thing. So, from a performance perspective, these are on par. Another question is around ingress and network plugin - is it a seamless "batteries included" experience ? Because these are two of the biggest pains in k8s to decide and setup .
- pas 8y ago
  Based on the README [0], quickly comparing to Swarm: - this has the same great K8S API, that Swarm lacks. (Deployment is a first class citizen in kube land, but you only have to make do with the service YMLs in the Swarm sphere.) - k3s lacks some in-tree plugins, that swarm might have (mount cloud provider managed block device), but there are out of tree addons - sqlite instead of etcd3 [but available], so out of the box k3s is not HA ready (if I interpret this[1] right, there's one APIserver, and it's something in development ) - this seems more lightweight than a docker setup in some sense (it uses containerd, not full docker), but of course this needs a proper evaluation - ingress plugins: good question. it should work with traefik [2] [0] https://github.com/rancher/k3s/blob/master/README.md#what-is-this https://github.com/rancher/k3s/blob/master/README.md#what-is... [1] https://github.com/rancher/k3s/blob/master/README.md#server-ha https://github.com/rancher/k3s/blob/master/README.md#server-... [2] https://github.com/rancher/k3s/search?q=ingress&type=Commits https://github.com/rancher/k3s/search?q=ingress&type=Commits
  - sandGorgon 8y ago
    >this has the same great K8S API, that Swarm lacks. (Deployment is a first class citizen in kube land, but you only have to make do with the service YMLs in the Swarm sphere.) With all due respect, some (like me) consider that a feature of Swarm. If k3s comes with the same API (and implicit yaml complexity) of k8s, but just reduces RAM usage...well that's not very interesting for me. Docker Swarm is very lean - there are zillions of videos of how people are building interesting raspberry Pi stacks using it (https://youtu.be/mMpZpa7uUSk https://youtu.be/mMpZpa7uUSk). But for most people using Swarm - they are using it for one reason only : simplicity.
Numberwang 8y ago
test
Jedi72 8y ago
"Any sufficiently complicated concurrent program in another language contains an ad hoc informally-specified bug-ridden slow implementation of half of Erlang." - Virding's Law
- Ruddle 8y ago
  IMO any program with a sufficiently high number of entities in a low level language contains an ad hoc informally-specified bug-ridden slow implementation of a garbage collector
  - omeid2 8y ago
    Not exactly, a garbage collector is by definition a collector (there is sweeps rather than per-reference and more importantly has runtime cost). I think, manual memory management is a lot more similar to a poorman's ARC implementation.
    - pjmlp 8y ago
      A poorman's ARC implementation might not collect cycles, but it does not do double free() or use-after-free().
- cdoxsey 8y ago
  Are you talking about Go, the language this is written in? Go's concurrency features are formally specified, are not ad-hoc (they are a major feature of the language, and the product of multiple language design iterations) and are certainly not bug-ridden. Or are you talking about using Kubernetes to manage workloads on machines? How does Erlang solve running my python webapp or my SQL database on my ec2 nodes, or in the case of this project, on an IoT device.
  - Jedi72 8y ago
    I am not talking about Go. I dont care what your language is, the true underlying reasoning for your engineering choices is independent of language. It just so happens that these problems have been around for ~30 years, and there has been significant progress in developing tooling to face them, in the domains that the BEAM languages sphere of influence. Just like how Ecto, despite being a great tool, does not free us of knowing about Relational Databases. If you approach the tooling infependently though, as a true knowledge expert does, you will see the progress made by other giants in these fields who are not afraid of specialised tools.
exabrial 8y ago
> Only Uses only 512 MB of RAM. One can literally run a full fledged Java EE server in 64mb of Ram. What?
- jcadam 8y ago
  > One can literally ruin a full fledged Java EE server in 64mb of Ram. What? Most definitely.
  - exabrial 8y ago
    That was hilarious. Thanks
- black-tea 8y ago
  My thought exactly. I did a double take initially thinking they must mean 512kB.
- oskapt 8y ago
  The site was updated to reflect that you can run K3s on hardware with only 512MB of RAM. You'll get to use most of that for your workloads.
tecleandor 8y ago
I've been laughing myself of for some minutes, not because the project is uninteresting (I'll be testing it next week on ARM, probably), but because I've decided to call this "Kubernetres" (Being 'TRES' three in Spanish)
- dajohnson89 8y ago
  why isn’t it called k7?
  - tyingq 8y ago
    They have a "5 less than K8S" tagline. I don't know if that means 5 fewer features, 5 fewer dependencies, or just "less than half the size".
  - dankohn1 8y ago
    I believe it goes: Kubernetes -> K8s -> "kates" -> k3s
    - ericandrewlewis 8y ago
      for some reason i thought it would be short for "kubes" (pronounced coobs)
      - GuyPostington 8y ago
        Not to be confused by Qubes (pronounced koobs)
    - caffeineaholic 8y ago
      This is correct we shortened it to "kates" -> k3s (Rancher employee)
      - deleted 8y ago
        [deleted]
  - 8y ago
meddlepal 8y ago
Technically speaking: cool Practically speaking: I worry this will become a parallel but subtlety different implementation of Kubernetes with it's own quirks.
- dankohn1 8y ago
  That's why CNCF is thrilled that Rancher passed our Certified Kubernetes conformance tests for k3s prior to the public announcement. https://landscape.cncf.io/category=certified-kubernetes-distribution,certified-kubernetes-hosted,certified-kubernetes-installer&format=card-mode&grouping=category&selected=k3s https://landscape.cncf.io/category=certified-kubernetes-dist...
  - meddlepal 8y ago
    Ah, interesting! Cool. Thanks.
- nvarsj 8y ago
  > Practically speaking: I worry this will become a parallel but subtlety different implementation of Kubernetes with it's own quirks. Hello, we're already there. Despite the CNCF certification process, there are many flavours of k8s, all with their own quirks. Being on the product side, it is quite painful to support.
  - meddlepal 8y ago
    I am aware of this problem (getambassador.io dev as well as infra engineer). The differences usually manifest at the edges rather than the internals right now. This has the potential to be even more disruptive problem. I agree the CNCF certification process isn't comprehensive enough.
    - nvarsj 8y ago
      Well, even the internals have issues. Take Openshift as the most extreme example - it's a fork of K8s, with a lot of custom functionality like scc and routes, and also a lot of custom behavior like its admission controllers that manage pod security contexts. It's almost like a different version of k8s - things that will run on GKE or vanilla k8s won't work on Openshift. Yet, it is CNCF compliant.
mleonard 8y ago
Anyone know if this will run on a Pixelbook? (inside the Crostini linux virtual machine in ChromeOS). Thanks
- excieve 8y ago
  Got excited about this too, but unfortunately no, it doesn't run on Crostini. Fails with querying netfilter. Also tried through docker-compose from the GH repository. It starts the k3s server fine but fails on starting the nodes as they require privileged container mode, which doesn't work on Crostini at the moment.
  - smw355 8y ago
    Mentioned this elsewhere, but if you create an issue, Darren mentioned on the CNCF webinar that they would be looking into ChromeOS soon.
    - excieve 8y ago
      There's also a feature request in ChromeOS bugtracker about adding the missing bits for minikube, part of which seem to be similar to what k3s is using: https://bugs.chromium.org/p/chromium/issues/detail?id=878034 https://bugs.chromium.org/p/chromium/issues/detail?id=878034
thinkersilver 8y ago
This looks great, we're planning to test with the intention of moving our CI/CD orchestration onto this. We use Calico internally, is there a plan in the future to allow other SDNs?
- granra 8y ago
  https://github.com/rancher/k3s#flannel https://github.com/rancher/k3s#flannel
rcarmo 8y ago
This is very nicely done. It annoys me a bit to have to set up a registry and have a build cycle prior to actually deploying things, but gitkube might help there...
- jpeeler 8y ago
  I hadn't heard of gitkube before. These blog posts helped me compare and contrast with similar solutions I had heard of (mainly skaffold and draft): https://blog.hasura.io/draft-vs-gitkube-vs-helm-vs-ksonnet-vs-metaparticle-vs-skaffold-f5aa9561f948/ https://blog.hasura.io/draft-vs-gitkube-vs-helm-vs-ksonnet-v... https://kubernetes.io/blog/2018/05/01/developing-on-kubernetes/ https://kubernetes.io/blog/2018/05/01/developing-on-kubernet...
johnmarcus 8y ago
"Added sqlite3 as the default storage mechanism. etcd3 is still available, but not the default." I want to use and support this product for this reason alone. Etcd was always an un-necessary complexity added for god knows what reason. Later cluster management solutions have abstracted etcd creation and management away (thankfully), but it's always irksome that it is there. Thank you to the K3s development team for taking on that challenge!
- geofft 8y ago
  AIUI it gets you away from a single point of failure, right? Unless you have a reliable NFS server (and non-SPOF NFS servers are rare and pricy), running k8s on SQLite sounds like you can only have one master. Of course that's totally fine for many k8s deployments and might even increase reliability for some use cases, but still, moving from a distributed system to a local one is a significant change.
  - johnmarcus 8y ago
    meh. Practically speaking, we're talking about a single point of disk failure - which certainly happens but at a rate that is sufficiently low. Plus, the actual amount of data stored is tiny, you can replicate it in seconds. Amazon has solutions for this if it's truly of concern, I would guess google does as well. IMHO, the operation of etcd - and the fact the data became unreadable if you lost quorum - was a much higher risk factor than possible disk failure. It was impractical to backup as well, you either have quorum or you don't. Even without NFS, I could backup that sqlite db every 5 minutes via a cron job and have most of my cluster state perfectly preserved.
    - geofft 8y ago
      Disk failure happens quite frequently for me at scale, but so do other things like RAM going bad or network cards dying or entire mainboards just acting weird or top-of-rack switches silently dropping packets because of memory corruption (all of these have happened to machines I'm responsible for in the last six months). Again I think this is a matter of scale. If you've got enough machines that disk failure is a concern, you can also run a 9-node etcd cluster and have a big enough pager rotation that keeping 5 of them up 99.999% if not 100% of the time isn't a challenge. If you have less than a rack of machines and you're not at the point where you're worried about having a SPOF in your network switches or power supplies, running etcd a bunch of overhead for a problem you don't have and you are genuinely better served by a robust non-distributed system whose availability is the availability of your hardware.
buckhx 8y ago
Would love this or something similar for local development. Currently using the Docker for Mac k8s and it shreds my machine and end up with an hour of battery life. Had similar experience with minikube.
- xur17 8y ago
  I would as well. Curious if anyone has tried this for local dev. Also curious if this can run on osx.
  - Ramiro 8y ago
    I tried it on my mac with the docker-compose from master and works pretty well.
milkers 8y ago
kubes
rb808 8y ago
I'm going to try this - I really hope its good. I've spent a lot of days trying to get k8s working with a cheap home server. With VMs and docker conflicting networks its a nightmare. Upgrading is worse.
- heroprotagonist 8y ago
  kubespray is great for multi-vms on home server. Just install base OS, run their ansible playbook (though might need to tell it the OS type in config, it will install and configure node requirements like docker).
- geerlingguy 8y ago
  I've tuned a Kubernetes setup to run pretty well on a Pi cluster with 1 GB RAM per node here: http://www.pidramble.com http://www.pidramble.com — there's a configuration for testing in Vagrant/VirtualBox machines as well.
  - rb808 8y ago
    Hey that looks pretty cool. I bought an old Xeon workstation for $200 off ebay that I think is more flexible. My own rack of blades does sound awesome. :)
diehunde 8y ago
The "Great for" section doesn't mentioned dev environment. Could this replace stuff like minikube or microk8s?
tasubotadas 8y ago
If it is as easy to deploy, as they say, it is bound to become the most used implementation as Ubuntu became the most popular Linux distro for servers. I am looking forward to this.
linuxdude314 8y ago
I find it humorous how many people in this thread seem so willing to use this in production. If you can't make Vanilla upstream Kubernetes work, it is very ill advised to think you will be met with success using a heavily modified fork that has a fraction of the support of K8S.
- oskapt 8y ago
  In an environment where only a fraction of support is needed, it doesn't make sense to run a fat distro with features that you'll never use. K3s is designed for low-power, low-resource environments that don't need things like EBS or NFS volume support or alpha/beta API features. Imagine edge environments like traffic management networks, wind farms, telco substations, or other places where environmental constraints make it more reliable to run something fanless with flash storage and 1GB of RAM. Stock Kubernetes will choke under those circumstances, but does that mean that those environments are forbidden from using the benefits of Kubernetes?
antranigv 8y ago
I mean, I've built a Kubernetes-y thing for my own needs, mostly "scheduling" LXC and FreeBSD Jails, and it needs hella less than 256MB of RAM and it's around 20MBs. Maybe I should clean the code and publish it.
- oskapt 8y ago
  The site has been updated to reflect that the nodes only need 512MB of RAM to run K3s _and Kubernetes workloads_. K3s itself doesn't consume 512MB of RAM.
- biggestlou 8y ago
  That's literally not Kubernetes, though. Does it support custom resource definitions? Services? Deployments? ConfigMaps? Multiple containers on a single host, i.e. Pods? Your project does sound very cool but I don't think provides a fair basis of comparison re: size and RAM usage.
  - antranigv 8y ago
    very good question! source definition -> no. services -> yes. deployments -> yes. configmaps -> no. multiple containers on single host -> yes. I should add the most common features I think. I also haven't integrated it with any reverse-proxy service yet, but will try that too.
deleted 8y ago
[deleted]
mbrumlow 8y ago
And we have done it! Solutions for the solutions!
crawshaw 8y ago
It is nice to see people working on making software smaller! It leaves me wondering though, what is Kubernetes doing with all of these resources? When I saw "lightweight" in the title I expected 5.12MB of RAM, not 512MB.
dividedbyzero 8y ago
This looks great, playing around with it right now. They seem to have removed PersistentVolumes, too. I'd like to run a MySQL database on K3S, how would I go about making sure that my database's data survives when its pod dies?

458 points by kadel 8y ago

pictur 8y ago

Do you think kubernetes are necessary for small businesses?

redmorphium 8y ago
No, for the same reason that Martin Fowler argues you should start with a monolith first before moving to a microservices architecture.
- rhizome31 8y ago
  What is currently a good alternative for a small business who needs to distribute containers on a handful of nodes? Docker swarm mode?
  - MediumD 8y ago
    If you need to distribute containers on a handful of nodes, kubernetes is great (and far better than Docker swarm imho). However, most small businesses probably don't need to have a microservice setup and can simply deploy a simple monolith while they focus on getting their business off the ground.
    - rhizome31 8y ago
      Thank you. I was thinking that maybe swarm mode would be simpler to set up and maintain. Just to give more context, our use case is spawning Jupyter notebook in containers using JupyterHub and DockerSpawner so that each data scientist gets a personal ready-to-use environment to work with. With 4 data scientists we're beginning to reach the limits of the largest instance provided by our cloud provider and we're expecting to have more users soon. Actually our current cloud provider has just released a managed k8s service so that might be the way to go.
      - MediumD 8y ago
        Gotcha. Yeah, I would much rather use a managed k8s service over deploying and maintaining my own. GCP's GKE has been great for us so far and very reasonably priced.
  - scarejunba 8y ago
    Kube is fine for this, but don't roll your own, use GKE. Perhaps you'd be better served using more managed services, though?
    - rhizome31 8y ago
      What do you mean by more managed services?
      - icebraining 8y ago
        I'm guessing Heroku, Google App Engine, AWS Beanstalk and such.
  - djsumdog 8y ago
    A single baremetal server with just docker containers. It's much easier to migrate from just containers running on nodes to an orchestration system (marathon, nomad OR k3s/k8s; there are other options) than to start with the orchestration system.
    - rhizome31 8y ago
      This is the situation we're in. We're beginning to reach the limits of what can be handled on a single instance, mostly regarding memory.
    - noir_lord 8y ago
      In my case it's a single bare metal server with KVM guests and ansible. Docker is on my long term list (maybe) but the current system is fast, stable and requires little intervention so docker would have to offer a massive change not a step change to make it worth it. I always default to things I know until the benefits are clear and it's worked out well so far.
  - peterwwillis 8y ago
    Docker swarm, Consul+Nomad, AWS Fargate, Azure ACS, Google GCP, etc. Even DC/OS+Marathon is probably simpler. If you really think you need K8s, use a managed service provider like GKE, AKS, or a platform manager like Rancher. You need to pay a specialized company to build and operate it for you, or you aren't going to have a fun time. I highly recommend leveraging AWS, Google, Azure, etc services. Especially if you're a small business, spending a little for off the shelf tools that someone else operates and supports is going to help you big time.
danaur 8y ago
It wasn't before it existed and isn't now it's convenient though
whazor 8y ago
I think maybe eventually with cheaper ARM CPUs.
sliken 8y ago
Necessary, no. But with the hardware almost free, a 3 x Raspberry Pi cluster, switch, access pointer, and a small UPS would be a few $100 which allows for some interesting possibilities. Pretty much any business will want inventory tracking, tracking staff hours, monitoring cameras, sensors for detecting flooding/doors left open/freezers dying, running point of sales terminals, keeping the supply chain running smoothly, handling customer loyalty cards, handling returns, etc. An on site cluster can help with much of that, even if there's a hardware failure or the network is out. Question is how will these business find the apps they need that run turn key on some small k8s cluster?
- vinceguidry 8y ago
  Hardware was never the bottleneck there. The systems could cost a hundred times that and it would still be a rounding error on the capex of even the smallest "real" business. The obstacle to using custom hardware and software in local biz is and has always been, the fact that by relying on it, the one guy who knows how to administer it all becomes the single point of failure for the whole business. No amount of tooling can save them from that. So local biz will always be beholden to the Oracles of the world, because smart individuals just can't be relied on. Businesses need to rely on other businesses. A business with a customer service departments and spare stock on hand. K3s or k8s might be used in the technology product offered by such a business, but this by itself won't change the economics of that market.
  - sliken 8y ago
    Dunno. I've know several business owners that didn't take credit cards until recently, and even then only did so because the newer companies have significantly lowered the investment cost. Maybe you don't consider small family owned restaurants "real". Seems like there's an opportunity for a cheap 3 node cluster, sold as a value add to businesses and supported a platform that application writers could target. They might well come with a support contract, some basic functionality, and then users could pay for inventory, staff scheduling, point of sales terminal support, integration with food delivery services, etc. Much like how some home/small business NASs these days have support for 100s of integrations to various online services.
outime 8y ago
Most likely not, as it adds quite a bit of complexity. It might be easier to start with simpler abstractions such as e.g. AWS ECS or similar alternatives (cloud or on-prem) if you want to use containers.
sigi45 8y ago
Nope but it solves problems you have to fix yourself otherwise anyway: - Rolling update? Build it yourself or get it with k8s - Infrastructure as code? Do it somehow yourself with ansible, awx or other setup OR just use k8s - Running different workloads on a bunch of hardware with resource scheduler -> some trickery or x different kind of vms I like it very much and i hope and guess that the k8s way will take over a lot. And since i'm working with k8s i really wanna have it at home. k3s sounds like a very good way forward. If you have a good working little setup, you don't need to replace it with k8s.
mrweasel 8y ago
Kubernetes isn't even necessary for medium and large businesses. It's more a question of what problem you're trying to solve. Kubernetes is only relevant if you have the need to scale your service quickly and/or often. If you run it on your own hardware the value is even more tricky to work out, it can still be a benefit. Whether or not it help you save on hardware cost is extremely dependent on your usage.
- base698 8y ago
  I see it as decoupling the system administrator tasks from the software tasks. Sys admins can setup the OS, volumes, kubernetes, and monitoring processes. Application developers can do their own thing and deploy apps without having to care about the details of which mount their app needs for disks or the intricacies of whichever process runner they use.
peterwwillis 8y ago
No, in capital letters, with exclamation marks. (We're not allowed to be visibly dramatic on HN)

erulabs 8y ago

This is extremely welcome. Kube needs to glom on to all available architectures to fulfill its destiny, to some extent, and the memory/cpu usage of kubeapi et al. can be prohibitive for small setups. Having to revert to ansible/systemd in "some cases" really weakens the story of a universal datacenter O/S. My hope is for more competition for development k8s (or k3s!) - minikube and docker-for-desktop is plagued with high CPU usage problems, and OSX-wielding developers are still doubtful. It will require a lot of work and elbow grease to convince them. I've built a hosted platform that tries to address this, but it's a complex education problem, which I am not well suited to solve :(

geerlingguy 8y ago
Agreed. Kubernetes (installed via kubeadm) on 1 GB of RAM is a shaky proposition at best; I've been managing a cluster on Raspberry Pis and the master often hits swap managing three Nodes with a dozen or so Pods.
- vhost- 8y ago
  K8s doesn’t support swap, you have to disable swap to even run kubelet.
  - phire 8y ago
    There is an override flag. It works, but it kind of invalidates all assumptions k8s makes about free memory on a node
- moondev 8y ago
  What workloads are you running? With only 1gb, have you looked into setting resource limits for your pods?
xur17 8y ago
I really want to do this as well - we've just started looking at minikube for our group, and I'd love to use k3s instead, but does anyone know if it works on OSX? I was able to get a cluster running on my laptop in under a minute, which is pretty amazing.

detaro 8y ago

Looks quite interesting for small setups, curious to read more about the limitations (E.g. "Added sqlite3 as the default storage mechanism. etcd3 is still available, but not the default." - what availability promises can this make?)

omeid2 8y ago
In terms of etcd3 vs sqlite3, it is as reliable as most airplane systems that depend on it. https://www.sqlite.org/famous.html https://www.sqlite.org/famous.html I think the "high availability by redundancy" story is oversold.
- philips 8y ago
  Alternatively the K3s authors could have embedded a single node etcd process into Kubernetes using the embed package instead of introducing sqlite. https://godoc.org/github.com/etcd-io/etcd/embed https://godoc.org/github.com/etcd-io/etcd/embed This is something the Kubernetes community might consider as well.
  - smarterclayton 8y ago
    Yeah, OpenShift did this from the very first version. It worked pretty well. Memory use was very reasonable from etcd 3.0 on.
- rickycook 8y ago
  pretty sure critical airline systems have redundancy
- tyingq 8y ago
  If someone wanted to do it, the lxc/lxd team has a distributed sqlite, so you could adapt that to k3s. Though I suppose that's not a good match for the stated purpose of k3s. https://github.com/lxc/lxd/blob/master/doc/database.md https://github.com/lxc/lxd/blob/master/doc/database.md https://github.com/CanonicalLtd/dqlite https://github.com/CanonicalLtd/dqlite
- 8y ago

lprd 8y ago

A bit off topic, but has anyone used K8s along with Proxmox?

Nux 8y ago
Not a k8s user, but no reason it shouldn't work from what I read about it. Spawn a few CoreOS or RancherOS VMs and off you go. Some people even used it with LXC, though it seems a bit of a work. e.g. https://medium.com/@kvaps/run-kubernetes-in-lxc-container-f04aa94b6c9c https://medium.com/@kvaps/run-kubernetes-in-lxc-container-f0...
- lprd 8y ago
  Gonna check this out, thank you! I tried installing CoreOS a couple years ago and I recall encountering some issues using LXC, but maybe it'll work now :)
erinnh 8y ago
Yes. I run my k8s cluster on a Proxmox host. Any specific questions you have?
- lprd 8y ago
  I'm mostly curious how you got it all set up? Did you install something like CoreOS as the host?
  - erinnh 8y ago
    Sorry for the late answer. Host (inside the VM) is a plain Ubuntu 18.04 + Docker 18.06 and Kubernetes 1.13. Currently I have 1 Master and 2 Workers. I tried to use LXC (and it worked), but for some reason the same setup resulted in 50% (yes, 50%) IOWAIT with LXC in comparison to 4-5% in the 3 KVMs. I didnt really care to look too far into the reason. The server I run it on only has 2 7200rpm SATA drives in Raid 1, so its obviously not the best thing youd want for massive load or an actual production setup. But its basically my playpen + some private "production" servers, so with 1 User the performance doesnt really matter.

comboy 8y ago

Not trying to be snarky, just a sincere question - why would you use kubernetes for something like IoT?

outime 8y ago
I attended KubeCon China last year and there was a very good example of a K8s setup for IoT that scales pretty well. Check these links if you’re curious: - https://schd.ws/hosted_files/kccncchina2018english/18/ShengLiang-En.pdf https://schd.ws/hosted_files/kccncchina2018english/18/ShengL... - https://thenewstack.io/rancher-takes-kubernetes-management-to-china/ https://thenewstack.io/rancher-takes-kubernetes-management-t... Disclaimer: not affiliated with any of the companies above. Edit: formatting.
- deleted 8y ago
  [deleted]
- perone 8y ago
  Is there any link with the talk ?
sliken 8y ago
Because I want my doors, windows, garage door, smoke alarms, security system, and any other smart thing in my house even when the IoT controller dies. I also want as much functionality as possible if my ISP (Comcast) decides on another "maintenance" window. So that basically means local smarts and fall over in the case of a hardware failure.
- comboy 8y ago
  Same. What does it have to do with kubernetes?
  - juiced 8y ago
    I like your username.
  - 8y ago

lostmsu 8y ago

Would it work on Android? That would be interesting.

smw355 8y ago
Darren mentioned on the CNCF webinar that he has plans to test it on chromeOS, which is similar to android, but that it hasn’t been done yet.

dankohn1 8y ago

Here's the recording of the webinar announcement: https://www.youtube.com/watch?v=5-5t672vFi4 https://www.youtube.com/watch?v=5-5t672vFi4

captn3m0 8y ago

@mods: This should be merged with https://news.ycombinator.com/item?id=19257675 https://news.ycombinator.com/item?id=19257675

zoobab 8y ago

I used k3s a while ago on my laptop to do some CI/CD with gitlab and arduino-cli to flash an ESP8266 device: https://fosdem.org/2019/schedule/event/hw_gitlab_ci_arduino/ https://fosdem.org/2019/schedule/event/hw_gitlab_ci_arduino/

zoobab 8y ago

Welcome to 2019 where an "embedded" system has 1GB RAM. The problem of GO is that all those unused libraries are loaded in RAM instead of staying on disk, and loaded when needed. An helloworld in GO (around 1.6MB) is still too large for my openwrt router.

ithkuil 8y ago
Like every process, the text segment is demand paged so it doesn't consume ram unless it's actually needed. What you probably mean is that multiple copies of the same library cannot share the same physical memory pages.
- 05 8y ago
  Nope, they mean there's no flash memory available on the resource constrained device to store megabytes of useless runtime. Routers/other IOT Linux devices regularly use SPI ROM chips sized 4MB..16MB, you can't just install a 100MB Ubuntu/Arm rootfs on those.
  - londons_explore 8y ago
    I think the days of the small SPI flash chip are numbered. A a few hundreds of megabytes of flash will soon cost less than the plastic and manufacturing costs of the chip itself. At that point, manufacturers will be putting in 128megabyte flash chips simply because 16 megabyte flash chips cost more. The only place that argument won't apply is in microcontrollers where the flash is on the same die as the CPU.
    - 05 8y ago
      SPI chips are unreasonably expensive per MB compared to NAND, some reasons on why: https://electronics.stackexchange.com/questions/32200/why-is-spi-flash-memory-so-limited-in-max-size-and-cost-way-more-per-mb-than https://electronics.stackexchange.com/questions/32200/why-is...
    - mgoetzke 8y ago

sandGorgon 8y ago

Oh nice - this is by the Rancher guys. Would love to compare this not with k8s, but with Docker Swarm. Docker Swarm on Raspberry Pi is a very common thing. So, from a performance perspective, these are on par. Another question is around ingress and network plugin - is it a seamless "batteries included" experience ? Because these are two of the biggest pains in k8s to decide and setup .

pas 8y ago
Based on the README [0], quickly comparing to Swarm: - this has the same great K8S API, that Swarm lacks. (Deployment is a first class citizen in kube land, but you only have to make do with the service YMLs in the Swarm sphere.) - k3s lacks some in-tree plugins, that swarm might have (mount cloud provider managed block device), but there are out of tree addons - sqlite instead of etcd3 [but available], so out of the box k3s is not HA ready (if I interpret this[1] right, there's one APIserver, and it's something in development ) - this seems more lightweight than a docker setup in some sense (it uses containerd, not full docker), but of course this needs a proper evaluation - ingress plugins: good question. it should work with traefik [2] [0] https://github.com/rancher/k3s/blob/master/README.md#what-is-this https://github.com/rancher/k3s/blob/master/README.md#what-is... [1] https://github.com/rancher/k3s/blob/master/README.md#server-ha https://github.com/rancher/k3s/blob/master/README.md#server-... [2] https://github.com/rancher/k3s/search?q=ingress&type=Commits https://github.com/rancher/k3s/search?q=ingress&type=Commits
- sandGorgon 8y ago
  >this has the same great K8S API, that Swarm lacks. (Deployment is a first class citizen in kube land, but you only have to make do with the service YMLs in the Swarm sphere.) With all due respect, some (like me) consider that a feature of Swarm. If k3s comes with the same API (and implicit yaml complexity) of k8s, but just reduces RAM usage...well that's not very interesting for me. Docker Swarm is very lean - there are zillions of videos of how people are building interesting raspberry Pi stacks using it (https://youtu.be/mMpZpa7uUSk https://youtu.be/mMpZpa7uUSk). But for most people using Swarm - they are using it for one reason only : simplicity.

Numberwang 8y ago

test

Jedi72 8y ago

"Any sufficiently complicated concurrent program in another language contains an ad hoc informally-specified bug-ridden slow implementation of half of Erlang." - Virding's Law

Ruddle 8y ago
IMO any program with a sufficiently high number of entities in a low level language contains an ad hoc informally-specified bug-ridden slow implementation of a garbage collector
- omeid2 8y ago
  Not exactly, a garbage collector is by definition a collector (there is sweeps rather than per-reference and more importantly has runtime cost). I think, manual memory management is a lot more similar to a poorman's ARC implementation.
  - pjmlp 8y ago
    A poorman's ARC implementation might not collect cycles, but it does not do double free() or use-after-free().
cdoxsey 8y ago
Are you talking about Go, the language this is written in? Go's concurrency features are formally specified, are not ad-hoc (they are a major feature of the language, and the product of multiple language design iterations) and are certainly not bug-ridden. Or are you talking about using Kubernetes to manage workloads on machines? How does Erlang solve running my python webapp or my SQL database on my ec2 nodes, or in the case of this project, on an IoT device.
- Jedi72 8y ago
  I am not talking about Go. I dont care what your language is, the true underlying reasoning for your engineering choices is independent of language. It just so happens that these problems have been around for ~30 years, and there has been significant progress in developing tooling to face them, in the domains that the BEAM languages sphere of influence. Just like how Ecto, despite being a great tool, does not free us of knowing about Relational Databases. If you approach the tooling infependently though, as a true knowledge expert does, you will see the progress made by other giants in these fields who are not afraid of specialised tools.

exabrial 8y ago

> Only Uses only 512 MB of RAM. One can literally run a full fledged Java EE server in 64mb of Ram. What?

jcadam 8y ago
> One can literally ruin a full fledged Java EE server in 64mb of Ram. What? Most definitely.
- exabrial 8y ago
  That was hilarious. Thanks
black-tea 8y ago
My thought exactly. I did a double take initially thinking they must mean 512kB.
oskapt 8y ago
The site was updated to reflect that you can run K3s on hardware with only 512MB of RAM. You'll get to use most of that for your workloads.

tecleandor 8y ago

I've been laughing myself of for some minutes, not because the project is uninteresting (I'll be testing it next week on ARM, probably), but because I've decided to call this "Kubernetres" (Being 'TRES' three in Spanish)

dajohnson89 8y ago
why isn’t it called k7?
- tyingq 8y ago
  They have a "5 less than K8S" tagline. I don't know if that means 5 fewer features, 5 fewer dependencies, or just "less than half the size".
- dankohn1 8y ago
  I believe it goes: Kubernetes -> K8s -> "kates" -> k3s
  - ericandrewlewis 8y ago
    for some reason i thought it would be short for "kubes" (pronounced coobs)
    - GuyPostington 8y ago
      Not to be confused by Qubes (pronounced koobs)
  - caffeineaholic 8y ago
    This is correct we shortened it to "kates" -> k3s (Rancher employee)
    - deleted 8y ago
      [deleted]
- 8y ago

meddlepal 8y ago

Technically speaking: cool Practically speaking: I worry this will become a parallel but subtlety different implementation of Kubernetes with it's own quirks.

dankohn1 8y ago
That's why CNCF is thrilled that Rancher passed our Certified Kubernetes conformance tests for k3s prior to the public announcement. https://landscape.cncf.io/category=certified-kubernetes-distribution,certified-kubernetes-hosted,certified-kubernetes-installer&format=card-mode&grouping=category&selected=k3s https://landscape.cncf.io/category=certified-kubernetes-dist...
- meddlepal 8y ago
  Ah, interesting! Cool. Thanks.
nvarsj 8y ago
> Practically speaking: I worry this will become a parallel but subtlety different implementation of Kubernetes with it's own quirks. Hello, we're already there. Despite the CNCF certification process, there are many flavours of k8s, all with their own quirks. Being on the product side, it is quite painful to support.
- meddlepal 8y ago
  I am aware of this problem (getambassador.io dev as well as infra engineer). The differences usually manifest at the edges rather than the internals right now. This has the potential to be even more disruptive problem. I agree the CNCF certification process isn't comprehensive enough.
  - nvarsj 8y ago
    Well, even the internals have issues. Take Openshift as the most extreme example - it's a fork of K8s, with a lot of custom functionality like scc and routes, and also a lot of custom behavior like its admission controllers that manage pod security contexts. It's almost like a different version of k8s - things that will run on GKE or vanilla k8s won't work on Openshift. Yet, it is CNCF compliant.

mleonard 8y ago

Anyone know if this will run on a Pixelbook? (inside the Crostini linux virtual machine in ChromeOS). Thanks

excieve 8y ago
Got excited about this too, but unfortunately no, it doesn't run on Crostini. Fails with querying netfilter. Also tried through docker-compose from the GH repository. It starts the k3s server fine but fails on starting the nodes as they require privileged container mode, which doesn't work on Crostini at the moment.
- smw355 8y ago
  Mentioned this elsewhere, but if you create an issue, Darren mentioned on the CNCF webinar that they would be looking into ChromeOS soon.
  - excieve 8y ago
    There's also a feature request in ChromeOS bugtracker about adding the missing bits for minikube, part of which seem to be similar to what k3s is using: https://bugs.chromium.org/p/chromium/issues/detail?id=878034 https://bugs.chromium.org/p/chromium/issues/detail?id=878034

thinkersilver 8y ago

This looks great, we're planning to test with the intention of moving our CI/CD orchestration onto this. We use Calico internally, is there a plan in the future to allow other SDNs?

granra 8y ago
https://github.com/rancher/k3s#flannel https://github.com/rancher/k3s#flannel

rcarmo 8y ago

This is very nicely done. It annoys me a bit to have to set up a registry and have a build cycle prior to actually deploying things, but gitkube might help there...

jpeeler 8y ago
I hadn't heard of gitkube before. These blog posts helped me compare and contrast with similar solutions I had heard of (mainly skaffold and draft): https://blog.hasura.io/draft-vs-gitkube-vs-helm-vs-ksonnet-vs-metaparticle-vs-skaffold-f5aa9561f948/ https://blog.hasura.io/draft-vs-gitkube-vs-helm-vs-ksonnet-v... https://kubernetes.io/blog/2018/05/01/developing-on-kubernetes/ https://kubernetes.io/blog/2018/05/01/developing-on-kubernet...

johnmarcus 8y ago

"Added sqlite3 as the default storage mechanism. etcd3 is still available, but not the default." I want to use and support this product for this reason alone. Etcd was always an un-necessary complexity added for god knows what reason. Later cluster management solutions have abstracted etcd creation and management away (thankfully), but it's always irksome that it is there. Thank you to the K3s development team for taking on that challenge!

geofft 8y ago
AIUI it gets you away from a single point of failure, right? Unless you have a reliable NFS server (and non-SPOF NFS servers are rare and pricy), running k8s on SQLite sounds like you can only have one master. Of course that's totally fine for many k8s deployments and might even increase reliability for some use cases, but still, moving from a distributed system to a local one is a significant change.
- johnmarcus 8y ago
  meh. Practically speaking, we're talking about a single point of disk failure - which certainly happens but at a rate that is sufficiently low. Plus, the actual amount of data stored is tiny, you can replicate it in seconds. Amazon has solutions for this if it's truly of concern, I would guess google does as well. IMHO, the operation of etcd - and the fact the data became unreadable if you lost quorum - was a much higher risk factor than possible disk failure. It was impractical to backup as well, you either have quorum or you don't. Even without NFS, I could backup that sqlite db every 5 minutes via a cron job and have most of my cluster state perfectly preserved.
  - geofft 8y ago
    Disk failure happens quite frequently for me at scale, but so do other things like RAM going bad or network cards dying or entire mainboards just acting weird or top-of-rack switches silently dropping packets because of memory corruption (all of these have happened to machines I'm responsible for in the last six months). Again I think this is a matter of scale. If you've got enough machines that disk failure is a concern, you can also run a 9-node etcd cluster and have a big enough pager rotation that keeping 5 of them up 99.999% if not 100% of the time isn't a challenge. If you have less than a rack of machines and you're not at the point where you're worried about having a SPOF in your network switches or power supplies, running etcd a bunch of overhead for a problem you don't have and you are genuinely better served by a robust non-distributed system whose availability is the availability of your hardware.

buckhx 8y ago

Would love this or something similar for local development. Currently using the Docker for Mac k8s and it shreds my machine and end up with an hour of battery life. Had similar experience with minikube.

xur17 8y ago
I would as well. Curious if anyone has tried this for local dev. Also curious if this can run on osx.
- Ramiro 8y ago
  I tried it on my mac with the docker-compose from master and works pretty well.

milkers 8y ago

kubes

rb808 8y ago

I'm going to try this - I really hope its good. I've spent a lot of days trying to get k8s working with a cheap home server. With VMs and docker conflicting networks its a nightmare. Upgrading is worse.

heroprotagonist 8y ago
kubespray is great for multi-vms on home server. Just install base OS, run their ansible playbook (though might need to tell it the OS type in config, it will install and configure node requirements like docker).
geerlingguy 8y ago
I've tuned a Kubernetes setup to run pretty well on a Pi cluster with 1 GB RAM per node here: http://www.pidramble.com http://www.pidramble.com — there's a configuration for testing in Vagrant/VirtualBox machines as well.
- rb808 8y ago
  Hey that looks pretty cool. I bought an old Xeon workstation for $200 off ebay that I think is more flexible. My own rack of blades does sound awesome. :)

diehunde 8y ago

The "Great for" section doesn't mentioned dev environment. Could this replace stuff like minikube or microk8s?

tasubotadas 8y ago

If it is as easy to deploy, as they say, it is bound to become the most used implementation as Ubuntu became the most popular Linux distro for servers. I am looking forward to this.

linuxdude314 8y ago

I find it humorous how many people in this thread seem so willing to use this in production. If you can't make Vanilla upstream Kubernetes work, it is very ill advised to think you will be met with success using a heavily modified fork that has a fraction of the support of K8S.

oskapt 8y ago
In an environment where only a fraction of support is needed, it doesn't make sense to run a fat distro with features that you'll never use. K3s is designed for low-power, low-resource environments that don't need things like EBS or NFS volume support or alpha/beta API features. Imagine edge environments like traffic management networks, wind farms, telco substations, or other places where environmental constraints make it more reliable to run something fanless with flash storage and 1GB of RAM. Stock Kubernetes will choke under those circumstances, but does that mean that those environments are forbidden from using the benefits of Kubernetes?

antranigv 8y ago

I mean, I've built a Kubernetes-y thing for my own needs, mostly "scheduling" LXC and FreeBSD Jails, and it needs hella less than 256MB of RAM and it's around 20MBs. Maybe I should clean the code and publish it.

oskapt 8y ago
The site has been updated to reflect that the nodes only need 512MB of RAM to run K3s _and Kubernetes workloads_. K3s itself doesn't consume 512MB of RAM.
biggestlou 8y ago
That's literally not Kubernetes, though. Does it support custom resource definitions? Services? Deployments? ConfigMaps? Multiple containers on a single host, i.e. Pods? Your project does sound very cool but I don't think provides a fair basis of comparison re: size and RAM usage.
- antranigv 8y ago
  very good question! source definition -> no. services -> yes. deployments -> yes. configmaps -> no. multiple containers on single host -> yes. I should add the most common features I think. I also haven't integrated it with any reverse-proxy service yet, but will try that too.

deleted 8y ago

[deleted]

mbrumlow 8y ago

And we have done it! Solutions for the solutions!

crawshaw 8y ago

It is nice to see people working on making software smaller! It leaves me wondering though, what is Kubernetes doing with all of these resources? When I saw "lightweight" in the title I expected 5.12MB of RAM, not 512MB.

dividedbyzero 8y ago

This looks great, playing around with it right now. They seem to have removed PersistentVolumes, too. I'd like to run a MySQL database on K3S, how would I go about making sure that my database's data survives when its pod dies?