14 ms·
A Gentle introduction to Kubernetes with more than just the basics
- ggregoire 7y agoI'm wondering if Kubernetes would be the right choice in our use case. Or if something like Ansible would be better suited and easier to setup and use? I'll soon have to manage 50 remote bare-metal servers, all of them in different cities. Each one of them will run the same dockerized application. It's not 1 application distributed in 50 servers; it's 50 instances of the same application running in 50 servers, independently. A frequent use case is to deploy a new version of this application on every server (nowadays I do it manually, it's OK since I manage only like 10 servers). A nice-to-have would be to avoid downtimes also, when I update the application (nowadays I have a downtime of 2-5 minutes (when it goes well), which matters for us).
- aprdm 7y agoI would personally go with ansible and consul if your application is something like a rest api, should be trivial to setup.
- clhodapp 7y agoAt a high level, your use-case maps well to a Kubernetes DaemonSet, which is a resource that represents the desire to run a copy of something containerized on every node within your cluster. Kubernetes would give you some things that might be useful: * A unified control plane (monitoring, statusing) * An automated rolling-upgrade system * "self-healing" in the form of automatically restarting your containers if they go awry However, for your use-case, it is likely more effort than it's worth: * Settin up and maintaining Kubernetes on bare metal servers is a huge amount of work * A lot of Kubernetes features are not needed for your use-case * The Kubernetes networking model essentially requires you to set up private network spanning all of your sites in order to work but provides almost no benefit for your use-case
- kissgyorgy 7y agoTechnically you could use a DaemonSet for that, but the primary use-case is not that for DaemonSets. It should be used for "daemon" containers, like logging daemons (e.g. fluentd), network plugin daemons like calico, flannel or monitoring like prometheus. If you want to run multiple instances of your application, you should define a replicaset or deployment with setting Pod affinity instead.
- clhodapp 7y agoThat's super fiddly though: you will be leaning on (anti) affinity and counts to retain the 1:1 relationship between hosts and pods. I would argue that daemonsets aren't for any specific workload, they're more about how you want scheduling to happen (1 per node, on some subset of your nodes).
- manigandham 7y agoThis is where K8S becomes messy because of its rapid development history. Applications should just be a `Deployment` manifest with the type of distribution being a child setting (X replicas with affinity or 1-per-host). There's no reason to have competing ReplicaSets and DaemonSets just for this detail.
- kissgyorgy 7y agoI disagree on this one, DaemonSets are for really a different use-case. There is no replicas field you can set, and strictly 1 Pod will run on 1 node. A Deployment is more flexible, because you can run even more Pods in different cities if necessary. What if there are more/less nodes are in different cities or different specs, which could and should run more Pods? You can't fine-tune DaemonSets for those cases, it really is for daemons.
- manigandham 7y agoIt's not about use-case, it's that "X number of replicas with affinity" OR "1-per-node" should just be a setting within a Deployment. No need to use a completely separate DaemonSet definition just to run 1-per-node.
- dragonsh 7y agoNo kubernetes will be a bad choice for your use case. But if you learn all the intricacies and manage a behemoth besides your own application it will help in your resume to brag that you are a modern devops and a fashionable one using kubernetes. If you need a practical suggestion use simple solution like LXD [1]. But as you are already using docker stay with it and ansible. As you are using ansible you are already doing infrastructure as a code, you are way ahead of curve already. Probably when you grow a bit bigger use packer [2] and terraform [3], but I think ansible will do just fine. Kubernetes was designed for Google kind of problem the burden to maintain is quite high with lots of moving parts, unless you use Google GKE or Amazon or managed kubernetes service. So don't fall for it unless you need it for your resume. [1] https://linuxcontainers.org/ https://linuxcontainers.org/ [2] https://www.packer.io/intro/ https://www.packer.io/intro/ [3] https://www.terraform.io/ https://www.terraform.io/
- samvher 7y agoGKE and EKS are very expensive which in my opinion makes using them unattractive for smaller scale projects. But Digital Ocean now offers managed Kubernetes for free, I've played around with it and it actually seems really easy to set up and manage. Would be great if the competition would follow suit.
- jorams 7y ago> GKE and EKS are very expensive > Digital Ocean now offers managed Kubernetes for free GKE is just as "free" as DO, in that you only pay for the nodes you actually use (those nodes are probably a bit more expensive, admittedly). EKS is the outlier in this trio in that you also have to pay (a lot) for the master. Could you clarify what GKE could do to "follow suit"?
- samvher 7y agoIt seems, nothing! I was just wrong about GKE being expensive, not sure why I thought that (though it seems that until late 2017 they did have costs per cluster).
- deleted 7y ago[deleted]
- kissgyorgy 7y agoKubernetes is supposed to installed nodes close to each other, typically in the same server room, or datacenter, otherwise the latency would kill the etcd cluster. Ansible is perfectly fine and simple for your use case, why would you bother even thinking about overcomplicating it? Simplicity is king!
- rochacon 7y agoNo, as @dragonsh said, Kubernetes will be a bad choice for you. Kubernetes is complex, specially its network stack, you'll need to setup an VPN from each site to your control plane site to keep some basic functionality working (kubectl logs, for example). If you don't care about a centralized API to probe status and manage each instance, Ansible should be enough to orchestrate these installations and with little effort (that also depends on the application at hand) getting zero-downtime rollouts with Docker can be easily done with it. However, if you want a single control place to probe status and want to avoid writing your own rollout scripts, Hashicorp's Nomad [1] might be a good solution for this. It is a lot simpler than Kubernetes while still giving you nice primitives to describe jobs/services, health checks, rollouts strategies and etc. Treat every site as a datacenter of its own, setup a job of type "system" (akin to Kubernetes DaemonSets) and all you need on these sites is internet access to your HTTPS endpoint of the Nomad control plane. If you want to talk more about this, hit me up on Twitter or Telegram, I use @rochacon as my handle virtually everywhere. [1] https://www.nomadproject.io https://www.nomadproject.io Edit: grammar and typos
- SomaticPirate 7y agoI would also investigate Terraform if the Ansible path appeals to you.
- MuffinFlavored 7y agoHow can zero downtime deploys be done with Docker? I had to write my own custom blue/green deploy script to hot reload traffic to proxy_pass definitions in nginx upstream configs since I don’t use Docker.
- jeremija 7y agoIIRC, if you use docker swarm it wil handle the routing for you. So if an app is running on port 8080 on two swarm nodes, if you access port 8080 on each node, you might actually be accessing another node. Since docker swarm knows if an instance is down, it will know to not use that instance. See https://docs.docker.com/engine/swarm/ingress/ https://docs.docker.com/engine/swarm/ingress/
- yashap 7y agoVery curious about what kind of service has these requirements! Servers in 50 different cities suggest you likely care A LOT about latency. Going bare metal likely means you’re dealing with huge amounts of computation. But tolerating 2-5 mins of downtime on every deploy (until now) means availability isn’t a big deal. I’ve never seen this combo of requirements on any service before, what is it? Or are my assumptions just wrong about what these things suggest about your business?
- ggregoire 7y agoWe detect and monitor commercials airing on terrestrial TV networks. :)
- cfors 7y agoPretty cool stuff. I'm assuming you have to set up Antenna's where you have your servers colocated and monitoring the transport stream's going through? Obviously you don't have to spill the company sauce but I'm trying to imagine the benefits of having all of this data.
- mherdeg 7y agoCool, so, looks like the product generates daily reports for advertisers about performance of ad spots over a trailing 1 day, lining up info about (a) what ad ran in what markets at what times plus (b) what increase in mobile/web traffic/conversion events happened from those markets coincidental with advertising, if I understand https://www.decidata.tv/advertisers.html https://www.decidata.tv/advertisers.html right. Looks like a complementary product (maybe running as a second process on the same nodes, or maybe running as part of the logic of the thing that parses the video) also produces quality reports about aired content ( https://www.decidata.tv/operators.html https://www.decidata.tv/operators.html ). I don't really know video very well, probably some very clever stuff you can do here to parallelize the processing. So hmmmm, a few things to think about for the tooling: () How quickly can you notice a single-node service outage and react / restore service? () How much downtime is ok? Is it ever ok to fail to process data for an entire day for one of the ~50 markets (can you just say "this report is incomplete" and reprocess it later)? How much wiggle room do you have (how long does it take 1 instance of the application to do its local processing job on 24 hours of data, and how long does the downline data pipeline have?) () Are you responsible for the part that watches TV + saves it to disk, and ALSO the part that batch-processes the saved video and produces the ad metadata / quality metadata? Are those different services deployed separately? () Yikes, is there, like, a person on call in each of 50 places who can unplug the thing that's watching TV and plug it back in, if a wire burns out or if the machine that's watching TV fails? That sounds like it could be an operational nightmare but maybe this is a solved problem / maybe you can buy watch-TV-and-save-it-to-disk as a service? () How do you make sure that every node is running the same version of the software with the same version of all the configuration? () How do you release new versions of the software? Do you (a) have a non-production environment which runs side-by-side at every node, processing the same data, and measure that the output is equally correct or better, the performance is the same or better, and there are no new errors or failures -- then promote the non-production code to production automatically after a while? Or do you (b) deploy a "canary" to one of the 50 nodes and watch how it performs for a while, then deploy to the rest if they all behave okay? Or do you (c) just ship the latest code to all 50 nodes on Friday nights when no one's looking, and check some health metrics over the weekend, and roll it back on Monday if a metric looks bad? () How do you keep track of what software versions you released when? If someone asks "What changed on 'X date'" does your tooling let you tell them pretty quickly and pretty accurately? () If you discover a bad bug (the parser is corrupting data; the new parsing job has a slow memory leak and fails catastrophically after 21 days of uptime) and you need to make a change in a hurry, do your runbooks let you make that kind of change quickly? Doesn't seem like you really need k8s or ansible or chef or whatever for this, as much as you need to write down your operational requirements and decide what technology will meet them.
- wmf 7y agoThis sounds like Chick-fil-A's use case where they decided to put a k8s cluster in each restaurant. https://medium.com/@cfatechblog/edge-computing-at-chick-fil-a-7d67242675e2 https://medium.com/@cfatechblog/edge-computing-at-chick-fil-... https://medium.com/@cfatechblog/bare-metal-k8s-clustering-at-chick-fil-a-scale-7b0607bd3541 https://medium.com/@cfatechblog/bare-metal-k8s-clustering-at... If you have a single node not a cluster in each location then k8s buys you a lot less. Some people still advocate single-node k8s "clusters" for benefits like rolling deployment.
- 19ylram49 7y agoI’m still stuck on the Chick-fil-A part. Haha.
- jbigelow76 7y agoIt's handy to have all those k8s instances scale to zero on Sundays.
- GhettoMaestro 7y agoI am fairly certain the second the old dude dies they are going to be like "Motion to open on Sundays..." in the board room.
- mr_cyborg 7y agoS. Truett Cathy (who I assume is the old dude you mentioned) passed away in September of 2014.
- GhettoMaestro 7y agoAh I did not know that. I figured he was still hanging out and getting a kick out of refusing to open on Sundays, given how much of meme it has become.
- merb 7y ago
- yongjik 7y agoEchoing others, you really don't want Kubernetes here. I only have limited experience, but Kubernetes works very hard to abstract away the actual machines, and it works best that way: you just say "Deploy 20 instances of job X", and k8s will somehow find the machines to deploy them. You don't care where they are running - k8s handles that. Once you start to care about actual machines and which jobs are running where, k8s starts to make less sense. You're paying for huge complexity (required to abstract away the machines), but you're getting none of the benefits - it just becomes a glorified wrapper around Docker daemon.
- thawkins 7y agoDocker in production is a potential disaster, it runs everything as root from the daemon, we are heavy k8s users, and we are transitioning to rootless cri-o containers specificaly because of this. Investigate podman which is a docker cli clone that builds cri-o containers from docker files and allows you to run them under a designated user id, the only restriction is that in rootless mode you cant map ports that are less than 1024, for that you need to run them under sudo which defeats tbe purpose. Howver we use an ingress controller to map inbound requests to containers so it can do the port mapping. We used to use traefik, but now we use nginx as ibgress, becuase we can run the excellent lua-waf module. We run on multiple clouds, so vendor apis and k8s implementations are not usefull to us.
- deleted 7y ago[deleted]
- MuffinFlavored 7y agoI struggle to understand when Docker is supposed to be introduced to a stack. If you have a 100 user SaaS platform, do you run Linux containers on top of Linux just for isolation?
- inapis 7y agoIf you need strong guarantees of isolation and if every user has their own dedicated installation. If you are a general service where isolation is enforced at the application level then just deploy straight away. Docker is useful when you don't want to care about the underlying operating system or you might be deploying on different operating systems etc. Also if you want to quickly switch providers (like going from Ubuntu 18.04 on digital ocean to Windows Server on Azure) since it's a self contained image. This self contained image is useful during development too. I can cleanly separate dependencies between applications and dev/test/prod. For example I develop a node app based on node 12 + pg 11 but I tinker with ghost too which requires node 10 + mysql. Having this in Docker keeps cruft and complexity off my macOS installation at the cost of space. Having multiple versions of a runtime/database etc can quickly become a nightmare to manage on local operating systems. Operationally if you can control deployment and you don't need many servers, deploying to the base operating system makes sense. Docker/Kubernetes shine when you have scale + need to provision many servers across different platform providers.
- halbritt 7y agoCheck out BalenaCloud. It's a container orchestration service for edge devices.
- gchamonlive 7y agoYou could also go full docker and experiment with docker swarm. I have little experience with swarm itself, but AWS ECS has deployment strategy. You could say you want 50 container instances on a fleet of 50 machines with deployment strategy of spreading across all machines and you would guarantee that every single machine is populated with a docker container. Maybe try to look for a similar functionality thinking about you bare metal machines as a single fleet of instances. Do you manage your containers configs with environment variables? I think docker also let's you share the hosts environment with the docker container, so you can pre provision the server with the necessary information and once the swarm is deployed, the container figures out what to do with the hosts env. Applying that to the spread across machines deployment strategy and you solve your problem.
- dgoog 7y agoAnsible all the way. I'm really upset with the way Google let their marketing team run roughshod all over the place with that software. Kubernetes is almost never the tool to use. It's entirely insecure, overly complicated and almost never fits the intended supposed benefit. Worse it feels that the entire CNCF ecosystem is ran entirely by marketing people with "developer evangelists" that have never coded a single line of code in their day - it's a real shame and quite honestly an insult to professional engineers.
- MuffinFlavored 7y agoI don’t understand when to use Ansible, Puppet, Chef, or Terraform...
- wmf 7y agoTerraform: First you use this to create VMs. Ansible/Puppet/Chef/Salt: Then you use this to install your stuff in the VMs. Just pick one of these and stick with it.
- jordanbeiber 7y agoInstall k8s with a/p/c/s. :) K8s is not about configuration management, it’s about dynamic application management. Some parts infringe on areas where CM tools work as well, but k8s is all about managing containerized applications. Trying to set up flows for ”works on my laptop”-dev, ci/cd, loadtesting, a/b tests, canary releases, autoscaling and rollbacks for multiple teams of devs? K8s really simplifies these things. The idea is that you have one api spec to rule the whole stack (from a dev perspective). If you go down a more light-weight stack, a lot can still be achieved. More duct tape required though. That being said- I love duct tape!
- gerbilly 7y agoAnsible can be used to create VMs too. I used to use it to provision AWS instances and to configure them after. https://docs.ansible.com/ansible/latest/modules/list_of_cloud_modules.html https://docs.ansible.com/ansible/latest/modules/list_of_clou...
- deleted 7y ago[deleted]
- ninetax 7y agoI've used SaltStack in a similar setup with some success, it's easy to keep all the servers in sync and prevent configuration drift.
- jordanbeiber 7y agoYou could rather easily manage this flow with ansible. K8s abstracts a lot of the pieces in your ansible plays away, but comes back at you in form of a rather complex stack. Say you want to host instances of this app in a DC/cloud, k8s would be more appropriate, and perhaps then it would make sense to go single node “micro-k8s” style (for single server locations), to give you a common api for managing deployments, network configuration, security etc. Win with k8s is that once you’re tuned, a lot of “config crap” is abstracted away, basically. But - there's a reason ansible is supported specifically through the “operators” SDK, something we use to manage seamless upgrades of a k8s deployed app: https://github.com/operator-framework/operator-sdk/blob/master/doc/proposals/ansible-operator.md https://github.com/operator-framework/operator-sdk/blob/mast... Edit - forgot that if meteics based autoscaling of the app is a requirement my experience is that k8s makes this a breeze. Probably not a req if using single node hosts though...
- mattbillenstein 7y agoI haven't used ansible in a few years, but I sorta loathed the transport after a time with even dozens of boxes -- constantly creating ssh connections was slowing things down significantly. I switched to saltstack some time back and it's much better while having a similar sorta interface. The docs are worse than ansible imo tho. ymmv
- pferde 7y agoYou shouldn't be creating ssh connections all the time, but instead using persistent connection to each server that ansible opens once at the beginning, and reuses for all subsequent tasks. See ControlPersist/ControlPath OpenSSH client configuration.
- mattbillenstein 7y agoNever found this incredibly reliable - and if you change networks, sleep, or power off your machine, those connections die. Salt is just better at all of this imho.
- mattbillenstein 7y agoI mean, I guess if you use a dedicated host, it's more reliable - but I've always just pushed from the client using Ansible whereas with Salt you have the master running the cloud running somewhere most usually.
- pferde 7y agoDo you often do these things in the middle of running a playbook? I would hope not. :) It's not meant to have the connections persist forever, just long enough for repeated short ssh connections to have almost no overhead, and therefore be much faster.
- mattbillenstein 7y agoYeah, good point.
- speedplane 7y agoKubernetes is complicated, it's a whole new world of terminology and abstractions that you need to learn, and it changes quickly. That said, maintaining a large cluster is complex too. Often, there is much more involved than making sure each image's config files are up to date. Kubernetes shines when you have a number of different docker images, where each needs its own unique network, scaling, cpu, and disk settings, i.e., hardware requirements that can't really be controlled with just config files. In your case, the fact that all of your images are identical probably cuts against using Kubernetes. However, the fact that these images are running on different networks where each may require their own tweaking leans towards it. If you expect to start adding many different docker images to your cluster, with different hardware requirements, it's probably worth the learning curve to get on to Kubernetes. Disclaimer: I've never run Kubernetes on bare metal, just hosted environments. Bare metal may make Kubernetes much less attractive.
- OJFord 7y agoSummary: 50 production instances of a single application. No, you don't need Kubernetes (though fwiw if you used it what you'd have would be 50 nodes with a single daemonset, 1 pod of the app per node, if that helps understand) - but I'd suggest not using Ansible. Instead, use Packer (or similar) to create a machine image/snapshot/whatever your server provider calls it, and then deploy that same image on all 50. You might have some small amount of host-specific things left to configure, and by all means use ansible for that if you want, but there's no need for error-prone running of the same large playbook 50 times. Personally, I find that final setup is little enough to do it with Terraform (which is provisioning the servers anyway).
- hackermailman 7y agoI do this on ~30 servers with GuixSD, so there is a single file to configure all of them. To update the application I wrote a Guix pkg for it that pulls from our repository (simple to set up with Guix) everything is controlled in emacs by me to change/start/stop all the servers. https://news.ycombinator.com/item?id=17084561 https://news.ycombinator.com/item?id=17084561 is similar to what I'm doing. Caveat: you have to go on google scholar to read guix and nix papers to understand the system architecture, then their documentation also downtimes aren't an issue for us since these are internal servers, and this is done during non critical time but it's usually just restarting the new app so a few seconds. Other caveats, security fixes we just edit in changes not run guix pull but I also have that automated now after mailing list help. You may have totally different req otherwise look into nix/guix, Guix the pkg manager runs on any distro.
- peterwwillis 7y agoIs there a really abbreviated reference to all of core kubernetes? Like, one page with just all the categories, services, objects, attributes, schemas, etc, for every component. I know it would be 1000 pages long, but it would allow quickly skimming as a reference, as well as getting a very quick low-level glance at everything involved. Guides like OP's are great to get started, but I want to know that weird_flag_that_almost_nobody_ever_sets exists in random_service_parameter.
- rochacon 7y agoSomething like this? https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.16/ https://kubernetes.io/docs/reference/generated/kubernetes-ap...
- Yen 7y ago+1 for the reference documentation. I've often been able to write complex and correct manifests, just by recursively following the links and filling in the YAML as appropriate.
- rochacon 7y ago`kubectl explain` is also very nice.
- blondin 7y agoi mean reference documentation is nice if you understand the core concepts. my struggle is with grasping those core concepts and how they all fit together to solve the orchestration problem. for now, a few books (kubernetes patterns is the current one i am reading) and articles are helpful. i would like to note that some of these concepts had their names changed overtime. i suppose there will be more changes as the platform stabilizes. mind you guys that kubernetes is only 5 years old...
- peterwwillis 7y agoYes! If this could be mashed up with different levels of diagrams I think this would be extremely useful for visualizing the components
- th3iedkid 7y agoIs mesos dead ?
- nih0 7y agono, It's very much alive, there's just no hype around it anymore
- Apocalypse_666 7y agoAfter having just spend most of the day yesterday trying to nurse a failing Kubernetes cluster back to health (taking down all of our production websites in the process), I’ve come to completely loathe it. It felt more like practicing medicine than engineering: try different actions to suppress symptoms, instead of figuring out a root cause to actually prevent this from happening again. While it is a pretty sweet system if it does run, I would strongly advice against anyone trying to manage their own cluster, as it is simply to complex to debug on your own, and there is preciously little information out there to help you
- deleted 7y ago[deleted]
- reacharavindh 7y agoWould you say that you would be happier if you put a bunch of websites/web applications behind a load balancer like haproxy - all in a few VMs or even bare-metal servers instead of taking on the complexity of Kubernetes? No snark or pushing opinion; I’m genuinely wondering how it is from someone who went through this path. As a sysadmin who cares more about the reliability of services, still managing critical services outside of Kubernetes, I’m wondering what I’m missing out with Kubernetes.
- Apocalypse_666 7y agoInfinitely happier, because if something goes wrong, you can usually figure out what it was, and fix it or even prevent it from happening again. Sure, the blue-green automatic deployment in k8s is cool, but a bit of clever Ansible scripting should get you there as well. It might be more busywork, but the amount of time spent nursing my k8s cluster in no way amount to time saving
- Apocalypse_666 7y agoI wrote this whole thing below as a reply to someone stating that I should just stop complaining and figure it out, but that comment has since been deleted. Figured sharing my frustration might be cathartic anyways :) If only I could!! That’s exactly the frustrating part: there seems to be no way of grokking what goes on under the hood, and there are so many different ways of setting up a cluster and very few have any information about them online whatsoever. As a practical example, what happened yesterday was that all of a sudden my pods could no longer resolve DNS lookups (took a while to figure out that that was what was going on, no fun when all your sites are down and customers are on the phone). Logging into the nodes, we found out about half of them had iptables disabled (but still worked somehow?). You try to figure out what’s going on, but there’s about 12 containers running in tandem to enable networking in the first place (what’s Calico again? KubeDNS? CoreDNS? I set it up a year ago, can’t remember now...) and no avail in Googling, because your setup is unique and nobody else was harebrained enough to set up their own cluster and blog about it. Commence the random sequence of commands I’ll never remember until by some miracle things seems to fix themselves. Now it’s just waiting for this to happen again, and being not one step closer to fixing it
- antocv 7y agoThis is yet another one of those "introductory" articles. The whole field it seems everywhere is filled with "introductory" "gentle" books/articles, and then "this is how you do reusable rocket science with x". Pro tip: to understand kubernetes in between, go read the manual pages of Linux networking and get a really good grip on iptables. Go read the manual pages for linux namespaces, cgroups and containers with lxc. Why dont people get the basics of the parts of the tool they are going to use, first, instead of trying to "understand a tool"? You wont succeed doing anything with kubernetes, if you come from say macosx or windows envrionment, and have no clue how/what kubernetes is built on.
- pjmlp 7y agoBecause Kubernetes is the new NoSQL, there are consulting services, conference talks and books to be sold.
- StreamBright 7y agoAnd SREs to resist. :) I refuse to put Kubernetes into our infra.
- baq 7y agoI wanted to, then I read about it.
- StreamBright 7y agoSure, I know iptables, namespaces, cgroups and containers and lxc. I stopped right here and did not bother to learn Kubernetes because all of this knowledge gives me enough to run anything I want, websites, applications, hadoop clusters and so on without it. I can debug my stack and resolve any issues with confidence, have upgrades and do autoscaling (using AWS). Why should I even think about Kubernetes? Now the real question is, if you need to learn all these to run Kubernetes than is it really tool they are trying to sell it is? One of the promise I often hear is that you don’t need systems knowledge or SREs to run it. Yet I read these stories and many customers reach out for help because they have an undebugable problem in production and nobody can help. I learned in the last 20 years working as a systems engineer that the best tools or stacks are ones easy to understand, debug and fix. Black box computing is the worst, when things happen for no good reason and it is hard to find out the root cause.
- vasco 7y agoThere are advantages to using minikube for local development if you are interested in replicating and testing the whole flow. I think these are most important for small teams of 1-5 people where everyone needs to understand the whole infrastructure stack and regularly performs changes down to the kubernetes resources. That said, I've found that for bigger teams where you have a few people concerned with keeping k8s stable, and most people developing the actual application logic, docker-compose is still the most no-frills-easy-to-understand option for local development.
- mfbx9da4 7y agoI hate the fact that most of my time is spent configuring rather than building.
- kn100 7y agoI've seen a deployment of a large number of ec2 instances + other Amazon services handled with Puppet+Ansible, and I've also seen a somewhat smaller but still fairly sizeable deployment using a custom Kube cluster. The custom Kube cluster was cool and definitely seemed to make those working with it happy, but I was much happier working with the Puppet+Ansible bare metal setup. It's just easier to figure out what's wrong. I am now working with hosted Kubernetes, which seems to be much less of a mental headache.
- skboosh 7y agoA major problem I've seen using Kubernetes is it's difficult to bring up an entire cluster from scratch, play with it and tear it down. So I wrote Sugarkube [1] which lets you launch an ephemeral cluster, install all your applications into it (including creating any dependent cloud infrastructure) and tear it down again. This means each dev can have their own isolated K8s cluster that's in parity with production. And you can release the same code through different environments. In fact your prod environments can become ephemeral too - instead of doing complicated in-place upgrades you could spin up a sister cluster, direct traffic to it and then tear down the original. Or you could spin up a cluster to test the upgrade before repeating it in your live environment. Based on my own experience I believe ephemeral clusters can solve a huge number of problems dev teams face using Kubernetes. Sugarkube currently supports launching clusters using Minikube, EKS and Kops and we'll be adding provisioners for GKE and Azure in future. Sugarkube also works with existing clusters, so you can use it deploy your applications. It's a sane way of managing how to ship infrastrucuture code (e.g. Terraform configs) with your applications that need them. Sugarkube also supports parameterising applications differently per environment - you could almost view it as something like Ansible but that was written with Kubernetes in mind. And it's in Go so it's way faster than Ansible (an early POC was actually written in Ansible and it was very slow). I've just finished intro tutorials for deploying Wordpress to Minikube [2] and EKS [3]. I'd be keen to hear feedback. We just tagged our first proper release earlier this week and it's ready to try now. [1] https://docs.sugarkube.io/introduction/ephemeral-clusters/ https://docs.sugarkube.io/introduction/ephemeral-clusters/ [2] https://docs.sugarkube.io/getting-started/tutorials/local-web/ https://docs.sugarkube.io/getting-started/tutorials/local-we... [3] https://docs.sugarkube.io/getting-started/tutorials/dev-web/ https://docs.sugarkube.io/getting-started/tutorials/dev-web/
- mcstafford 7y agoThis sounds like one of those "Why didn't somebody do this sooner?" moments. TYVM
- vitalus 7y agoI definitely agree with you - my team switched to an "ephemeral cluster" model which allows us to very quickly spin up an entirely new cluster and drain traffic to it as needed. It's something that we've ended up implementing on our own with a lot of Terraform, but that's had its own obstacles and is something of a small maintenance burden. I'll be taking a look at sugarkube!
- MichaelMoser123 7y agoI have a short text on kubernetes operators. https://github.com/MoserMichael/cstuff/releases/download/kbook-upd/kubernetes-book.pdf https://github.com/MoserMichael/cstuff/releases/download/kbo... learning this stuff got me something of an overview of what is going on there.
- auslander 7y agoA generic engineering rule: fight complexity, a.k.a KISS principle. It takes additional effort and courage. You'd need to fight for that project time too. You'll live longer (and your systems too), but probably on your next job :)
- domenukk 7y agoTried to set up a Kubernetes cluster with IPv6 the other day. These hipster tools don't care about IPv6 :(
- deleted 7y ago[deleted]