13 ms·

Kubernetes at GitHub

413 points by darwhy 9y ago

dookahku 9y ago
Any favorted training for learning Kubernetes? I found this one so far: https://classroom.udacity.com/courses/ud615 https://classroom.udacity.com/courses/ud615 But any extra courses/trainings is always appreciated
- Omnipresent 9y ago
  I'm interested in this as well. I found Hepito https://www.heptio.com/support-services-and-training https://www.heptio.com/support-services-and-training
- pdelgallego 9y ago
  Udemy: https://www.udemy.com/learn-devops-the-complete-kubernetes-course/ https://www.udemy.com/learn-devops-the-complete-kubernetes-c... Pluralsight: https://www.pluralsight.com/courses/getting-started-kubernetes https://www.pluralsight.com/courses/getting-started-kubernet...
  - timrichard 9y ago
    I'm got that Pluralsight one earmarked for 'soon'. I thought Nigel Poulton's Docker courses were excellent, so looking forward to it...
DDub 9y ago
We're currently looking at moving our applications to k8s, and was wondering what deployment tools people are using? This week we are evaluating spinnaker, helm and bash wrappers for kubectl. There is concern over adding too many layers of abstraction and that KISS is the best approach.
- dguaraglia 9y ago
  I feel you. About a month ago I was fighting with the same feeling. In the end, decided to use Kubernetes only for a single piece of infrastructure so it's all pretty manageable through scripts. Managing secrets in particular is a pain in the ass. One route I started checking but didn't commit to was using Ansible. They have a relatively good Kubernetes playbook and a facility to store secrets. That said, every damn task needs to be pointed to the K8S API endpoint, which is not the greatest.
  - smarterclayton 9y ago
    Agree - we've been talking about how we can more natively tie the inventory into clusters, contexts, and apps. The host focus of Ansible doesn't always map to other domains, but I think it has a real chance with Kube.
- MattRogish 9y ago
  We (ReactiveOps) use a combination of CircleCI and some scripts wrapping kubectl https://github.com/reactiveops/rok8s-scripts https://github.com/reactiveops/rok8s-scripts
- jameskilton 9y ago
  I've got a simple setup that makes use of YAML files, Rake tasks, and raw kubectl. I've yet to take a look at helm or spinnaker but it's on my list. You really can go a long way just with K8s' own tooling.
  - jomakasi 9y ago
    I've started with creating Rake tasks. Then I found https://github.com/CommercialTribe/psykube https://github.com/CommercialTribe/psykube which is opinionated. Then I've ended up creating my own simple ruby tool to manage kubernetes with my own directory structure and configuration.
  - danielodio 9y ago
    We've written a bunch about Spinnaker & K8s at http://go.Armory.io/kubernetes http://go.Armory.io/kubernetes ; hope that's helpful!
- foxylion 9y ago
  We also did some evaluation and then decided to stick to KISS an chose kubectl commands combined with cat and kexpand. Really simple approach to allow dynamic kubernetes deployments. Example command can be cat service.yml | kexpand expand -v image-tag=git-135afed4 | kubectl apply -f - The service.yml contains the full deployment configuration, service definition and ingress rules. So this works without preconfiguring anything in kubernetes when deploying an new service. An engineer only has to create the service.yml and Jenkins does deploy it automatically on every master build. *kexpand is a small tool which does something similar to sed, but in a simpler and less powerful way (keep it simple): https://github.com/kopeio/kexpand https://github.com/kopeio/kexpand
  - nikon 9y ago
    I do something similar, but envsubst does the job.
    - ff_ 9y ago
      +1 on envsubst, it's the minimal solution to the problem of templating kubernetes (of course YMMV, we are a small team and don't need more complex stuff)
    - brianwawok 9y ago
      Ditto me too to get my git sha into my image names and deploy with deployments.
  - howinator 9y ago
    We basically did the same thing except we used Ansible for our templating. This allows us to store all our shared "environment configuration data", e.g., name of RDS for services in prod environment, name of backup S3 bucket for services in dev environment, in an Ansible role then just pull that information into our templated deployment manifest file. So far, it's worked out pretty well for us.
- deadmik3 9y ago
  I'd recommend looking into openshift. it's basically kubectl + cool deployment features. there's also free, paid, and dedicated online hosted options. disclaimer: I work on openshift
  - awill 9y ago
    You had a big announcement about openshift.io. Everyone on HN signed up, but it's been months, but I'm still 'awaiting approval' Have you let anyone in? What's the value in all the marketing hype if you then don't let people in.
    - yebyen 9y ago
      Seconded. The last e-mail I got from OpenShift.io indicated they haven't let anyone in. I've already practically lost interest. It was evidently only "coming soon" but that announcement really looked like "coming tomorrow." The original OpenShift Developer Preview made you sign up, but you would be allowed into the platform within hours or days.
      - deadmik3 9y ago
        OpenShift.io is a different product than hosted OpenShift on its own. If you were to sign up for the Pro tier at openshift.com your account would be immediately provisioned. We are working on provisioning enqueued users for our Starter tier as well but again, these are separate from OpenShift.io
        yebyen 9y ago
        Yeah, OpenShift Pro is also quite expensive... starting at $10k for 5 nodes right? I just checked out your pricing page and I'm 100% wrong about this. You have a $50/mo tier now! That's fantastic, thanks for pointing me at that. You should definitely spam everyone on the OpenShift.io waiting list and let us know about your new pricing /s No seriously, I would have liked to get some spam about this. Maybe you sent it and I missed it. I am a lot more interested at $50/mo than I would have been at $10k/yr!
- bryanlarsen 9y ago
  We're using helm, but that was chosen mostly based on gut feel. It's an official project, and has momentum. We didn't want to spend too much time choosing a tool until we knew our requirements, and we don't really have a firm grasp of requirements until we've used something for a while. It seems to be working well for us so far, but it's still early. There are lots of answers here that aren't helm, so I'm curious if there are any particular reasons that people ruled out helm?
  - nikon 9y ago
    Are you doing what Gitlab claims below "We (GitLab) use it mostly as a templating system as well" or really using it to manage complex apps? I think it doesn't really offer much for normal microservices. I do use it for things like nginx ingress but for stuff I've built a service.yml / deployment.yml are fine.
  - henridf 9y ago
    We ran into a number of issues with Helm when deploying - failures leading us to have to rollback, with rollbacks then failing, requiring manual changes to unblock. I think that for third-party packages and related templating (which seems like the original use-case) it works well, but I would be wary of using it for high-res deploys of our own stuff.
- yebyen 9y ago
  You should take a look at Deis Workflow. I say this in spite of the fact that it was announced last week[1], the next release of Deis Workflow will be the last (under the current stewardship, and probably under that name.) It's just such a solid system, I would even more strongly recommend the (already EOL'ed early last year)[2] Deis v1 PaaS, except that you've already indicated you're moving to K8S, and Deis v2 is designed for K8S. I still recommend the v1 PaaS for people learning about principles of HA clusters. (Another disclosure: I have published[3] about how to do this, a work on how to do a cheap HA cluster using Deis v1 PaaS.) I have a strong suspicion that Deis will live on after March under stewardship of new leadership from the community. In the mean time, you have roughly 6 months of support from Microsoft, maybe I am overstating to say that they have committed to keeping the lights on for that long, but they have committed to merging critical fixes for that long (and we hope that in 6 months, Kubernetes will have solidified enough that we don't have to worry too much about breaking changes from upstream release mongers anymore.) Personally I don't buy commercial support and it would not be the deal maker or breaker for me. [1]: https://deis.com/blog/2017/deis-workflow-final-release/#future https://deis.com/blog/2017/deis-workflow-final-release/#futu... [2]: https://deis.com/blog/2016/deis-1-13-lts/ https://deis.com/blog/2016/deis-1-13-lts/ [3]: https://deis.com/blog/2016/cheapest-fault-tolerant-cluster-deis/ https://deis.com/blog/2016/cheapest-fault-tolerant-cluster-d...
  - gtirloni 9y ago
    I'd advise against choosing core infrastructure components (that have a clear EOL deadline) based on strong suspicion. Even more so in a landscape that's constantly changing like Kubernetes. You have zero guarantees that it'll be maintained and will keep up with new breaking changes.
    - yebyen 9y ago
      You know it's open source, right? I have zero guarantees that any of my open source projects that I use for business critical infrastructure aren't going to pack up shop and quit maintaining their stuff tomorrow. You should know how your infrastructure works well enough to maintain it for yourself. I (personally) will be maintaining this one in the future, if necessary! We're working it out now. What do you mean by "strong suspicion?" Please don't downvote because you read a few words you didn't like, I was upfront about this EOL date because I don't want it coming back later that I was dishonest about it, but my perception is not that "EOL" means it's dead, it is that "EOL" means it's done. Stability is a good thing. Microsoft also EOL'ed MSPaint.exe, and I remember how the community reacted. I think the quote was about "works for 99% of users and has been stable for over a decade? sounds like a good candidate for deletion!" The project is cancelled because it's not strategically important to Microsoft, not because it's not viable or having technical issues. The core devs have chosen to work on more kubernetes-native tooling. They aren't abandoning Kubernetes, and I'll bet you don't have a competing product you can show me that has guaranteed to keep the lights on for the next 6 months.
- lobster_johnson 9y ago
  We wrote an internal tool that wraps Helm and GPG. But we're really using Helm as a glorified templating system; since we deploy from git, Helm's release management is useless to us, and is even somewhat in the way. We might decide to drop Helm at some point, I think.
  - xur17 9y ago
    We went down a similar path and ended up using helm-template [0] to render our helm charts without tiller. We also use an internal tool that: - maps applications to namespaces within clusters for different environments (since we have 1 cluster per environment) - does some magic around secrets to make them easier to interface with [0] https://github.com/technosophos/helm-template https://github.com/technosophos/helm-template
  - Snappy 9y ago
    That's a good point. We (GitLab) use it mostly as a templating system as well. It's a step up from piping `sed` output to `kubectl`. But we have our own tools for managing redeploys and rollbacks.
- sytse 9y ago
  If you're using our GitLab consider using Auto Deploy. Our CTO recently made a quick start guide for it https://docs.gitlab.com/ee/ci/autodeploy/quick_start_guide.html https://docs.gitlab.com/ee/ci/autodeploy/quick_start_guide.h...
  - Snappy 9y ago
    FWIW, the next version of GitLab's Auto Deploy will use Helm under the hood (and let you bring-your-own-chart).
- majewsky 9y ago
  At SAP, we're using Helm [1] to deploy OpenStack (plus a bunch of extra services like a custom Rails-based dashboard [2]) on baremetal Kubernetes. For image building, testing and deployment, we use Concourse CI [3], and OpenStack assets (like service users, service projects and roles) are deployed with a custom Kubernetes operator [4]. Our charts are at [5] if you want to take a look. [1] https://github.com/kubernetes/helm https://github.com/kubernetes/helm [2] https://github.com/sapcc/elektra https://github.com/sapcc/elektra [3] https://concourse.ci https://concourse.ci [4] https://github.com/sapcc/kubernetes-operators https://github.com/sapcc/kubernetes-operators in the "openstack-operator" directory [5] https://github.com/sapcc/openstack-helm https://github.com/sapcc/openstack-helm and https://github.com/sapcc/helm-charts https://github.com/sapcc/helm-charts (two different repos since we're in the middle of modularizing the original monolithic OpenStack chart in the first repo into per-service charts in the second one)
- invisible 9y ago
  We are using ecfg (from Shopify) and Jenkins+kubectl. We use ansible for a couple of things but it's largely only because of some parameters of our architecture for a legacy monolith.
- theptip 9y ago
  A lot of folks are using Helm, but I find it very opaque to debug when templates go wrong (and I feel quite strongly that we shouldn't be writing untyped templates for our models). Also I found writing reusable spec components to be very difficult, e.g. a reverse proxy that I add to a number of pods. I use pykube (also worth looking at the incubator project client-python) to write deploy scripts in Python; client-python is particularly nice as it uses OpenAPI specs to give you type hinting on the API objects/fields that you're writing. Much more civilized than typing bare yaml into a text editor. If Python isn't your thing you can generate your own client from the OpenAPI specs, though I've found the client generation process to be a bit buggy.
- rubenv 9y ago
  I recently spoke about the approach we use at Ticketmatic: https://rocketeer.be/articles/coreos-fest-2017/ https://rocketeer.be/articles/coreos-fest-2017/
  - kalev 9y ago
    That must have been an awesome talk. Thanks for the write-up!
shock 9y ago
> During this migration, we encountered an issue that persists to this day: during times of high load and/or high rates of container churn, some of our Kubernetes nodes will kernel panic and reboot. Considering that Kubernetes doesn't modify the kernel, this issue sounds like is present in mainline and kernel devs should be involved.
- cyphar 9y ago
  I would be interested to know what storage driver they're using for their nodes. High container churn puts a lot of stress on the VFS subsystem of Linux, and we've seen cases where customers have trigger lots of mount/umounts which results in filesystems causing panics. At SUSE, we do have some kernel devs debugging the issues, but the workaround is almost always "rate limit all the things". There are a few other kernel areas that are stressed with high container churn (like networking), but VFS is the most likely candidate from my experience. While on paper containers are very lightweight, spawning a lot of them exercises kernel codepaths that probably haven't been exerted to that type of stress during development.
  - AaronBBrown 9y ago
    Hey, this is Aaron from GitHub. We're using devicemapper w/ LVM backed pools. Would love to hear about your experience there. We definitely see this problem during periods of high container churn.
    - cyphar 9y ago
      That's funny, we have an internal bug open right now about a kernel panics that happen with devicemapper (with XFS as the base filesystem). We found that the issue was exacerbated if you used loopback devices, but on paper it should still happen in non-loopback mode (the current theory is that it's a bug in XFS). Our kernel team is still investigating the issue, but they cannot seem to reproduce the issue with direct-lvm (and loop-lvm is inconsistent in reproducing the issue). If you can consistently reproduce the issue, would you mind providing the backtrace and/or coredump? Is it possible for you to reproduce the issue on a machine without needing to be hit by GitHub-levels of traffic, and if so can you provide said reproducer? For reference, our backtraces show that the kernel dies at Xfs_vm_writepage. Though of course different kernel versions may have varying backtraces. You can reach me on the email in my profile, or asarai(at)suse.com.
Yeroc 9y ago
One thing I would have liked to have seen addressed in the article is whether the new architecture requires additional hardware (presumably) to operate and if so how much more.
- ben_jones 9y ago
  I've only dabbled in K8s and it strikes me that using it in production is a long term investment and, as it stands currently, a long term project to implement properly. You'll want to do exactly what Github did: setup a "review lab" or similarly comprehensive dev and test environment until you are absolutely comfortable with it in production. This will lead to the provisioning (and cost) of quite a bit of hardware - and when it is finally in production it'll likely be over-provisioned for quite some time until norms can be established and excess cut. So basically its a traditional devops migration. But you get quite a few goodies and arguably much better practices at the end of it.
  - majewsky 9y ago
    I agree very much, and I'd like to add one point: When you build a lab environment for testing Kubernetes deployments (and verifying Kubernetes upgrades), make sure it's on the same hardware as your production environment. When my team did the first Kubernetes deployment, we made the mistake of building a lab environment that did not match the anticipated production environment. (Two reasons: The BOM for the production environment was not yet decided upon at that time, and the lab was frankensteined together by taking hardware out of existing labs.) We learned the hard way that, just because the Kubernetes upgrade worked in the lab, it need not work on the production hardware. Right now, we're stuck on last year's (i.e., ancient) Kubernetes 1.4 release because no one dares to upgrade production. (There's light at the end of the tunnel, though. A new lab is being built up in the datacenter around now.)
  - joshuatalb 9y ago
    If you're on AWS you can use kops (https://github.com/kubernetes/kops https://github.com/kubernetes/kops) to significantly reduce the amount of time to get a cluster up. It took me about 1hr to get a basic k8s cluster up and running with it.
lobster_johnson 9y ago
I'd be interested in hearing what kind of autoscaling system they use for their Ruby pods. We're running a few (legacy — we're moving to Go) Ruby apps in production on Kubernetes. We're using Puma, which is very similar to Unicorn, and it's unclear what the optimal strategy here is. I've not benchmarked this in any systematic way. For example, in theory you could make a single deployment run a single Unicorn worker, then set resources:requests:cpu and resources:limits:cpu both to 1.0, and then add a horizontal pod autoscaler that's set to scale the deployment up on, say, 80% CPU. But that gives you terrible request rates, and will be choking long before it's reaching 80% CPU. So it's better to give it, say, 4 workers. At the same time, it's counter-productive to allocate it 4 CPUs, because Ruby will generally not be able to utilize them fully. At the same time, more workers mean a lot more memory usage, obviously. I did some quick benchmarking, and found I could give them 4 workers but still constrain to 1 CPU, and that would still give me a decent qps.
- sytse 9y ago
  At GitLab we recommend to use CPU cores + 1 as the number of unicorn workers https://docs.gitlab.com/ce/install/requirements.html#unicorn-workers https://docs.gitlab.com/ce/install/requirements.html#unicorn...
  - lobster_johnson 9y ago
    How do you configure that? A pod doesn't know what machine it's running on ahead of time. You can create nodepools and use node selectors to pin the pod to that nodepool, but I'm not sure I love the idea.
    - ecnahc515 9y ago
      Using the downward API, your application can get the number of CPU cores or the millicore value via environment variables or volumes. This would let you configure your number of workers based on what resource limits you set on your container.
    - twk3 9y ago
sandGorgon 9y ago
> We enhanced GLB, our internal load balancing service, to support Kubernetes NodePort Services. Everyone does this - because Kubernetes Achilles heel is its ingress. It is still built philosophically as a post-loadbalancing system . This is the single biggest reason why using Docker Swarm is so pleasant.
- alpb 9y ago
  Can you elaborate? What are the issues you're seeing with Ingress?
  - mnutt 9y ago
    The service I'm working on porting to kubernetes has bandwidth requirements that far exceed a single machine, and though we're on AWS, we can't use ELB for various reasons. We ended up with a ReplicaSet of haproxies using HostPort, and a separate app that watches for haproxy service changes and updates Route53 round-robin DNS. We're somewhat fortunate that we only have a single service that needs to use the http/https HostPort. We could have just left our haproxies outside of kubernetes, and may eventually end up doing so if the network performance doesn't meet our needs. As it is, it all works but there are a ton of sidecar services all over the place.
    - yebyen 9y ago
      This sounds like what Deis Router was created to do. It sounds like you've already got it nailed down, but maybe like to have a look at this: https://github.com/deis/router https://github.com/deis/router (It's probably tightly coupled to Deis Controller, but something to look at anyway!)
- orf 9y ago
  Can you elaborate? I found Kubernetes ingresses to be one of the most pleasant parts - we use the `nginx-ingress` from helm and it works very well. Not exactly an industrial github-strength load balancer, but it will get you a long way surely?
philips 9y ago
Really exciting stuff, happy to see the Github team launch this. Kubernetes is becoming the goto for folks needing both their own physical metal presence and cloud footprint too. And the magic of Kubernetes is that it has APIs that can actually give teams the confidence to run and reuse deployment strategies in all environments. Even across clouds. If you are like Github and want to use Kubernetes across clouds (AWS, Azure, etc) & bare metal and do deploy/customize that infra using Terraform checkout CoreOS Tectonic[1]. It also tackles more of the subtle things that aren't covered in this article like cluster management, LDAP/SAML authentication, user authorization, etc. [1] https://coreos.com/tectonic https://coreos.com/tectonic
- robotmay 9y ago
  I'm still utterly perplexed as to what Tectonic actually -is-. I kinda get that it's a kubernetes setup, but is it a GUI over the top of it? The website is pretty confusing and I think I gave up really quickly when trying to set it up.
  - philips 9y ago
    Tectonic is Enterprise Kubernetes. We start with pure upstream Kubernetes at the core and install it in a production ready setup with the Tectonic Installer[1] across clouds or bare metal. On top of those basics Tectonic provides things most organizations need: - Authentication backed by LDAP/SAML/etc - One-click automated updates of the entire cluster - Pre-configured cluster monitoring/alerting There is a bunch more in there and in the roadmap too but that gives you a taste. The other thing is that we provide professional services, training, and support to customers on the whole stack from the VM or machine on up to the Kubernetes API. We have done neat collaborations with customers like the ALB Ingress Controller[2] too. [1] https://github.com/coreos/tectonic-installer https://github.com/coreos/tectonic-installer [2] https://github.com/coreos/alb-ingress-controller https://github.com/coreos/alb-ingress-controller
    - tachion 9y ago
      I'm currently deploying Tectonic flavoured Kubernetes at a large organisation and I can vouch for how great you guys are at supporting users (who are not yet customers) at any stage of the process - love that, and can't recommend you guys for that more. However, as the comment above says, Tectonic (and Quay for that matter) documentation is... just horrible and if not the engineers support, I'd be pretty much stuck on quite few things. Why don't you push your docs to a public repo, so I could do some writing and send some PR's? ;)
Alan01252 9y ago
I'm curious to what this means for the existing puppet code base, is it now irrelevant, or are there still usages for it in the k8s world?
- spmurrayzzz 9y ago
  They could easily still use standalone puppet to handle the config management for individual container images. I currently do this with salt-minion. It reduces the burden on the Dockerfile itself, and lets you embrace a declarative configuration state at build time.
  - brown9-2 9y ago
    This introduces a dependency on which pod runs on which host, unless you have puppet write the config for every service to every host. People tend to use Puppet less to configure their applications as they move into containers, as a configuration change can just be made by rolling out a new image.
  - ninkendo 9y ago
    It definitely seems like the wrong approach to me to have puppet manage your base images. They're not VM's, they shouldn't have multiple services, they shouldn't require any complex configuration management, they should just be the minimum requirements to support your application's local runtime dependencies, and that's it. From previous experience migrating from a puppet setup to one that used containers, puppet's vestigial use case ends up being to get the orchestration control plane itself setup (ie. kubernetes, networking configs, etc) and that's about it.
    - dkh99 9y ago
      There's nothing inherently about Puppet that means it has to manage multi-service "whole OS"-like installations. It can just as easily be put to the task of a Dockerfile: install dependencies and deployables for a single application. Its robust ability to manage things like user accounts, packages, scheduled jobs (e.g. for alerting, though you would have to install at least a second service for this: _crond) and the like makes it vastly superior to Dockerfile shell scripts for complex tasks. Think of puppet more like a way of simplifying your Dockerfiles to have fewer crazy shell commands in total, rather than hiding the craziness in layers and hoping it all composes properly. If you do use lots of layers, Puppet can make your life much easier, since it can be better at detecting previous layers' changes and working around them (think redundant package install commands. Even the no-op "already installed!" command takes time; if you're installing hundreds of packages--many people are, for better or worse--that can eat up build time). Puppet isn't a VM provisioner; it can also be used as a replacement for large parts of your Dockerfile, or a better autoconf to set up the environment/deps for your application to run in. Edit: syntax.
iagooar 9y ago
Love to see more Kubernetes success stories. I work for an ISP and we are trying to write another success story ;) As an ISP, we have tons of constraints in terms of infrastructure. We're not allowed to use any public cloud services. At the same time, the in-house infrastructure is either too limited, or managed via spreadsheets by a bunch of dysfunctional teams. For my team, Kubernetes has been truly a life saver when it comes to deploying applications. We're still working on making our cluster production-ready, but we're getting there very fast. Some people are already queuing up to get to deploy their applications on Kubernetes :D What I especially love about Kubernetes is how solid the different concepts are and how they make you think differently about (distributed) systems. It sure takes a lot of time to truly grasp it, and even more so to be confident managing and deploying it as Ops / SRE. But once you get it, it starts to feel like second nature. Plus the benefits, in almost any possible way, are huge.
- lima 9y ago
  Red Hat's OpenShift makes it a lot easier by providing all of the infrastructure around it (docker registry, docker build from Git, Ansible integration and so on). Best docs of all open source projects I've seen.
  - EtienneK 9y ago
    I second this. Have been PoCing OpenShift for a couple of months now and it's been a joy to use.
    - Valien 9y ago
      3rd for sure. We're a RH partner and specialize in OpenShift work. Tons of excitement with customers on using OpenShift. I love it.
    - smarterclayton 9y ago
      Thanks! Hearing this makes it all worthwhile. We love helping make Kube awesome.
- 9y ago
dmart 9y ago
> Enhancements to our internal deployment application to support deploying Kubernetes resources from a repository into a Kubernetes namespace, as well as the creation of Kubernetes secrets from our internal secret store. Would love to hear more about this was accomplished. I'm currently exploring a similar issue (pulling per-namespace Vault secrets into a cluster). From what I've found, it looks like more robust secrets management is scheduled for the next few k8s releases, but in the meantime have been thinking about a custom solution that would poll Vault and update secrets in k8s when necessary.
skewart 9y ago
> Several qualities of Kubernetes stood out from the other platforms we evaluated: the vibrant open source community supporting the project, the first run experience (which allowed us to deploy a small cluster and an application in the first few hours of our initial experiment), and a wealth of information available about the experience that motivated its design. It's interesting that the reasons they cite for choosing Kubernetes over alternatives are entirely driven by 'developer experience' and not at all technical. It shows how critical community development, good documentation, and marketing are to building a successful open source project.
- gu4x 9y ago
  I believe developer experience on being introduced to a tool is paramount to its success. It gives a lot of confidence in what you're doing and keeps things moving forward. To me appear that the application is built on a solid simple concept instead of a convoluted complex architecture. Some tools sin on the opposite though, very simple to setup but very complicated to understand how to scale.
erulabs 9y ago
If you're running Kube on AWS, make sure you install the proper drivers! For Ubuntu, that's the `linux-aws` apt package. https://github.com/kubernetes/kops/issues/1558 https://github.com/kubernetes/kops/issues/1558 Missing ENA and ixgbevf can be a real performance killer!
- joombaga 9y ago
  Is this used by vanilla docker / ECS, or just k8s?
  - TheDong 9y ago
    that advice holds regardless what software you're using, so long as the software does network traffic. It holds for just running plain old nginx websites. It doesn't really matter for small instance sizes where your networking is already rate-limited by amazon so much that ENA drivers won't matter, but on beefy instances it's always good advice to make sure you're using ENA supported driverse. See https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/enhanced-networking-ena.html#test-enhanced-networking-ena https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/enhanced...
- bmoyles 9y ago
  FWIW, the stock kernel (and HWE/HWE-edge kernels) recently picked up current ENA drivers. ixgbevf, unfortunately, doesn't look like it's been updated in-tree so it still lags behind Amazon's recommendation (currently 2.14.2, whereas Xenial's in-tree driver claims to be 2.12.1 and Trusty has 2.11.3).
drdaeman 9y ago
How hard (and how realistic) it is to actually get a reasonable understanding (and then stay up-to-date) with Kubernetes internals? Is there any go-to reading material? We had ran another large-footprint container management system (not K8s, but also popular), and when its DNS component started to eat all the CPU on all nodes, best I was able to do fast,was just scrapping the whole thing and quickly replacing it with some quick-and-dirty Compose files and manual networking. At least, we were back to normal in an hour or so. Obvious steps (recreating nodes) failed, logs looked perfectly normal, quick strace/ltrace gave no insights, and trying to debug the problem in detail would've taken more time. But that was only possible because all we ran was small 2.5-node system, not even a proper full HA or anything. And it had resembled Compose close enough. Since then I'm really wary about using larger black boxes for critical parts. Just Linux kernel and Docker can bring enough headache, and K8s on top of this looks terrifying. Simplicity has value. GitHub can afford to deal with a lot of complexity, but a tiny startup probably can't. Or am I just unnecessarily scaring myself?
- AlexB138 9y ago
  I wouldn't say that you're unnecessarily scaring yourself at all. Kubernetes is extremely complex. I've been running it for a few months and I'm just starting to get my hands around it. Things will just stop working for what seems like no reason, and there are so many places to investigate you can easily burn most of a day troubleshooting. It's a great system, but it's also relatively new, and most issues aren't well documented. You'll spend a lot of time in github issues or asking for help in the (very active, and often very helpful) community. If you have a valid use case, I wouldn't steer you away from it, but your fears are well founded.
daxfohl 9y ago
Curious, being a RoR app, did github ever run on Heroku? (Obviously googling "github heroku" is just a million tutorials on how to integrate.)
- ejholmes 9y ago
  I think I remember hearing that they were on EngineYard back in the day, but never Heroku.
- jbarnette 9y ago
  EY, Rackspace, and then our own metal.
lukaskroepfl 9y ago
hey! we at bitmovin have been using k8s for quite a while for our infrastructure and on premise deployments. In case you're interested in how we do multi stage canary deployments, check out: http://blog.kubernetes.io/2017/04/multi-stage-canary-deployments-with-kubernetes-in-the-cloud-onprem.html http://blog.kubernetes.io/2017/04/multi-stage-canary-deploym...

413 points by darwhy 9y ago

Any favorted training for learning Kubernetes? I found this one so far: https://classroom.udacity.com/courses/ud615 https://classroom.udacity.com/courses/ud615 But any extra courses/trainings is always appreciated

Omnipresent 9y ago
I'm interested in this as well. I found Hepito https://www.heptio.com/support-services-and-training https://www.heptio.com/support-services-and-training
pdelgallego 9y ago
Udemy: https://www.udemy.com/learn-devops-the-complete-kubernetes-course/ https://www.udemy.com/learn-devops-the-complete-kubernetes-c... Pluralsight: https://www.pluralsight.com/courses/getting-started-kubernetes https://www.pluralsight.com/courses/getting-started-kubernet...
- timrichard 9y ago
  I'm got that Pluralsight one earmarked for 'soon'. I thought Nigel Poulton's Docker courses were excellent, so looking forward to it...

DDub 9y ago

We're currently looking at moving our applications to k8s, and was wondering what deployment tools people are using? This week we are evaluating spinnaker, helm and bash wrappers for kubectl. There is concern over adding too many layers of abstraction and that KISS is the best approach.

dguaraglia 9y ago
I feel you. About a month ago I was fighting with the same feeling. In the end, decided to use Kubernetes only for a single piece of infrastructure so it's all pretty manageable through scripts. Managing secrets in particular is a pain in the ass. One route I started checking but didn't commit to was using Ansible. They have a relatively good Kubernetes playbook and a facility to store secrets. That said, every damn task needs to be pointed to the K8S API endpoint, which is not the greatest.
- smarterclayton 9y ago
  Agree - we've been talking about how we can more natively tie the inventory into clusters, contexts, and apps. The host focus of Ansible doesn't always map to other domains, but I think it has a real chance with Kube.
MattRogish 9y ago
We (ReactiveOps) use a combination of CircleCI and some scripts wrapping kubectl https://github.com/reactiveops/rok8s-scripts https://github.com/reactiveops/rok8s-scripts
jameskilton 9y ago
I've got a simple setup that makes use of YAML files, Rake tasks, and raw kubectl. I've yet to take a look at helm or spinnaker but it's on my list. You really can go a long way just with K8s' own tooling.
- jomakasi 9y ago
  I've started with creating Rake tasks. Then I found https://github.com/CommercialTribe/psykube https://github.com/CommercialTribe/psykube which is opinionated. Then I've ended up creating my own simple ruby tool to manage kubernetes with my own directory structure and configuration.
- danielodio 9y ago
  We've written a bunch about Spinnaker & K8s at http://go.Armory.io/kubernetes http://go.Armory.io/kubernetes ; hope that's helpful!
foxylion 9y ago
We also did some evaluation and then decided to stick to KISS an chose kubectl commands combined with cat and kexpand. Really simple approach to allow dynamic kubernetes deployments. Example command can be cat service.yml | kexpand expand -v image-tag=git-135afed4 | kubectl apply -f - The service.yml contains the full deployment configuration, service definition and ingress rules. So this works without preconfiguring anything in kubernetes when deploying an new service. An engineer only has to create the service.yml and Jenkins does deploy it automatically on every master build. *kexpand is a small tool which does something similar to sed, but in a simpler and less powerful way (keep it simple): https://github.com/kopeio/kexpand https://github.com/kopeio/kexpand
- nikon 9y ago
  I do something similar, but envsubst does the job.
  - ff_ 9y ago
    +1 on envsubst, it's the minimal solution to the problem of templating kubernetes (of course YMMV, we are a small team and don't need more complex stuff)
  - brianwawok 9y ago
    Ditto me too to get my git sha into my image names and deploy with deployments.
- howinator 9y ago
  We basically did the same thing except we used Ansible for our templating. This allows us to store all our shared "environment configuration data", e.g., name of RDS for services in prod environment, name of backup S3 bucket for services in dev environment, in an Ansible role then just pull that information into our templated deployment manifest file. So far, it's worked out pretty well for us.
deadmik3 9y ago
I'd recommend looking into openshift. it's basically kubectl + cool deployment features. there's also free, paid, and dedicated online hosted options. disclaimer: I work on openshift
- awill 9y ago
  You had a big announcement about openshift.io. Everyone on HN signed up, but it's been months, but I'm still 'awaiting approval' Have you let anyone in? What's the value in all the marketing hype if you then don't let people in.
  - yebyen 9y ago
    Seconded. The last e-mail I got from OpenShift.io indicated they haven't let anyone in. I've already practically lost interest. It was evidently only "coming soon" but that announcement really looked like "coming tomorrow." The original OpenShift Developer Preview made you sign up, but you would be allowed into the platform within hours or days.
    - deadmik3 9y ago
      OpenShift.io is a different product than hosted OpenShift on its own. If you were to sign up for the Pro tier at openshift.com your account would be immediately provisioned. We are working on provisioning enqueued users for our Starter tier as well but again, these are separate from OpenShift.io
      - yebyen 9y ago
        Yeah, OpenShift Pro is also quite expensive... starting at $10k for 5 nodes right? I just checked out your pricing page and I'm 100% wrong about this. You have a $50/mo tier now! That's fantastic, thanks for pointing me at that. You should definitely spam everyone on the OpenShift.io waiting list and let us know about your new pricing /s No seriously, I would have liked to get some spam about this. Maybe you sent it and I missed it. I am a lot more interested at $50/mo than I would have been at $10k/yr!
bryanlarsen 9y ago
We're using helm, but that was chosen mostly based on gut feel. It's an official project, and has momentum. We didn't want to spend too much time choosing a tool until we knew our requirements, and we don't really have a firm grasp of requirements until we've used something for a while. It seems to be working well for us so far, but it's still early. There are lots of answers here that aren't helm, so I'm curious if there are any particular reasons that people ruled out helm?
- nikon 9y ago
  Are you doing what Gitlab claims below "We (GitLab) use it mostly as a templating system as well" or really using it to manage complex apps? I think it doesn't really offer much for normal microservices. I do use it for things like nginx ingress but for stuff I've built a service.yml / deployment.yml are fine.
- henridf 9y ago
  We ran into a number of issues with Helm when deploying - failures leading us to have to rollback, with rollbacks then failing, requiring manual changes to unblock. I think that for third-party packages and related templating (which seems like the original use-case) it works well, but I would be wary of using it for high-res deploys of our own stuff.
yebyen 9y ago
You should take a look at Deis Workflow. I say this in spite of the fact that it was announced last week[1], the next release of Deis Workflow will be the last (under the current stewardship, and probably under that name.) It's just such a solid system, I would even more strongly recommend the (already EOL'ed early last year)[2] Deis v1 PaaS, except that you've already indicated you're moving to K8S, and Deis v2 is designed for K8S. I still recommend the v1 PaaS for people learning about principles of HA clusters. (Another disclosure: I have published[3] about how to do this, a work on how to do a cheap HA cluster using Deis v1 PaaS.) I have a strong suspicion that Deis will live on after March under stewardship of new leadership from the community. In the mean time, you have roughly 6 months of support from Microsoft, maybe I am overstating to say that they have committed to keeping the lights on for that long, but they have committed to merging critical fixes for that long (and we hope that in 6 months, Kubernetes will have solidified enough that we don't have to worry too much about breaking changes from upstream release mongers anymore.) Personally I don't buy commercial support and it would not be the deal maker or breaker for me. [1]: https://deis.com/blog/2017/deis-workflow-final-release/#future https://deis.com/blog/2017/deis-workflow-final-release/#futu... [2]: https://deis.com/blog/2016/deis-1-13-lts/ https://deis.com/blog/2016/deis-1-13-lts/ [3]: https://deis.com/blog/2016/cheapest-fault-tolerant-cluster-deis/ https://deis.com/blog/2016/cheapest-fault-tolerant-cluster-d...
- gtirloni 9y ago
  I'd advise against choosing core infrastructure components (that have a clear EOL deadline) based on strong suspicion. Even more so in a landscape that's constantly changing like Kubernetes. You have zero guarantees that it'll be maintained and will keep up with new breaking changes.
  - yebyen 9y ago
    You know it's open source, right? I have zero guarantees that any of my open source projects that I use for business critical infrastructure aren't going to pack up shop and quit maintaining their stuff tomorrow. You should know how your infrastructure works well enough to maintain it for yourself. I (personally) will be maintaining this one in the future, if necessary! We're working it out now. What do you mean by "strong suspicion?" Please don't downvote because you read a few words you didn't like, I was upfront about this EOL date because I don't want it coming back later that I was dishonest about it, but my perception is not that "EOL" means it's dead, it is that "EOL" means it's done. Stability is a good thing. Microsoft also EOL'ed MSPaint.exe, and I remember how the community reacted. I think the quote was about "works for 99% of users and has been stable for over a decade? sounds like a good candidate for deletion!" The project is cancelled because it's not strategically important to Microsoft, not because it's not viable or having technical issues. The core devs have chosen to work on more kubernetes-native tooling. They aren't abandoning Kubernetes, and I'll bet you don't have a competing product you can show me that has guaranteed to keep the lights on for the next 6 months.
lobster_johnson 9y ago
We wrote an internal tool that wraps Helm and GPG. But we're really using Helm as a glorified templating system; since we deploy from git, Helm's release management is useless to us, and is even somewhat in the way. We might decide to drop Helm at some point, I think.
- xur17 9y ago
  We went down a similar path and ended up using helm-template [0] to render our helm charts without tiller. We also use an internal tool that: - maps applications to namespaces within clusters for different environments (since we have 1 cluster per environment) - does some magic around secrets to make them easier to interface with [0] https://github.com/technosophos/helm-template https://github.com/technosophos/helm-template
- Snappy 9y ago
  That's a good point. We (GitLab) use it mostly as a templating system as well. It's a step up from piping `sed` output to `kubectl`. But we have our own tools for managing redeploys and rollbacks.
sytse 9y ago
If you're using our GitLab consider using Auto Deploy. Our CTO recently made a quick start guide for it https://docs.gitlab.com/ee/ci/autodeploy/quick_start_guide.html https://docs.gitlab.com/ee/ci/autodeploy/quick_start_guide.h...
- Snappy 9y ago
  FWIW, the next version of GitLab's Auto Deploy will use Helm under the hood (and let you bring-your-own-chart).
majewsky 9y ago
At SAP, we're using Helm [1] to deploy OpenStack (plus a bunch of extra services like a custom Rails-based dashboard [2]) on baremetal Kubernetes. For image building, testing and deployment, we use Concourse CI [3], and OpenStack assets (like service users, service projects and roles) are deployed with a custom Kubernetes operator [4]. Our charts are at [5] if you want to take a look. [1] https://github.com/kubernetes/helm https://github.com/kubernetes/helm [2] https://github.com/sapcc/elektra https://github.com/sapcc/elektra [3] https://concourse.ci https://concourse.ci [4] https://github.com/sapcc/kubernetes-operators https://github.com/sapcc/kubernetes-operators in the "openstack-operator" directory [5] https://github.com/sapcc/openstack-helm https://github.com/sapcc/openstack-helm and https://github.com/sapcc/helm-charts https://github.com/sapcc/helm-charts (two different repos since we're in the middle of modularizing the original monolithic OpenStack chart in the first repo into per-service charts in the second one)
invisible 9y ago
We are using ecfg (from Shopify) and Jenkins+kubectl. We use ansible for a couple of things but it's largely only because of some parameters of our architecture for a legacy monolith.
theptip 9y ago
A lot of folks are using Helm, but I find it very opaque to debug when templates go wrong (and I feel quite strongly that we shouldn't be writing untyped templates for our models). Also I found writing reusable spec components to be very difficult, e.g. a reverse proxy that I add to a number of pods. I use pykube (also worth looking at the incubator project client-python) to write deploy scripts in Python; client-python is particularly nice as it uses OpenAPI specs to give you type hinting on the API objects/fields that you're writing. Much more civilized than typing bare yaml into a text editor. If Python isn't your thing you can generate your own client from the OpenAPI specs, though I've found the client generation process to be a bit buggy.
rubenv 9y ago
I recently spoke about the approach we use at Ticketmatic: https://rocketeer.be/articles/coreos-fest-2017/ https://rocketeer.be/articles/coreos-fest-2017/
- kalev 9y ago
  That must have been an awesome talk. Thanks for the write-up!

shock 9y ago

> During this migration, we encountered an issue that persists to this day: during times of high load and/or high rates of container churn, some of our Kubernetes nodes will kernel panic and reboot. Considering that Kubernetes doesn't modify the kernel, this issue sounds like is present in mainline and kernel devs should be involved.

cyphar 9y ago
I would be interested to know what storage driver they're using for their nodes. High container churn puts a lot of stress on the VFS subsystem of Linux, and we've seen cases where customers have trigger lots of mount/umounts which results in filesystems causing panics. At SUSE, we do have some kernel devs debugging the issues, but the workaround is almost always "rate limit all the things". There are a few other kernel areas that are stressed with high container churn (like networking), but VFS is the most likely candidate from my experience. While on paper containers are very lightweight, spawning a lot of them exercises kernel codepaths that probably haven't been exerted to that type of stress during development.
- AaronBBrown 9y ago
  Hey, this is Aaron from GitHub. We're using devicemapper w/ LVM backed pools. Would love to hear about your experience there. We definitely see this problem during periods of high container churn.
  - cyphar 9y ago
    That's funny, we have an internal bug open right now about a kernel panics that happen with devicemapper (with XFS as the base filesystem). We found that the issue was exacerbated if you used loopback devices, but on paper it should still happen in non-loopback mode (the current theory is that it's a bug in XFS). Our kernel team is still investigating the issue, but they cannot seem to reproduce the issue with direct-lvm (and loop-lvm is inconsistent in reproducing the issue). If you can consistently reproduce the issue, would you mind providing the backtrace and/or coredump? Is it possible for you to reproduce the issue on a machine without needing to be hit by GitHub-levels of traffic, and if so can you provide said reproducer? For reference, our backtraces show that the kernel dies at Xfs_vm_writepage. Though of course different kernel versions may have varying backtraces. You can reach me on the email in my profile, or asarai(at)suse.com.

Yeroc 9y ago

One thing I would have liked to have seen addressed in the article is whether the new architecture requires additional hardware (presumably) to operate and if so how much more.

ben_jones 9y ago
I've only dabbled in K8s and it strikes me that using it in production is a long term investment and, as it stands currently, a long term project to implement properly. You'll want to do exactly what Github did: setup a "review lab" or similarly comprehensive dev and test environment until you are absolutely comfortable with it in production. This will lead to the provisioning (and cost) of quite a bit of hardware - and when it is finally in production it'll likely be over-provisioned for quite some time until norms can be established and excess cut. So basically its a traditional devops migration. But you get quite a few goodies and arguably much better practices at the end of it.
- majewsky 9y ago
  I agree very much, and I'd like to add one point: When you build a lab environment for testing Kubernetes deployments (and verifying Kubernetes upgrades), make sure it's on the same hardware as your production environment. When my team did the first Kubernetes deployment, we made the mistake of building a lab environment that did not match the anticipated production environment. (Two reasons: The BOM for the production environment was not yet decided upon at that time, and the lab was frankensteined together by taking hardware out of existing labs.) We learned the hard way that, just because the Kubernetes upgrade worked in the lab, it need not work on the production hardware. Right now, we're stuck on last year's (i.e., ancient) Kubernetes 1.4 release because no one dares to upgrade production. (There's light at the end of the tunnel, though. A new lab is being built up in the datacenter around now.)
- joshuatalb 9y ago
  If you're on AWS you can use kops (https://github.com/kubernetes/kops https://github.com/kubernetes/kops) to significantly reduce the amount of time to get a cluster up. It took me about 1hr to get a basic k8s cluster up and running with it.

lobster_johnson 9y ago

I'd be interested in hearing what kind of autoscaling system they use for their Ruby pods. We're running a few (legacy — we're moving to Go) Ruby apps in production on Kubernetes. We're using Puma, which is very similar to Unicorn, and it's unclear what the optimal strategy here is. I've not benchmarked this in any systematic way. For example, in theory you could make a single deployment run a single Unicorn worker, then set resources:requests:cpu and resources:limits:cpu both to 1.0, and then add a horizontal pod autoscaler that's set to scale the deployment up on, say, 80% CPU. But that gives you terrible request rates, and will be choking long before it's reaching 80% CPU. So it's better to give it, say, 4 workers. At the same time, it's counter-productive to allocate it 4 CPUs, because Ruby will generally not be able to utilize them fully. At the same time, more workers mean a lot more memory usage, obviously. I did some quick benchmarking, and found I could give them 4 workers but still constrain to 1 CPU, and that would still give me a decent qps.

sytse 9y ago
At GitLab we recommend to use CPU cores + 1 as the number of unicorn workers https://docs.gitlab.com/ce/install/requirements.html#unicorn-workers https://docs.gitlab.com/ce/install/requirements.html#unicorn...
- lobster_johnson 9y ago
  How do you configure that? A pod doesn't know what machine it's running on ahead of time. You can create nodepools and use node selectors to pin the pod to that nodepool, but I'm not sure I love the idea.
  - ecnahc515 9y ago
    Using the downward API, your application can get the number of CPU cores or the millicore value via environment variables or volumes. This would let you configure your number of workers based on what resource limits you set on your container.
  - twk3 9y ago

sandGorgon 9y ago

> We enhanced GLB, our internal load balancing service, to support Kubernetes NodePort Services. Everyone does this - because Kubernetes Achilles heel is its ingress. It is still built philosophically as a post-loadbalancing system . This is the single biggest reason why using Docker Swarm is so pleasant.

alpb 9y ago
Can you elaborate? What are the issues you're seeing with Ingress?
- mnutt 9y ago
  The service I'm working on porting to kubernetes has bandwidth requirements that far exceed a single machine, and though we're on AWS, we can't use ELB for various reasons. We ended up with a ReplicaSet of haproxies using HostPort, and a separate app that watches for haproxy service changes and updates Route53 round-robin DNS. We're somewhat fortunate that we only have a single service that needs to use the http/https HostPort. We could have just left our haproxies outside of kubernetes, and may eventually end up doing so if the network performance doesn't meet our needs. As it is, it all works but there are a ton of sidecar services all over the place.
  - yebyen 9y ago
    This sounds like what Deis Router was created to do. It sounds like you've already got it nailed down, but maybe like to have a look at this: https://github.com/deis/router https://github.com/deis/router (It's probably tightly coupled to Deis Controller, but something to look at anyway!)
orf 9y ago
Can you elaborate? I found Kubernetes ingresses to be one of the most pleasant parts - we use the `nginx-ingress` from helm and it works very well. Not exactly an industrial github-strength load balancer, but it will get you a long way surely?

philips 9y ago

Really exciting stuff, happy to see the Github team launch this. Kubernetes is becoming the goto for folks needing both their own physical metal presence and cloud footprint too. And the magic of Kubernetes is that it has APIs that can actually give teams the confidence to run and reuse deployment strategies in all environments. Even across clouds. If you are like Github and want to use Kubernetes across clouds (AWS, Azure, etc) & bare metal and do deploy/customize that infra using Terraform checkout CoreOS Tectonic[1]. It also tackles more of the subtle things that aren't covered in this article like cluster management, LDAP/SAML authentication, user authorization, etc. [1] https://coreos.com/tectonic https://coreos.com/tectonic

robotmay 9y ago
I'm still utterly perplexed as to what Tectonic actually -is-. I kinda get that it's a kubernetes setup, but is it a GUI over the top of it? The website is pretty confusing and I think I gave up really quickly when trying to set it up.
- philips 9y ago
  Tectonic is Enterprise Kubernetes. We start with pure upstream Kubernetes at the core and install it in a production ready setup with the Tectonic Installer[1] across clouds or bare metal. On top of those basics Tectonic provides things most organizations need: - Authentication backed by LDAP/SAML/etc - One-click automated updates of the entire cluster - Pre-configured cluster monitoring/alerting There is a bunch more in there and in the roadmap too but that gives you a taste. The other thing is that we provide professional services, training, and support to customers on the whole stack from the VM or machine on up to the Kubernetes API. We have done neat collaborations with customers like the ALB Ingress Controller[2] too. [1] https://github.com/coreos/tectonic-installer https://github.com/coreos/tectonic-installer [2] https://github.com/coreos/alb-ingress-controller https://github.com/coreos/alb-ingress-controller
  - tachion 9y ago
    I'm currently deploying Tectonic flavoured Kubernetes at a large organisation and I can vouch for how great you guys are at supporting users (who are not yet customers) at any stage of the process - love that, and can't recommend you guys for that more. However, as the comment above says, Tectonic (and Quay for that matter) documentation is... just horrible and if not the engineers support, I'd be pretty much stuck on quite few things. Why don't you push your docs to a public repo, so I could do some writing and send some PR's? ;)

Alan01252 9y ago

I'm curious to what this means for the existing puppet code base, is it now irrelevant, or are there still usages for it in the k8s world?

spmurrayzzz 9y ago
They could easily still use standalone puppet to handle the config management for individual container images. I currently do this with salt-minion. It reduces the burden on the Dockerfile itself, and lets you embrace a declarative configuration state at build time.
- brown9-2 9y ago
  This introduces a dependency on which pod runs on which host, unless you have puppet write the config for every service to every host. People tend to use Puppet less to configure their applications as they move into containers, as a configuration change can just be made by rolling out a new image.
- ninkendo 9y ago
  It definitely seems like the wrong approach to me to have puppet manage your base images. They're not VM's, they shouldn't have multiple services, they shouldn't require any complex configuration management, they should just be the minimum requirements to support your application's local runtime dependencies, and that's it. From previous experience migrating from a puppet setup to one that used containers, puppet's vestigial use case ends up being to get the orchestration control plane itself setup (ie. kubernetes, networking configs, etc) and that's about it.
  - dkh99 9y ago
    There's nothing inherently about Puppet that means it has to manage multi-service "whole OS"-like installations. It can just as easily be put to the task of a Dockerfile: install dependencies and deployables for a single application. Its robust ability to manage things like user accounts, packages, scheduled jobs (e.g. for alerting, though you would have to install at least a second service for this: _crond) and the like makes it vastly superior to Dockerfile shell scripts for complex tasks. Think of puppet more like a way of simplifying your Dockerfiles to have fewer crazy shell commands in total, rather than hiding the craziness in layers and hoping it all composes properly. If you do use lots of layers, Puppet can make your life much easier, since it can be better at detecting previous layers' changes and working around them (think redundant package install commands. Even the no-op "already installed!" command takes time; if you're installing hundreds of packages--many people are, for better or worse--that can eat up build time). Puppet isn't a VM provisioner; it can also be used as a replacement for large parts of your Dockerfile, or a better autoconf to set up the environment/deps for your application to run in. Edit: syntax.

iagooar 9y ago

Love to see more Kubernetes success stories. I work for an ISP and we are trying to write another success story ;) As an ISP, we have tons of constraints in terms of infrastructure. We're not allowed to use any public cloud services. At the same time, the in-house infrastructure is either too limited, or managed via spreadsheets by a bunch of dysfunctional teams. For my team, Kubernetes has been truly a life saver when it comes to deploying applications. We're still working on making our cluster production-ready, but we're getting there very fast. Some people are already queuing up to get to deploy their applications on Kubernetes :D What I especially love about Kubernetes is how solid the different concepts are and how they make you think differently about (distributed) systems. It sure takes a lot of time to truly grasp it, and even more so to be confident managing and deploying it as Ops / SRE. But once you get it, it starts to feel like second nature. Plus the benefits, in almost any possible way, are huge.

lima 9y ago
Red Hat's OpenShift makes it a lot easier by providing all of the infrastructure around it (docker registry, docker build from Git, Ansible integration and so on). Best docs of all open source projects I've seen.
- EtienneK 9y ago
  I second this. Have been PoCing OpenShift for a couple of months now and it's been a joy to use.
  - Valien 9y ago
    3rd for sure. We're a RH partner and specialize in OpenShift work. Tons of excitement with customers on using OpenShift. I love it.
  - smarterclayton 9y ago
    Thanks! Hearing this makes it all worthwhile. We love helping make Kube awesome.
9y ago

dmart 9y ago

> Enhancements to our internal deployment application to support deploying Kubernetes resources from a repository into a Kubernetes namespace, as well as the creation of Kubernetes secrets from our internal secret store. Would love to hear more about this was accomplished. I'm currently exploring a similar issue (pulling per-namespace Vault secrets into a cluster). From what I've found, it looks like more robust secrets management is scheduled for the next few k8s releases, but in the meantime have been thinking about a custom solution that would poll Vault and update secrets in k8s when necessary.

skewart 9y ago

> Several qualities of Kubernetes stood out from the other platforms we evaluated: the vibrant open source community supporting the project, the first run experience (which allowed us to deploy a small cluster and an application in the first few hours of our initial experiment), and a wealth of information available about the experience that motivated its design. It's interesting that the reasons they cite for choosing Kubernetes over alternatives are entirely driven by 'developer experience' and not at all technical. It shows how critical community development, good documentation, and marketing are to building a successful open source project.

gu4x 9y ago
I believe developer experience on being introduced to a tool is paramount to its success. It gives a lot of confidence in what you're doing and keeps things moving forward. To me appear that the application is built on a solid simple concept instead of a convoluted complex architecture. Some tools sin on the opposite though, very simple to setup but very complicated to understand how to scale.

erulabs 9y ago

If you're running Kube on AWS, make sure you install the proper drivers! For Ubuntu, that's the `linux-aws` apt package. https://github.com/kubernetes/kops/issues/1558 https://github.com/kubernetes/kops/issues/1558 Missing ENA and ixgbevf can be a real performance killer!

joombaga 9y ago
Is this used by vanilla docker / ECS, or just k8s?
- TheDong 9y ago
  that advice holds regardless what software you're using, so long as the software does network traffic. It holds for just running plain old nginx websites. It doesn't really matter for small instance sizes where your networking is already rate-limited by amazon so much that ENA drivers won't matter, but on beefy instances it's always good advice to make sure you're using ENA supported driverse. See https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/enhanced-networking-ena.html#test-enhanced-networking-ena https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/enhanced...
bmoyles 9y ago
FWIW, the stock kernel (and HWE/HWE-edge kernels) recently picked up current ENA drivers. ixgbevf, unfortunately, doesn't look like it's been updated in-tree so it still lags behind Amazon's recommendation (currently 2.14.2, whereas Xenial's in-tree driver claims to be 2.12.1 and Trusty has 2.11.3).

drdaeman 9y ago

How hard (and how realistic) it is to actually get a reasonable understanding (and then stay up-to-date) with Kubernetes internals? Is there any go-to reading material? We had ran another large-footprint container management system (not K8s, but also popular), and when its DNS component started to eat all the CPU on all nodes, best I was able to do fast,was just scrapping the whole thing and quickly replacing it with some quick-and-dirty Compose files and manual networking. At least, we were back to normal in an hour or so. Obvious steps (recreating nodes) failed, logs looked perfectly normal, quick strace/ltrace gave no insights, and trying to debug the problem in detail would've taken more time. But that was only possible because all we ran was small 2.5-node system, not even a proper full HA or anything. And it had resembled Compose close enough. Since then I'm really wary about using larger black boxes for critical parts. Just Linux kernel and Docker can bring enough headache, and K8s on top of this looks terrifying. Simplicity has value. GitHub can afford to deal with a lot of complexity, but a tiny startup probably can't. Or am I just unnecessarily scaring myself?

AlexB138 9y ago
I wouldn't say that you're unnecessarily scaring yourself at all. Kubernetes is extremely complex. I've been running it for a few months and I'm just starting to get my hands around it. Things will just stop working for what seems like no reason, and there are so many places to investigate you can easily burn most of a day troubleshooting. It's a great system, but it's also relatively new, and most issues aren't well documented. You'll spend a lot of time in github issues or asking for help in the (very active, and often very helpful) community. If you have a valid use case, I wouldn't steer you away from it, but your fears are well founded.

daxfohl 9y ago

Curious, being a RoR app, did github ever run on Heroku? (Obviously googling "github heroku" is just a million tutorials on how to integrate.)

ejholmes 9y ago
I think I remember hearing that they were on EngineYard back in the day, but never Heroku.
jbarnette 9y ago
EY, Rackspace, and then our own metal.

lukaskroepfl 9y ago

hey! we at bitmovin have been using k8s for quite a while for our infrastructure and on premise deployments. In case you're interested in how we do multi stage canary deployments, check out: http://blog.kubernetes.io/2017/04/multi-stage-canary-deployments-with-kubernetes-in-the-cloud-onprem.html http://blog.kubernetes.io/2017/04/multi-stage-canary-deploym...