8 ms·

Right - I saw that. The problem I have is that most people feel like theyre doing 'serious work' so they need a serious solution when the reality is that it re

Right - I saw that. The problem I have is that most people feel like theyre doing 'serious work' so they need a serious solution when the reality is that it really is just a small percentage of companies - not usually even successful startups, for instance - that require this sort of automation. Mostly, it adds complexity rather than reduces it. I know that sounds like we're getting into the weeds, but if you consider that most companies are taking on additional complexity because of this, it becomes a problem.

hnarayanan 11y ago
Again, I don't generally disagree with you. But as our team is growing at my day job, I am finding more and more that we're getting bitten by some developer's "It works on my machine" or a random npm failure at deploy time. And this makes me believe that having a prebuilt (immutable, hermetically sealed) artefact that I can later deploy is the way to go. Containers solve this problem nicely. And once you're going down the container route, you need something like a Kubernetes to easily schedule and network them. My personal site -- the one you're reading the article on -- is literally just a static nginx server on an old Linux box (with the files copied in by rsync). Even the curmudgeon in me now thinks it makes sense to use solutions of different complexity for different situations.
jsmthrowaway 11y ago
I've run production at a 'successful startup,' a million node corporate fleet, and a personal toy project or six, and consider Kubernetes the single most important work being done in the field right now. Docker, no (rkt is better architected). Kubernetes, yes[0]. Don't think of Kubernetes as a replacement for Puppet. Think of it as distributed init/supervisord with service discovery, which you are clumsily building with Puppet. Your automation ends up writing init, units, monit, supervisor, and so on, right? (You shouldn't be deploying apps with Puppet anyway because CD is not CM, but nothing like Kubernetes or Mesos really existed in the open source world until now so a whole generation learned to do it with Puppet.) Kubernetes exists for when your scripted machine fails or you exceed the capacity of your scripted machine and need to horizontally scale. It is not a deployment tool per se; that's only part of what it does as a scheduler and resource manager. Every project, from personal to Fortune 500, needs to plan for those scenarios. I'm not saying Kubernetes is always the answer, I'm saying it solves a problem that every project universally has, despite your claims. The bonus of working that way is now you have an API to your machines and can treat them as a single unit of resources. Kubernetes is about half of building your own mini Heroku. You've heard snowflakes versus cattle, right? Your way is for snowflakes. Kubernetes is for cattle. The threshold where one becomes more productive than the other is the constant debate, and as an SRE, my opinion bucks must people who talk about this on where that threshold lies. If you self-identify as a sysadmin you'll likely have one opinion, devops another, and then you have my group of crazies that generally want to crush all snowflakes. I can speak with experience that our crazy SRE ways scale pretty well to the hundreds of thousands of nodes case but also work quite well for toy systems. Subscribe to everything CoreOS is doing. There's a spectrum of quality there, but they're running with the programmable infrastructure ideal and have largely bet the company on the Kubernetes ecosystem. [0]: Kubernetes makes some interesting choices versus Borg and I think they're going to have trouble scaling it to that, and Borg really shines at Google because of the global filesystem layer that exists on every machine (they don't realize this and are doing gymnastics around storage in the open source side), but if they can mature Kubernetes it'll be a solid building block for platform development and eliminate a lot of the clumsy scripts and automation that you are defending and push us toward programmable infrastructure.
- toomuchtodo 11y ago
  So Kubernetes is another layer on top of basic AWS primitives? I already have AMIs, autoscaling groups, health checks, ELBs, etc. Why am I adding another unnecessary layer of abstraction? Disclaimer: Devops/sysadmin who would like to get off the hype train and get real work done.
  - jsmthrowaway 11y ago
    Not everybody is on Amazon and some of us touch our gear. You should be excited because it gives you a rapidly-improving path off of Amazon if you've gone all in on their vendor stuff. Kubernetes helps build that stuff for any vendor. Multiple startups are working on AWS in a box. That was the promise of OpenStack but now you don't need virtualization so it's automatically an order of magnitude better. Maybe two. People don't realize how deep Amazon's hooks are at scale. We were dropping half a million USD a month at that successful startup I mentioned and could do the same work with four or five racks of hardware, storage included. I bet you could cut your opex in half on something else, and it's a safe bet because (a) I've seen it be resoundingly true even coming from three year reservations and (b) half is actually a conservative estimate. I am floored by how much money the industry throws at the fundamental inefficiency of multitenant virtualization atop AWS. What a strange argument, considering Amazon a primitive. That's flirting with vendor koolaid. Physical and DigitalOcean/cheapo deployments are an immediate counterexample, too. Hell, if network latency is low enough, you can straddle both and schedule Kubernetes pods on whichever provider is cheaper on a given day. Spot instances, too. Lots of possibilities.
    - toomuchtodo 11y ago
      > What a strange argument, considering Amazon a primitive. Note I said "AWS primitives", not AWS as a primitive. I think they're overly expensive unless you're using them for prototyping, as you should eventually move off to your own gear once you know what your load/compute/storage profiles look like. I appreciate you pointing out that this is most valuable when you're _not_ running in AWS (I work at a startup that wants to use this tech in AWS, hence my not understanding why we'd waste additional abstraction for little additional benefit). Disclaimer: I _want_ to move off of AWS to our own physical gear at some point, but our dev team is married to it because they're afraid of physical infrastructure (but its okay when critical services like EC2, autoscaling, and IAM aren't working for several hours apparently).

hnarayanan 11y ago

Again, I don't generally disagree with you. But as our team is growing at my day job, I am finding more and more that we're getting bitten by some developer's "It works on my machine" or a random npm failure at deploy time. And this makes me believe that having a prebuilt (immutable, hermetically sealed) artefact that I can later deploy is the way to go. Containers solve this problem nicely. And once you're going down the container route, you need something like a Kubernetes to easily schedule and network them. My personal site -- the one you're reading the article on -- is literally just a static nginx server on an old Linux box (with the files copied in by rsync). Even the curmudgeon in me now thinks it makes sense to use solutions of different complexity for different situations.

jsmthrowaway 11y ago

I've run production at a 'successful startup,' a million node corporate fleet, and a personal toy project or six, and consider Kubernetes the single most important work being done in the field right now. Docker, no (rkt is better architected). Kubernetes, yes[0]. Don't think of Kubernetes as a replacement for Puppet. Think of it as distributed init/supervisord with service discovery, which you are clumsily building with Puppet. Your automation ends up writing init, units, monit, supervisor, and so on, right? (You shouldn't be deploying apps with Puppet anyway because CD is not CM, but nothing like Kubernetes or Mesos really existed in the open source world until now so a whole generation learned to do it with Puppet.) Kubernetes exists for when your scripted machine fails or you exceed the capacity of your scripted machine and need to horizontally scale. It is not a deployment tool per se; that's only part of what it does as a scheduler and resource manager. Every project, from personal to Fortune 500, needs to plan for those scenarios. I'm not saying Kubernetes is always the answer, I'm saying it solves a problem that every project universally has, despite your claims. The bonus of working that way is now you have an API to your machines and can treat them as a single unit of resources. Kubernetes is about half of building your own mini Heroku. You've heard snowflakes versus cattle, right? Your way is for snowflakes. Kubernetes is for cattle. The threshold where one becomes more productive than the other is the constant debate, and as an SRE, my opinion bucks must people who talk about this on where that threshold lies. If you self-identify as a sysadmin you'll likely have one opinion, devops another, and then you have my group of crazies that generally want to crush all snowflakes. I can speak with experience that our crazy SRE ways scale pretty well to the hundreds of thousands of nodes case but also work quite well for toy systems. Subscribe to everything CoreOS is doing. There's a spectrum of quality there, but they're running with the programmable infrastructure ideal and have largely bet the company on the Kubernetes ecosystem. [0]: Kubernetes makes some interesting choices versus Borg and I think they're going to have trouble scaling it to that, and Borg really shines at Google because of the global filesystem layer that exists on every machine (they don't realize this and are doing gymnastics around storage in the open source side), but if they can mature Kubernetes it'll be a solid building block for platform development and eliminate a lot of the clumsy scripts and automation that you are defending and push us toward programmable infrastructure.

toomuchtodo 11y ago
So Kubernetes is another layer on top of basic AWS primitives? I already have AMIs, autoscaling groups, health checks, ELBs, etc. Why am I adding another unnecessary layer of abstraction? Disclaimer: Devops/sysadmin who would like to get off the hype train and get real work done.
- jsmthrowaway 11y ago
  Not everybody is on Amazon and some of us touch our gear. You should be excited because it gives you a rapidly-improving path off of Amazon if you've gone all in on their vendor stuff. Kubernetes helps build that stuff for any vendor. Multiple startups are working on AWS in a box. That was the promise of OpenStack but now you don't need virtualization so it's automatically an order of magnitude better. Maybe two. People don't realize how deep Amazon's hooks are at scale. We were dropping half a million USD a month at that successful startup I mentioned and could do the same work with four or five racks of hardware, storage included. I bet you could cut your opex in half on something else, and it's a safe bet because (a) I've seen it be resoundingly true even coming from three year reservations and (b) half is actually a conservative estimate. I am floored by how much money the industry throws at the fundamental inefficiency of multitenant virtualization atop AWS. What a strange argument, considering Amazon a primitive. That's flirting with vendor koolaid. Physical and DigitalOcean/cheapo deployments are an immediate counterexample, too. Hell, if network latency is low enough, you can straddle both and schedule Kubernetes pods on whichever provider is cheaper on a given day. Spot instances, too. Lots of possibilities.
  - toomuchtodo 11y ago
    > What a strange argument, considering Amazon a primitive. Note I said "AWS primitives", not AWS as a primitive. I think they're overly expensive unless you're using them for prototyping, as you should eventually move off to your own gear once you know what your load/compute/storage profiles look like. I appreciate you pointing out that this is most valuable when you're _not_ running in AWS (I work at a startup that wants to use this tech in AWS, hence my not understanding why we'd waste additional abstraction for little additional benefit). Disclaimer: I _want_ to move off of AWS to our own physical gear at some point, but our dev team is married to it because they're afraid of physical infrastructure (but its okay when critical services like EC2, autoscaling, and IAM aren't working for several hours apparently).