4 ms·

Often when I talk to founders who use Kubernetes, they could simply use docker and a bit of shell scripting instead. Those of you who use Kubernetes: What is a

Often when I talk to founders who use Kubernetes, they could simply use docker and a bit of shell scripting instead. Those of you who use Kubernetes: What is a functionality it brings to the table you would miss if you automated your docker handling without it?

hueving 7y ago
Resume boosting. Using shell scripts is lame and straight forward. If you have a declarative configuration though you can scale to thousands of nodes with nary a thought.
- jrockway 7y ago
  What service did you scale across thousands of nodes with a shell script?
- marcinzm 7y ago
  If you have a single service that runs of thousands of nodes then okay. If you have dozens of services that run across even dozens of nodes, that interact with each other, that have resource constraints, that need rolling updates, depend on external volumes, require security policies, etc. then you eventually recreate enough k8s to make it simpler to just use k8s.
vitalus 7y ago
- Simple DNS based service discovery - namespace separations between resources - optimized resource utilization and container scheduling These are the biggest 3 for the smaller orgs I’ve been a part of
- founderling 7y ago
  Interesting. Can you give an example of why DNS based service discovery was needed?
  - markbnj 7y ago
    > Can you give an example of why DNS based service discovery was needed? Not to be flippant, but any case where things need to connect to other things. At scale they usually need to connect to other things through some sort of load balancer. You get that out of the box with kubernetes for services hosted inside the cluster, and there are straightforward solutions to ingress for clients outside it. Another important feature is pod scheduling. Yes you could wire up a few machines using docker compose and any of a few different networking approaches, but if one of your VMs dies are those workloads going to move to a healthy instance by themselves?
  - andrewstuart2 7y ago
    Because I can run `ping` in a busybox container that knows how to do all the same service discovery as a more complicated fully fledged microservice. No extra libraries. No smattering of supporting services (on top of DNS, which you need anyway, of course). Setup and debugging is 10x simpler than any of the more complicated service discovery solutions. And it just works. Simply and intuitively. With everything. Because it's been an IETF standard since 1983.
  - seabrookmx 7y ago
    Need is a strong word. But you don't need containers either. Service discovery just makes it easier to link up services if your architecture is truly microservice. We used to have an incredible amount of config that tightly coupled our API's.. now we use a combination of service discovery and an API gateway (Ambassador) to decouple the services, cut down on the number of random endpoints in our config, and we also get the added benefit of load balancing, rate limiting, and additional logging. There's always a tradeoff with scale. If you have four servers than obviously all of this stuff is overkill.
    - andrewstuart2 7y ago
      > If you have four servers than obviously all of this stuff is overkill. I disagree, actually. I have four servers at home, and have some pods that have been running little tinkery things, and a bunch of open source software, with ridiculous uptime and little or no effort, even when I reboot one of those "servers" to do some gaming on Windows. Now, do I need that uptime for all of those services? Not really. But for some of them, I want it, and it'd be annoying if I had to go figure out why they'd stopped running. The reality is, things just keep ticking without me worrying when they're on k8s. These skills transfer into very in-demand job skills as well, and if I ever build anything that gains traction, I already have all the tools, configs, and knowledge to deploy that app across 500 generic cloud servers.
jhgaylor 7y ago
I feel like getting to the point that my container restarts when it dies is hard from just userdata loaded in when the instance is created. Everyone will let me spawn a base image with docker installed and shove in a userdata script but I want to say "please run these 3 docker images and if any of them die restart it" Additionally in k8s I can say "oh and update my dns provider to point to this pod and by the way stick TLS in front of it". I like those last bits a lot too.
cyansmoker 7y ago
Adding to what was already said: - being able to force better design when engineers have to work with multiple services depending on each other (using start-up containers, readiness and liveliness, etc)
quickthrower2 7y ago
That's a question I ask myself. I use Kubernetes because I inherited a Terraform/Kubernetes project. I like the way it all works now I have learnt it, but I wouldn't have been brave enough to set that lot up on my own. I'd probably use a VM and Shell script. But I'm glad for the experience.
- speedplane 7y ago
  > I use Kubernetes because I inherited a Terraform/Kubernetes project... but I wouldn't have been brave enough to set that lot up on my own. I'd probably use a VM and Shell script. If you use a shell script to handle a cluster with a dozen docker images and nodes, with intermittent crashes, out-of-memory issues, running out of disk, or network failures, your shell script will be so complex that you will basically need to recreate kubernetes from scratch.
  - quickthrower2 7y ago
    Good point. The setup is quite simple in terms of number or containers/nodes. We are using Azure so would probably use cloud features to do the things you mention. A load balancer and a couple of VMs for example, with auto scaling set up.
    - speedplane 7y ago
      > The setup is quite simple ... We are using Azure so would probably use cloud features ... [a] load balancer and a couple of VMs for example, with auto scaling set up. Load balancing and auto-scaling are some of the more basic and popular services that kubernetes provides. If your platform already provides them for you, it makes the case for kubernetes weaker. That said, kubernetes can be run on multiple cloud providers, and provides far more features. Once the complexity of the cluster grows, and vendor lock-in issues increase, kubernetes makes more and more sense.
solatic 7y ago
We're a B2B company with a microservice-architecture SaaS that wants to start to sell an on-prem offering. Kubernetes is a no-brainer. Even if we had a monolith instead, and no plans for an on-prem offering, ultimately I still think that a managed Kubernetes offering makes sense. Efficient resource utilization is all the more important for small companies, and once you have more than a handful of servers, Kubernetes makes it much easier to right-size your fleet by handling all of the scheduling for you. If you have a couple of servers and that's it, then sure, Kubernetes isn't giving you much. But if you're building a professional offering then you're likely to outgrow that couple of servers pretty quickly.
- founderling 7y ago
  Are you sure you can't simply run it all from a single server? Computers are pretty fast these days.
  - marcinzm 7y ago
    And then the server goes down and you've got downtime. Or you want to release a new version and you have downtime (ie: no rolling upgrades). At some point you end up re-inventing enough of k8s to makes it sort of silly to not use the actual thing.
  - solatic 7y ago
    It's a big data company doing an amount of traffic per day that exceeds what you can fit in RAM, including those fancy x1.32xl's with 2 TB RAM and a monthly cost of ~$6,000 for a reserved instance. Yeah, I'm pretty sure we can't simply run it all from a single server.
skboosh 7y ago
Network policies, RBAC, the ecosystem...
jjeaff 7y ago
I only run on a 2+ node cluster. -Rolling deployments -auto scaling pods and nodes -health checks and self healing
Benjamin_Dobell 7y ago
Unless you're already a k8s guru, then I don't really recommend k8s for most start-ups. Also, even if you are contemplating moving to k8s, I'd suggest waiting until https://github.com/kubernetes-sigs/cluster-api-provider-aws https://github.com/kubernetes-sigs/cluster-api-provider-aws stabilises. In saying that, I've just performed a reasonably significant migration to k8s, and despite the significant time investment, I am quite happy with it. However, prior to this we spent 5 years on Dokku[1], which is freaking fantastic project and was for the most part more than enough to meet our needs. It solves pretty much all the same issues start-ups use k8s for, with a lot less overhead. The reason for the migration was simply that we've reached the point where our clients demand improved reliability (redundancy) and somewhat coincidentally we'd outgrown some other infrastructure; which on its own would require a large migration. So we moved to k8s at the same time as rejigging our infrastructure, for redundancy and future-proofing purposes. [1] https://github.com/dokku/dokku https://github.com/dokku/dokku
marcinzm 7y ago
I run a data infrastructure on terraform and kubernetes. Things it gives me out of the box: * DNS. This lets all the various components talk to each other across nodes (Airflow, Spark, Zeppelin, etc.). I can also, via VPN, connect to things to look at what's going on. * Load-balancing. I create pods and then they run on the node where there's capacity without me thinking about it. * Auto-scaling. I spin up pods and nodes get created to handle the load. * Helm charts for things so I don't have to figure out how to run them myself. * Built-in support in the tools I use. Airflow will spin up k8s pods to run tasks. Spark will spin up pods to run a job. Etc.

hueving 7y ago

Resume boosting. Using shell scripts is lame and straight forward. If you have a declarative configuration though you can scale to thousands of nodes with nary a thought.

jrockway 7y ago
What service did you scale across thousands of nodes with a shell script?
marcinzm 7y ago
If you have a single service that runs of thousands of nodes then okay. If you have dozens of services that run across even dozens of nodes, that interact with each other, that have resource constraints, that need rolling updates, depend on external volumes, require security policies, etc. then you eventually recreate enough k8s to make it simpler to just use k8s.

vitalus 7y ago

- Simple DNS based service discovery - namespace separations between resources - optimized resource utilization and container scheduling These are the biggest 3 for the smaller orgs I’ve been a part of

founderling 7y ago
Interesting. Can you give an example of why DNS based service discovery was needed?
- markbnj 7y ago
  > Can you give an example of why DNS based service discovery was needed? Not to be flippant, but any case where things need to connect to other things. At scale they usually need to connect to other things through some sort of load balancer. You get that out of the box with kubernetes for services hosted inside the cluster, and there are straightforward solutions to ingress for clients outside it. Another important feature is pod scheduling. Yes you could wire up a few machines using docker compose and any of a few different networking approaches, but if one of your VMs dies are those workloads going to move to a healthy instance by themselves?
- andrewstuart2 7y ago
  Because I can run `ping` in a busybox container that knows how to do all the same service discovery as a more complicated fully fledged microservice. No extra libraries. No smattering of supporting services (on top of DNS, which you need anyway, of course). Setup and debugging is 10x simpler than any of the more complicated service discovery solutions. And it just works. Simply and intuitively. With everything. Because it's been an IETF standard since 1983.
- seabrookmx 7y ago
  Need is a strong word. But you don't need containers either. Service discovery just makes it easier to link up services if your architecture is truly microservice. We used to have an incredible amount of config that tightly coupled our API's.. now we use a combination of service discovery and an API gateway (Ambassador) to decouple the services, cut down on the number of random endpoints in our config, and we also get the added benefit of load balancing, rate limiting, and additional logging. There's always a tradeoff with scale. If you have four servers than obviously all of this stuff is overkill.
  - andrewstuart2 7y ago
    > If you have four servers than obviously all of this stuff is overkill. I disagree, actually. I have four servers at home, and have some pods that have been running little tinkery things, and a bunch of open source software, with ridiculous uptime and little or no effort, even when I reboot one of those "servers" to do some gaming on Windows. Now, do I need that uptime for all of those services? Not really. But for some of them, I want it, and it'd be annoying if I had to go figure out why they'd stopped running. The reality is, things just keep ticking without me worrying when they're on k8s. These skills transfer into very in-demand job skills as well, and if I ever build anything that gains traction, I already have all the tools, configs, and knowledge to deploy that app across 500 generic cloud servers.

jhgaylor 7y ago

I feel like getting to the point that my container restarts when it dies is hard from just userdata loaded in when the instance is created. Everyone will let me spawn a base image with docker installed and shove in a userdata script but I want to say "please run these 3 docker images and if any of them die restart it" Additionally in k8s I can say "oh and update my dns provider to point to this pod and by the way stick TLS in front of it". I like those last bits a lot too.

cyansmoker 7y ago

Adding to what was already said: - being able to force better design when engineers have to work with multiple services depending on each other (using start-up containers, readiness and liveliness, etc)

quickthrower2 7y ago

That's a question I ask myself. I use Kubernetes because I inherited a Terraform/Kubernetes project. I like the way it all works now I have learnt it, but I wouldn't have been brave enough to set that lot up on my own. I'd probably use a VM and Shell script. But I'm glad for the experience.

speedplane 7y ago
> I use Kubernetes because I inherited a Terraform/Kubernetes project... but I wouldn't have been brave enough to set that lot up on my own. I'd probably use a VM and Shell script. If you use a shell script to handle a cluster with a dozen docker images and nodes, with intermittent crashes, out-of-memory issues, running out of disk, or network failures, your shell script will be so complex that you will basically need to recreate kubernetes from scratch.
- quickthrower2 7y ago
  Good point. The setup is quite simple in terms of number or containers/nodes. We are using Azure so would probably use cloud features to do the things you mention. A load balancer and a couple of VMs for example, with auto scaling set up.
  - speedplane 7y ago
    > The setup is quite simple ... We are using Azure so would probably use cloud features ... [a] load balancer and a couple of VMs for example, with auto scaling set up. Load balancing and auto-scaling are some of the more basic and popular services that kubernetes provides. If your platform already provides them for you, it makes the case for kubernetes weaker. That said, kubernetes can be run on multiple cloud providers, and provides far more features. Once the complexity of the cluster grows, and vendor lock-in issues increase, kubernetes makes more and more sense.

solatic 7y ago

We're a B2B company with a microservice-architecture SaaS that wants to start to sell an on-prem offering. Kubernetes is a no-brainer. Even if we had a monolith instead, and no plans for an on-prem offering, ultimately I still think that a managed Kubernetes offering makes sense. Efficient resource utilization is all the more important for small companies, and once you have more than a handful of servers, Kubernetes makes it much easier to right-size your fleet by handling all of the scheduling for you. If you have a couple of servers and that's it, then sure, Kubernetes isn't giving you much. But if you're building a professional offering then you're likely to outgrow that couple of servers pretty quickly.

founderling 7y ago
Are you sure you can't simply run it all from a single server? Computers are pretty fast these days.
- marcinzm 7y ago
  And then the server goes down and you've got downtime. Or you want to release a new version and you have downtime (ie: no rolling upgrades). At some point you end up re-inventing enough of k8s to makes it sort of silly to not use the actual thing.
- solatic 7y ago
  It's a big data company doing an amount of traffic per day that exceeds what you can fit in RAM, including those fancy x1.32xl's with 2 TB RAM and a monthly cost of ~$6,000 for a reserved instance. Yeah, I'm pretty sure we can't simply run it all from a single server.

skboosh 7y ago

Network policies, RBAC, the ecosystem...

jjeaff 7y ago

I only run on a 2+ node cluster. -Rolling deployments -auto scaling pods and nodes -health checks and self healing

Benjamin_Dobell 7y ago

Unless you're already a k8s guru, then I don't really recommend k8s for most start-ups. Also, even if you are contemplating moving to k8s, I'd suggest waiting until https://github.com/kubernetes-sigs/cluster-api-provider-aws https://github.com/kubernetes-sigs/cluster-api-provider-aws stabilises. In saying that, I've just performed a reasonably significant migration to k8s, and despite the significant time investment, I am quite happy with it. However, prior to this we spent 5 years on Dokku[1], which is freaking fantastic project and was for the most part more than enough to meet our needs. It solves pretty much all the same issues start-ups use k8s for, with a lot less overhead. The reason for the migration was simply that we've reached the point where our clients demand improved reliability (redundancy) and somewhat coincidentally we'd outgrown some other infrastructure; which on its own would require a large migration. So we moved to k8s at the same time as rejigging our infrastructure, for redundancy and future-proofing purposes. [1] https://github.com/dokku/dokku https://github.com/dokku/dokku

marcinzm 7y ago

I run a data infrastructure on terraform and kubernetes. Things it gives me out of the box: * DNS. This lets all the various components talk to each other across nodes (Airflow, Spark, Zeppelin, etc.). I can also, via VPN, connect to things to look at what's going on. * Load-balancing. I create pods and then they run on the node where there's capacity without me thinking about it. * Auto-scaling. I spin up pods and nodes get created to handle the load. * Helm charts for things so I don't have to figure out how to run them myself. * Built-in support in the tools I use. Airflow will spin up k8s pods to run tasks. Spark will spin up pods to run a job. Etc.