3 ms·

I haven't used Kubernetes yet since I have a Mesos scheduler that runs heavy workloads on-demand and wants to use all unused resources for a few minutes. I can'

I haven't used Kubernetes yet since I have a Mesos scheduler that runs heavy workloads on-demand and wants to use all unused resources for a few minutes. I can't see a way to configure a k8 service to use either zero or as many pods as it possibly could, without writing something to watch the cluster's metadata and update the service's config files

hardwaresofton 8y ago
Kubernetes has resource limiting built in, in the form of Resource Quotas[0] and more practically (or more straight forward to use) resources for pods[1]. The system's not perfect, so it's likely possible to overcommit machines, but there are QoS controls for critical workloads. Kubernetes is built in a way that it's easy for people to write something that watches cluster metadata and perform actions (controller pattern), as such a lot of functionality is built in gradually overtime just like that. I'm not sure when resource management first came on the scene but it's been around for a while. You can also use Kubernetes to manage resources completely unrelated to Kubernetes by bringing Custom Resource Definitions ("CRDs") into play -- that's when you create a "fake" Kubernetes resource that (ex. VirtualMachine) manages some resource that's actually on the machine. The combination of CRDs and controllers to manage them is called the "Operator pattern"[2] and it's gaining a lot of hype right now as people wrestle with it conceptually but it's been around the whole time. [0]: https://kubernetes.io/docs/concepts/policy/resource-quotas/ https://kubernetes.io/docs/concepts/policy/resource-quotas/ [1]: https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/ https://kubernetes.io/docs/concepts/configuration/manage-com... [2]: https://coreos.com/blog/introducing-operators.html https://coreos.com/blog/introducing-operators.html
- falsedan 8y ago
  Yeah with bare mesos I don't have to do any of that, I just accept resource offers and release them when I'm done. The scheduler decides how much it will scale, not mesos. We like to create a new framework for each instance of the job (so we can track slow runs, trial-run new versions), so we can have anywhere between 0 and 50 similar frameworks all running and vying for 100% of the cluster. Naturally, we have cluster-level scaling, so we add nodes until we hit our max spend per hour, and then jobs take a bit longer to complete.
  - hardwaresofton 8y ago
    It was kind of hard to understand what you were describing without reading the Mesos Architecture documentation[0], but I think I get it now. When you say "you" just accept resource offers, you mean the frameworks (in mesos terminology) you're deploying correct? Weirdly enough, Mesos resembles a system I was building in my head that I thought could compete with Kubernetes... Taking the resource supplier (agents in mesos-speak) and consumer (frameworks in mesos-speak) paradigm to the extreme. [0]: https://mesos.apache.org/documentation/latest/architecture/ https://mesos.apache.org/documentation/latest/architecture/
    - falsedan 8y ago
      Essentially, yeah. You don't really deploy a framework, you just start its scheduler. I recommend the technical paper, it's clear and short[0]. 0: https://people.eecs.berkeley.edu/~alig/papers/mesos.pdf https://people.eecs.berkeley.edu/~alig/papers/mesos.pdf
    - halbritt 8y ago
      I've been running a reasonably large kubernetes cluster (~600 cores) in production for the last 6 months or so. I've had to deal with hotspots, both IO, CPU and load (which could be either). Getting requests and limits dialed has been a bit of a challenge. I worked with Mesos a bit last year and miss the sophistication of the scheduler. Granted, I never ran it at the same scale, but I felt like the approach was much easier to grok. I've seen a few projects floating around meant help dynamically tune resource utilization in k8s. By this time next year, I imagine those will be fairly commonplace. I'm definitely looking forward to using descheduler: https://github.com/kubernetes-incubator/descheduler https://github.com/kubernetes-incubator/descheduler

hardwaresofton 8y ago

Kubernetes has resource limiting built in, in the form of Resource Quotas[0] and more practically (or more straight forward to use) resources for pods[1]. The system's not perfect, so it's likely possible to overcommit machines, but there are QoS controls for critical workloads. Kubernetes is built in a way that it's easy for people to write something that watches cluster metadata and perform actions (controller pattern), as such a lot of functionality is built in gradually overtime just like that. I'm not sure when resource management first came on the scene but it's been around for a while. You can also use Kubernetes to manage resources completely unrelated to Kubernetes by bringing Custom Resource Definitions ("CRDs") into play -- that's when you create a "fake" Kubernetes resource that (ex. VirtualMachine) manages some resource that's actually on the machine. The combination of CRDs and controllers to manage them is called the "Operator pattern"[2] and it's gaining a lot of hype right now as people wrestle with it conceptually but it's been around the whole time. [0]: https://kubernetes.io/docs/concepts/policy/resource-quotas/ https://kubernetes.io/docs/concepts/policy/resource-quotas/ [1]: https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/ https://kubernetes.io/docs/concepts/configuration/manage-com... [2]: https://coreos.com/blog/introducing-operators.html https://coreos.com/blog/introducing-operators.html

falsedan 8y ago
Yeah with bare mesos I don't have to do any of that, I just accept resource offers and release them when I'm done. The scheduler decides how much it will scale, not mesos. We like to create a new framework for each instance of the job (so we can track slow runs, trial-run new versions), so we can have anywhere between 0 and 50 similar frameworks all running and vying for 100% of the cluster. Naturally, we have cluster-level scaling, so we add nodes until we hit our max spend per hour, and then jobs take a bit longer to complete.
- hardwaresofton 8y ago
  It was kind of hard to understand what you were describing without reading the Mesos Architecture documentation[0], but I think I get it now. When you say "you" just accept resource offers, you mean the frameworks (in mesos terminology) you're deploying correct? Weirdly enough, Mesos resembles a system I was building in my head that I thought could compete with Kubernetes... Taking the resource supplier (agents in mesos-speak) and consumer (frameworks in mesos-speak) paradigm to the extreme. [0]: https://mesos.apache.org/documentation/latest/architecture/ https://mesos.apache.org/documentation/latest/architecture/
  - falsedan 8y ago
    Essentially, yeah. You don't really deploy a framework, you just start its scheduler. I recommend the technical paper, it's clear and short[0]. 0: https://people.eecs.berkeley.edu/~alig/papers/mesos.pdf https://people.eecs.berkeley.edu/~alig/papers/mesos.pdf
  - halbritt 8y ago
    I've been running a reasonably large kubernetes cluster (~600 cores) in production for the last 6 months or so. I've had to deal with hotspots, both IO, CPU and load (which could be either). Getting requests and limits dialed has been a bit of a challenge. I worked with Mesos a bit last year and miss the sophistication of the scheduler. Granted, I never ran it at the same scale, but I felt like the approach was much easier to grok. I've seen a few projects floating around meant help dynamically tune resource utilization in k8s. By this time next year, I imagine those will be fairly commonplace. I'm definitely looking forward to using descheduler: https://github.com/kubernetes-incubator/descheduler https://github.com/kubernetes-incubator/descheduler