4 ms·

Only Kubernetes has facilities for hosting external services reliably. I'm not sure why these other lower level tools are being compared to it. Especially Meso

Only Kubernetes has facilities for hosting external services reliably. I'm not sure why these other lower level tools are being compared to it. Especially Mesos and Marathon, which are much lower level.

manojlds 11y ago
Marathon and Kubernetes are in the same level, why do you call it lower level. Mesos, yes, and Kubernetes can run on Mesos too.
- KirinDave 11y ago
  Maraton schedules long running tasks, but doesn't offer any explicit facilities for binding a homogenous group into a relible responder set. At least, last time I checked. I can see components in the Marathon API to help you build such a thing. But Kubernetes deals with a fundamentally more abstract set of primitives: groups of containers (pods), replicated pods, and automatically managed dispatch points to said groups called services. The Marathon documentation literally recommends you hand-roll a periodically updated haproxy service to accomplish that last bit. While K8S introduces a fair amount of overhead to accomplish this feat reliably, that overhead pays off by having a very responsive and available system. So yeah. K8s is working one level up the stack, as I see it. I think you can make something reasonable with only Marathon, and that thought is backed up by the reality of people doing it. But my own experience suggests K8S does more for you here. Hence my excitement about K8S-mesos.
  - ersoft 11y ago
    I managed to achieve load balancing for a fleet of microservices using vulcand [0] and marathon callbacks [1]. It listens for marathon callbacks (for app create/update/scale), or when a task is killed/lost and updates the backends in etcd, propagating to all vulcand load balancers. The tool is available on github [2] [0] https://vulcand.io/proxy.html#backends-and-servers https://vulcand.io/proxy.html#backends-and-servers [1] https://mesosphere.github.io/marathon/docs/event-bus.html https://mesosphere.github.io/marathon/docs/event-bus.html [2] https://github.com/kuende/kameni https://github.com/kuende/kameni
    - KirinDave 11y ago
      This is a good thing. Perhaps you could get the Maraton folks to stop recommending such an awkward solution in preference to your work?
      - ersoft 11y ago
        Well, it depends. My solution is opinionated, using only one vulcand cluster for load balancing, multiple vulcand servers listening on one etcd namespace. For a small number of nodes (< 100) and requests (< 8000 req/s on each vulcand server) this vulcand approach is ok. For large scale, HAProxy/Nginx supports a lot more req/s than vulcand, and I think it can also be configured using confd [0] using a similar aproach: listen for marathon events, update etcd keys, then confd will listen for changes and reload HAProxy/Nginx [0] http://ox86.tumblr.com/post/90554410668/easy-scaling-with-docker-haproxy-and-confd http://ox86.tumblr.com/post/90554410668/easy-scaling-with-do...
- KirinDave 11y ago
  Just to double down on that previous point, check out the code for haproxy-marathon-bridge. https://github.com/mesosphere/marathon/blob/master/bin/haproxy-marathon-bridge https://github.com/mesosphere/marathon/blob/master/bin/hapro...
deleted 11y ago
[deleted]
amouat 11y ago
They are being compared as they are the main options for clustering and scheduling containers. I'd agree Kubernetes is at a higher level to the other options, or at least comes with more features. I'm not sure what you mean by "hosting external services reliably" - what's external and who is unreliable?
- amouat 11y ago
  Hmm, I'll answer my own question - you mean has a load-balancer built in, don't you?
  - KirinDave 11y ago
    And that doesn't recommend a comically brittle solution (as in marathon) or provide no facilities at all (as in swarm). There's nothing wrong with these decisions. It's just odd to compare this group. It makes me wonder how people are considering these options and in what context. You cannot field a reliable and available service to the outside world without this component. And the instrumentation and operationalization of this component is tricky.
    - amouat 11y ago
      Yeah, I agree that the load-balancer is important and not adequately addressed. I guess most people rig up their own set-up based on haproxy currently. I don't get why you think I can't compare that group though. What should I compare Kubernetes with? The group represents the various approaches to clustering and orchestration that most people will consider for container deployments. They are all slightly different, which is what makes for an interesting comparison.
      - KirinDave 11y ago
        K8S pretty much stands on its own in the Docker ecosystem right now. I'm not aware of good alternatives that aren't comically janky as you described.

manojlds 11y ago

Marathon and Kubernetes are in the same level, why do you call it lower level. Mesos, yes, and Kubernetes can run on Mesos too.

KirinDave 11y ago
Maraton schedules long running tasks, but doesn't offer any explicit facilities for binding a homogenous group into a relible responder set. At least, last time I checked. I can see components in the Marathon API to help you build such a thing. But Kubernetes deals with a fundamentally more abstract set of primitives: groups of containers (pods), replicated pods, and automatically managed dispatch points to said groups called services. The Marathon documentation literally recommends you hand-roll a periodically updated haproxy service to accomplish that last bit. While K8S introduces a fair amount of overhead to accomplish this feat reliably, that overhead pays off by having a very responsive and available system. So yeah. K8s is working one level up the stack, as I see it. I think you can make something reasonable with only Marathon, and that thought is backed up by the reality of people doing it. But my own experience suggests K8S does more for you here. Hence my excitement about K8S-mesos.
- ersoft 11y ago
  I managed to achieve load balancing for a fleet of microservices using vulcand [0] and marathon callbacks [1]. It listens for marathon callbacks (for app create/update/scale), or when a task is killed/lost and updates the backends in etcd, propagating to all vulcand load balancers. The tool is available on github [2] [0] https://vulcand.io/proxy.html#backends-and-servers https://vulcand.io/proxy.html#backends-and-servers [1] https://mesosphere.github.io/marathon/docs/event-bus.html https://mesosphere.github.io/marathon/docs/event-bus.html [2] https://github.com/kuende/kameni https://github.com/kuende/kameni
  - KirinDave 11y ago
    This is a good thing. Perhaps you could get the Maraton folks to stop recommending such an awkward solution in preference to your work?
    - ersoft 11y ago
      Well, it depends. My solution is opinionated, using only one vulcand cluster for load balancing, multiple vulcand servers listening on one etcd namespace. For a small number of nodes (< 100) and requests (< 8000 req/s on each vulcand server) this vulcand approach is ok. For large scale, HAProxy/Nginx supports a lot more req/s than vulcand, and I think it can also be configured using confd [0] using a similar aproach: listen for marathon events, update etcd keys, then confd will listen for changes and reload HAProxy/Nginx [0] http://ox86.tumblr.com/post/90554410668/easy-scaling-with-docker-haproxy-and-confd http://ox86.tumblr.com/post/90554410668/easy-scaling-with-do...
KirinDave 11y ago
Just to double down on that previous point, check out the code for haproxy-marathon-bridge. https://github.com/mesosphere/marathon/blob/master/bin/haproxy-marathon-bridge https://github.com/mesosphere/marathon/blob/master/bin/hapro...

deleted 11y ago

[deleted]

amouat 11y ago

They are being compared as they are the main options for clustering and scheduling containers. I'd agree Kubernetes is at a higher level to the other options, or at least comes with more features. I'm not sure what you mean by "hosting external services reliably" - what's external and who is unreliable?

amouat 11y ago
Hmm, I'll answer my own question - you mean has a load-balancer built in, don't you?
- KirinDave 11y ago
  And that doesn't recommend a comically brittle solution (as in marathon) or provide no facilities at all (as in swarm). There's nothing wrong with these decisions. It's just odd to compare this group. It makes me wonder how people are considering these options and in what context. You cannot field a reliable and available service to the outside world without this component. And the instrumentation and operationalization of this component is tricky.
  - amouat 11y ago
    Yeah, I agree that the load-balancer is important and not adequately addressed. I guess most people rig up their own set-up based on haproxy currently. I don't get why you think I can't compare that group though. What should I compare Kubernetes with? The group represents the various approaches to clustering and orchestration that most people will consider for container deployments. They are all slightly different, which is what makes for an interesting comparison.
    - KirinDave 11y ago
      K8S pretty much stands on its own in the Docker ecosystem right now. I'm not aware of good alternatives that aren't comically janky as you described.