5 ms·

Dunno this is really a huge deal -- they're different tools with different purposes. I've always viewed Mesos as a bare-metal orchestration platform. From ther

Dunno this is really a huge deal -- they're different tools with different purposes. I've always viewed Mesos as a bare-metal orchestration platform. From there, you can deploy k8s, VMware, Linux KVM, whatever to the nodes under management by Mesos. k8s is more of a container management platform that can be used for applications to tie arbitrary software architectures into a set of unified operations tooling. So you may host your cluster on k8s, but you would use Mesos to auto-scale the active servers in your clusters for say, power management. But Mesos is something you probably won't have a need for until you're well past managing one k8s cluster.

pram 9y ago
Does kubernetes actually work as a Mesos framework? I thought that was the point of marathon.
- nemothekid 9y ago
  I don't think k8s operates as a Mesos framework, but the use cases for k8s is a lot closer to Marathon than Mesos. For most use cases k8s and marathon satisfy the same needs. If the only framework you are running on Mesos is Marathon, then you may be better off with k8s.
  - SEJeff 9y ago
    "I don't think k8s operates as a Meoss framework" Have you ever used Mesos? It is the only way k8s could run ontop of Mesos. Also: https://github.com/kubernetes-incubator/kube-mesos-framework https://github.com/kubernetes-incubator/kube-mesos-framework Also, you could maybe compare Marathon to Kubernetes Dashboard + the Kubernetes apiserver maybe, but comparing the two directly is not remotely close. With Mesos + Marathon, you still need a load balancer for both North <---> South AND East <---> West cluster traffice. This is builtin to Kubernetes via Service and Ingress objects and their associated controllers. There is also Federation, and so much more. Mesos is just a resource scheduler, and a very good two level one at that, built to prove the theory of two level schedulers working well with interactive and batch workloads concurrently. Kubernetes is the entire package and includes a default scheduler, but allows you to plug in your own. Disclaimer: I spent a full year working on Mesos and went to MesosCon twice. After speaking in person with Kelsey Hightower at Monitorama, he convinced me to switch to Kubernetes on purely technical reasons. The stuff that you have to DIY on Mesos is built sensibly and pluggably in Kubernetes. From a cluster operator standpoint frankly, Kubernetes is leagues ahead. Lots of fortune 500s and enterprise customers find the lack if sensible RBAC and real to the core auditing support in Mesos to be seriously lacking. https://twitter.com/nirajtolia/status/903437346443378688 https://twitter.com/nirajtolia/status/903437346443378688
- hartem_ 9y ago
  It does work as a Mesos framework now.
- takeda 9y ago
  Marathon was just Mesosphere's (not even Mesos') way of running docker containers. It is very simple, and it gained more popularity than naive mesos' container solution - Aurora). At this point kubernetes' won in the container area, but that doesn't mean mesos is no longer useful. Running containers was just one of things it could do, its main purpose is to manage hardware. As for your original question, I don't see any other way mesos would run kubernetes than as a framework. That is the whole point, otherwise there would be no point of using mesos. BTW there already was kubernetes framework, but it seemd to be abandoned since k8s 1.3 or 1.2. I'm glad it was resurrected.
  - amazingman 9y ago
    >At this point kubernetes' won in the container area This seems like an overstatement. From where I’m standing, it seems like there’s plenty of room for many solutions to thrive in this space. I use and like Kubernetes in local development at my org, and we’re probably going to be testing production workloads on Kubernetes soon. The learning curve is quite steep and almost entirely front-loaded. There will be a significant portion of companies for which a different solution is more desirable, based on that fact alone.
yebyen 9y ago
It is a big deal, because I (my last employer) spent ten(s?) of thousands of dollars on VMware so that I would not have to put infrastructure nodes on bare metal. They created vSAN so it would be so. (Modern infrastructure I'm sure has better solutions than vSAN, but it worked great for us, and I'm out of there now. We never honestly needed that until we had large Windows servers to take care of. I personally have a distaste and distrust for Windows servers, but when the new owner came he was the boss, and that was where they were invested. (Yeah, let's go with invested...)) Some of those infrastructure nodes were Windows machines, and (most) of them were Linux VMs. I have not used Mesos yet (I am using Kubernetes in dev only, but my org has not drank any of the kool-aid yet), it kind of sounds like I want Mesos (and Marathon?) for the same reasons that I wanted VMware when I had only one Windows server and I literally couldn't live without it when I suddenly had dozens more Windows servers. (I heard that OpenShift will soon run Windows containers. I can't imagine it will be long after that before Kubernetes at-large also does.) I'm not sure I need Mesos, Kubernetes does an awful lot, but it still seems like Kubernetes Admin user could royally fuck things up for me on a multi-tenant system that I admin (I learned the ways from OpenShift), if it's just Kubernetes. But on Mesos, maybe I can get some guarantees that I couldn't without it... I'm not sure how much anyone has a "need" for Mesos. If I understand exactly what makes it different from Kubernetes, it's one of those things that you just kind of layer on top so that when your suspenders fall right off for some reason, your belt actually can still keep your pants up. I honestly don't know any more about Mesos than what I've been told, but as a belt-and-suspenders kind of guy I think it could actually be for me. What say you to that?
- haimez 9y ago
  Ahem... Millennials.
  - dang 9y ago
    Would you please not post unsubstantive comments to HN? https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html
- devonkim 9y ago
mgummelt 9y ago
How do you see Mesos as more bare-metal than k8s? Also, both Mesos and k8s deploy containers. I'm always confused when people call k8s a container orchestration tool, but not Mesos. What makes a container orchestration tool? Does everyone agree on this definition or are we talking past each other? At the very least, Mesos was considered a container orchestration tool a couple years ago. Something seems to have changed in the popular discourse such that now k8s and Marathon are considered "container orchestration", yet Mesos isn't.
- ianburrell 9y ago
  Mesos is a cluster manager. It coordinates running tasks on a cluster. It can run Docker container tasks but can also run local programs. Most people are interested in running containers because makes easier to deploy and isolate programs. It can be used for container orchestration but isn't only container orchestration.
  - justicezyx 9y ago
    The are the same thing.
    - yebyen 9y ago
      It is not. And I am hoping that someone will correct me if I am wrong about some of this (because I don't really want to learn Mesos for myself), but here is how it is not: In Kubernetes, you have nodes. Your containers run directly on nodes. This is very much the same as running your kernel inside of a VM. It's there, on the computer, running. One kernel, one VM. One node, many containers, but the containers are still each running on exactly one node. There are however five or more abstractions between your containers and the nodes that they run on (ingress routes traffic into service, which points at deployment, and under deployment is a ReplicaSet, that spawns pods, which may run containers in them). Through these you may arrange to provide Highly Available service guarantees. But if that node goes down, your container is down until the pod can be restarted on a different node, or possibly until that node comes back. So you ensure that your pods can be scaled out horizontally through some potentially painful process if your product is not new and you've made bad decisions, or if it's from a vendor and the vendor has made some bad decisions that are perhaps not easy to undo. (If you chose the wrong vendor. It happens. Time to update your resume perhaps.) In Mesos, you have Agents and physical machines, and the agents take care of "work units" that are roughly a share of some task in the same sense that on VMware DRS[1], your RAM and CPU cycles are drawn from a pool, and they may come from any of the physical nodes in your cluster at any time. I think this was called "non-locality" in vSAN. Do not think about your data as being on a disk, because enough hot copies of it may be scattered across a fleet of disks that at any given time, you might even have a local copy but an agent decides that you're going to read from and write to somewhere else across 10gbit backplane instead. That's where I'll draw the parallel, and that's where I'll stop trying to draw parallels because I'm not trying to tell you about VMware, and hope that someone comes along to correct me if I'm actually telling lies about Mesos. Because I don't really know about Mesos and could be talking right out of my ass. But from what I've read, I'm thinking not too far off. (I had access to full ESXi clustering and vSAN license for a year, and we turned on all those features before I left that company. This is why I think that I know something about VMware, at least.) I'd assume that you can get the same kind of guarantees from Mesos that I could get from VMware's DRS and HA solutions. In other words, if a node goes down and I have enabled all the, err, checkboxes... ...then I won't need to wait for that node to be rescheduled, I won't even notice an interruption in the continuous operation of the container(s) unless the loss of a node represents enough missing CPU cycles that it puts my cluster "over the edge." I don't need to have pod replicas in the sense that Kubernetes needed me to have them in order to provide that guarantee. Because they are not down. Their shares of work are just sent to another worker from the pool seamlessly. (They might not even have been scheduled onto one physical machine during normal operation, the agent decides where to send the shares and may change its mind about that any time based on new information about load. You're never supposed to notice that unless you're paying very close attention to it.) I'm extrapolating how I think Mesos must work based on what I've heard about it, and how I know you can get VMware to do if you're willing to spend millions of dollars scaling it out, or if your hardware is already big enough and you know the salesman that is willing to sell you a zero-support no-maintenance-included contract at 97% discount so you can use the HA and FT features when you need them. (I used ESXi for 4 years without any of these features and it was fine, but they are nice features and you need to arrange things a bit differently when you don't have them. vSAN is not RAID5.) [1]: "What is DRS" VMware DRS (Distributed Resource Scheduler) is a utility that balances computing workloads with available resources in a virtualized environment. The utility is part of a virtualization suite called VMware Infrastructure 3.
ianburrell 9y ago
Mesos isn't a bare metal platform but a cluster manager that runs on OS (usually Linux). It coordinates clients scheduling tasks on the cluster. Tasks can be programs, containers, or KVM VMs.

pram 9y ago

Does kubernetes actually work as a Mesos framework? I thought that was the point of marathon.

nemothekid 9y ago
I don't think k8s operates as a Mesos framework, but the use cases for k8s is a lot closer to Marathon than Mesos. For most use cases k8s and marathon satisfy the same needs. If the only framework you are running on Mesos is Marathon, then you may be better off with k8s.
- SEJeff 9y ago
  "I don't think k8s operates as a Meoss framework" Have you ever used Mesos? It is the only way k8s could run ontop of Mesos. Also: https://github.com/kubernetes-incubator/kube-mesos-framework https://github.com/kubernetes-incubator/kube-mesos-framework Also, you could maybe compare Marathon to Kubernetes Dashboard + the Kubernetes apiserver maybe, but comparing the two directly is not remotely close. With Mesos + Marathon, you still need a load balancer for both North <---> South AND East <---> West cluster traffice. This is builtin to Kubernetes via Service and Ingress objects and their associated controllers. There is also Federation, and so much more. Mesos is just a resource scheduler, and a very good two level one at that, built to prove the theory of two level schedulers working well with interactive and batch workloads concurrently. Kubernetes is the entire package and includes a default scheduler, but allows you to plug in your own. Disclaimer: I spent a full year working on Mesos and went to MesosCon twice. After speaking in person with Kelsey Hightower at Monitorama, he convinced me to switch to Kubernetes on purely technical reasons. The stuff that you have to DIY on Mesos is built sensibly and pluggably in Kubernetes. From a cluster operator standpoint frankly, Kubernetes is leagues ahead. Lots of fortune 500s and enterprise customers find the lack if sensible RBAC and real to the core auditing support in Mesos to be seriously lacking. https://twitter.com/nirajtolia/status/903437346443378688 https://twitter.com/nirajtolia/status/903437346443378688
hartem_ 9y ago
It does work as a Mesos framework now.
takeda 9y ago
Marathon was just Mesosphere's (not even Mesos') way of running docker containers. It is very simple, and it gained more popularity than naive mesos' container solution - Aurora). At this point kubernetes' won in the container area, but that doesn't mean mesos is no longer useful. Running containers was just one of things it could do, its main purpose is to manage hardware. As for your original question, I don't see any other way mesos would run kubernetes than as a framework. That is the whole point, otherwise there would be no point of using mesos. BTW there already was kubernetes framework, but it seemd to be abandoned since k8s 1.3 or 1.2. I'm glad it was resurrected.
- amazingman 9y ago
  >At this point kubernetes' won in the container area This seems like an overstatement. From where I’m standing, it seems like there’s plenty of room for many solutions to thrive in this space. I use and like Kubernetes in local development at my org, and we’re probably going to be testing production workloads on Kubernetes soon. The learning curve is quite steep and almost entirely front-loaded. There will be a significant portion of companies for which a different solution is more desirable, based on that fact alone.

yebyen 9y ago

It is a big deal, because I (my last employer) spent ten(s?) of thousands of dollars on VMware so that I would not have to put infrastructure nodes on bare metal. They created vSAN so it would be so. (Modern infrastructure I'm sure has better solutions than vSAN, but it worked great for us, and I'm out of there now. We never honestly needed that until we had large Windows servers to take care of. I personally have a distaste and distrust for Windows servers, but when the new owner came he was the boss, and that was where they were invested. (Yeah, let's go with invested...)) Some of those infrastructure nodes were Windows machines, and (most) of them were Linux VMs. I have not used Mesos yet (I am using Kubernetes in dev only, but my org has not drank any of the kool-aid yet), it kind of sounds like I want Mesos (and Marathon?) for the same reasons that I wanted VMware when I had only one Windows server and I literally couldn't live without it when I suddenly had dozens more Windows servers. (I heard that OpenShift will soon run Windows containers. I can't imagine it will be long after that before Kubernetes at-large also does.) I'm not sure I need Mesos, Kubernetes does an awful lot, but it still seems like Kubernetes Admin user could royally fuck things up for me on a multi-tenant system that I admin (I learned the ways from OpenShift), if it's just Kubernetes. But on Mesos, maybe I can get some guarantees that I couldn't without it... I'm not sure how much anyone has a "need" for Mesos. If I understand exactly what makes it different from Kubernetes, it's one of those things that you just kind of layer on top so that when your suspenders fall right off for some reason, your belt actually can still keep your pants up. I honestly don't know any more about Mesos than what I've been told, but as a belt-and-suspenders kind of guy I think it could actually be for me. What say you to that?

haimez 9y ago
Ahem... Millennials.
- dang 9y ago
  Would you please not post unsubstantive comments to HN? https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html
devonkim 9y ago

mgummelt 9y ago

How do you see Mesos as more bare-metal than k8s? Also, both Mesos and k8s deploy containers. I'm always confused when people call k8s a container orchestration tool, but not Mesos. What makes a container orchestration tool? Does everyone agree on this definition or are we talking past each other? At the very least, Mesos was considered a container orchestration tool a couple years ago. Something seems to have changed in the popular discourse such that now k8s and Marathon are considered "container orchestration", yet Mesos isn't.

ianburrell 9y ago
Mesos is a cluster manager. It coordinates running tasks on a cluster. It can run Docker container tasks but can also run local programs. Most people are interested in running containers because makes easier to deploy and isolate programs. It can be used for container orchestration but isn't only container orchestration.
- justicezyx 9y ago
  The are the same thing.
  - yebyen 9y ago
    It is not. And I am hoping that someone will correct me if I am wrong about some of this (because I don't really want to learn Mesos for myself), but here is how it is not: In Kubernetes, you have nodes. Your containers run directly on nodes. This is very much the same as running your kernel inside of a VM. It's there, on the computer, running. One kernel, one VM. One node, many containers, but the containers are still each running on exactly one node. There are however five or more abstractions between your containers and the nodes that they run on (ingress routes traffic into service, which points at deployment, and under deployment is a ReplicaSet, that spawns pods, which may run containers in them). Through these you may arrange to provide Highly Available service guarantees. But if that node goes down, your container is down until the pod can be restarted on a different node, or possibly until that node comes back. So you ensure that your pods can be scaled out horizontally through some potentially painful process if your product is not new and you've made bad decisions, or if it's from a vendor and the vendor has made some bad decisions that are perhaps not easy to undo. (If you chose the wrong vendor. It happens. Time to update your resume perhaps.) In Mesos, you have Agents and physical machines, and the agents take care of "work units" that are roughly a share of some task in the same sense that on VMware DRS[1], your RAM and CPU cycles are drawn from a pool, and they may come from any of the physical nodes in your cluster at any time. I think this was called "non-locality" in vSAN. Do not think about your data as being on a disk, because enough hot copies of it may be scattered across a fleet of disks that at any given time, you might even have a local copy but an agent decides that you're going to read from and write to somewhere else across 10gbit backplane instead. That's where I'll draw the parallel, and that's where I'll stop trying to draw parallels because I'm not trying to tell you about VMware, and hope that someone comes along to correct me if I'm actually telling lies about Mesos. Because I don't really know about Mesos and could be talking right out of my ass. But from what I've read, I'm thinking not too far off. (I had access to full ESXi clustering and vSAN license for a year, and we turned on all those features before I left that company. This is why I think that I know something about VMware, at least.) I'd assume that you can get the same kind of guarantees from Mesos that I could get from VMware's DRS and HA solutions. In other words, if a node goes down and I have enabled all the, err, checkboxes... ...then I won't need to wait for that node to be rescheduled, I won't even notice an interruption in the continuous operation of the container(s) unless the loss of a node represents enough missing CPU cycles that it puts my cluster "over the edge." I don't need to have pod replicas in the sense that Kubernetes needed me to have them in order to provide that guarantee. Because they are not down. Their shares of work are just sent to another worker from the pool seamlessly. (They might not even have been scheduled onto one physical machine during normal operation, the agent decides where to send the shares and may change its mind about that any time based on new information about load. You're never supposed to notice that unless you're paying very close attention to it.) I'm extrapolating how I think Mesos must work based on what I've heard about it, and how I know you can get VMware to do if you're willing to spend millions of dollars scaling it out, or if your hardware is already big enough and you know the salesman that is willing to sell you a zero-support no-maintenance-included contract at 97% discount so you can use the HA and FT features when you need them. (I used ESXi for 4 years without any of these features and it was fine, but they are nice features and you need to arrange things a bit differently when you don't have them. vSAN is not RAID5.) [1]: "What is DRS" VMware DRS (Distributed Resource Scheduler) is a utility that balances computing workloads with available resources in a virtualized environment. The utility is part of a virtualization suite called VMware Infrastructure 3.

ianburrell 9y ago

Mesos isn't a bare metal platform but a cluster manager that runs on OS (usually Linux). It coordinates clients scheduling tasks on the cluster. Tasks can be programs, containers, or KVM VMs.