5 ms·

I keep thinking about Mesos and I don't get what the big deal is here. With low-overhead virtualization like linux containers combined with configuration manage

by 23david 13y ago

I keep thinking about Mesos and I don't get what the big deal is here. With low-overhead virtualization like linux containers combined with configuration management tools like Salstack, Ansible, Chef or Puppet, it isn't a big deal to have multiple properly configured applications running on individual physical machines. And to then distribute out workloads to those machines isn't super hard if you use distributed frameworks or job managers such as gearman, celery or something built on top of ZeroMQ. Am I missing something here? You could even get all fancy by using openvswitch to configure your own virtual network topology. It just doesn't seem so complicated what they're trying to accomplish.

fourspace 13y ago
I managed a Google SRE team that helped run Borg for 5 years, so I'm pretty familiar with this. You can certainly run multiple application servers on one machine using virtualization. What isn't easy is automating where the applications are deployed, in an elastic, constantly shifting way that deals with massive numbers of machines and constantly changing hardware. Cheap hardware crashes. It crashes all the time. Furthermore, the application's needs themselves change all the time, depending on traffic curves and processing needs. The dynamic needs of Google's (and presumably Twitter's) completely heterogeneous application stacks don't lend themselves to simple virtualization and over-the-counter software. This is an incredibly tricky bin packing problem that was never quite solved in my 5 years at Google. I don't know a lot about the "distributed frameworks" you mentioned, so perhaps they do this too. I kind of doubt it. If it were as easy as you think, I'm sure my friends at Twitter would be using what you mentioned.
jsnell 13y ago
You seem to be describing essentially a static allocation of dedicated machines. That's fine if there are a couple of different apps you want to run. It'd be totally insane if a single data centre has 10k+ servers that are permanently running hundreds of different services and thousands of largely unpredictable and transient batch jobs. (Why would it be insane? Because with dedicated machines your utilization levels are going to be crap, reconfiguring and rebalancing the machine allocations will be painful, and you'd need massive over-provisioning to achieve a sufficient level of fault-tolerance.)
btilly 13y ago
Add in monitoring. Both generic monitoring, and monitoring of application specific information produced in application specific ways. (While you're at it, make it easy for applications that reuse standard libraries to get standard monitoring that goes with said libraries.) Add in spinning jobs up and down quickly on demand rather than making them long-running. (This matters for a map-reduce, for instance.) Add in making it easy to have dev/production running side by side with configurations that are as constant as possible, and not getting into each other's way. Add in automatic failover from machine to machine, or from data center to data center as needed. Add in making it easy to find/access applications set up and configured by someone else so that you can set rpcs their way. After you keep adding things in, you understand why people think of this as a "data center operating system". And you realize that solving the basic problem - get jobs to run on a distributed set of machines - is only the visible first step in a long list of features that you want.
fiatmoney 13y ago
My impression is that the secret sauce is in the allocation between machines - managing the allocation of high-CPU vs. high-IO vs. high-memory jobs, while still preserving things like rack locality and latency guarantees. A multidimensional knapsack problem, that changes over time.
nivertech 13y ago
Try to write Chef recipe to manage several distributed systems on the same cluster (i.e. Hadoop, HBase, Cassandra, Kaffka, etc.) Using just CM tools are not enough and you will need also deployment management, resource management, monitoring, etc. Another takeout from this, that virtualization is not Web-scale, it's more appropriate to IT and public cloud workloads, large companies like Google prefer bar-metal.

by 23david 13y ago

fourspace 13y ago

I managed a Google SRE team that helped run Borg for 5 years, so I'm pretty familiar with this. You can certainly run multiple application servers on one machine using virtualization. What isn't easy is automating where the applications are deployed, in an elastic, constantly shifting way that deals with massive numbers of machines and constantly changing hardware. Cheap hardware crashes. It crashes all the time. Furthermore, the application's needs themselves change all the time, depending on traffic curves and processing needs. The dynamic needs of Google's (and presumably Twitter's) completely heterogeneous application stacks don't lend themselves to simple virtualization and over-the-counter software. This is an incredibly tricky bin packing problem that was never quite solved in my 5 years at Google. I don't know a lot about the "distributed frameworks" you mentioned, so perhaps they do this too. I kind of doubt it. If it were as easy as you think, I'm sure my friends at Twitter would be using what you mentioned.

jsnell 13y ago

You seem to be describing essentially a static allocation of dedicated machines. That's fine if there are a couple of different apps you want to run. It'd be totally insane if a single data centre has 10k+ servers that are permanently running hundreds of different services and thousands of largely unpredictable and transient batch jobs. (Why would it be insane? Because with dedicated machines your utilization levels are going to be crap, reconfiguring and rebalancing the machine allocations will be painful, and you'd need massive over-provisioning to achieve a sufficient level of fault-tolerance.)

btilly 13y ago

Add in monitoring. Both generic monitoring, and monitoring of application specific information produced in application specific ways. (While you're at it, make it easy for applications that reuse standard libraries to get standard monitoring that goes with said libraries.) Add in spinning jobs up and down quickly on demand rather than making them long-running. (This matters for a map-reduce, for instance.) Add in making it easy to have dev/production running side by side with configurations that are as constant as possible, and not getting into each other's way. Add in automatic failover from machine to machine, or from data center to data center as needed. Add in making it easy to find/access applications set up and configured by someone else so that you can set rpcs their way. After you keep adding things in, you understand why people think of this as a "data center operating system". And you realize that solving the basic problem - get jobs to run on a distributed set of machines - is only the visible first step in a long list of features that you want.

fiatmoney 13y ago

My impression is that the secret sauce is in the allocation between machines - managing the allocation of high-CPU vs. high-IO vs. high-memory jobs, while still preserving things like rack locality and latency guarantees. A multidimensional knapsack problem, that changes over time.

nivertech 13y ago

Try to write Chef recipe to manage several distributed systems on the same cluster (i.e. Hadoop, HBase, Cassandra, Kaffka, etc.) Using just CM tools are not enough and you will need also deployment management, resource management, monitoring, etc. Another takeout from this, that virtualization is not Web-scale, it's more appropriate to IT and public cloud workloads, large companies like Google prefer bar-metal.