5 ms·

I think this it where we get back to the CPU vs memory bound tasks. If the task allows you to be running 1/8th of a newer Intel/AMD cpu at any given time (also

by druiid 14y ago

I think this it where we get back to the CPU vs memory bound tasks. If the task allows you to be running 1/8th of a newer Intel/AMD cpu at any given time (also accounting for bursting), why not look if you can squish some instances together? CPU bound I fully agree with you (although I wonder how hyper-scale type companies like Zynga did an analysis at this level and decided upon their stack... if there's a writeup out their about their decisions I'd love to see it actually... sorry, off-topic slightly).

jbooth 14y ago
Most 'cloud' software like Hadoop, Cassandra, clustered Mongo or whatever makes it a point to solve that problem in userspace software, giving you a view of 'one big app' across a bunch of machines. Since that's your level of interaction with the system, VMs tend to not really make a lot of sense, they just give you new opportunities to make mistakes while imposing a small performance tax. Now, if you have 10 legacy apps on old P4 xeons and want to lump them all onto a sandy bridge server and save a bunch of power and rackspace, then separate VMs on one machine makes sense. Basically, a cluster of 30 servers, all running exactly one VM doing the same thing is on-its-face silly. Why even have the VM then?
- travem 14y ago
  The advantage I see with virtualization is that you could use those servers to run your Hadoop workloads on those 30 servers over the weekend and then repurpose those machines for other workloads during the week.
  - mikescar 14y ago
    How long does it take in practice to swap out VM images in this way for different types of loads? How do you monitor that all 30 are ready to go and working right to spec?
  - avleenvig 14y ago
    That sounds really good in theory, but in practice it's less good. There are a number of different types of costs to consider: 1. The cost of server hardware 2. The cost of unused hardware capacity 3. The administration cost (people, skills, etc) 4. Opportunity cost The Cloud(tm) excels at addressing some of these. You don't have to pay /as/ highly for staff to manage you machines and network (the provider does much of that for you). You can run smaller, cheaper instances much closer to their limits. The downside is that you pay a premium to the provider (even if you are your own provider). Additionally you lose a great deal of opportunity cost. If you use the "excess" capacity on the weekend for hadoop jobs, and need to stop them because you have a large burst of traffic, you've hurt yourself. Your non-production hadoop jobs can also have unexpected and unintended impact on your production web servers, causing your users pain. Given these things, if you focus is on keeping the best experience for users then you should split things apart. The actual gain you get from cramming as much as you can onto one piece of hardware is much, much lower than you might expect. It ends up being easier just to get more hardware and dedicate that hardware for specific tasks. - Avleen Vig, Staff Operations Engineer, Etsy
prodigal_erik 14y ago
A hypervisor is just a kernel with a more clumsy API dictated by legacy hardware design. If you have eight services that can fit on the box together, eight processes on one good kernel should offer better visibility and control over what's going on than would eight VMs each hosting one process (along with some uninteresting crud). VMs shine when you're stuck with proprietary software that can't all be ported to run on the same kernel, then what you're paying for is a compatibility shim that's cheaper than another box. I don't see much sense for software you wrote in-house.

by druiid 14y ago

jbooth 14y ago

Most 'cloud' software like Hadoop, Cassandra, clustered Mongo or whatever makes it a point to solve that problem in userspace software, giving you a view of 'one big app' across a bunch of machines. Since that's your level of interaction with the system, VMs tend to not really make a lot of sense, they just give you new opportunities to make mistakes while imposing a small performance tax. Now, if you have 10 legacy apps on old P4 xeons and want to lump them all onto a sandy bridge server and save a bunch of power and rackspace, then separate VMs on one machine makes sense. Basically, a cluster of 30 servers, all running exactly one VM doing the same thing is on-its-face silly. Why even have the VM then?

travem 14y ago
The advantage I see with virtualization is that you could use those servers to run your Hadoop workloads on those 30 servers over the weekend and then repurpose those machines for other workloads during the week.
- mikescar 14y ago
  How long does it take in practice to swap out VM images in this way for different types of loads? How do you monitor that all 30 are ready to go and working right to spec?
- avleenvig 14y ago
  That sounds really good in theory, but in practice it's less good. There are a number of different types of costs to consider: 1. The cost of server hardware 2. The cost of unused hardware capacity 3. The administration cost (people, skills, etc) 4. Opportunity cost The Cloud(tm) excels at addressing some of these. You don't have to pay /as/ highly for staff to manage you machines and network (the provider does much of that for you). You can run smaller, cheaper instances much closer to their limits. The downside is that you pay a premium to the provider (even if you are your own provider). Additionally you lose a great deal of opportunity cost. If you use the "excess" capacity on the weekend for hadoop jobs, and need to stop them because you have a large burst of traffic, you've hurt yourself. Your non-production hadoop jobs can also have unexpected and unintended impact on your production web servers, causing your users pain. Given these things, if you focus is on keeping the best experience for users then you should split things apart. The actual gain you get from cramming as much as you can onto one piece of hardware is much, much lower than you might expect. It ends up being easier just to get more hardware and dedicate that hardware for specific tasks. - Avleen Vig, Staff Operations Engineer, Etsy

prodigal_erik 14y ago

A hypervisor is just a kernel with a more clumsy API dictated by legacy hardware design. If you have eight services that can fit on the box together, eight processes on one good kernel should offer better visibility and control over what's going on than would eight VMs each hosting one process (along with some uninteresting crud). VMs shine when you're stuck with proprietary software that can't all be ported to run on the same kernel, then what you're paying for is a compatibility shim that's cheaper than another box. I don't see much sense for software you wrote in-house.