8 ms·

It's very cool to see these kinds of posts. I always learn something from them and it's always neat to see what others have built to deal with a specific proble

by druiid 14y ago

It's very cool to see these kinds of posts. I always learn something from them and it's always neat to see what others have built to deal with a specific problem. I wonder in this case why, if any specific reason Etsy is not utilizing virtualization of one kind or another? Basically they're building dumb-boxes that seem a much better fit to have a good number of slightly beefier VM systems and just rebuild bad instances from images (this would be especially good since they're already utilizing Chef). I know that as an e-commerce firm this has helped us substantially to move toward this kind of a structure.

jcampbell1 14y ago
Several years ago, 37 signals wrote a post showing that moving away from virtualization improved response times. http://37signals.com/svn/posts/1819-basecamp-now-with-more-vroom http://37signals.com/svn/posts/1819-basecamp-now-with-more-v...
- druiid 14y ago
  Well at least one thing to note there, is that post is from several years ago... both virtualization technologies have gotten better since then, as well as processors. I think there are certainly specific web and db loads that aren't going to work as well under VM scenarios, but on the whole our experience has been positive. There will, I imagine, always be a slight 'slowdown' due to VM but with the gigantic push toward 'the cloud' right now, it seems to me one of the last things on the minds of most (and obviously Xen is highly used in that field, which is also what we use). The bigger question I think is if you are using a highly CPU bound task which VM isn't always very good at. According to Etsy they are very CPU bound, so it makes some amount of sense.
  - jbooth 14y ago
    I think, generally speaking, once you start having >10 or >50 servers, VMs just stop making sense. You're getting a lot of hardware because you have a problem that requires it, and for most cloud systems with built-in redundancy, you'd only be running a single VM per machine anyways. Why not just run the machine? You should be using puppet or chef or something so it's not much/any harder to refresh a raw machine than it would be to refresh a VM image.
    - druiid 14y ago
      I think this it where we get back to the CPU vs memory bound tasks. If the task allows you to be running 1/8th of a newer Intel/AMD cpu at any given time (also accounting for bursting), why not look if you can squish some instances together? CPU bound I fully agree with you (although I wonder how hyper-scale type companies like Zynga did an analysis at this level and decided upon their stack... if there's a writeup out their about their decisions I'd love to see it actually... sorry, off-topic slightly).
      - jbooth 14y ago
        Most 'cloud' software like Hadoop, Cassandra, clustered Mongo or whatever makes it a point to solve that problem in userspace software, giving you a view of 'one big app' across a bunch of machines. Since that's your level of interaction with the system, VMs tend to not really make a lot of sense, they just give you new opportunities to make mistakes while imposing a small performance tax. Now, if you have 10 legacy apps on old P4 xeons and want to lump them all onto a sandy bridge server and save a bunch of power and rackspace, then separate VMs on one machine makes sense. Basically, a cluster of 30 servers, all running exactly one VM doing the same thing is on-its-face silly. Why even have the VM then?
        travem 14y ago
        The advantage I see with virtualization is that you could use those servers to run your Hadoop workloads on those 30 servers over the weekend and then repurpose those machines for other workloads during the week.
        mikescar 14y ago
        How long does it take in practice to swap out VM images in this way for different types of loads? How do you monitor that all 30 are ready to go and working right to spec?
        avleenvig 14y ago
        That sounds really good in theory, but in practice it's less good. There are a number of different types of costs to consider: 1. The cost of server hardware 2. The cost of unused hardware capacity 3. The administration cost (people, skills, etc) 4. Opportunity cost The Cloud(tm) excels at addressing some of these. You don't have to pay /as/ highly for staff to manage you machines and network (the provider does much of that for you). You can run smaller, cheaper instances much closer to their limits. The downside is that you pay a premium to the provider (even if you are your own provider). Additionally you lose a great deal of opportunity cost. If you use the "excess" capacity on the weekend for hadoop jobs, and need to stop them because you have a large burst of traffic, you've hurt yourself. Your non-production hadoop jobs can also have unexpected and unintended impact on your production web servers, causing your users pain. Given these things, if you focus is on keeping the best experience for users then you should split things apart. The actual gain you get from cramming as much as you can onto one piece of hardware is much, much lower than you might expect. It ends up being easier just to get more hardware and dedicate that hardware for specific tasks. - Avleen Vig, Staff Operations Engineer, Etsy
runako 14y ago
The OP says they can spin up a new bare metal machine in 10 minutes. It's likely the performance hit from running in a virtualized environment isn't worth shaving time off the 10 minute build.
- druiid 14y ago
  I'd say there are other reasons behind using virtualization than simply quick system spin-up, such as easier/better integration with (Dev|Sys)Op tools, higher hardware->instance densities than you'll ever see from physical... etc. I think the performance hit for many people is overstated. Unless you're at the long-tail of CPU bound instances I imagine it won't become very apparent. That said, if I'm missing something/incorrect then someone please correct me!
  - tedunangst 14y ago
    It sounds like the reason they have so many machines is because they need a lot of CPU. Compressing those machines onto fewer CPUs would not be a win.
  - runako 14y ago
    I don't work at Etsy or have any specific knowledge, but it sounds like they already have the tool integration they need with their current setup. Increasing densities through virtualization allows you to perform more discrete functions, but obviously reduces the aggregate capacity to do so. Etsy specifically has a goal of minimizing instance types, but virtualization would enable them go in a direction counter to their goals. In my experience, the biggest weakness of virtualization is subpar I/O, and I would imagine that this would be an issue for Etsy as well. Virtualization has a very real cost. If the performance hit (CPU & I/O) is (say) 20%, that's 20% of your capacity you pay to keep operational but don't get to use. The benefits don't outweigh that cost for every situation.
  - avleenvig 14y ago
    So, this is somewhat untrue for us. For example, let's look at some common web stack infrastructure: * Web servers * Database servers * Monitoring servers * Asynchronous task processors * Scheduled task servers (eg, cron jobs) In all of these cases, we want the service on that hardware to be able to use 100% CPU. For example, it doesn't make sense to get N web servers, and then run M virtual instances with apache on them, when each instance will max out its available resources. Just run N dedicated web servers :-) The same is true for pretty much everything else too, sometimes for different reasons. Eg, you don't want your database servers or monitoring system to be impacted by multi-tenancy issues. - Avleen Vig, Staff Operations Engineer, Etsy
jgoulah 14y ago
FWIW we do use virtualization for our dev infrastructure (everyone gets a vm) which uses KVM/QEMU and our CI cluster which is LXC based A bit about the dev infrastructure here: http://codeascraft.etsy.com/2012/03/13/making-it-virtually-easy-to-deploy-on-day-one/ http://codeascraft.etsy.com/2012/03/13/making-it-virtually-e...
- druiid 14y ago
  Thanks for the reply. Also interesting to see dev side... we run something similar although a bit more free-form with the internal developers, but that's being pushed into a more controlled environment ;). I should also say it's a bit heartening to know of other e-com groups out there benefiting from an in-house stack rather than attempting to go 'TO THE CLOUD!'.
Dobbs 14y ago
Your point about rebuilding bad instances is just as easy to do with PXE and Chef/Puppet on bare metal.
- avleenvig 14y ago
  sudo koan --replace-self sudo reboot -f
- e12e 14y ago
  "Just as easy" might be an overstatement. Equally doable might be closer to the truth. One example where virtualization is easier in practice is where you're not in control of the network intfrastructure -- ie: you subcontract management of switches etc to someone else. For someone the size of Etsy that wouldn't matter much -- they have an obvious need to manage the entire stack. But for smaller institutions that might not target "scaling up to serve the entire public web" -- it is a factor. Another example is if you have heterogeneous hardware -- it is much easier to just install a hypervisor that runs on every box and then manage that via your vm-toolset, that set up many different images. It also makes it easier if you need to run different os' -- do all your os images have drivers for all your hardware? Granted, in theory OS' should abstract away hardware -- in practice, if you need to change out a few of your mysql instances running GNU/Linux to Microsoft SQL servers for a new project/change in workload -- being able to just spin down the old vms and up the new ones helps a lot in terms of agility. But yes, it's absolutely possible to provision physical hardware as well.

by druiid 14y ago

jcampbell1 14y ago

Several years ago, 37 signals wrote a post showing that moving away from virtualization improved response times. http://37signals.com/svn/posts/1819-basecamp-now-with-more-vroom http://37signals.com/svn/posts/1819-basecamp-now-with-more-v...

druiid 14y ago
Well at least one thing to note there, is that post is from several years ago... both virtualization technologies have gotten better since then, as well as processors. I think there are certainly specific web and db loads that aren't going to work as well under VM scenarios, but on the whole our experience has been positive. There will, I imagine, always be a slight 'slowdown' due to VM but with the gigantic push toward 'the cloud' right now, it seems to me one of the last things on the minds of most (and obviously Xen is highly used in that field, which is also what we use). The bigger question I think is if you are using a highly CPU bound task which VM isn't always very good at. According to Etsy they are very CPU bound, so it makes some amount of sense.
- jbooth 14y ago
  I think, generally speaking, once you start having >10 or >50 servers, VMs just stop making sense. You're getting a lot of hardware because you have a problem that requires it, and for most cloud systems with built-in redundancy, you'd only be running a single VM per machine anyways. Why not just run the machine? You should be using puppet or chef or something so it's not much/any harder to refresh a raw machine than it would be to refresh a VM image.
  - druiid 14y ago
    I think this it where we get back to the CPU vs memory bound tasks. If the task allows you to be running 1/8th of a newer Intel/AMD cpu at any given time (also accounting for bursting), why not look if you can squish some instances together? CPU bound I fully agree with you (although I wonder how hyper-scale type companies like Zynga did an analysis at this level and decided upon their stack... if there's a writeup out their about their decisions I'd love to see it actually... sorry, off-topic slightly).
    - jbooth 14y ago
      Most 'cloud' software like Hadoop, Cassandra, clustered Mongo or whatever makes it a point to solve that problem in userspace software, giving you a view of 'one big app' across a bunch of machines. Since that's your level of interaction with the system, VMs tend to not really make a lot of sense, they just give you new opportunities to make mistakes while imposing a small performance tax. Now, if you have 10 legacy apps on old P4 xeons and want to lump them all onto a sandy bridge server and save a bunch of power and rackspace, then separate VMs on one machine makes sense. Basically, a cluster of 30 servers, all running exactly one VM doing the same thing is on-its-face silly. Why even have the VM then?
      - travem 14y ago
        The advantage I see with virtualization is that you could use those servers to run your Hadoop workloads on those 30 servers over the weekend and then repurpose those machines for other workloads during the week.
        mikescar 14y ago
        How long does it take in practice to swap out VM images in this way for different types of loads? How do you monitor that all 30 are ready to go and working right to spec?
        avleenvig 14y ago
        That sounds really good in theory, but in practice it's less good. There are a number of different types of costs to consider: 1. The cost of server hardware 2. The cost of unused hardware capacity 3. The administration cost (people, skills, etc) 4. Opportunity cost The Cloud(tm) excels at addressing some of these. You don't have to pay /as/ highly for staff to manage you machines and network (the provider does much of that for you). You can run smaller, cheaper instances much closer to their limits. The downside is that you pay a premium to the provider (even if you are your own provider). Additionally you lose a great deal of opportunity cost. If you use the "excess" capacity on the weekend for hadoop jobs, and need to stop them because you have a large burst of traffic, you've hurt yourself. Your non-production hadoop jobs can also have unexpected and unintended impact on your production web servers, causing your users pain. Given these things, if you focus is on keeping the best experience for users then you should split things apart. The actual gain you get from cramming as much as you can onto one piece of hardware is much, much lower than you might expect. It ends up being easier just to get more hardware and dedicate that hardware for specific tasks. - Avleen Vig, Staff Operations Engineer, Etsy

runako 14y ago

The OP says they can spin up a new bare metal machine in 10 minutes. It's likely the performance hit from running in a virtualized environment isn't worth shaving time off the 10 minute build.

druiid 14y ago
I'd say there are other reasons behind using virtualization than simply quick system spin-up, such as easier/better integration with (Dev|Sys)Op tools, higher hardware->instance densities than you'll ever see from physical... etc. I think the performance hit for many people is overstated. Unless you're at the long-tail of CPU bound instances I imagine it won't become very apparent. That said, if I'm missing something/incorrect then someone please correct me!
- tedunangst 14y ago
  It sounds like the reason they have so many machines is because they need a lot of CPU. Compressing those machines onto fewer CPUs would not be a win.
- runako 14y ago
  I don't work at Etsy or have any specific knowledge, but it sounds like they already have the tool integration they need with their current setup. Increasing densities through virtualization allows you to perform more discrete functions, but obviously reduces the aggregate capacity to do so. Etsy specifically has a goal of minimizing instance types, but virtualization would enable them go in a direction counter to their goals. In my experience, the biggest weakness of virtualization is subpar I/O, and I would imagine that this would be an issue for Etsy as well. Virtualization has a very real cost. If the performance hit (CPU & I/O) is (say) 20%, that's 20% of your capacity you pay to keep operational but don't get to use. The benefits don't outweigh that cost for every situation.
- avleenvig 14y ago
  So, this is somewhat untrue for us. For example, let's look at some common web stack infrastructure: * Web servers * Database servers * Monitoring servers * Asynchronous task processors * Scheduled task servers (eg, cron jobs) In all of these cases, we want the service on that hardware to be able to use 100% CPU. For example, it doesn't make sense to get N web servers, and then run M virtual instances with apache on them, when each instance will max out its available resources. Just run N dedicated web servers :-) The same is true for pretty much everything else too, sometimes for different reasons. Eg, you don't want your database servers or monitoring system to be impacted by multi-tenancy issues. - Avleen Vig, Staff Operations Engineer, Etsy

jgoulah 14y ago

FWIW we do use virtualization for our dev infrastructure (everyone gets a vm) which uses KVM/QEMU and our CI cluster which is LXC based A bit about the dev infrastructure here: http://codeascraft.etsy.com/2012/03/13/making-it-virtually-easy-to-deploy-on-day-one/ http://codeascraft.etsy.com/2012/03/13/making-it-virtually-e...

druiid 14y ago
Thanks for the reply. Also interesting to see dev side... we run something similar although a bit more free-form with the internal developers, but that's being pushed into a more controlled environment ;). I should also say it's a bit heartening to know of other e-com groups out there benefiting from an in-house stack rather than attempting to go 'TO THE CLOUD!'.

Dobbs 14y ago

Your point about rebuilding bad instances is just as easy to do with PXE and Chef/Puppet on bare metal.

avleenvig 14y ago
sudo koan --replace-self sudo reboot -f
e12e 14y ago
"Just as easy" might be an overstatement. Equally doable might be closer to the truth. One example where virtualization is easier in practice is where you're not in control of the network intfrastructure -- ie: you subcontract management of switches etc to someone else. For someone the size of Etsy that wouldn't matter much -- they have an obvious need to manage the entire stack. But for smaller institutions that might not target "scaling up to serve the entire public web" -- it is a factor. Another example is if you have heterogeneous hardware -- it is much easier to just install a hypervisor that runs on every box and then manage that via your vm-toolset, that set up many different images. It also makes it easier if you need to run different os' -- do all your os images have drivers for all your hardware? Granted, in theory OS' should abstract away hardware -- in practice, if you need to change out a few of your mysql instances running GNU/Linux to Microsoft SQL servers for a new project/change in workload -- being able to just spin down the old vms and up the new ones helps a lot in terms of agility. But yes, it's absolutely possible to provision physical hardware as well.