6 ms·

Faster Web Development With Virtual Machines at deviantART

65 points by kemayo 16y ago

sammcd 16y ago
After talking with a few small PHP teams this seems like the logical conclusion whenever growing a team of PHP developers. Its just very hard to perfectly recreate your server environment locally on multiple OS's. Its a great solution, but I much prefer working with RoR or Python/Django where recreating the server environment locally is much easier.
- drwxrwxrwt 16y ago
  even for RoR or python - might make sense for those working on multiple projects simultaneously, using different generations of frameworks
  - Groxx 16y ago
    There's RVM in the Ruby world. Anything similar for Python?
    - marte 16y ago
      virtualenv
      - Groxx 16y ago
        I'd say that seems to cover both bases for not needing a VM. Every language should have something as simple as these. Heck, it should be part of the OS, IMO; why I can't (easily) run two different versions of Firefox with two different profiles at the same time is beyond comprehension.
        sprout 16y ago
        A VM is a constant time investment, doing it for every language is linear with the number of languages. Unless there's some serious increases in clock speeds, I'm not sure doing it for every language is worth it. As it is, VMs are a great, general purpose system that are easy to set up, use, and replicate.
        StavrosK 16y ago
        You do need a VM if you use memcached/redis/some other thing and don't want to install it on your machine. Other than that (for 90% of cases), virtualenv has you covered.
        andrewf 16y ago
        It's been a while since I've done this; but: * Install your different Firefox versions in different locations (so one doesn't overwrite the other) * Use "firefox -ProfileManager" to create at least one different 'profile', which is Firefox's user data set * Use "firefox -P <profilename> -no-remote" to launch Firefox. -no-remote allows multiple Firefox instances to run on the same machine. -P profilename is necessary because you need a separate profile for each running instance. More info at http://kb.mozillazine.org/Command_line_arguments http://kb.mozillazine.org/Command_line_arguments
        Groxx 16y ago
        Ooh, I'll have to try that. I've been using different profiles, but didn't know about the last step. Still; mindboggling. Different version = different binary launched, which accesses a different and non-locked profile. Why the hell does it instead open a new window of the currently-running version? As a heads-up to anyone considering this: Sync doesn't (currently) like multiple versions. 3.6.10 won't sync now that I've hooked up some 4 betas.
- evilhackerdude 16y ago
  Bundler was supposed to remedy dependency hell problems and I believe it eventually managed to do that. But boy can I tell you stories about migrating from pre-1.0 bundler versions. RVM is just great and simple. What I’m trying to say though: Having done quite some side projects with node, the ruby stuff suddenly feels very enterprisey to me. More and more conventions might make it harder to keep track of everything in the future.
- leftnode 16y ago
  How is creating the local server environment on RoR or Python/Django easier than PHP/anything? I've been developing locally for years and while I can't mimic the exact hardware of the live server, I wouldn't be able to do that regardless of stack I chose.
  - Pewpewarrows 16y ago
    For the Python world, virtualenvs and pip requirement files make deploying a local copy of a project take about 30 seconds, and it's completely independent of your overall system and all other projects you happen to be working on.
    - leftnode 16y ago
      Gotcha. With git and a Phing build script, I can do the same in PHP :)
  - StavrosK 16y ago
    As Pewpewarrows said, you can use pip with a virtualenv to deploy with one command. If your settings.py is any good, you can load your data dump in an sqlite database in one more command, and you're ready to go. We used to develop historious like this, until we switched to redis/celery. Then it became too much hassle to install these in every machine, plus we wanted to test on the db we used (postgres) rather than sqlite, so we made a VM and now everything's dandy.
- thefreshteapot 16y ago
  Your last sentence is an example of when I would like to vote "my displeasure" with a down arrow. As others have mentioned, you make a bold claim, yet fail to give anything concrete as proof that it is "much easier".
kemayo 16y ago
I do acknowledge that this is something that looks very "oh, how obvious!" to the modern crowd, but deviantART is a 10 year old PHP app... Of course, even if you're running a modern web framework in Ruby/Python/somethingtrendy, a VM makes a lot of sense. It lets you run the exact versions of everything that your production environment is, without having to worry about it.
- Supermighty 16y ago
  Developing in the exact same environment as you deploy in should be standard practice.
tsmith 16y ago
I wonder how long this took to set up? And why they don't run their production servers as VMs as well? Edit: I think it's an awesome idea, btw, and something we try to push to web service shops at GridCentric.
kristianp 16y ago
It's not clear to me why they don't just run development on their local machines. I'm not familiar with PHP, though. I'm all for having a build/test vm, though.
- sammcd 16y ago
  I good PHP environment that models the server is very hard to get going locally. At least on OS X. The only way I could get a good environment for Nginx+MySQL+PHP was to locally run a debian VM in VirtualBox. Now this doesn't even take into account things like memcached that require a lot more customization.
  - thefreshteapot 16y ago
    Im confused, this post of yours seems a tad contradictory to your post about "setting up environments in RoR etc".
- Legion 16y ago
  Developers aren't server administrators (usually). It's not the best use of their time. I wish I had thought of this when we ran into issues with Windows people and Ruby. Too much of their time was spent fussing with stuff and crying about it. :)
  - evilhackerdude 16y ago
    “So now it should work, right?” -“I’m getting an error” “Which error?” -“Unnamed method in [...]” Then, a mountain of pain and torture later. "But now it works, ok?” -“Ok.” --- Ever had to fix someone else’s system via Skype or phone? I personally believe writing a rails clone in brainfuck is more fun.
  - aaronblohowiak 16y ago
    JRuby cn also be a good way to run ruby on windows.
- eli 16y ago
evilhackerdude 16y ago
I only skimmed the article but I assume it’s about the benefits of running VMs to have solid dev environments with zero setup cost. Let me ask another question: Will it become feasible to move code, tools and everything else into web services? Imagine you have your favorite editor (emacs, TM, vim) running in a browser window. Only that it’s live-updating like, e.g., Ethercodes or Wave. And you have very straight-forward access to SCM features like staging hunks of code. Also the code deploys straight to heroku or whatever. It could, of course, also go through closure or sproutcore in case of a fancy js client app. Do you think it’ll all evaporate (I refuse to say cloud :) eventually? The thing is I’m really going through a lot of pain setting up my system and other’s systems over and over again. Yes, there are ways to automate that but the effort is too high for most or we would all be doing it right now.
- ryanhuff 16y ago
  I remember Heroku had something like this a few years ago. It was pretty easy to get a taste of Rails.
- die_sekte 16y ago
  I'm sure it will happen one day, but it's quite a big challenge and you're up against all those bad habits build over the last 20 years. For example, developers really like customization. It's quite hard to provide a hosted environment that is fully customizable. Also, parenthesis mismatch in paragraph 3.
mgrouchy 16y ago
I do most of my work in python/django so virtualenv essentially solves much of these problems. The problem I am running into now is that our production environment is starting to get complicated enough(mysql, redis, 2 different django projects) that testing all the components working together gets more and more difficult all the time.
- fleitz 16y ago
  Start scripting your environment, the more repeatable your deploy procedure is the better your deploys will be. The easier your deploys are the more frequently you do them, the more frequently you do them, the better your deploys will be. Rinse wash repeat. Once you have your env scripted with EC2 you can setup / teardown environments easily for testing.
The_Igor 16y ago
Great Article. As QA Analyst and Engineer, I know first hand about the pains of replicating production environment. Thanks for submission.
notmyname 16y ago
I've found that developing against a VM is a great tool for a dev. For OpenStack swift, we develop all of our code against a VM. Since we deploy on debian and I develop on a Mac, I can use OS X tools to develop (TextMate, etc) and code against an environment similar to our production. After code works on a VM, we can move to a lab or a staging environment to do more scale testing before deploying to production. Developing against a VM is not something I did before working at Rackspace, and I find the practice so useful that I will do what I can to continue doing it no matter where I work in the future.
dnaquin 16y ago
Only use VMs or sandboxes on production-like machines. Local development doesn't scale.
mitchellh 16y ago
Disclaimer: I am the creator of what I'm about to plug here. If anyone is looking for an easy, low-risk, documented method of beginning development in VMs, check out Vagrant[1], which is what I wrote this for. Creating dev VMs from scratch is pretty tedious and there are a lot of steps involved. Vagrant abstracts this away and provides a tool to automate all of this in a reproducible manner. [1]: http://vagrantup.com http://vagrantup.com
- gnubardt 16y ago
  I love vagrant! It was super easy to create an archlinux base box and i've been using chef to automatically provision environments. Setting up the environment for a new project takes minutes and is both totally automated and separated from my actual OS. Thanks!
- denimboy 16y ago
  I started using vagrant last week to debug my Chef scripts and it is indeed awesome. Vagrant uses Chef (chef-solo) configure your VM from bare image to working server. I see that is has a plug in system for other provision systems like Puppet, but the chef support works great out of the box. I was hoping to contribute to the project by integrating fog the ruby library which talks to many cloud systems including EC2 so someone could build an image via Chef or Puppet, dev and test in their image, then provision and upload to the cloud. Could probably do this without integrating with vagrant by building a clever rake or capistrano script.
chewbranca 16y ago
This reminds me of facebook development, where you need to have a live fqdn for all of your environments. We used to use a staging server and had a similar problem with huge amounts of small unnatural commits for things you would normally test locally. I ended up moving to using ssh tunnels to a server, where each dev has a subdomain redirecting to their dev box through the tunnel. This has the added benefit that you can work locally from anywhere you have wifi. Whether you use tunnels or VMs, both are a solid way to do development.
drwxrwxrwt 16y ago
Another benefit of this approach is that it takes seconds to revert to a good state if you destroyed your dataset in a mishap. use snapshots or just restart VM from original image. all while using your preferred code editor (the app source is mounted from the host OS)
Jach 16y ago
If commits were an issue before this, they should really have considered using git or something. You shouldn't aim for fewer commits, but less expensive commits, i.e. nothing goes over a network. Plus you can squash later if you really need to. Anyway, VMs are cool but has anyone tried developing with NixOS? It seems even more suited for the tasks they want since it has rollbacks and so on: http://nixos.org/ http://nixos.org/
- yycom 16y ago
  Isn't the expense of frequent commits the cost of composing an intelligible commit message?
prodigal_erik 16y ago
Having a VM you can trivially reimage is very helpful because you can test that (re)deploying packages to it is going to work, you didn't miss any dependencies that happen to be present on your desktop but not on your production servers, and uninstall won't leave cruft behind.
hdeshev 16y ago
This all VM thing is fine and dandy if you want to have an accessible mirror of your staging machine and reduce unnecessary stages. Although that in itself looks fishy - if developers want to stage something just to see if it works, then something is seriously wrong with your process or developers. I'm amazed at the level of sophistication in this setup - using virtual machines and TCP proxies that do text replaces seems too much. They could have just built the configurability in the damn application. I have a .NET app that normally gets deployed on three servers and uses various remote resources like Amazon SQS queues, but I easily built a configuration abstraction similar to Rails' environments that lets me deploy on a single machine and even test locally with a debugger attached. My point is that this problem looks as if it got solved at the wrong level of abstraction. Which, obviously, made it unnecessarily hard and complex. If I had to tackle it, I'd make sure developers do the right thing and try to make the app configurable enough first.
- tomedme 16y ago
  The mind boggles. There's probably a rockstar in there looking to put something on their CV/resume. Whatever the case, it smells wrong.

65 points by kemayo 16y ago

After talking with a few small PHP teams this seems like the logical conclusion whenever growing a team of PHP developers. Its just very hard to perfectly recreate your server environment locally on multiple OS's. Its a great solution, but I much prefer working with RoR or Python/Django where recreating the server environment locally is much easier.

drwxrwxrwt 16y ago
even for RoR or python - might make sense for those working on multiple projects simultaneously, using different generations of frameworks
- Groxx 16y ago
  There's RVM in the Ruby world. Anything similar for Python?
  - marte 16y ago
    virtualenv
    - Groxx 16y ago
      I'd say that seems to cover both bases for not needing a VM. Every language should have something as simple as these. Heck, it should be part of the OS, IMO; why I can't (easily) run two different versions of Firefox with two different profiles at the same time is beyond comprehension.
      - sprout 16y ago
        A VM is a constant time investment, doing it for every language is linear with the number of languages. Unless there's some serious increases in clock speeds, I'm not sure doing it for every language is worth it. As it is, VMs are a great, general purpose system that are easy to set up, use, and replicate.
      - StavrosK 16y ago
        You do need a VM if you use memcached/redis/some other thing and don't want to install it on your machine. Other than that (for 90% of cases), virtualenv has you covered.
      - andrewf 16y ago
        It's been a while since I've done this; but: * Install your different Firefox versions in different locations (so one doesn't overwrite the other) * Use "firefox -ProfileManager" to create at least one different 'profile', which is Firefox's user data set * Use "firefox -P <profilename> -no-remote" to launch Firefox. -no-remote allows multiple Firefox instances to run on the same machine. -P profilename is necessary because you need a separate profile for each running instance. More info at http://kb.mozillazine.org/Command_line_arguments http://kb.mozillazine.org/Command_line_arguments
        Groxx 16y ago
        Ooh, I'll have to try that. I've been using different profiles, but didn't know about the last step. Still; mindboggling. Different version = different binary launched, which accesses a different and non-locked profile. Why the hell does it instead open a new window of the currently-running version? As a heads-up to anyone considering this: Sync doesn't (currently) like multiple versions. 3.6.10 won't sync now that I've hooked up some 4 betas.
evilhackerdude 16y ago
Bundler was supposed to remedy dependency hell problems and I believe it eventually managed to do that. But boy can I tell you stories about migrating from pre-1.0 bundler versions. RVM is just great and simple. What I’m trying to say though: Having done quite some side projects with node, the ruby stuff suddenly feels very enterprisey to me. More and more conventions might make it harder to keep track of everything in the future.
leftnode 16y ago
How is creating the local server environment on RoR or Python/Django easier than PHP/anything? I've been developing locally for years and while I can't mimic the exact hardware of the live server, I wouldn't be able to do that regardless of stack I chose.
- Pewpewarrows 16y ago
  For the Python world, virtualenvs and pip requirement files make deploying a local copy of a project take about 30 seconds, and it's completely independent of your overall system and all other projects you happen to be working on.
  - leftnode 16y ago
    Gotcha. With git and a Phing build script, I can do the same in PHP :)
- StavrosK 16y ago
  As Pewpewarrows said, you can use pip with a virtualenv to deploy with one command. If your settings.py is any good, you can load your data dump in an sqlite database in one more command, and you're ready to go. We used to develop historious like this, until we switched to redis/celery. Then it became too much hassle to install these in every machine, plus we wanted to test on the db we used (postgres) rather than sqlite, so we made a VM and now everything's dandy.
thefreshteapot 16y ago
Your last sentence is an example of when I would like to vote "my displeasure" with a down arrow. As others have mentioned, you make a bold claim, yet fail to give anything concrete as proof that it is "much easier".

kemayo 16y ago

I do acknowledge that this is something that looks very "oh, how obvious!" to the modern crowd, but deviantART is a 10 year old PHP app... Of course, even if you're running a modern web framework in Ruby/Python/somethingtrendy, a VM makes a lot of sense. It lets you run the exact versions of everything that your production environment is, without having to worry about it.

Supermighty 16y ago
Developing in the exact same environment as you deploy in should be standard practice.

tsmith 16y ago

I wonder how long this took to set up? And why they don't run their production servers as VMs as well? Edit: I think it's an awesome idea, btw, and something we try to push to web service shops at GridCentric.

kristianp 16y ago

It's not clear to me why they don't just run development on their local machines. I'm not familiar with PHP, though. I'm all for having a build/test vm, though.

sammcd 16y ago
I good PHP environment that models the server is very hard to get going locally. At least on OS X. The only way I could get a good environment for Nginx+MySQL+PHP was to locally run a debian VM in VirtualBox. Now this doesn't even take into account things like memcached that require a lot more customization.
- thefreshteapot 16y ago
  Im confused, this post of yours seems a tad contradictory to your post about "setting up environments in RoR etc".
Legion 16y ago
Developers aren't server administrators (usually). It's not the best use of their time. I wish I had thought of this when we ran into issues with Windows people and Ruby. Too much of their time was spent fussing with stuff and crying about it. :)
- evilhackerdude 16y ago
  “So now it should work, right?” -“I’m getting an error” “Which error?” -“Unnamed method in [...]” Then, a mountain of pain and torture later. "But now it works, ok?” -“Ok.” --- Ever had to fix someone else’s system via Skype or phone? I personally believe writing a rails clone in brainfuck is more fun.
- aaronblohowiak 16y ago
  JRuby cn also be a good way to run ruby on windows.
eli 16y ago

evilhackerdude 16y ago

I only skimmed the article but I assume it’s about the benefits of running VMs to have solid dev environments with zero setup cost. Let me ask another question: Will it become feasible to move code, tools and everything else into web services? Imagine you have your favorite editor (emacs, TM, vim) running in a browser window. Only that it’s live-updating like, e.g., Ethercodes or Wave. And you have very straight-forward access to SCM features like staging hunks of code. Also the code deploys straight to heroku or whatever. It could, of course, also go through closure or sproutcore in case of a fancy js client app. Do you think it’ll all evaporate (I refuse to say cloud :) eventually? The thing is I’m really going through a lot of pain setting up my system and other’s systems over and over again. Yes, there are ways to automate that but the effort is too high for most or we would all be doing it right now.

ryanhuff 16y ago
I remember Heroku had something like this a few years ago. It was pretty easy to get a taste of Rails.
die_sekte 16y ago
I'm sure it will happen one day, but it's quite a big challenge and you're up against all those bad habits build over the last 20 years. For example, developers really like customization. It's quite hard to provide a hosted environment that is fully customizable. Also, parenthesis mismatch in paragraph 3.

mgrouchy 16y ago

I do most of my work in python/django so virtualenv essentially solves much of these problems. The problem I am running into now is that our production environment is starting to get complicated enough(mysql, redis, 2 different django projects) that testing all the components working together gets more and more difficult all the time.

fleitz 16y ago
Start scripting your environment, the more repeatable your deploy procedure is the better your deploys will be. The easier your deploys are the more frequently you do them, the more frequently you do them, the better your deploys will be. Rinse wash repeat. Once you have your env scripted with EC2 you can setup / teardown environments easily for testing.

The_Igor 16y ago

Great Article. As QA Analyst and Engineer, I know first hand about the pains of replicating production environment. Thanks for submission.

notmyname 16y ago

I've found that developing against a VM is a great tool for a dev. For OpenStack swift, we develop all of our code against a VM. Since we deploy on debian and I develop on a Mac, I can use OS X tools to develop (TextMate, etc) and code against an environment similar to our production. After code works on a VM, we can move to a lab or a staging environment to do more scale testing before deploying to production. Developing against a VM is not something I did before working at Rackspace, and I find the practice so useful that I will do what I can to continue doing it no matter where I work in the future.

dnaquin 16y ago

Only use VMs or sandboxes on production-like machines. Local development doesn't scale.

mitchellh 16y ago

Disclaimer: I am the creator of what I'm about to plug here. If anyone is looking for an easy, low-risk, documented method of beginning development in VMs, check out Vagrant[1], which is what I wrote this for. Creating dev VMs from scratch is pretty tedious and there are a lot of steps involved. Vagrant abstracts this away and provides a tool to automate all of this in a reproducible manner. [1]: http://vagrantup.com http://vagrantup.com

gnubardt 16y ago
I love vagrant! It was super easy to create an archlinux base box and i've been using chef to automatically provision environments. Setting up the environment for a new project takes minutes and is both totally automated and separated from my actual OS. Thanks!
denimboy 16y ago
I started using vagrant last week to debug my Chef scripts and it is indeed awesome. Vagrant uses Chef (chef-solo) configure your VM from bare image to working server. I see that is has a plug in system for other provision systems like Puppet, but the chef support works great out of the box. I was hoping to contribute to the project by integrating fog the ruby library which talks to many cloud systems including EC2 so someone could build an image via Chef or Puppet, dev and test in their image, then provision and upload to the cloud. Could probably do this without integrating with vagrant by building a clever rake or capistrano script.

chewbranca 16y ago

This reminds me of facebook development, where you need to have a live fqdn for all of your environments. We used to use a staging server and had a similar problem with huge amounts of small unnatural commits for things you would normally test locally. I ended up moving to using ssh tunnels to a server, where each dev has a subdomain redirecting to their dev box through the tunnel. This has the added benefit that you can work locally from anywhere you have wifi. Whether you use tunnels or VMs, both are a solid way to do development.

drwxrwxrwt 16y ago

Another benefit of this approach is that it takes seconds to revert to a good state if you destroyed your dataset in a mishap. use snapshots or just restart VM from original image. all while using your preferred code editor (the app source is mounted from the host OS)

Jach 16y ago

If commits were an issue before this, they should really have considered using git or something. You shouldn't aim for fewer commits, but less expensive commits, i.e. nothing goes over a network. Plus you can squash later if you really need to. Anyway, VMs are cool but has anyone tried developing with NixOS? It seems even more suited for the tasks they want since it has rollbacks and so on: http://nixos.org/ http://nixos.org/

yycom 16y ago
Isn't the expense of frequent commits the cost of composing an intelligible commit message?

prodigal_erik 16y ago

Having a VM you can trivially reimage is very helpful because you can test that (re)deploying packages to it is going to work, you didn't miss any dependencies that happen to be present on your desktop but not on your production servers, and uninstall won't leave cruft behind.

hdeshev 16y ago

This all VM thing is fine and dandy if you want to have an accessible mirror of your staging machine and reduce unnecessary stages. Although that in itself looks fishy - if developers want to stage something just to see if it works, then something is seriously wrong with your process or developers. I'm amazed at the level of sophistication in this setup - using virtual machines and TCP proxies that do text replaces seems too much. They could have just built the configurability in the damn application. I have a .NET app that normally gets deployed on three servers and uses various remote resources like Amazon SQS queues, but I easily built a configuration abstraction similar to Rails' environments that lets me deploy on a single machine and even test locally with a debugger attached. My point is that this problem looks as if it got solved at the wrong level of abstraction. Which, obviously, made it unnecessarily hard and complex. If I had to tackle it, I'd make sure developers do the right thing and try to make the app configurable enough first.

tomedme 16y ago
The mind boggles. There's probably a rockstar in there looking to put something on their CV/resume. Whatever the case, it smells wrong.