9 ms·

If you love automation, why is your development environment manual?

137 points by timruffles 13y ago

campbellmorgan 13y ago
I think the general idea of running a vm with all your dev setup in is great, but I've often found the small niggles (such as speed of switching quickly between browser and vm terminal, the fact you need a fairly good computer to run even quite a modest server with a gui and if you don't use a gui, you're limited to 256 colours) all a bit more trouble than the benefits. My solution is to set up my perfect dev environment in linux (I use Fedora) on a partitioned usb pen drive. Then either boot from it (if i'm on a crappy computer) or boot a vm direct from the usb if the computer is more competent. It's also quite nice if there's an emergency and you're away from your computer -- you can just rush into an internet cafe, plead/bribe them to let you boot from your usb then have everything, ssh keys inlcuded ready for urgent repairs!
- scaryclam 13y ago
  I think the point here is that it took you time to set up that dev environment. It shouldn't matter where your environment is (a VM, a laptop, etc), you should be able to recreate it easily and reliably. This is what the automation tools are for. I prefer salt stack and have a salt config for my work laptop. If something happens to the laptop or I get a new one I just need to run salt, wonder off for an hour and when I get back my entire environment is set up. I also use it with vagrant so that when a new dev joins the team they can run one command and have a copy of the project setup in a vagrant box, databases and all, ready to go. You could set up a similar thing so that if you lose that USB key you can have your prefect setup reinstalled with minimal fuss.
  - pestaa 13y ago
    Would you be able to share at least some of your current salt config? This is one idea I've been thinking a lot about lately, but never managed to get beyond the obvious package names, which is trivially installable with one command on a good Linux distro.
  - campbellmorgan 13y ago
    very good suggestion. At the moment I keep an LVM snapshot, but that is a bit cumbersome. I hadn't heard of salt - thanks! Will look into it.
- jstanley 13y ago
  "such as speed of switching quickly between browser and vm terminal, the fact you need a fairly good computer to run even quite a modest server with a gui and if you don't use a gui, you're limited to 256 colours" This implies you are running a terminal emulator in X on the VM and doing your coding there? I use VMs for development all the time, and a headless VM with SSH access is perfect. It's almost identical to doing development locally.
  - campbellmorgan 13y ago
    the options as I see it are either do everything over ssh to a headless vm as you suggest which definitely has its advantages or working inside a full-screened vm/ booted environment. Perhaps not the best justification, but I prefer the latter because I can include my gvim setup and use the mouse for resizing columns, browsing code and using NERDTree. I miss that when running vim over ssh.
    - SpenserJ 13y ago
      Bring your configs with you - http://dotfiles.github.io/ http://dotfiles.github.io/ Vim should be able to do everything that GVim can do, if you add a few lines to your configs. I'm not sure what specifically though. I bring my entire development setup with me when using a server, and preparing it is as simple as cloning Homeshick (or Homesick if you have Ruby), cloning your dotfiles repository, and installing any applications that you want to use.
    - gtaylor 13y ago
      You can edit and browse on your host VM. Just adjust the VM's network settings and use a shared host-guest folder for the code (what exactly this ends up being varies based on the virtualization). You really can make it a near exact match for a local development environment. Bridge your ports to your expected ones on the host machine, share the code folders, and SSH in if you need to punt a dev server when you make changes (or set your Editor/IDE up to do that for you). Browser can still point at local:8000 or whatever you expect.
- eli 13y ago
  Why are you switching between a browser and your VM terminal? I run my VM headless -- all the files I need to edit are in shared directory.
  - jabbernotty 13y ago
    You don't run any commands on your VM? Such as compiling/restarting an application?
    - gtaylor 13y ago
      You can run those through your console on the host machine like normal, with the added SSH piece. Or you could set your editor/IDE up to do that after changes (if required).
    - eli 13y ago
      Well, I suspect SSH is going to be faster than using the VM terminal in any case. But I'd set it up to watch the directory and compile/restart automatically, if you can.
- tga 13y ago
  My primary development environment is a VMware virtual machine with Ubuntu LTS getting 2GB out of my MacBook Air's 4GB of RAM. I usually run a database, a web server, a couple of browsers and PyCharm (Java) in the virtual machine and another browser with many tabs in the host OS. What I wanted to point out is that everything works perfectly and I can instantly switch between applications and in/out of the VM just fine. Backing up the VM is of course as simple as copying ~40GB to an external drive or the desktop at home (gigabit ethernet ftw) and the nice thing is that I can have multiple versions of it on various machines.
xentronium 13y ago
For OS X tool see http://boxen.github.com/ http://boxen.github.com/
- davefp 13y ago
  Boxen is rad. We've started using it for new machines at Shopify and it's been really helpful.
- druiid 13y ago
  Well, but the article was actually speaking out against Boxen a bit. Essentially, you're still shoehorning all of the dev stuff onto your local machine with Boxen. It's much better than installing it all manually, but still not a separate dev environment.
rb2k_ 13y ago
Any comments on what people use to manage the chef cookbooks (and why)? Librarian or Berkshelf seem to be the two main contenders to make updating cookbooks similar to updating gems with bundler.
- timruffles 13y ago
  I used Berkshelf and it works nicely! The only confusion comes when actually deploying via Chef solo and I forget to ensure the cookbooks are actually present. Haven't tried Librarian - only because I saw Berkshelf first and it integrated nicely with Vagrant.
- tomku 13y ago
  I've tried both Librarian and Berkshelf to manage cookbooks for my Vagrant VMs for use with Vagrant's chef-solo provisioner. They both have nearly identical core features. Both can grab cookbooks from the Chef community site, local path, generic Git repository (which includes getting a single cookbook from a subdir) or Github. Their syntax is nearly identical. The only difference that I can think of is that Berkshelf has pessimistic version matching support (a la Bundler), while I'm not sure that librarian-chef does. Where they differ is in where they put your cookbooks, and what additional features they offer. Librarian-chef puts your cookbooks in a "cookbooks" folder in the project root, while Berkshelf keeps a global cache of all the versions of each cookbook you've installed outside of the project. Both provide support for uploading your cookbooks to a Chef server, librarian-chef through plain knife and Berkshelf through it's own upload command. Berkshelf also features scaffolding support for Vagrantfiles and new cookbooks, and generally has extra features specifically for rapidly iterating on cookbooks themselves. It also provides a vagrant plugin to manage bundling/uploading cookbooks to a Vagrant VM, since they need to be copied out of the central repo and made available. Overall, I've found that I like Berkshelf better. It feels a little more idiomatic, and I like having my cookbook versions shared between projects rather than duplicated everywhere. If you have any specific questions about either, I'd be happy to answer them.
  - bryanlarsen 13y ago
    I've never tried Berkshelf, didn't even know about it. librarian-chef works great for our needs; I doubt it's worth investigating alternatives. I will note that the librarian-chef also has a vagrant plugin: vagrant-librarian-chef.
dugmartin 13y ago
Your don't need to give up your GUI editor to do this. Vagrant sets up a shared folder automatically in the virtual environment under /vagrant that maps to the folder where your Vagrantfile is located on the host machine. I do all my development now using Vagrant running Ubuntu VMs but I still do all my editing with the same Windows editor I've used for years running on the host machine.
- jaredmcateer 13y ago
  I also prefer to edit on my local machine but I still prefer terminal vim, I'd rather not have the environment import my settings because I work in a team and don't want to fight with them over the one true config.
  - npsimons 13y ago
    If your team is checking in to the central repository such personal things as config files for development environments, you've got bigger problems. Sure, occasional snippets (such as a .dir-locals.el for style), but whole configs? WTF?
    - jaredmcateer 13y ago
      The environment is shared for a consistency, if I'm sshing into the environment to do my work then that means I need the environment updated with my config and that either means that we need to share configs on the vagrant user or setup the environment to provision settings for every user of that environment which sounds unmaintainable. It's easier to just work locally through the share directory.
      - sikhnerd 13y ago
        Use can easily do something like this for one-off edits "vim scp://remoteuser@server.tld//path/to/document" or use netrw for a more robust solution to using your local vim to edit remotly. http://www.drchip.org/astronaut/vim/index.html#NETRW http://www.drchip.org/astronaut/vim/index.html#NETRW
roel_v 13y ago
It's funny that this article proclaims to combat yak shaving while it actually promotes it.
- hobs 13y ago
  I take your point, but I would say that the article really encourages shaving the yak one time, instead of using scissors to beat it back occasionally.
markeganfuller 13y ago
I prefer having a VM built with Vagrant (and in our case puppet) which runs the same configs as our servers while having my development tools (editors etc) installed on the local machine. That way all my apps run at native speed, and the server replicates (almost exactly) the production environment.
qwerta 13y ago
>If you love automation why is your development environment manual? Because I usually do it once in 5 years :-) My setup: Linux, Kubuntu LTS. Entire system is backed-up with rsync, so I have daily snapshots for last 6 months. There is also weekly backup to cloud and external hdd. I have primary 250GB SSD which usually sits in my workstation. Whan I travel I physically move SSD to laptop (takes 10 minutes). If something went wrong and my primary SSD would die: I just boot from Livecd, copy 200GB, setup bootloader and reboot. No need to 'reinstall' system from scratch. Every six months I take one afternoon and update all my tools (IDE, OS, apps...). I dont use automated updates except critical stuff such as browser.
- peterwwillis 13y ago
  Ditto. Backing up your dev environment is less time-consuming and more reliable than scripting it.
  - StavrosK 13y ago
    I don't know, I script production provisioning anyway. Dev provisioning is free, I just change the hostname to the VM.
- fideloper 13y ago
  I think an important point in this post was more about not trying to setup a dev environment in your base Mac install (or similar). Use Virtual Machines.
  - qwerta 13y ago
    In that case I just find it better to wipe MacOS and replace it with Linux.
    - gtaylor 13y ago
      It's not always a good solution for your entire team. I run Linux on my desktop, but have found that the touchpad drivers (even the experimental ones) just really suck on Linux+Macbook Pro to the point where it's either external mouse (not always possible) or decreased productivity. I have a feeling this is going to devolve into a "well, just use vim and a tiling WM or X other thing", but the point is that just replacing Mac OS with Linux isn't always going to be good for your whole team.
dmytton 13y ago
We do this using Vagrant and Puppet and actually just blogged about it yesterday: http://blog.serverdensity.com/many-projects-with-vagrant-and-puppet/ http://blog.serverdensity.com/many-projects-with-vagrant-and...
andrewcooke 13y ago
if you use an ide you can do something similar, but use the vm client directly. i do this all the time, and virtualbox's seamless mode means that you can mix windows from client and host on the same screen (except ubuntu unity?). it's particularly good when different clients have different OSs. and you can even do hardware development - i have tested usb drivers in a vm client that talk to hardware connected to the host. the only drawback is initial startup time (particularly pulling latest updates after install) and archiving the vms (they're large, so fill up a laptop ssd). i export to ovas on my backup system and then wipe the vm. another worry is that virtualbox has been flakey recently (http://www.acooke.org/cute/VirtualBox1.html http://www.acooke.org/cute/VirtualBox1.html http://www.acooke.org/cute/UbuntuonVi0.html http://www.acooke.org/cute/UbuntuonVi0.html) - but ovas can be imported into other hosts...
hleszek 13y ago
obligatory xkcd: http://xkcd.com/1205/ http://xkcd.com/1205/
- medecau 13y ago
  "xkcd #1205 says I don't need it."
VLM 13y ago
The author missed a golden opportunity to discuss battery life. Your laptop will run longer if you SSH/VNC/whatever into the virtualized development machine if you don't host the virtualized image on the laptop. Also I can SSH/VNC/Xwindow/whatever into a server machine with performance stats far beyond any currently imaginable laptop. Its like owning a laptop from 2023 today in 2013.
- 23david 13y ago
  The latency is a pain. I like being able to work from anywhere by tethering my iphone5's LTE connection. Easier and more reliable to have everything working locally (With local dev environments using vagrant/virtualbox) and use the internet connection for syncing git changes or doing research.
  - archivator 13y ago
    There's the Mosh project (http://mosh.mit.edu http://mosh.mit.edu) which seems fit for your use case. I don't have personal experience with it but it looks promising.
    - 23david 13y ago
      wow looks very cool. never heard of it... http://mosh.mit.edu/#techinfo http://mosh.mit.edu/#techinfo How Mosh works Remote-shell protocols traditionally work by conveying a byte-stream from the server to the client, to be interpreted by the client's terminal. (This includes TELNET, RLOGIN, and SSH.) Mosh works differently and at a different layer. With Mosh, the server and client both maintain a snapshot of the current screen state. The problem becomes one of state-synchronization: getting the client to the most recent server-side screen as efficiently as possible. This is accomplished using a new protocol called the State Synchronization Protocol, for which Mosh is the first application. SSP runs over UDP, synchronizing the state of any object from one host to another. Datagrams are encrypted and authenticated using AES-128 in OCB mode. While SSP takes care of the networking protocol, it is the implementation of the object being synchronized that defines the ultimate semantics of the protocol.
  - 13y ago
orclev 13y ago
I tried this on a recent couple projects and it failed miserably on both of them for completely different reasons. On one, the architecture of the project dictated really really fast network connections and the overhead from the VM choked the project. That's probably an unfair example because this was actually a toy project that was never meant to run in a serious environment (hence the requirement of a huge bandwidth virtually 0 latency network connection). The other failure however is more serious. Our company (and at least one project) dictates a Windows based development environment, and the abysmal performance of NTFS combined with a massive project code base meant it was taking 2+ minutes just to run a status check on the version controlled code. I think given a Linux host environment this might work better, or a smaller code base, but at least in these two situations it was a failure.
- bryanlarsen 13y ago
  Note that the recent release of VMWare support for Vagrant may at least partially mitigate the problems in your second scenario.
cpursley 13y ago
Cloud development like www.nitrous.io (previously action.io) is the future. I've played around with it am quite impressed. Basically no setup for standard platforms like rails and node - choose your dev environment, install your dependencies if not already there and get coding. It's a Linux environment so your dev is same or close to production. And the IDE is quite nice. Caveat: Yes, you have to have a connection. But, for those few precious unconnected moments, I don't want to be coding anyways.
CatMtKing 13y ago
I've heard of Salt Stack[1], though I've never used it. From what I can tell, it can be used to do the same thing. [1] http://docs.saltstack.com/ http://docs.saltstack.com/
- jsdalton 13y ago
  The Salt Stack team also has a plugin built specifically for provisioning environments in Vagrant called Salty Vagrant: https://github.com/saltstack/salty-vagrant https://github.com/saltstack/salty-vagrant (I haven't used it, but I have used -- and love using -- Salt Stack.)
jpitz 13y ago
Monoculture used to be a dirty word.
khitchdee 13y ago
Points towards generative programming?
mwcampbell 13y ago
I've often thought that it would be nice to have a script that rebuilt the software installation on a developer's workstation from scratch, completely automatically, using only configuration files and scripts checked into version control somewhere. Such a process could run every night while the developer sleeps. This would ensure that it's dead easy to bring a new developer fully up to speed on a team.
- hosh 13y ago
  I know a couple guys who do that. Their Steam app is also managed by Chef. We got our dev setup partially automated like that. Even the partial setup was better than what I heard -- took a week to set up a laptop with the tools and the whole platform. When I was onboarded, it took about day, more or less hands off.
zerr 13y ago
Couldn't this be shown without a blasphemy?
- taybin 13y ago
  Where is the blasphemy? The terrible restoration of the fresco?
  - zerr 13y ago
    Shown in the context of this article.
    - taybin 13y ago
      I re-read the article and still don't know what you're talking about.
druiid 13y ago
I fully agree with this. 'Before', the dev team used a mix of, well, I'd rather not say. It was just not very pretty and eventually everyone got their work done. A while back I built a vagrant+puppet configuration for everyone and the team has since transitioned to using that. It's nice to have at least semi-parity with the production environment. This is possible because we can utilize most (not all, due to no central Puppet server) of the same Puppet modules which we use (or I built), in the VM. Essentially what you gain is not worrying about if a developer will break their machine/VM and go 'Whoops, can you fixor it?'. Additionally, you no longer have to worry about such things as php/node/ruby version issues with the dev/production side of things. We've gone from having sometimes issues with certain code not running the same as in development, to just throwing it up on the staging environment through deployment tools and it just runs!
karlkatzke 13y ago
Issue: What about VPN? If you're connecting to your corporate network with OpenVPN, and you are restricted to one connection (common sense security policy) then you need to get your infrastructure team to issue you a separate key for each VM that you're running that you want to be able to access a tunnel and get on your corporate network. I realize that this isn't an issue for 99.9% of startups, but in the non-startup corporate world where there are security implications to keeping everything in a publicly accessible code repo (healthcare, government, education in some states) then you've got another complication. The only thing I've been able to work up so far is sharing a folder on my desktop through to the VM and running commits from my desktop.
- druiid 13y ago
  I would say that you probably shouldn't be doing VPN to your development boxes in general. I'd see this as a security risk. I generally don't let anything 'production' touch directly anything which might not have the same security standards as the production gear.
- mwcampbell 13y ago
  Using NAT-based networking for the VMs should neatly solve the VPN issue.
hosh 13y ago
Vagrant + Chef is great! Vagrant + Chef + Berkshelf is better! You can define the cookbook dependencies in the Berksfile and version it along with Vagrantfile. Berkshelf is like bundler for your cookbooks. It assembles cookbooks from a variety of sources and loads them into your Vagrant-managed vm. Check it out at http://berkshelf.com http://berkshelf.com To help get things going in that end, I've been working on Kamino (https://github.com/hosh/kamino https://github.com/hosh/kamino). It's intended to generate a Vagrantfile+Berkshelf file for use in your project. Right now, it only supports a basic Rails stack. Pull requests welcomed :-)
pixelcort 13y ago
How about the best of both worlds? A common VM image with CLI tools and dev server inside the guest OS, with VirtualBox folder sharing to permit a GUI IDE and the git repo to live on the host OS?
- tjbiddle 13y ago
  This is what we do now - It gets to be a bit of a pain later. I'm looking forward to switching to Vagrant in the future (On the backburner for now as I have other things on my plate) - Vagrant allows you to easily nuke and re-setup your entire environment, and change things easily for the next go around - while if you're using a VM image you'd need to re-image every time you make changes that you want to stay persistent.
kstrauser 13y ago
I don't want dev to be exactly like stage and prod. Yep, you read that right. I deploy to Debian servers but develop on Linux Mint or OS X. The key is that the environments are very similar but not identical. The payoff comes when you break yourself of the habit of relying on accidents of deployment instead of building a general-case solution that works in multiple environments. You scan /proc to look at running processes? That's great, until you're on a machine without /proc. Better to spend an hour learning how your dev platform (Python, Java, whatever) abstracts that away for you. Trying to send an email by shelling out to /usr/bin/sendmail? Oops! That's broken in lots of places; better learn how your dev platform handles it! The big win comes when you upgrade your stage and prod environments to a newer distro - or a different one altogether - and your stuff keeps working because you'd relentlessly whittled away all the OS-specific dependencies.
- mwcampbell 13y ago
  You describe a pay-off when upgrading stage and prod. But if dev, stage, and prod are identical, then you simply upgrade dev first, find the things that broke, do the same for stage, and then it's no big deal when you upgrade prod.
- blake8086 13y ago
  It seems odd to have an environment that you are 100% in control of, and then relish the times when it does unexpected things.
- snowwrestler 13y ago
  This strikes me as an example of an unnecessary/premature optimization. Unless you know that your product will need to deploy on an unknown variety of server environments, there's no advantage in taking the extra time to 100% abstract away the server environment. And really, how trustworthy is that abstraction? You've basically outsourced responsibility for your server environment to your language community. What if they create a bug that is exposed on Debian, but not on OS X?
  - gtaylor 13y ago
doki_pen 13y ago
There is something to be said for running your software on multiple environments. There is also something to be said for stubbing out pieces of your stack for testing and development purposes. There is often a lot to be learned. That said, I've used the VM approach on projects with high turnover. It is usually not the same as production deployment, so it does take effort, but depending on the team it might be worth the effort.
vinitool76 13y ago
How is this different from Boxen (http://boxen.github.com/ http://boxen.github.com/)? You have exactly copied the central idead around it and made a blog post.
- hoov 13y ago
  The two ideas are very different, but complementary. Boxen is about getting your MacOS environment set up with tooling. The author of the original blog posting is talking about how to automate the creation of a production-like environment in which to do your development. The creation of that environment requires a few tools installed on your physical Mac, which could be managed via Boxen. For example, you might use Boxen to automate the installation/upgrade of Vagrant or your IDE of choice.
rbsn 13y ago
I have used Vagrant in the past, and I had to stop using it. Shared folders were just too slow. 2 seconds to refresh a page was too much when you're trying to be productive.
- hosh 13y ago
  Huh, that's interesting. How old is your hardware?
EliRivers 13y ago
No no no, I don't like automation; I like automating things :)
graycat 13y ago
The first reason is, people who build tools mostly don't think that the tools will or should be used as components of other tools for more in 'automation'. The result is that it's tough to build on such tools. The second, and more specific, reason is that location of one of my most intense love/hate dilemmas, Xerox PARC that pushed the first 'graphical user interface' (GUI) instead of what came before, usually typing text into a command line. Command lines are mostly easy to automate. GUIs are mostly a total PAIN to automate. I'm wanting to automate, willing to automate, waiting to automate.
sandGorgon 13y ago
has anybody tried Vagabond (https://github.com/chrisroberts/vagabond https://github.com/chrisroberts/vagabond) to do the same thing without a VM ?

137 points by timruffles 13y ago

I think the general idea of running a vm with all your dev setup in is great, but I've often found the small niggles (such as speed of switching quickly between browser and vm terminal, the fact you need a fairly good computer to run even quite a modest server with a gui and if you don't use a gui, you're limited to 256 colours) all a bit more trouble than the benefits. My solution is to set up my perfect dev environment in linux (I use Fedora) on a partitioned usb pen drive. Then either boot from it (if i'm on a crappy computer) or boot a vm direct from the usb if the computer is more competent. It's also quite nice if there's an emergency and you're away from your computer -- you can just rush into an internet cafe, plead/bribe them to let you boot from your usb then have everything, ssh keys inlcuded ready for urgent repairs!

scaryclam 13y ago
I think the point here is that it took you time to set up that dev environment. It shouldn't matter where your environment is (a VM, a laptop, etc), you should be able to recreate it easily and reliably. This is what the automation tools are for. I prefer salt stack and have a salt config for my work laptop. If something happens to the laptop or I get a new one I just need to run salt, wonder off for an hour and when I get back my entire environment is set up. I also use it with vagrant so that when a new dev joins the team they can run one command and have a copy of the project setup in a vagrant box, databases and all, ready to go. You could set up a similar thing so that if you lose that USB key you can have your prefect setup reinstalled with minimal fuss.
- pestaa 13y ago
  Would you be able to share at least some of your current salt config? This is one idea I've been thinking a lot about lately, but never managed to get beyond the obvious package names, which is trivially installable with one command on a good Linux distro.
- campbellmorgan 13y ago
  very good suggestion. At the moment I keep an LVM snapshot, but that is a bit cumbersome. I hadn't heard of salt - thanks! Will look into it.
jstanley 13y ago
"such as speed of switching quickly between browser and vm terminal, the fact you need a fairly good computer to run even quite a modest server with a gui and if you don't use a gui, you're limited to 256 colours" This implies you are running a terminal emulator in X on the VM and doing your coding there? I use VMs for development all the time, and a headless VM with SSH access is perfect. It's almost identical to doing development locally.
- campbellmorgan 13y ago
  the options as I see it are either do everything over ssh to a headless vm as you suggest which definitely has its advantages or working inside a full-screened vm/ booted environment. Perhaps not the best justification, but I prefer the latter because I can include my gvim setup and use the mouse for resizing columns, browsing code and using NERDTree. I miss that when running vim over ssh.
  - SpenserJ 13y ago
    Bring your configs with you - http://dotfiles.github.io/ http://dotfiles.github.io/ Vim should be able to do everything that GVim can do, if you add a few lines to your configs. I'm not sure what specifically though. I bring my entire development setup with me when using a server, and preparing it is as simple as cloning Homeshick (or Homesick if you have Ruby), cloning your dotfiles repository, and installing any applications that you want to use.
  - gtaylor 13y ago
    You can edit and browse on your host VM. Just adjust the VM's network settings and use a shared host-guest folder for the code (what exactly this ends up being varies based on the virtualization). You really can make it a near exact match for a local development environment. Bridge your ports to your expected ones on the host machine, share the code folders, and SSH in if you need to punt a dev server when you make changes (or set your Editor/IDE up to do that for you). Browser can still point at local:8000 or whatever you expect.
eli 13y ago
Why are you switching between a browser and your VM terminal? I run my VM headless -- all the files I need to edit are in shared directory.
- jabbernotty 13y ago
  You don't run any commands on your VM? Such as compiling/restarting an application?
  - gtaylor 13y ago
    You can run those through your console on the host machine like normal, with the added SSH piece. Or you could set your editor/IDE up to do that after changes (if required).
  - eli 13y ago
    Well, I suspect SSH is going to be faster than using the VM terminal in any case. But I'd set it up to watch the directory and compile/restart automatically, if you can.
tga 13y ago
My primary development environment is a VMware virtual machine with Ubuntu LTS getting 2GB out of my MacBook Air's 4GB of RAM. I usually run a database, a web server, a couple of browsers and PyCharm (Java) in the virtual machine and another browser with many tabs in the host OS. What I wanted to point out is that everything works perfectly and I can instantly switch between applications and in/out of the VM just fine. Backing up the VM is of course as simple as copying ~40GB to an external drive or the desktop at home (gigabit ethernet ftw) and the nice thing is that I can have multiple versions of it on various machines.

xentronium 13y ago

For OS X tool see http://boxen.github.com/ http://boxen.github.com/

davefp 13y ago
Boxen is rad. We've started using it for new machines at Shopify and it's been really helpful.
druiid 13y ago
Well, but the article was actually speaking out against Boxen a bit. Essentially, you're still shoehorning all of the dev stuff onto your local machine with Boxen. It's much better than installing it all manually, but still not a separate dev environment.

rb2k_ 13y ago

Any comments on what people use to manage the chef cookbooks (and why)? Librarian or Berkshelf seem to be the two main contenders to make updating cookbooks similar to updating gems with bundler.

timruffles 13y ago
I used Berkshelf and it works nicely! The only confusion comes when actually deploying via Chef solo and I forget to ensure the cookbooks are actually present. Haven't tried Librarian - only because I saw Berkshelf first and it integrated nicely with Vagrant.
tomku 13y ago
I've tried both Librarian and Berkshelf to manage cookbooks for my Vagrant VMs for use with Vagrant's chef-solo provisioner. They both have nearly identical core features. Both can grab cookbooks from the Chef community site, local path, generic Git repository (which includes getting a single cookbook from a subdir) or Github. Their syntax is nearly identical. The only difference that I can think of is that Berkshelf has pessimistic version matching support (a la Bundler), while I'm not sure that librarian-chef does. Where they differ is in where they put your cookbooks, and what additional features they offer. Librarian-chef puts your cookbooks in a "cookbooks" folder in the project root, while Berkshelf keeps a global cache of all the versions of each cookbook you've installed outside of the project. Both provide support for uploading your cookbooks to a Chef server, librarian-chef through plain knife and Berkshelf through it's own upload command. Berkshelf also features scaffolding support for Vagrantfiles and new cookbooks, and generally has extra features specifically for rapidly iterating on cookbooks themselves. It also provides a vagrant plugin to manage bundling/uploading cookbooks to a Vagrant VM, since they need to be copied out of the central repo and made available. Overall, I've found that I like Berkshelf better. It feels a little more idiomatic, and I like having my cookbook versions shared between projects rather than duplicated everywhere. If you have any specific questions about either, I'd be happy to answer them.
- bryanlarsen 13y ago
  I've never tried Berkshelf, didn't even know about it. librarian-chef works great for our needs; I doubt it's worth investigating alternatives. I will note that the librarian-chef also has a vagrant plugin: vagrant-librarian-chef.

dugmartin 13y ago

Your don't need to give up your GUI editor to do this. Vagrant sets up a shared folder automatically in the virtual environment under /vagrant that maps to the folder where your Vagrantfile is located on the host machine. I do all my development now using Vagrant running Ubuntu VMs but I still do all my editing with the same Windows editor I've used for years running on the host machine.

jaredmcateer 13y ago
I also prefer to edit on my local machine but I still prefer terminal vim, I'd rather not have the environment import my settings because I work in a team and don't want to fight with them over the one true config.
- npsimons 13y ago
  If your team is checking in to the central repository such personal things as config files for development environments, you've got bigger problems. Sure, occasional snippets (such as a .dir-locals.el for style), but whole configs? WTF?
  - jaredmcateer 13y ago
    The environment is shared for a consistency, if I'm sshing into the environment to do my work then that means I need the environment updated with my config and that either means that we need to share configs on the vagrant user or setup the environment to provision settings for every user of that environment which sounds unmaintainable. It's easier to just work locally through the share directory.
    - sikhnerd 13y ago
      Use can easily do something like this for one-off edits "vim scp://remoteuser@server.tld//path/to/document" or use netrw for a more robust solution to using your local vim to edit remotly. http://www.drchip.org/astronaut/vim/index.html#NETRW http://www.drchip.org/astronaut/vim/index.html#NETRW

roel_v 13y ago

It's funny that this article proclaims to combat yak shaving while it actually promotes it.

hobs 13y ago
I take your point, but I would say that the article really encourages shaving the yak one time, instead of using scissors to beat it back occasionally.

markeganfuller 13y ago

I prefer having a VM built with Vagrant (and in our case puppet) which runs the same configs as our servers while having my development tools (editors etc) installed on the local machine. That way all my apps run at native speed, and the server replicates (almost exactly) the production environment.

qwerta 13y ago

>If you love automation why is your development environment manual? Because I usually do it once in 5 years :-) My setup: Linux, Kubuntu LTS. Entire system is backed-up with rsync, so I have daily snapshots for last 6 months. There is also weekly backup to cloud and external hdd. I have primary 250GB SSD which usually sits in my workstation. Whan I travel I physically move SSD to laptop (takes 10 minutes). If something went wrong and my primary SSD would die: I just boot from Livecd, copy 200GB, setup bootloader and reboot. No need to 'reinstall' system from scratch. Every six months I take one afternoon and update all my tools (IDE, OS, apps...). I dont use automated updates except critical stuff such as browser.

peterwwillis 13y ago
Ditto. Backing up your dev environment is less time-consuming and more reliable than scripting it.
- StavrosK 13y ago
  I don't know, I script production provisioning anyway. Dev provisioning is free, I just change the hostname to the VM.
fideloper 13y ago
I think an important point in this post was more about not trying to setup a dev environment in your base Mac install (or similar). Use Virtual Machines.
- qwerta 13y ago
  In that case I just find it better to wipe MacOS and replace it with Linux.
  - gtaylor 13y ago
    It's not always a good solution for your entire team. I run Linux on my desktop, but have found that the touchpad drivers (even the experimental ones) just really suck on Linux+Macbook Pro to the point where it's either external mouse (not always possible) or decreased productivity. I have a feeling this is going to devolve into a "well, just use vim and a tiling WM or X other thing", but the point is that just replacing Mac OS with Linux isn't always going to be good for your whole team.

dmytton 13y ago

We do this using Vagrant and Puppet and actually just blogged about it yesterday: http://blog.serverdensity.com/many-projects-with-vagrant-and-puppet/ http://blog.serverdensity.com/many-projects-with-vagrant-and...

andrewcooke 13y ago

if you use an ide you can do something similar, but use the vm client directly. i do this all the time, and virtualbox's seamless mode means that you can mix windows from client and host on the same screen (except ubuntu unity?). it's particularly good when different clients have different OSs. and you can even do hardware development - i have tested usb drivers in a vm client that talk to hardware connected to the host. the only drawback is initial startup time (particularly pulling latest updates after install) and archiving the vms (they're large, so fill up a laptop ssd). i export to ovas on my backup system and then wipe the vm. another worry is that virtualbox has been flakey recently (http://www.acooke.org/cute/VirtualBox1.html http://www.acooke.org/cute/VirtualBox1.html http://www.acooke.org/cute/UbuntuonVi0.html http://www.acooke.org/cute/UbuntuonVi0.html) - but ovas can be imported into other hosts...

hleszek 13y ago

obligatory xkcd: http://xkcd.com/1205/ http://xkcd.com/1205/

medecau 13y ago
"xkcd #1205 says I don't need it."

VLM 13y ago

The author missed a golden opportunity to discuss battery life. Your laptop will run longer if you SSH/VNC/whatever into the virtualized development machine if you don't host the virtualized image on the laptop. Also I can SSH/VNC/Xwindow/whatever into a server machine with performance stats far beyond any currently imaginable laptop. Its like owning a laptop from 2023 today in 2013.

23david 13y ago
The latency is a pain. I like being able to work from anywhere by tethering my iphone5's LTE connection. Easier and more reliable to have everything working locally (With local dev environments using vagrant/virtualbox) and use the internet connection for syncing git changes or doing research.
- archivator 13y ago
  There's the Mosh project (http://mosh.mit.edu http://mosh.mit.edu) which seems fit for your use case. I don't have personal experience with it but it looks promising.
  - 23david 13y ago
    wow looks very cool. never heard of it... http://mosh.mit.edu/#techinfo http://mosh.mit.edu/#techinfo How Mosh works Remote-shell protocols traditionally work by conveying a byte-stream from the server to the client, to be interpreted by the client's terminal. (This includes TELNET, RLOGIN, and SSH.) Mosh works differently and at a different layer. With Mosh, the server and client both maintain a snapshot of the current screen state. The problem becomes one of state-synchronization: getting the client to the most recent server-side screen as efficiently as possible. This is accomplished using a new protocol called the State Synchronization Protocol, for which Mosh is the first application. SSP runs over UDP, synchronizing the state of any object from one host to another. Datagrams are encrypted and authenticated using AES-128 in OCB mode. While SSP takes care of the networking protocol, it is the implementation of the object being synchronized that defines the ultimate semantics of the protocol.
- 13y ago

orclev 13y ago

I tried this on a recent couple projects and it failed miserably on both of them for completely different reasons. On one, the architecture of the project dictated really really fast network connections and the overhead from the VM choked the project. That's probably an unfair example because this was actually a toy project that was never meant to run in a serious environment (hence the requirement of a huge bandwidth virtually 0 latency network connection). The other failure however is more serious. Our company (and at least one project) dictates a Windows based development environment, and the abysmal performance of NTFS combined with a massive project code base meant it was taking 2+ minutes just to run a status check on the version controlled code. I think given a Linux host environment this might work better, or a smaller code base, but at least in these two situations it was a failure.

bryanlarsen 13y ago
Note that the recent release of VMWare support for Vagrant may at least partially mitigate the problems in your second scenario.

cpursley 13y ago

Cloud development like www.nitrous.io (previously action.io) is the future. I've played around with it am quite impressed. Basically no setup for standard platforms like rails and node - choose your dev environment, install your dependencies if not already there and get coding. It's a Linux environment so your dev is same or close to production. And the IDE is quite nice. Caveat: Yes, you have to have a connection. But, for those few precious unconnected moments, I don't want to be coding anyways.

CatMtKing 13y ago

I've heard of Salt Stack[1], though I've never used it. From what I can tell, it can be used to do the same thing. [1] http://docs.saltstack.com/ http://docs.saltstack.com/

jsdalton 13y ago
The Salt Stack team also has a plugin built specifically for provisioning environments in Vagrant called Salty Vagrant: https://github.com/saltstack/salty-vagrant https://github.com/saltstack/salty-vagrant (I haven't used it, but I have used -- and love using -- Salt Stack.)

jpitz 13y ago

Monoculture used to be a dirty word.

khitchdee 13y ago

Points towards generative programming?

mwcampbell 13y ago

I've often thought that it would be nice to have a script that rebuilt the software installation on a developer's workstation from scratch, completely automatically, using only configuration files and scripts checked into version control somewhere. Such a process could run every night while the developer sleeps. This would ensure that it's dead easy to bring a new developer fully up to speed on a team.

hosh 13y ago
I know a couple guys who do that. Their Steam app is also managed by Chef. We got our dev setup partially automated like that. Even the partial setup was better than what I heard -- took a week to set up a laptop with the tools and the whole platform. When I was onboarded, it took about day, more or less hands off.

zerr 13y ago

Couldn't this be shown without a blasphemy?

taybin 13y ago
Where is the blasphemy? The terrible restoration of the fresco?
- zerr 13y ago
  Shown in the context of this article.
  - taybin 13y ago
    I re-read the article and still don't know what you're talking about.

druiid 13y ago

I fully agree with this. 'Before', the dev team used a mix of, well, I'd rather not say. It was just not very pretty and eventually everyone got their work done. A while back I built a vagrant+puppet configuration for everyone and the team has since transitioned to using that. It's nice to have at least semi-parity with the production environment. This is possible because we can utilize most (not all, due to no central Puppet server) of the same Puppet modules which we use (or I built), in the VM. Essentially what you gain is not worrying about if a developer will break their machine/VM and go 'Whoops, can you fixor it?'. Additionally, you no longer have to worry about such things as php/node/ruby version issues with the dev/production side of things. We've gone from having sometimes issues with certain code not running the same as in development, to just throwing it up on the staging environment through deployment tools and it just runs!

karlkatzke 13y ago

Issue: What about VPN? If you're connecting to your corporate network with OpenVPN, and you are restricted to one connection (common sense security policy) then you need to get your infrastructure team to issue you a separate key for each VM that you're running that you want to be able to access a tunnel and get on your corporate network. I realize that this isn't an issue for 99.9% of startups, but in the non-startup corporate world where there are security implications to keeping everything in a publicly accessible code repo (healthcare, government, education in some states) then you've got another complication. The only thing I've been able to work up so far is sharing a folder on my desktop through to the VM and running commits from my desktop.

druiid 13y ago
I would say that you probably shouldn't be doing VPN to your development boxes in general. I'd see this as a security risk. I generally don't let anything 'production' touch directly anything which might not have the same security standards as the production gear.
mwcampbell 13y ago
Using NAT-based networking for the VMs should neatly solve the VPN issue.

hosh 13y ago

Vagrant + Chef is great! Vagrant + Chef + Berkshelf is better! You can define the cookbook dependencies in the Berksfile and version it along with Vagrantfile. Berkshelf is like bundler for your cookbooks. It assembles cookbooks from a variety of sources and loads them into your Vagrant-managed vm. Check it out at http://berkshelf.com http://berkshelf.com To help get things going in that end, I've been working on Kamino (https://github.com/hosh/kamino https://github.com/hosh/kamino). It's intended to generate a Vagrantfile+Berkshelf file for use in your project. Right now, it only supports a basic Rails stack. Pull requests welcomed :-)

pixelcort 13y ago

How about the best of both worlds? A common VM image with CLI tools and dev server inside the guest OS, with VirtualBox folder sharing to permit a GUI IDE and the git repo to live on the host OS?

tjbiddle 13y ago
This is what we do now - It gets to be a bit of a pain later. I'm looking forward to switching to Vagrant in the future (On the backburner for now as I have other things on my plate) - Vagrant allows you to easily nuke and re-setup your entire environment, and change things easily for the next go around - while if you're using a VM image you'd need to re-image every time you make changes that you want to stay persistent.

kstrauser 13y ago

I don't want dev to be exactly like stage and prod. Yep, you read that right. I deploy to Debian servers but develop on Linux Mint or OS X. The key is that the environments are very similar but not identical. The payoff comes when you break yourself of the habit of relying on accidents of deployment instead of building a general-case solution that works in multiple environments. You scan /proc to look at running processes? That's great, until you're on a machine without /proc. Better to spend an hour learning how your dev platform (Python, Java, whatever) abstracts that away for you. Trying to send an email by shelling out to /usr/bin/sendmail? Oops! That's broken in lots of places; better learn how your dev platform handles it! The big win comes when you upgrade your stage and prod environments to a newer distro - or a different one altogether - and your stuff keeps working because you'd relentlessly whittled away all the OS-specific dependencies.

mwcampbell 13y ago
You describe a pay-off when upgrading stage and prod. But if dev, stage, and prod are identical, then you simply upgrade dev first, find the things that broke, do the same for stage, and then it's no big deal when you upgrade prod.
blake8086 13y ago
It seems odd to have an environment that you are 100% in control of, and then relish the times when it does unexpected things.
snowwrestler 13y ago
This strikes me as an example of an unnecessary/premature optimization. Unless you know that your product will need to deploy on an unknown variety of server environments, there's no advantage in taking the extra time to 100% abstract away the server environment. And really, how trustworthy is that abstraction? You've basically outsourced responsibility for your server environment to your language community. What if they create a bug that is exposed on Debian, but not on OS X?
- gtaylor 13y ago

doki_pen 13y ago

There is something to be said for running your software on multiple environments. There is also something to be said for stubbing out pieces of your stack for testing and development purposes. There is often a lot to be learned. That said, I've used the VM approach on projects with high turnover. It is usually not the same as production deployment, so it does take effort, but depending on the team it might be worth the effort.

vinitool76 13y ago

How is this different from Boxen (http://boxen.github.com/ http://boxen.github.com/)? You have exactly copied the central idead around it and made a blog post.

hoov 13y ago
The two ideas are very different, but complementary. Boxen is about getting your MacOS environment set up with tooling. The author of the original blog posting is talking about how to automate the creation of a production-like environment in which to do your development. The creation of that environment requires a few tools installed on your physical Mac, which could be managed via Boxen. For example, you might use Boxen to automate the installation/upgrade of Vagrant or your IDE of choice.

rbsn 13y ago

I have used Vagrant in the past, and I had to stop using it. Shared folders were just too slow. 2 seconds to refresh a page was too much when you're trying to be productive.

hosh 13y ago
Huh, that's interesting. How old is your hardware?

EliRivers 13y ago

No no no, I don't like automation; I like automating things :)

graycat 13y ago

The first reason is, people who build tools mostly don't think that the tools will or should be used as components of other tools for more in 'automation'. The result is that it's tough to build on such tools. The second, and more specific, reason is that location of one of my most intense love/hate dilemmas, Xerox PARC that pushed the first 'graphical user interface' (GUI) instead of what came before, usually typing text into a command line. Command lines are mostly easy to automate. GUIs are mostly a total PAIN to automate. I'm wanting to automate, willing to automate, waiting to automate.

sandGorgon 13y ago

has anybody tried Vagabond (https://github.com/chrisroberts/vagabond https://github.com/chrisroberts/vagabond) to do the same thing without a VM ?