We've been doing this for ages where I work. New EC2 AMIs are built inside a disk image. We can pretty easily use all of the standard tools through chroot, so i

We've been doing this for ages where I work. New EC2 AMIs are built inside a disk image. We can pretty easily use all of the standard tools through chroot, so installing packages is relatively simple and flexible. Instances are ready for use immediately after booting, and deploying new code is as simple as rebuilding the image and booting it. We can also easily scale up most forms of servers by simply booting more and adding them to the proxy as they come up. The only downside here is that firefighting still kind of remains a two step process. Even after investing a decent bit of time optimizing the process, it still takes 5-10 minutes to build and upload an image. From there it takes another 5-30 minutes to get EC2 to boot it, so there's a strong temptation to tweak things on the server. I've taken to patching, testing, and committing the fix on the tag for that release so I can simply apply a diff to get the server up to date. Not perfect, but it goes a long way to prevent regressions.