> why in the world are there so many "slight" variations of a build? That's what happens if you tell all your developers to stop using capistrano, fabric, ftp

> why in the world are there so many "slight" variations of a build? That's what happens if you tell all your developers to stop using capistrano, fabric, ftp and heroku, and tell them to create .debs out of EVERYTHING instead. They're going to (unhappily) build and install dozens of packages per hour, because that's how developers increasingly work: by deploying their stuff very, very frequently. In our case it was the combination of 1) rapid development and staging, 2) different environments for different customers, and 3) numerous, fast-moving and precise dependencies. 1) Rapid development and staging: there are several developers, building from their respective branches. They want to deploy and test their work, sometimes in isolation (dev), sometimes as a team before releasing (staging). The overhead of deploying a new machine every time (virtual or not, in this particular case bare metal was part of the equation) is considered overkill. But that's OK, debian package can be named, flavored and versioned, so deployment in parallel should work out, right? Wrong, as it turns out... having to manually "scope" each deployment by generating different package names, flavors and versions basically breaks every assumption of system packaging. And if you want to get a little fancy and, say map particular versions/flavors to particular git commits... Forget it. 2) Different environments for different customers. All of the above was multiplied by the fact that this was appliance software - each customer got its own build, which on a good day was stock, and when dealing with more demanding customers, involved various customizations, build options etc. This is business as usual in many "vertical" software markets where the most valuable customers always have some legacy crap software and workflows they need to integrate with. 3) Numerous, fast-moving and precise dependencies. This software had lots of dependencies, which exact version of those dependencies, and even how they were built (and the exact build environment) mattered immensely, and the whole thing was a moving target because we frequently patched those dependencies. In this particular company the dependencies had mostly to do with video processing (custom ffmpeg builds etc), but this is also business as usual in many software-intensive organizations. The project rarely fits in a neat Rails or Spring app. There's all sorts of libraries, binaries and rpc services floating around - and deb packaging forces you to break all of those down into dependencies, further multiplying the dependency hell that your life already is. It's also increasingly typical (and good!) for developers to be very specific about the exact version and build of all their dependencies (from mysql to libssl) because it affects the behavior of their application. So you can't just wing it by swapping out their particular postgres setup with something "kind of the same thing". "It's the same version, what else do you want?" --> not good enough. > And if they're so slight, why do they break things? I meant that they break traditional system packaging. Simply put, system packaging is not well designed for applications which A) are made of many components, B) are updated and shipped several times per hour, C) on a large set of heterogeneous machines. ... And I didn't even mention the case where sales come back saying "the customer wants RPMs...". Good times. So yeah, I'm sticking to containers and not looking back :)