Previous job, we were running almost everything in "normal" jails (as opposed to fat). We even stayed with ezjail, which is dead simple, because we were so comf

Previous job, we were running almost everything in "normal" jails (as opposed to fat). We even stayed with ezjail, which is dead simple, because we were so comfortable with it and could understand how to manually fiddle with stuff when there was a need. There are very very few things that cannot be told to bind to the real jail IP address instead of localhost. Most stuff will just work. And that's it. But there are also added benefits here: * Parts of the filesystem cannot be altered, period. You update your base from your jailhost and every jail takes the change. That was an added security benefit. * Pkg software can be individually installed per jail, goes into /usr/local. FreeBSD has already been brilliant in separating the base system from the third party software. * For minor and major version bumps, we developed a simple set of steps and tools to upgrade a jailhost but leave the jails in their previous versions, maintaining multiple versions of base, like /var/jails/basejail104, /var/jails/basejail113, and so on. So the downtime is minimized, and each jail can be upgraded later in its own pace. This was a huge relief compared to having to upgrade everything only to find that some piece of software doesn't quite work correctly in the next version. And rollback is possible, by just copying the jail. * Jails can likewise be very easily copied between jailhosts with a couple of rsync commands and some simple script. Have used iocage and zfs and it is brilliant, but even without the benefits of zfs clone and snapshot, jails were (and I believe still are) a very pleasant experience to work with.