10 ms·

Iocage – A FreeBSD jail manager

100 points by HugoDaniel 10y ago

baobrien 10y ago
I'd love to see a combined FreeBSD jail and bhyve manager. Can libvirt do that?
- ketralnis 10y ago
  Have you looked at https://github.com/pr1ntf/iohyve https://github.com/pr1ntf/iohyve?
- soupbowl 10y ago
  CBSd https://www.bsdstore.ru/en/about.html https://www.bsdstore.ru/en/about.html It's a little overwhelming though. FreeBSD really needs more/better jail and bhyve tools.
smkelly 10y ago
iocage is great, but according to the README in GitHub it is not being developed anymore. The author is working on a rewrite in Go. The rewrite isn't done yet last I checked.
- NuSkooler 10y ago
  This. I still use iocage over alternatives though. If anyone knows a updated status on the Go rewrite, please chime in!
- synchronise 10y ago
  How does it compare to ezjail?
  - patricius 10y ago
    I've found iocage's user interface to be more intuitive. I started using iocage because ezjail broke something when trying to upgrade a jail, but that was probably me doing something wrong.
  - crest 10y ago
    Ezjail is fairly old. It works with the older less capable API to jails and is restricted to features supported by the old API. It was also designed for use with UFS on tiny disks. Lets say you are a webhoster around the year 2000 and your servers have a handfull 36GB or 72GB SCSI disks. You want to protect each customer from all other customers and protect yourself from all customer scripts. This was before IA32 CPUs offered the features to support efficient transparent virtualisation and even if they did the resource demand per VM would have been too high. As long as your customers are happy with static file hosting everything is fine, but as soon as way want to execute some useful server side scripts you have a problem. FreeBSD offers a way to run one HTTP server per customer inside jail, but keeping a full FreeBSD userland (base + http server + databases + scripting language + customer code) per customer would quickly fill your puny little disks. Ezjails offers a neat solution to the problem: store a template just once and instantiate it with a nullfs read-only mount. Now your storage requirements are manageable at a reasonable price with hardware of the day and your buffer cache hitrates are better too. All of these indirection and aliasing hacks make ezjail more complicated than modern jail managers, because ezjail had to work around the operating system limitations instead of taking advantage yet to be invented operation system features.
    - citrin_ru 10y ago
      Ezjail now supports ZFS. Sharing one basejail via nullfs is useful feature, can Iocage do this? Nullfs not only allows to save space on disk but also allows faster updates (extract new basejail then switch all jails to it, without full upgrade of each jail). Also in software old doesn't mean bad, and newer is not automatically better.
      - crest 10y ago
        I didn't mean to imply that. I just explained why ezjail looks so convoluted to a new user. Ezjail still solves the same problems today as it did 10 years ago. On the other hand since not even bloatware did keep up with storage cost decreases users can afford more comfortable trade-offs today.
- tachion 10y ago
  Due to this very fact I've forked iocage into iocell (https://github.com/bartekrutkowski/iocell https://github.com/bartekrutkowski/iocell) where I am fixing numerous critical bugs currently present in iocage, that most likely won't be fixed, due to iocage support being on hold (if not completely abandoned, until rewrite is available). Feel free to check it out, especially the `develop` branch! Once I have most of the annoying bugs fixed, I'll merge `develop` into `master` and create a FreeBSD port for iocell.
  - voltagex_ 10y ago
    Is there a way I can try iocell while I've got existing iocage jails?
    - tachion 10y ago
      'It depends' ;) First of all, if you're using 'stable iocage' then you're out of luck, since too much changed in devel iocage, and iocell is based on devel branch. Second, it would either involve renaming your datasets in few places (from iocage to iocell) or changing few strings in iocell (from iocell to iocage). Other than that, iocell aims to work exactly the same way as iocage, with same principles, workflow and so on. Check out the commits history in iocell `develop` branch to find the one where iocage name is being changed, it should give you an idea on what and where to do. I am also planning some migration guide for existing iocage installations, but it might not be very easy/very stable, I am afraid.
      - voltagex_ 10y ago
        Okay, let me ask it a different way then. If I had to move away from iocage right away, how would I preserve my jail? I don't really care what the jail is called - I just want Plex or whatever to start up when I start the jail, and to be able to keep upgrading packages inside it.
        tachion 10y ago
        That's entirely different question, and doesn't really have much in common with iocage/iocell. I, for example, have every single thing I host automated in a way where it doesn't matter wether it runs in jail provided by iocage, jail manually built, vm in AWS or a physical machine. This way, my migration path would be: wipe existing jails, create new ones with new tool, launch automation, done. You, however, might need entirely different approach, based on how your environment/setup looks like right now. One very primitive way would be to simply archive (rsync, tar, whatever else) jail contents and deploy it in jails created with new tool.
- rhabarba 10y ago
  I'm not sure if Go is the best language for this task, but he might have his reasons.
  - 4ad 10y ago
    Why do you think Go is not the best language for this task?
    - rhabarba 10y ago
      As iocage pretty much only wraps FreeBSD system tasks, I can't see why a language different than "shell scripting" would be an improvement here - does it use API calls instead?
      - 4ad 10y ago
        I have no idea how iocage is being rewritten, but if I were to rewrite iocage in Go I would call the relevant system calls directly. I would not wrap exiting programs or shell scripts. The iocage shell script was totally unmaintainable. I know because I forked it and used it for my own purposes, until I stopped and wrote my own thing (coincidentally, also in Go). Implementing state machines correctly in shell script is painful.
jzelinskie 10y ago
I'm deeply involved in the Linux container ecosystem (docker, rkt) and I'd like to understand the difference in workflow between that and this. Is anyone familiar enough to speak to both?
- crest 10y ago
  Jails are a general mechanism to run multiple userlands on one kernel. They started with a change root directory, an IPv4 address and no access to anything dangerous (e.g. /dev/mem). Jails grew a lot features with the biggest bump in FreeBSD 8. Starting with FreeBSD 8 jails can be nested, have multiple IP addresses etc. but they still lacked System V IPC. This changed with FreeBSD 11. Now the only thing jails can't have is their own IP stack. Jails share the hosts IP stack which improves efficiency and simplifies most deployments, but it prevents them having administrative access to the IP stack. There is an experimental kernel feature (VIMAGE) to jails to run their own instance of the IP stack but there are still some nasty bugs hiding in this code, because nobody thought about how to tear down the IP stack. After all it was initialised once during the boot process and kept running until the power went out. The largest difference is in the mindset behind jails. Jails are designed as secure operating system level virtualisation. Docker on the other hand is fairly fragile and offers neither secure isolation between containers nor between containers and the host. Jails can contain a complete userland and this a very common setup. A full FreeBSD userland + some ports/packages to make it useful is about one 1GB. This used to be a lot 15+ years ago when jails where created and the older jail management tools like ezjail reduce the per jail storage requirements with nullfs and unionfs hacks. These days 1GB isn't that much for a simple container and most FreeBSD servers run on ZFS. ZFS offers a much simpler and cleaner way to reduce storage requirements: just clone a snapshot (the template) create a new jail and copy a few config files into the clone. The only problem is that you can't rebase your clone. Docker is designed around the idea of single purpose containers without stable storage. You can use jails to implement this idea, but FreeBSD jails support more than that. Also all the FreeBSD jail managers I used try to stay out of your network configuration as far as possible and at most configure alias IP addresses on existing interfaces. Docker is very opinionated software fighting against limitations imposed on it by the Linux kernel. Jails are FreeBSD kernel feature touching multiple parts of the kernel with a minimal userland interface in the FreeBSD base system . Multiple higher level jail managers are available in the FreeBSD ports tree. There is no reason why you couldn't implement a docker like jail manager and the jetpack projects started doing exactly this. Keep in mind that docket images are the worlds new statically linked binaries for people who can't figure out how to define and reproduce the relevant parts of their development environment in their production environment. Executing existing docker images with their Linux binaries would probably require a massive update to the Linux compatibility layer (a reimplementation the Linux syscall ABI).
voltagex_ 10y ago
iocage is easier to use than ezjail or warden, but I still managed to end up with a broken system where the system thought it was FreeBSD 11 but still expected to use packages from FreeBSD 10 (I get an ABI error when doing pkg upgrade). The distinction between jails and basejails is tricky to follow. I don't know where the rewrite went.
- jlgaddis 10y ago
  What do your pkg.conf entries look like? Do they point at 10 specifically?
  - voltagex_ 10y ago
    FreeBSD: { url: "pkg+http://pkg0.nyi.FreeBSD.org/${ABI}/quarterly", mirror_type: "srv", signature_type: "fingerprints", fingerprints: "/usr/share/keys/pkg", enabled: yes }
cyphar 10y ago
Is there anyone from the FreeBSD community interested in standardising FreeBSD containers inside the OCI specification? Currently only Linux, Solaris and Windows have been included in the standard -- which is a bit disappointing (I've always been fond of FreeBSD). If anyone is interested, please contribute to the Open Container Initiative. https://github.com/opencontainers https://github.com/opencontainers
- tachion 10y ago
  Not exactly OCI, but as an example of what can be done with FreeBSD, check out https://github.com/3ofcoins/jetpack https://github.com/3ofcoins/jetpack that's a APC implementation using Jails/ZFS and other FreeBSD goodies.
t_tsonev 10y ago
Works well for me. The only thing I dislike is the GUIDs for the containers that can't be changed to something shorter. At the time I had to decide ezjail didn't work with FreeBSD 10, not sure if it has been updated.
- kchoudhu 10y ago
  ...Am using ezjail on a mix of 10.3 and 11 now. Works flawlessly.
_paulc 10y ago
There are lots of jail wrappers but in most cases it is just as easily to just take the time to understand how jail.conf works and use this (see FreeBSD jails the hard way [1]). It's fairly easy to just use zfs clones to create jails and customise the exec.prestart/exec.start functions to do automatic provisioning and configuration (eg. I store the port forwards in a variable for each jail and process this and automatically setup pf rules when the jail starts). [1] https://clinta.github.io/freebsd-jails-the-hard-way/ https://clinta.github.io/freebsd-jails-the-hard-way/
deleted 10y ago
[deleted]

100 points by HugoDaniel 10y ago

baobrien 10y ago

I'd love to see a combined FreeBSD jail and bhyve manager. Can libvirt do that?

ketralnis 10y ago
Have you looked at https://github.com/pr1ntf/iohyve https://github.com/pr1ntf/iohyve?
soupbowl 10y ago
CBSd https://www.bsdstore.ru/en/about.html https://www.bsdstore.ru/en/about.html It's a little overwhelming though. FreeBSD really needs more/better jail and bhyve tools.

smkelly 10y ago

iocage is great, but according to the README in GitHub it is not being developed anymore. The author is working on a rewrite in Go. The rewrite isn't done yet last I checked.

NuSkooler 10y ago
This. I still use iocage over alternatives though. If anyone knows a updated status on the Go rewrite, please chime in!
synchronise 10y ago
How does it compare to ezjail?
- patricius 10y ago
  I've found iocage's user interface to be more intuitive. I started using iocage because ezjail broke something when trying to upgrade a jail, but that was probably me doing something wrong.
- crest 10y ago
  Ezjail is fairly old. It works with the older less capable API to jails and is restricted to features supported by the old API. It was also designed for use with UFS on tiny disks. Lets say you are a webhoster around the year 2000 and your servers have a handfull 36GB or 72GB SCSI disks. You want to protect each customer from all other customers and protect yourself from all customer scripts. This was before IA32 CPUs offered the features to support efficient transparent virtualisation and even if they did the resource demand per VM would have been too high. As long as your customers are happy with static file hosting everything is fine, but as soon as way want to execute some useful server side scripts you have a problem. FreeBSD offers a way to run one HTTP server per customer inside jail, but keeping a full FreeBSD userland (base + http server + databases + scripting language + customer code) per customer would quickly fill your puny little disks. Ezjails offers a neat solution to the problem: store a template just once and instantiate it with a nullfs read-only mount. Now your storage requirements are manageable at a reasonable price with hardware of the day and your buffer cache hitrates are better too. All of these indirection and aliasing hacks make ezjail more complicated than modern jail managers, because ezjail had to work around the operating system limitations instead of taking advantage yet to be invented operation system features.
  - citrin_ru 10y ago
    Ezjail now supports ZFS. Sharing one basejail via nullfs is useful feature, can Iocage do this? Nullfs not only allows to save space on disk but also allows faster updates (extract new basejail then switch all jails to it, without full upgrade of each jail). Also in software old doesn't mean bad, and newer is not automatically better.
    - crest 10y ago
      I didn't mean to imply that. I just explained why ezjail looks so convoluted to a new user. Ezjail still solves the same problems today as it did 10 years ago. On the other hand since not even bloatware did keep up with storage cost decreases users can afford more comfortable trade-offs today.
tachion 10y ago
Due to this very fact I've forked iocage into iocell (https://github.com/bartekrutkowski/iocell https://github.com/bartekrutkowski/iocell) where I am fixing numerous critical bugs currently present in iocage, that most likely won't be fixed, due to iocage support being on hold (if not completely abandoned, until rewrite is available). Feel free to check it out, especially the `develop` branch! Once I have most of the annoying bugs fixed, I'll merge `develop` into `master` and create a FreeBSD port for iocell.
- voltagex_ 10y ago
  Is there a way I can try iocell while I've got existing iocage jails?
  - tachion 10y ago
    'It depends' ;) First of all, if you're using 'stable iocage' then you're out of luck, since too much changed in devel iocage, and iocell is based on devel branch. Second, it would either involve renaming your datasets in few places (from iocage to iocell) or changing few strings in iocell (from iocell to iocage). Other than that, iocell aims to work exactly the same way as iocage, with same principles, workflow and so on. Check out the commits history in iocell `develop` branch to find the one where iocage name is being changed, it should give you an idea on what and where to do. I am also planning some migration guide for existing iocage installations, but it might not be very easy/very stable, I am afraid.
    - voltagex_ 10y ago
      Okay, let me ask it a different way then. If I had to move away from iocage right away, how would I preserve my jail? I don't really care what the jail is called - I just want Plex or whatever to start up when I start the jail, and to be able to keep upgrading packages inside it.
      - tachion 10y ago
        That's entirely different question, and doesn't really have much in common with iocage/iocell. I, for example, have every single thing I host automated in a way where it doesn't matter wether it runs in jail provided by iocage, jail manually built, vm in AWS or a physical machine. This way, my migration path would be: wipe existing jails, create new ones with new tool, launch automation, done. You, however, might need entirely different approach, based on how your environment/setup looks like right now. One very primitive way would be to simply archive (rsync, tar, whatever else) jail contents and deploy it in jails created with new tool.
rhabarba 10y ago
I'm not sure if Go is the best language for this task, but he might have his reasons.
- 4ad 10y ago
  Why do you think Go is not the best language for this task?
  - rhabarba 10y ago
    As iocage pretty much only wraps FreeBSD system tasks, I can't see why a language different than "shell scripting" would be an improvement here - does it use API calls instead?
    - 4ad 10y ago
      I have no idea how iocage is being rewritten, but if I were to rewrite iocage in Go I would call the relevant system calls directly. I would not wrap exiting programs or shell scripts. The iocage shell script was totally unmaintainable. I know because I forked it and used it for my own purposes, until I stopped and wrote my own thing (coincidentally, also in Go). Implementing state machines correctly in shell script is painful.

jzelinskie 10y ago

I'm deeply involved in the Linux container ecosystem (docker, rkt) and I'd like to understand the difference in workflow between that and this. Is anyone familiar enough to speak to both?

crest 10y ago
Jails are a general mechanism to run multiple userlands on one kernel. They started with a change root directory, an IPv4 address and no access to anything dangerous (e.g. /dev/mem). Jails grew a lot features with the biggest bump in FreeBSD 8. Starting with FreeBSD 8 jails can be nested, have multiple IP addresses etc. but they still lacked System V IPC. This changed with FreeBSD 11. Now the only thing jails can't have is their own IP stack. Jails share the hosts IP stack which improves efficiency and simplifies most deployments, but it prevents them having administrative access to the IP stack. There is an experimental kernel feature (VIMAGE) to jails to run their own instance of the IP stack but there are still some nasty bugs hiding in this code, because nobody thought about how to tear down the IP stack. After all it was initialised once during the boot process and kept running until the power went out. The largest difference is in the mindset behind jails. Jails are designed as secure operating system level virtualisation. Docker on the other hand is fairly fragile and offers neither secure isolation between containers nor between containers and the host. Jails can contain a complete userland and this a very common setup. A full FreeBSD userland + some ports/packages to make it useful is about one 1GB. This used to be a lot 15+ years ago when jails where created and the older jail management tools like ezjail reduce the per jail storage requirements with nullfs and unionfs hacks. These days 1GB isn't that much for a simple container and most FreeBSD servers run on ZFS. ZFS offers a much simpler and cleaner way to reduce storage requirements: just clone a snapshot (the template) create a new jail and copy a few config files into the clone. The only problem is that you can't rebase your clone. Docker is designed around the idea of single purpose containers without stable storage. You can use jails to implement this idea, but FreeBSD jails support more than that. Also all the FreeBSD jail managers I used try to stay out of your network configuration as far as possible and at most configure alias IP addresses on existing interfaces. Docker is very opinionated software fighting against limitations imposed on it by the Linux kernel. Jails are FreeBSD kernel feature touching multiple parts of the kernel with a minimal userland interface in the FreeBSD base system . Multiple higher level jail managers are available in the FreeBSD ports tree. There is no reason why you couldn't implement a docker like jail manager and the jetpack projects started doing exactly this. Keep in mind that docket images are the worlds new statically linked binaries for people who can't figure out how to define and reproduce the relevant parts of their development environment in their production environment. Executing existing docker images with their Linux binaries would probably require a massive update to the Linux compatibility layer (a reimplementation the Linux syscall ABI).

voltagex_ 10y ago

iocage is easier to use than ezjail or warden, but I still managed to end up with a broken system where the system thought it was FreeBSD 11 but still expected to use packages from FreeBSD 10 (I get an ABI error when doing pkg upgrade). The distinction between jails and basejails is tricky to follow. I don't know where the rewrite went.

jlgaddis 10y ago
What do your pkg.conf entries look like? Do they point at 10 specifically?
- voltagex_ 10y ago
  FreeBSD: { url: "pkg+http://pkg0.nyi.FreeBSD.org/${ABI}/quarterly", mirror_type: "srv", signature_type: "fingerprints", fingerprints: "/usr/share/keys/pkg", enabled: yes }

cyphar 10y ago

Is there anyone from the FreeBSD community interested in standardising FreeBSD containers inside the OCI specification? Currently only Linux, Solaris and Windows have been included in the standard -- which is a bit disappointing (I've always been fond of FreeBSD). If anyone is interested, please contribute to the Open Container Initiative. https://github.com/opencontainers https://github.com/opencontainers

tachion 10y ago
Not exactly OCI, but as an example of what can be done with FreeBSD, check out https://github.com/3ofcoins/jetpack https://github.com/3ofcoins/jetpack that's a APC implementation using Jails/ZFS and other FreeBSD goodies.

t_tsonev 10y ago

Works well for me. The only thing I dislike is the GUIDs for the containers that can't be changed to something shorter. At the time I had to decide ezjail didn't work with FreeBSD 10, not sure if it has been updated.

kchoudhu 10y ago
...Am using ezjail on a mix of 10.3 and 11 now. Works flawlessly.

_paulc 10y ago

There are lots of jail wrappers but in most cases it is just as easily to just take the time to understand how jail.conf works and use this (see FreeBSD jails the hard way [1]). It's fairly easy to just use zfs clones to create jails and customise the exec.prestart/exec.start functions to do automatic provisioning and configuration (eg. I store the port forwards in a variable for each jail and process this and automatically setup pf rules when the jail starts). [1] https://clinta.github.io/freebsd-jails-the-hard-way/ https://clinta.github.io/freebsd-jails-the-hard-way/

deleted 10y ago

[deleted]