6 ms·

I’ve been using FreeBSD in production on AWS for a few years now and it’s been great, thanks to ZFS, jails and pf. There are lots of different ways to configur

I’ve been using FreeBSD in production on AWS for a few years now and it’s been great, thanks to ZFS, jails and pf. There are lots of different ways to configure jails, which unfortunately means there are lots of places to get stuck. I typically deploy to x1.* instances, which have NVMe drives. I create a zpool of all the NVMe drives, and then mount a zfs filesystem on the zpool at /mnt/jails. I run all my services in jails. Each jail gets its own IP address on the host, and services are made available to the outside world using pf rdr rules. Services in jails are allowed to access the outside network using pf nat rules. pf makes it easy to monitor and restrict network activity for each service. Once you’ve configured a jail, you can snapshot it with zfs, compress the snapshot, archive it and rsync it to other FreeBSD hosts. Once copied to other hosts, starting the jail simply requires registering it in /etc/jail.conf and running `service jail start $JailName`. I’ve started documenting this and have been thinking hard about how to provide a more generic container interface on top of this implementation. If you’re interested in the shell scripts, I’ve started posting them here: https://github.com/cieplak/jz https://github.com/cieplak/jz

krn 8y ago
A honest question: why would one choose FreeBSD with jails over something like CoreOS with rkt for a stateless cloud application? If I am not mistaken, Netflix uses FreeBSD on a bare metal for CDN, but prefers Linux for applications on AWS.
- lmm 8y ago
  Maturity; an organisation that's been going for 25 years (and maintained backwards compatibility for most of that), a decade or more of documentation/tutorials/... that one can be confident will still work. rkt might offer the same thing but you need a compelling reason to pick something newer and less established.
  - deleted 8y ago
    [deleted]
- tannhaeuser 8y ago
  A good question indeed considering "the cloud" thrives on Linux, and increasingly Docker/Linux containers. What I've seen happening is that there are lots of Docker images installing (parts of) Ubuntu/Debian or Alpine on first startup (coreutils, bash, gawk, package managers and what not). Not saying it's against the letter of GPL, but to me this, if not Docker in general, had always the smell of a GPL-circumvention technique. I'm not a GPL zealot or something; if anything, GPL (except AGPL) hasn't so far prevented concentration in the hands of very few cloud providers, and in becoming Android (largest Linux install base by far) a blatant privacy invasion instrument. Given that BSD doesn't require you to go through such hoops for a POSIX API, it's a mystery to me why the BSDs aren't more used in the enterprise/SaaS space.
  - solatic 8y ago
    > it's a mystery to me why the BSDs aren't more used in the enterprise/SaaS space There needs to be a name for the fallacy that the technically best solution will always win. BSDs aren't more popular in commercial spaces because it's too difficult to find engineers with BSD experience, and no pre-existing supporting ecosystem. There needs to be a compelling reason for companies to ignore the conventional unpopularity, which means that using a BSD must solve a specific pain point that's visible and understandable enough to executives to get approval for going outside the mold. I hate to break it to you, but your boss (probably) doesn't appreciate why a POSIX API is even a thing in the first place, let alone how it applies to a business case.
  - kjeetgill 8y ago
    I've honestly never heard of anyone bring up the GPL or GPL-circumvention in the same breath as Docker. I have no idea what hoops you're referring to. In my experience, the biggest reason most people don't use jails is because people don't use or get exposure to FreeBSD. Nine times out of ten, the servers at your first job or university were centos or ubuntu, and your desktop tower was rhel, fedora, or again, ubuntu. If you're an enthusiast maybe you had an arch, debian, or gentoo install. (shrugs slackware predates me). Without some degree of exposure or evangelism you'll choose what you've seen work over something you suspect might not. And that rationale applies to a lesser agree to docker vs rkt. Nothing profound. It just has popular mindshare so it's the default. You'll have to beat the pants off something to overcome that.
JdeBP 8y ago
I took the approach of splitting jails into two services. * http://jdebp.info./Softwares/nosh/guide/jailing-services.html http://jdebp.info./Softwares/nosh/guide/jailing-services.htm... * http://jdebp.info./Softwares/nosh/guide/converting-jails.html http://jdebp.info./Softwares/nosh/guide/converting-jails.htm...
olavgg 8y ago
> There are lots of different ways to configure jails, which unfortunately means there are lots of places to get stuck. This is something that annoys me too, do you have any recommandations? Should I learn to setup jails from scratch? Or settle on a single tool like ezjails?
- dorian-graph 8y ago
  As someone who has just gone through this, I would say from scratch. ezjails and some of the others really do seem like the definition of leaky abstractions if you expect a nice layer on top. They're more like handy reusable shell functions. If you learn from scratch (which again, isn't far removed from those helper packages), you'll have a better understanding for when you get stuck.
- Grauwolf 8y ago
  The base system utilities have gotten better with the years. I recommend using them. https://www.skyforge.at/posts/an-introduction-to-jails-and-jail-networking/ https://www.skyforge.at/posts/an-introduction-to-jails-and-j...
soupbowl 8y ago
I use a similar setup to yours and it has been a joy to work with. Your jz scripts are a little different to mine, I think I might borrow some ideas from them!
314 8y ago
We have somewhat different use-cases, but there might be something useful for you in here: https://git.cse.bth.se/system/bertha/tree/master/usr/sj https://git.cse.bth.se/system/bertha/tree/master/usr/sj My target use-case is deploying backend servers into minimal jails. The servers are single binaries written in golang and the goal is to minimize the attack-surface as far as possible in case I’ve left something exploitable in the server code.
Cieplak 8y ago
Edit: I wrote x1.* instances. I meant i3.* instances.

krn 8y ago

A honest question: why would one choose FreeBSD with jails over something like CoreOS with rkt for a stateless cloud application? If I am not mistaken, Netflix uses FreeBSD on a bare metal for CDN, but prefers Linux for applications on AWS.

lmm 8y ago
Maturity; an organisation that's been going for 25 years (and maintained backwards compatibility for most of that), a decade or more of documentation/tutorials/... that one can be confident will still work. rkt might offer the same thing but you need a compelling reason to pick something newer and less established.
- deleted 8y ago
  [deleted]
tannhaeuser 8y ago
A good question indeed considering "the cloud" thrives on Linux, and increasingly Docker/Linux containers. What I've seen happening is that there are lots of Docker images installing (parts of) Ubuntu/Debian or Alpine on first startup (coreutils, bash, gawk, package managers and what not). Not saying it's against the letter of GPL, but to me this, if not Docker in general, had always the smell of a GPL-circumvention technique. I'm not a GPL zealot or something; if anything, GPL (except AGPL) hasn't so far prevented concentration in the hands of very few cloud providers, and in becoming Android (largest Linux install base by far) a blatant privacy invasion instrument. Given that BSD doesn't require you to go through such hoops for a POSIX API, it's a mystery to me why the BSDs aren't more used in the enterprise/SaaS space.
- solatic 8y ago
  > it's a mystery to me why the BSDs aren't more used in the enterprise/SaaS space There needs to be a name for the fallacy that the technically best solution will always win. BSDs aren't more popular in commercial spaces because it's too difficult to find engineers with BSD experience, and no pre-existing supporting ecosystem. There needs to be a compelling reason for companies to ignore the conventional unpopularity, which means that using a BSD must solve a specific pain point that's visible and understandable enough to executives to get approval for going outside the mold. I hate to break it to you, but your boss (probably) doesn't appreciate why a POSIX API is even a thing in the first place, let alone how it applies to a business case.
- kjeetgill 8y ago
  I've honestly never heard of anyone bring up the GPL or GPL-circumvention in the same breath as Docker. I have no idea what hoops you're referring to. In my experience, the biggest reason most people don't use jails is because people don't use or get exposure to FreeBSD. Nine times out of ten, the servers at your first job or university were centos or ubuntu, and your desktop tower was rhel, fedora, or again, ubuntu. If you're an enthusiast maybe you had an arch, debian, or gentoo install. (shrugs slackware predates me). Without some degree of exposure or evangelism you'll choose what you've seen work over something you suspect might not. And that rationale applies to a lesser agree to docker vs rkt. Nothing profound. It just has popular mindshare so it's the default. You'll have to beat the pants off something to overcome that.

JdeBP 8y ago

I took the approach of splitting jails into two services. * http://jdebp.info./Softwares/nosh/guide/jailing-services.html http://jdebp.info./Softwares/nosh/guide/jailing-services.htm... * http://jdebp.info./Softwares/nosh/guide/converting-jails.html http://jdebp.info./Softwares/nosh/guide/converting-jails.htm...

olavgg 8y ago

> There are lots of different ways to configure jails, which unfortunately means there are lots of places to get stuck. This is something that annoys me too, do you have any recommandations? Should I learn to setup jails from scratch? Or settle on a single tool like ezjails?

dorian-graph 8y ago
As someone who has just gone through this, I would say from scratch. ezjails and some of the others really do seem like the definition of leaky abstractions if you expect a nice layer on top. They're more like handy reusable shell functions. If you learn from scratch (which again, isn't far removed from those helper packages), you'll have a better understanding for when you get stuck.
Grauwolf 8y ago
The base system utilities have gotten better with the years. I recommend using them. https://www.skyforge.at/posts/an-introduction-to-jails-and-jail-networking/ https://www.skyforge.at/posts/an-introduction-to-jails-and-j...

soupbowl 8y ago

I use a similar setup to yours and it has been a joy to work with. Your jz scripts are a little different to mine, I think I might borrow some ideas from them!

314 8y ago

We have somewhat different use-cases, but there might be something useful for you in here: https://git.cse.bth.se/system/bertha/tree/master/usr/sj https://git.cse.bth.se/system/bertha/tree/master/usr/sj My target use-case is deploying backend servers into minimal jails. The servers are single binaries written in golang and the goal is to minimize the attack-surface as far as possible in case I’ve left something exploitable in the server code.

Cieplak 8y ago

Edit: I wrote x1.* instances. I meant i3.* instances.