4 ms·

I run FreeBSD in my homelab, too! One reason is the stellar ZFS support, but the simple fun of doing stuff differently is definitely a thing, too. And I like Fr

I run FreeBSD in my homelab, too! One reason is the stellar ZFS support, but the simple fun of doing stuff differently is definitely a thing, too. And I like FreeBSD jails. For me, the balance between all the overhead of the "cattle, not pets" approach and the manual way is the a README.md file for basic setup, and then having Ansible stand up the rest of the configuration. The host is configured as a Jail host, then individual services live inside the jails. Creating and configuring the jails is also done through Ansible. Overall, I really like the setup. I can individually SSH into each jail to allow easy debugging, I can snapshot the jails, and data lives on a special ZFS subvolume that I mount into each jail at "/bucket". This way, I can throw away the jail at any time, fire up Ansible, and have everything up and running again in no time.

donw 11mo ago
If I didn't need CUDA support, I'd be on FreeBSD all the way. No systemd, built-in "containers" before they were cool, basically just good 'ole UNIX.
- seg_lol 11mo ago
  You can pass a CUDA device through to a bhyve vm.
  - agapon 11mo ago
    In practice, it's not as easy as it may sound.
throw0101a 11mo ago
> And I like FreeBSD jails. If you don't know about them already, you may be interested in service jails (forthcoming[1] in 15): > A service jail shares the complete filesystem tree directly with the host (the jail root path is /) and as such can access and modify any file on the host, and shares the same user accounts with the host. By default it has no access to the network or other resources which are restricted in jails, but they can be configured to re-use the network of the host and to remove some of the jail-restrictions. * https://docs.freebsd.org/en/books/handbook/jails/#service-jails https://docs.freebsd.org/en/books/handbook/jails/#service-ja... * https://docs.freebsd.org/en/books/handbook/jails/#service-jails-config https://docs.freebsd.org/en/books/handbook/jails/#service-ja... * https://man.freebsd.org/cgi/man.cgi?query=rc.conf&manpath=FreeBSD+15.0-STABLE#SERVICE%09JAILS https://man.freebsd.org/cgi/man.cgi?query=rc.conf&manpath=Fr... [1] https://www.freebsd.org/releases/15.0R/schedule/ https://www.freebsd.org/releases/15.0R/schedule/
- avhception 11mo ago
  Sounds interesting, but it sounds like that would mean installing the service software and it's dependencies into the root filesystem. I'm relatively sure I don't want that, as it would create a big mess on the host. I have stuff like Nextcloud in my jails, and wouldn't want to install PHP and all of it's deps outside the dedicated filesystem of the jail. But it's very cool to see continued development, jails are such an awesome feature!
ahepp 11mo ago
I've done something like this in the past, it works really well. Have you used Poudriere? I never tried it, but it sounds promising. Ansible is a good idea as well. I just wrote some shell scripts that parsed a file with some packages and hooks to set up the jails. These days I have my FreeBSD server providing NFS for a k3s instance on a different box.
- avhception 11mo ago
  Yes, I have used Poudriere. Every time I want to talk about it, I have to look up how exactly that is spelled! My usecase was that I needed stuff to be linked against newer PostgreSQL client libs than were in quarterly. I thought about using Linux and simply providing storage from FreeBSD using NFS, but that would require 2 machines or a VM that connects to the host using NFS or something... meh.
  - ahepp 11mo ago
    Sorry, I don’t know why I googled around and thought “Poudriere” was the one. I was thinking of Bastille, which is for managing jails
    - avhception 11mo ago
      Ah! No, I haven't used Bastille. I didn't want to rely on fancy wrappers that might go away or change their behavior, so I wrote a shell script that orchestrates some shell tools and ZFS into creating a jail, and that in turn is consumed from Ansible. It doesn't have all the bells and whistles, but it's guaranteed to still be there tomorrow - and that was one of my goals, getting away from all the churn that came from the Linux / Docker ecosystem and finally settle on something!

donw 11mo ago

If I didn't need CUDA support, I'd be on FreeBSD all the way. No systemd, built-in "containers" before they were cool, basically just good 'ole UNIX.

seg_lol 11mo ago
You can pass a CUDA device through to a bhyve vm.
- agapon 11mo ago
  In practice, it's not as easy as it may sound.

throw0101a 11mo ago

> And I like FreeBSD jails. If you don't know about them already, you may be interested in service jails (forthcoming[1] in 15): > A service jail shares the complete filesystem tree directly with the host (the jail root path is /) and as such can access and modify any file on the host, and shares the same user accounts with the host. By default it has no access to the network or other resources which are restricted in jails, but they can be configured to re-use the network of the host and to remove some of the jail-restrictions. * https://docs.freebsd.org/en/books/handbook/jails/#service-jails https://docs.freebsd.org/en/books/handbook/jails/#service-ja... * https://docs.freebsd.org/en/books/handbook/jails/#service-jails-config https://docs.freebsd.org/en/books/handbook/jails/#service-ja... * https://man.freebsd.org/cgi/man.cgi?query=rc.conf&manpath=FreeBSD+15.0-STABLE#SERVICE%09JAILS https://man.freebsd.org/cgi/man.cgi?query=rc.conf&manpath=Fr... [1] https://www.freebsd.org/releases/15.0R/schedule/ https://www.freebsd.org/releases/15.0R/schedule/

avhception 11mo ago
Sounds interesting, but it sounds like that would mean installing the service software and it's dependencies into the root filesystem. I'm relatively sure I don't want that, as it would create a big mess on the host. I have stuff like Nextcloud in my jails, and wouldn't want to install PHP and all of it's deps outside the dedicated filesystem of the jail. But it's very cool to see continued development, jails are such an awesome feature!

ahepp 11mo ago

I've done something like this in the past, it works really well. Have you used Poudriere? I never tried it, but it sounds promising. Ansible is a good idea as well. I just wrote some shell scripts that parsed a file with some packages and hooks to set up the jails. These days I have my FreeBSD server providing NFS for a k3s instance on a different box.

avhception 11mo ago
Yes, I have used Poudriere. Every time I want to talk about it, I have to look up how exactly that is spelled! My usecase was that I needed stuff to be linked against newer PostgreSQL client libs than were in quarterly. I thought about using Linux and simply providing storage from FreeBSD using NFS, but that would require 2 machines or a VM that connects to the host using NFS or something... meh.
- ahepp 11mo ago
  Sorry, I don’t know why I googled around and thought “Poudriere” was the one. I was thinking of Bastille, which is for managing jails
  - avhception 11mo ago
    Ah! No, I haven't used Bastille. I didn't want to rely on fancy wrappers that might go away or change their behavior, so I wrote a shell script that orchestrates some shell tools and ZFS into creating a jail, and that in turn is consumed from Ansible. It doesn't have all the bells and whistles, but it's guaranteed to still be there tomorrow - and that was one of my goals, getting away from all the churn that came from the Linux / Docker ecosystem and finally settle on something!