4 ms·

I run FreeBSD and ZFS on my RPi-B as a cheap and cheerful destination for remote zfs send replication. I wrote up a guide: https://github.com/hughobrien/zfs-re

I run FreeBSD and ZFS on my RPi-B as a cheap and cheerful destination for remote zfs send replication. I wrote up a guide: https://github.com/hughobrien/zfs-remote-mirror https://github.com/hughobrien/zfs-remote-mirror

dpedu 11y ago
Nice work!
forinti 11y ago
Great job. How does it perform? I thought ZFS needed tons of RAM.
- jacquesm 11y ago
  1G is not a lot of RAM?
  - forinti 11y ago
    Apparently, ZFS needs 1GB of RAM per 1TB of storage.
    - masklinn 11y ago
      You're probably thinking of deduplication (whose memory requirements is ~320 bytes per deduplicated block, and which generally requires way more than 1GB per TB, it's highly variable depending on storage load but I've seen quotes from 5 to 20GB/TB). Most users don't benefit from deduplication and it's usually recommended not to enable it unless you do demonstrably get large benefits from it (on FreeBSD, `zdb -S` will simulate and estimate deduplication benefits). 1GB is the recommended conf for ZFS on FreeBSD, it will benefit from more (and may need some tuning below 2GB) but even that's not a requirement, the FreeBSD wiki's ZFS tuning page quotes "ZFS running nicely on a laptop with 768 Megs of physical RAM" with tuning (and provides configuration values).
      - hengheng 11y ago
        I've always been wondering if file system RAM caches couldn't be moved to a fast SSD nowadays. Do you think this would be possible with, say, an Intel 750?
        woodman 11y ago
        I'm pretty sure that is already a thing. Are you talking about something like L2ARC or swap disks?
        masklinn 11y ago
        ZFS (and probably others) let you do that, kinda: it always has a primary in-memory cache but you can add level 2 cache devices (L2ARC) to the pool. In fact you can extend two different caches to storage devices: L2ARC is a read cache, and you can add a separate set of Log Devices used by the ZFS Intent Log, where writes are cached.
  - anotherhue 11y ago
    This model only has 512MB, minus 16MB for the video system.
- awqrre 11y ago
  The Raspberry-Pi can't perform that good because it lacks USB-3 or something else with good performance (it only has USB-2)... It is not a very good choice for a file server. The ODROID-XU4 seems like a much better choice even if it is double the price.
- bespoke_engnr 11y ago
  Aside from the "needs many gigs of RAM, even without deduplication turned on" angle, the lack of ECC memory would prevent me from using ZFS on raspi. Sooner or later you will trash your data unless you're using ECC memory (which the Raspberry Pi RAM chips are not): http://louwrentius.com/please-use-zfs-with-ecc-memory.html http://louwrentius.com/please-use-zfs-with-ecc-memory.html For the record, I also think it's cool to get this compiled and running on RasPi; just wanted to add the ECC info for anyone now thinking of doing something with this in production :-D.
  - jlgaddis 11y ago
    ZFS without ECC is no worse than any other file system which doesn't do checksumming.
    - lorenzfx 11y ago
      See a quote from Matt(hew) Ahrens (zfs developer)[0]: There's nothing special about ZFS that requires/encourages the use of ECC RAM more so than any other filesystem. [...] I would simply say: if you love your data, use ECC RAM. Additionally, use a filesystem that checksums your data, such as ZFS. [0] http://arstechnica.com/civis/viewtopic.php?f=2&t=1235679&p=26303271#p26303271 http://arstechnica.com/civis/viewtopic.php?f=2&t=1235679&p=2...
klapinat0r 11y ago
While I haven't tried crochet, I have tried building my own RPi-B+ image. You mention in your guide that you'll post updates "when you have something", and judging by your post it seems you have. Since I had difficulties compiling the zfs (and one other, which I don't recall) module, I'm wondering if you'd care to share your experience in your guide? Kudos to your very well thoughtout and thorough guide.
- zyngaro 11y ago
  Give crochet a try. I've used it recently to build a FreeBSD 11 image for RPi B+ recently and it just works.
  - klapinat0r 11y ago
    Thanks. Yes it did build just fine. Now I just have to figure out what's "stalling" during boot. It compiled with zfs and all fine though :)
- anotherhue 11y ago
  I got it going a few days ago, have been meaning to update the guide and post some images so people can skip the compilation. Star the repo for updates. You're probably thinking of the opensolaris module. There's a small patch to apply and some KERNCONF to trigger the module build but otherwise its straightforward -- once you know where to look!
rleigh 11y ago
Interesting, I thought ZFS required a 64-bit system. Is this no longer the case? (I am using UFS on powerpc32 since it's not supported for 10.2.)
- Freaky 11y ago
  64-bit is very strongly recommended, but it does work on 32-bit with tuning: https://wiki.freebsd.org/ZFSTuningGuide https://wiki.freebsd.org/ZFSTuningGuide

dpedu 11y ago

Nice work!

forinti 11y ago

Great job. How does it perform? I thought ZFS needed tons of RAM.

jacquesm 11y ago
1G is not a lot of RAM?
- forinti 11y ago
  Apparently, ZFS needs 1GB of RAM per 1TB of storage.
  - masklinn 11y ago
    You're probably thinking of deduplication (whose memory requirements is ~320 bytes per deduplicated block, and which generally requires way more than 1GB per TB, it's highly variable depending on storage load but I've seen quotes from 5 to 20GB/TB). Most users don't benefit from deduplication and it's usually recommended not to enable it unless you do demonstrably get large benefits from it (on FreeBSD, `zdb -S` will simulate and estimate deduplication benefits). 1GB is the recommended conf for ZFS on FreeBSD, it will benefit from more (and may need some tuning below 2GB) but even that's not a requirement, the FreeBSD wiki's ZFS tuning page quotes "ZFS running nicely on a laptop with 768 Megs of physical RAM" with tuning (and provides configuration values).
    - hengheng 11y ago
      I've always been wondering if file system RAM caches couldn't be moved to a fast SSD nowadays. Do you think this would be possible with, say, an Intel 750?
      - woodman 11y ago
        I'm pretty sure that is already a thing. Are you talking about something like L2ARC or swap disks?
      - masklinn 11y ago
        ZFS (and probably others) let you do that, kinda: it always has a primary in-memory cache but you can add level 2 cache devices (L2ARC) to the pool. In fact you can extend two different caches to storage devices: L2ARC is a read cache, and you can add a separate set of Log Devices used by the ZFS Intent Log, where writes are cached.
- anotherhue 11y ago
  This model only has 512MB, minus 16MB for the video system.
awqrre 11y ago
The Raspberry-Pi can't perform that good because it lacks USB-3 or something else with good performance (it only has USB-2)... It is not a very good choice for a file server. The ODROID-XU4 seems like a much better choice even if it is double the price.
bespoke_engnr 11y ago
Aside from the "needs many gigs of RAM, even without deduplication turned on" angle, the lack of ECC memory would prevent me from using ZFS on raspi. Sooner or later you will trash your data unless you're using ECC memory (which the Raspberry Pi RAM chips are not): http://louwrentius.com/please-use-zfs-with-ecc-memory.html http://louwrentius.com/please-use-zfs-with-ecc-memory.html For the record, I also think it's cool to get this compiled and running on RasPi; just wanted to add the ECC info for anyone now thinking of doing something with this in production :-D.
- jlgaddis 11y ago
  ZFS without ECC is no worse than any other file system which doesn't do checksumming.
  - lorenzfx 11y ago
    See a quote from Matt(hew) Ahrens (zfs developer)[0]: There's nothing special about ZFS that requires/encourages the use of ECC RAM more so than any other filesystem. [...] I would simply say: if you love your data, use ECC RAM. Additionally, use a filesystem that checksums your data, such as ZFS. [0] http://arstechnica.com/civis/viewtopic.php?f=2&t=1235679&p=26303271#p26303271 http://arstechnica.com/civis/viewtopic.php?f=2&t=1235679&p=2...

klapinat0r 11y ago

While I haven't tried crochet, I have tried building my own RPi-B+ image. You mention in your guide that you'll post updates "when you have something", and judging by your post it seems you have. Since I had difficulties compiling the zfs (and one other, which I don't recall) module, I'm wondering if you'd care to share your experience in your guide? Kudos to your very well thoughtout and thorough guide.

zyngaro 11y ago
Give crochet a try. I've used it recently to build a FreeBSD 11 image for RPi B+ recently and it just works.
- klapinat0r 11y ago
  Thanks. Yes it did build just fine. Now I just have to figure out what's "stalling" during boot. It compiled with zfs and all fine though :)
anotherhue 11y ago
I got it going a few days ago, have been meaning to update the guide and post some images so people can skip the compilation. Star the repo for updates. You're probably thinking of the opensolaris module. There's a small patch to apply and some KERNCONF to trigger the module build but otherwise its straightforward -- once you know where to look!

rleigh 11y ago

Interesting, I thought ZFS required a 64-bit system. Is this no longer the case? (I am using UFS on powerpc32 since it's not supported for 10.2.)

Freaky 11y ago
64-bit is very strongly recommended, but it does work on 32-bit with tuning: https://wiki.freebsd.org/ZFSTuningGuide https://wiki.freebsd.org/ZFSTuningGuide