4 ms·

A big user of RAM in ZFS is dedup and compression - which can both be disabled.

alblue 10y ago
Compression does not use a significant amount of memory - it's the deduplication which uses a lot of memory. In fact compression can make it run faster as there is less I/O data.
snuxoll 10y ago
Compression in ZFS doesn't use RAM, dedup tables, the arc and l2arc mapping tables are all that matters with regards to memory usage with ZFS. l2arc is where a lot of newbies end up shooting themselves in the foot, they go toss in a mirrored pair of 512GB SSD's for their l2arc on a system with 32GB of RAM and wonder why performance got worse or it crashed, or they enabled dedup on 24TB of data with 64GB of ram and wonder why it crashed and their zpool is beyond recovery. If you are using ZFS without an l2arc or dedup enabled even a measly 1GB of RAM is "sufficient", you just won't get the best performance since it's going to have to constantly fetch from disk depending on your active data set (which is literally no different a problem than OS-level file caching in any other operating system).
- voltagex_ 10y ago
  Ah crap. The advice I got was just about the opposite - I wanted an L2ARC device in my system for performance. Atom C2758, 32GB ECC RAM, 4x4TB in RAIDZ2 (losing that much storage was painful, and the $1200AUD upgrade path to 4x8TB even more so) with a 128GB SSD as L2ARC. I found out some time later that I probably wanted an SLOG device instead but I'm really too afraid to touch my config. FWIW it's pretty stable on FreeBSD 10.3 with around 5GB RAM in use.
  - snuxoll 10y ago
    Your L2ARC shouldn't be bigger than 5x your system memory, ZFS has to keep mapping tables in-memory to determine where data on the L2ARC is stored, the bigger it gets the more memory you take away from your in-memory ARC, which can lead to worse performance than before since more data is having to come from your much higher-latency and lower bandwidth L2ARC or straight from spinning disks. RAIDZ is a performance killer right off the bat, I would switch to using mirrored vdev's instead if throughput is an issue for you. Parity calculation kills write speeds, especially on slower CPU's like a C2758, and RAIDZ doesn't give you any extra throughput on reads since there's only one usable copy of a stripe to read from. I have 2x4TB disks in my pool as a mirrored vdev (with an additional 2x3TB coming after I get my 128GB flash drive for XenServer to free up my second 3TB disk again) and I get reads over 500MB/sec using 10Gig-E to my XenServer host, and equally fast writes. Disks are fairly cheap, RAIDZ is not a good solution if you need performance, and if you are using four disks in RAIDZ-2 you would lose the same amount of storage using two mirrored vdev's anyway (and save the increased potential of a rebuild failing due to multiple disk failures that is increasingly common on higher capacity drives). If writes are a bigger problem for you, then, yes, a SLOG device will help - but any random SSD is not going to do. If you are using a SLOG ZFS expects it will not corrupt in-flight data following a power loss, even a "high-end" SSD like a Samsung 850 Pro or a Crucial MX200 will lose data in the event of a power failure. You also don't want a SLOG that isn't mirrored, if your SLOG is corrupted you just lost your entire pool. In addition you need a SSD with proper power-failure protection like the Intel DC series. Also, large writes skip the SLOG (>64KB) entirely, so if you are bandwidth bound (either from network or to disk) it is not going to help, if you are IOPS bound it can help tremendously. Also, you mention 4x4TB and 4x8TB, I assume your issue is the available disk bays in your system? I personally ignore high-capacity drives as they are far too expensive, and my HP ML10 that I run FreeNAS off only has 1 internal drive bay (with a $50 add-on I can buy to install an extra 3), instead of dealing with internal drives as well as limited capacity I bought a cheap DAS array (Lenovo SA120, here's a picture of mine and my cheap TP-Link layer 3 switch http://i.imgur.com/eEvtP6Z.jpg http://i.imgur.com/eEvtP6Z.jpg) that cost me $200USD and connected it with an equally cheap LSI 9200-8e SAS HBA ($40), I now have 12 hot-swap drive bays and can replace my FreeNAS box without worrying about how many bays it has, if I need more I just need to buy an additional enclosure and daisy chain it off the first. This isn't a dirt cheap solution by any means (my homelab gear is easily worth over $1500 at this point, including my Lenovo TD340 acting as my XenServer host), but it will save you money in the long run by allowing you to easily buy many cheaper drives than fewer more expensive (but high capacity) ones.
    - qmr 10y ago
      Are you quite sure that is the best solution long term? All of those extra drives are using additional power.
    - voltagex_ 10y ago
      And at this point I give up and go back to an off the shelf solution. I do have mirrored vdevs but now you've made me doubt what configuration I put them in. This stuff is so advanced it's not funny.
    - voltagex_ 10y ago
      Also, where should I read up about DAS arrays? I didn't even know that was an option when I started building. Edit: thanks for your help here - you don't have an email address listed in your profile but I'd like to discuss this further if you have the time.
      - snuxoll 10y ago
        DAS arrays are extremely easy to understand, they're just "expanders" - SAS has special support built in for this, in servers you typically have the front drive bays connected to a single SAS port on the board instead of individual cables running to each drive, a DAS is the same thing just as an external enclosure with separate power. You connect it to a basic SAS HBA with an external SAS port (SFF-8088), like the LSI 9200-8e I mentioned and it just shows up as a bunch of drives to your operating system. There's some special control services you have to the enclosure depending on the model through a protocol called SAS Enclosure Services (SES), my Lenovo SA120 uses this to manage things like fan speeds (very important, because every time it loses power it resets to HIGH which is VERY LOUD, I have a boot script on my FreeNAS box to set this back down to LOW which is much more manageable). This seems like a lot, but you can ignore most of the technical details I just posted. Buy a SAS HBA with external ports, buy a SAS DAS array, plug it in and you see a bunch of drives - no fuss. Since SAS controllers can also support SATA drives you can save yourself $10-20 a drive and buy normal SATA disks, or you can get some added reliability (multi-pathing and error handling) and buy near-line SAS drives for a small premium (I don't bother personally, but I only have one controller installed in my SA120 so I have no second path for data to travel in the event of a failure anyway). Feel free to hit me up anytime, I'm /u/snuxoll on reddit (pretty active on /r/homelab) and you can email me at stefan [a] nuxoll.me.

Compression does not use a significant amount of memory - it's the deduplication which uses a lot of memory. In fact compression can make it run faster as there is less I/O data.

snuxoll 10y ago

Compression in ZFS doesn't use RAM, dedup tables, the arc and l2arc mapping tables are all that matters with regards to memory usage with ZFS. l2arc is where a lot of newbies end up shooting themselves in the foot, they go toss in a mirrored pair of 512GB SSD's for their l2arc on a system with 32GB of RAM and wonder why performance got worse or it crashed, or they enabled dedup on 24TB of data with 64GB of ram and wonder why it crashed and their zpool is beyond recovery. If you are using ZFS without an l2arc or dedup enabled even a measly 1GB of RAM is "sufficient", you just won't get the best performance since it's going to have to constantly fetch from disk depending on your active data set (which is literally no different a problem than OS-level file caching in any other operating system).

voltagex_ 10y ago
Ah crap. The advice I got was just about the opposite - I wanted an L2ARC device in my system for performance. Atom C2758, 32GB ECC RAM, 4x4TB in RAIDZ2 (losing that much storage was painful, and the $1200AUD upgrade path to 4x8TB even more so) with a 128GB SSD as L2ARC. I found out some time later that I probably wanted an SLOG device instead but I'm really too afraid to touch my config. FWIW it's pretty stable on FreeBSD 10.3 with around 5GB RAM in use.
- snuxoll 10y ago
  Your L2ARC shouldn't be bigger than 5x your system memory, ZFS has to keep mapping tables in-memory to determine where data on the L2ARC is stored, the bigger it gets the more memory you take away from your in-memory ARC, which can lead to worse performance than before since more data is having to come from your much higher-latency and lower bandwidth L2ARC or straight from spinning disks. RAIDZ is a performance killer right off the bat, I would switch to using mirrored vdev's instead if throughput is an issue for you. Parity calculation kills write speeds, especially on slower CPU's like a C2758, and RAIDZ doesn't give you any extra throughput on reads since there's only one usable copy of a stripe to read from. I have 2x4TB disks in my pool as a mirrored vdev (with an additional 2x3TB coming after I get my 128GB flash drive for XenServer to free up my second 3TB disk again) and I get reads over 500MB/sec using 10Gig-E to my XenServer host, and equally fast writes. Disks are fairly cheap, RAIDZ is not a good solution if you need performance, and if you are using four disks in RAIDZ-2 you would lose the same amount of storage using two mirrored vdev's anyway (and save the increased potential of a rebuild failing due to multiple disk failures that is increasingly common on higher capacity drives). If writes are a bigger problem for you, then, yes, a SLOG device will help - but any random SSD is not going to do. If you are using a SLOG ZFS expects it will not corrupt in-flight data following a power loss, even a "high-end" SSD like a Samsung 850 Pro or a Crucial MX200 will lose data in the event of a power failure. You also don't want a SLOG that isn't mirrored, if your SLOG is corrupted you just lost your entire pool. In addition you need a SSD with proper power-failure protection like the Intel DC series. Also, large writes skip the SLOG (>64KB) entirely, so if you are bandwidth bound (either from network or to disk) it is not going to help, if you are IOPS bound it can help tremendously. Also, you mention 4x4TB and 4x8TB, I assume your issue is the available disk bays in your system? I personally ignore high-capacity drives as they are far too expensive, and my HP ML10 that I run FreeNAS off only has 1 internal drive bay (with a $50 add-on I can buy to install an extra 3), instead of dealing with internal drives as well as limited capacity I bought a cheap DAS array (Lenovo SA120, here's a picture of mine and my cheap TP-Link layer 3 switch http://i.imgur.com/eEvtP6Z.jpg http://i.imgur.com/eEvtP6Z.jpg) that cost me $200USD and connected it with an equally cheap LSI 9200-8e SAS HBA ($40), I now have 12 hot-swap drive bays and can replace my FreeNAS box without worrying about how many bays it has, if I need more I just need to buy an additional enclosure and daisy chain it off the first. This isn't a dirt cheap solution by any means (my homelab gear is easily worth over $1500 at this point, including my Lenovo TD340 acting as my XenServer host), but it will save you money in the long run by allowing you to easily buy many cheaper drives than fewer more expensive (but high capacity) ones.
  - qmr 10y ago
    Are you quite sure that is the best solution long term? All of those extra drives are using additional power.
  - voltagex_ 10y ago
    And at this point I give up and go back to an off the shelf solution. I do have mirrored vdevs but now you've made me doubt what configuration I put them in. This stuff is so advanced it's not funny.
  - voltagex_ 10y ago
    Also, where should I read up about DAS arrays? I didn't even know that was an option when I started building. Edit: thanks for your help here - you don't have an email address listed in your profile but I'd like to discuss this further if you have the time.
    - snuxoll 10y ago
      DAS arrays are extremely easy to understand, they're just "expanders" - SAS has special support built in for this, in servers you typically have the front drive bays connected to a single SAS port on the board instead of individual cables running to each drive, a DAS is the same thing just as an external enclosure with separate power. You connect it to a basic SAS HBA with an external SAS port (SFF-8088), like the LSI 9200-8e I mentioned and it just shows up as a bunch of drives to your operating system. There's some special control services you have to the enclosure depending on the model through a protocol called SAS Enclosure Services (SES), my Lenovo SA120 uses this to manage things like fan speeds (very important, because every time it loses power it resets to HIGH which is VERY LOUD, I have a boot script on my FreeNAS box to set this back down to LOW which is much more manageable). This seems like a lot, but you can ignore most of the technical details I just posted. Buy a SAS HBA with external ports, buy a SAS DAS array, plug it in and you see a bunch of drives - no fuss. Since SAS controllers can also support SATA drives you can save yourself $10-20 a drive and buy normal SATA disks, or you can get some added reliability (multi-pathing and error handling) and buy near-line SAS drives for a small premium (I don't bother personally, but I only have one controller installed in my SA120 so I have no second path for data to travel in the event of a failure anyway). Feel free to hit me up anytime, I'm /u/snuxoll on reddit (pretty active on /r/homelab) and you can email me at stefan [a] nuxoll.me.