4 ms·

My 'storage solution' up to this point has been 'random disks in a desktop tower' and I was doing some research just this morning for upgrading to a more comple

My 'storage solution' up to this point has been 'random disks in a desktop tower' and I was doing some research just this morning for upgrading to a more complete local solution. My use-cases: primary storage device for home lab, computer/phone backup server for myself and extended family, media server, self-hosted web archive, general self-hosted services. My preferences: I don't like the direction that Synology is trending in (i.e. the same direction as Ubiquiti) / I don't like that it's proprietary. I really like the idea of using ZFS. I don't want to spend a ton of time designing building and debugging a custom system. That leads me to the IX Systems Mini X [0], a 5+2-bay integrated system with FreeBSD-based TrueNAS OS (successor/rename of FreeNAS). One of the X+ models comes with dual 10GbE which is pretty nice for future-proofing. So my current homelab plan is: Mini X+: 8 core, 64 GB, 2x10GbE - $1600 5x 6TB WD Red Plus (in zfs-raidz1 for 24TB net capacity) - $650 1x 480GB Kingston DC1000B M.2 PCIE-3.0 enterprise SSD with power loss protection (PLP) as a SLOG/write-cache - $130 MikroTik RB5009UG+S+IN router-switch [1] with 1x10Gb SFP+, 1x2.5GbE, 7x1GbE - $185 Encrypted backup from NAS to rsync.net using Borg [2][3], maybe raw ZFS snapshots for certain workloads (databases). Total: $2565 Monthly: internet - $70 (1GB) - $100 (2GB), remote backup: $10 (initial) - $80 (half capacity) That's my current plan anyways. Feedback welcome. [0]: https://www.truenas.com/truenas-mini/ https://www.truenas.com/truenas-mini/ [1]: https://mikrotik.com/product/rb5009ug_s_in#fndtn-gallery https://mikrotik.com/product/rb5009ug_s_in#fndtn-gallery [2]: https://www.borgbackup.org/ https://www.borgbackup.org/ [3]: https://news.ycombinator.com/item?id=27997856 https://news.ycombinator.com/item?id=27997856

mindslight 5y ago
Similar, but different: I'd recommend assembling your own machine (you get to pick your own case), Supermicro A2SDI for a low power motherboard, CRS326/CRS328/CRS309/CRS317 for switches (listed in increasing number of 10GbE ports), and routing on the A2SDI or another low power amd64 board. Storage wise I'd either shrink your requirements for simplicity (eg two 14TB spindles in a simple mirror), or go for larger capacity with larger drives. 6TB drives seem small for a new deployment. Also, I'd do more than raidz1 to prevent against cascading failures on rebuild. I'm coming from a different place where I view vanilla Linux as a positive (hence the router), but my constraints give you a different datapoint to consider. If I could find an inexpensive quiet 10GbE switch that was OpenFlow capable to punt switching policy to my Linux router, I'd be using that instead of Mikrotik.
- infogulch 5y ago
  Maybe I'm not weighting the management complexity of running different OSes for different devices high enough. If I upgrade to 2 Gbit internet (recently offered by Google Fiber in my area) a motherboard with only 1GbE would be annoyingly limiting (though I suppose a PCIE card could fix that). I was looking at the RB5009UG because it's relatively powerful (cpu,mem,storage) and has that 2.5GbE to cover my line upgrade, and I could pair it with a CRS305 or CRS309 if I need more 10G switching in the future. Though I admit that a clearer delineation between router and switch might be nice. I chose 6TB because it's enough to cover my immediate use and it's the lowest $/TB right now (net 24TB is ~4x my current capacity and I've been kinda cramped). The idea is to upgrade to 14TB drives when it's full for a net 56TB. I guess the questions are: How much do I care about the upfront cost difference? ($650 vs $2000) How long before I fill up 24TB? (1/2y?) What will the price of 14TB be when I get there? (<$1300 total?) How much value am I getting from upgrading to new drives? (MTBF) How much trouble will it be to upgrade? I'm honestly not sure how to weigh these. Is it 'get to pick your own case' or 'have to pick your own case, power supply, memory, motherboard, cooling fans, assemble wire and test it, ...', though I guess these are two sides of the same coin. I'm not afraid of assembling PCs from parts (I've used my own PCs for 10+ years), but if I'm paying a $400 premium for eliminating 20+h of the above tasks + a 1y warranty that doesn't seem like a bad deal to me. Though now that you bring it up I might downgrade to the $1200 Mini X and add a $200 SFP+ card myself. Thanks for weighing in. :)
  - mindslight 5y ago
    Weigh it however you feel is necessary. I've just grown to hate clicking around in arbitrary web UIs to input arbitrary configuration models, where the only way to make things easier the next time is to create free form notes. And then when having a problem with a device, or needed additional functionality, the only solution is to throw it out and buy a new one. When you're doing price calculations on the drives, don't forget to take into account the cost of a drive bay, and additional electricity used. In other words, it helps to prefer larger capacity drives that are similar $/GB. I was just surprised by 6TB drives because I built my array 4 years ago out of 8TB drives and have seen that even larger drives are coming down in price. In your situation I might opt for a two 14TB drive mirror, with the idea that you could reuse those drives as part of a raidz later. Although that's easy to say for someone else's array. And yeah, having to build the machine is a bit of work. But I find it less work to know what went into it than to having to modify it a few years down the line and discover what the builders did. In general I favor spending time up front on my own schedule, rather than having to spend it later when I just need to get something to work so I can accomplish something else. At any rate, enjoy your new hardware and plentiful storage space!

mindslight 5y ago

Similar, but different: I'd recommend assembling your own machine (you get to pick your own case), Supermicro A2SDI for a low power motherboard, CRS326/CRS328/CRS309/CRS317 for switches (listed in increasing number of 10GbE ports), and routing on the A2SDI or another low power amd64 board. Storage wise I'd either shrink your requirements for simplicity (eg two 14TB spindles in a simple mirror), or go for larger capacity with larger drives. 6TB drives seem small for a new deployment. Also, I'd do more than raidz1 to prevent against cascading failures on rebuild. I'm coming from a different place where I view vanilla Linux as a positive (hence the router), but my constraints give you a different datapoint to consider. If I could find an inexpensive quiet 10GbE switch that was OpenFlow capable to punt switching policy to my Linux router, I'd be using that instead of Mikrotik.

infogulch 5y ago
Maybe I'm not weighting the management complexity of running different OSes for different devices high enough. If I upgrade to 2 Gbit internet (recently offered by Google Fiber in my area) a motherboard with only 1GbE would be annoyingly limiting (though I suppose a PCIE card could fix that). I was looking at the RB5009UG because it's relatively powerful (cpu,mem,storage) and has that 2.5GbE to cover my line upgrade, and I could pair it with a CRS305 or CRS309 if I need more 10G switching in the future. Though I admit that a clearer delineation between router and switch might be nice. I chose 6TB because it's enough to cover my immediate use and it's the lowest $/TB right now (net 24TB is ~4x my current capacity and I've been kinda cramped). The idea is to upgrade to 14TB drives when it's full for a net 56TB. I guess the questions are: How much do I care about the upfront cost difference? ($650 vs $2000) How long before I fill up 24TB? (1/2y?) What will the price of 14TB be when I get there? (<$1300 total?) How much value am I getting from upgrading to new drives? (MTBF) How much trouble will it be to upgrade? I'm honestly not sure how to weigh these. Is it 'get to pick your own case' or 'have to pick your own case, power supply, memory, motherboard, cooling fans, assemble wire and test it, ...', though I guess these are two sides of the same coin. I'm not afraid of assembling PCs from parts (I've used my own PCs for 10+ years), but if I'm paying a $400 premium for eliminating 20+h of the above tasks + a 1y warranty that doesn't seem like a bad deal to me. Though now that you bring it up I might downgrade to the $1200 Mini X and add a $200 SFP+ card myself. Thanks for weighing in. :)
- mindslight 5y ago
  Weigh it however you feel is necessary. I've just grown to hate clicking around in arbitrary web UIs to input arbitrary configuration models, where the only way to make things easier the next time is to create free form notes. And then when having a problem with a device, or needed additional functionality, the only solution is to throw it out and buy a new one. When you're doing price calculations on the drives, don't forget to take into account the cost of a drive bay, and additional electricity used. In other words, it helps to prefer larger capacity drives that are similar $/GB. I was just surprised by 6TB drives because I built my array 4 years ago out of 8TB drives and have seen that even larger drives are coming down in price. In your situation I might opt for a two 14TB drive mirror, with the idea that you could reuse those drives as part of a raidz later. Although that's easy to say for someone else's array. And yeah, having to build the machine is a bit of work. But I find it less work to know what went into it than to having to modify it a few years down the line and discover what the builders did. In general I favor spending time up front on my own schedule, rather than having to spend it later when I just need to get something to work so I can accomplish something else. At any rate, enjoy your new hardware and plentiful storage space!