9 ms·
Building a fast all-SSD NAS on a budget
- deleted 4y ago[deleted]
- liuliu 4y agoTBH, not sure if spending $3500 on 40TB storage v.s. ~$800 with rotating disks at the same storage capacity. You can put $200 on top with a 2TB NVMe SSD as cache. The reason to question this is that 40TB seems small if you want to have a NAS for small video editing studios. And for personal use, you probably not going to need more than 2TB work set paged in at any given moment.
- walrus01 4y agopersonally if I had to do this I would go with rotating disk for bulk storage in a NAS, and something like two 2TB to 4TB size NVME SSDs in a proper video editing workstation motherboard directly attached to pci-e 4.0 bus. This will be considerably faster for working with "immediate" needs of video files rather than over a 10GbE network. like, a difference of 900MBps over network vs 2500MBps with local sequential read/writes on NVME SSD on same motherboard.
- TheDudeMan 4y ago"I find myself needing a central storage solution disconnected from my main workstation."
- walrus01 4y agothere would still be such in what I described, just that it would have a much higher capacity composed of 16-18TB each spinning disks.
- geerlingguy 4y agoIn this case the storage is used directly for the editing, though. I am able to close a project on one computer, hop over to another and within seconds start editing at full speed (without requiring proxy footage). HDDs can't provide the latency that I need, unless I was running a dozen or two to overcome the horrible seek speeds (or had at least double the RAM so the entire project would be cached in RAM).
- derefr 4y agoWhat's wrong with proxy footage, though? Especially if the NAS was hyperconverged-in-the-small (i.e. had the spare compute to render out proxy footage for uploaded files in the background.)
- geerlingguy 4y agoNothing really, but it's just one extra layer of inconvenience. If you can edit full res, it's nice to be able to (especially for color grading and cropping purposes).
- jjcm 4y agoPossibly for smaller projects, for anything remotely sizable, 2TB is likely not going to cut it. 5k prores is 1TB for every 30min of footage, which means you're only getting an hour out of a 2TB drive. Storage needs for any pro video workflow get very large, very quick.
- tiernano 4y agoThat is true... But in that case, double or quadruple your SSD...
- Dylan16807 4y agoWell you're talking about 60fps ProRes 4444 (XQ?) footage, and the article is talking about 30fps ProRes 422 LT footage. That's 6.5x (10x?) less data for the same size project, and not much difference in how "pro" it is.
- rektide 4y agoFor compare $16/TB is pretty awesome[1], which for that budget would be about 217GB. ~5.5X. [1] https://diskprices.com/?locale=us&condition=new&capacity=12-&disk_types=internal_hdd https://diskprices.com/?locale=us&condition=new&capacity=12-...
- sorenjan 4y agoI'm not familiar with NAS file systems. Is it fairly straight forward to use hard drives with SSDs as transparent cache, and make it look like a single file share?
- js2 4y agoI had written a long comment about ZFS caching and still didn't do a good job, so let me link you to this instead: https://www.truenas.com/docs/references/slog/ https://www.truenas.com/docs/references/slog/ The short version is: - ZFS caches reads into RAM. Gobs of RAM helps. You can also add a secondary read cache device (L2ARC). - ZFS can caches synchronous writes to an SLOG device, but if you've disabled synchronous writes it won't make a difference. An SLOG device makes no difference for asynchronous writes. I use ZFS for my Time Machine backups (among other things) and have synchronous writes disabled for the Time Machine datasets. Before we had ZFS, the traditional way to speed up NAS was a raid controller with a battery-backed RAM cache.
- pdimitar 4y agoHow do you use your ZFS NAS for Time Machine backups? Very curious, I'd like to do the same.
- js2 4y agoI'm using TrueNAS which makes it pretty easy. I'm doing it via SMB (Samba). Previously I was doing it via AFP (Netatalk). This looks more or less correct: https://www.reddit.com/r/MacOS/comments/lh0yjc/configure_a_truenas_core_share_as_a_time_machine/ https://www.reddit.com/r/MacOS/comments/lh0yjc/configure_a_t... Basically it creates a dataset and a Samba share, and the configures Samba with a couple plugins so that: 1) a dataset is created for each user that connects to the share, so there's per-user datasets; 2) a ZFS snapshot is created automatically whenever Time Machine disconnects from the share. (2) is so that you can roll back if need be. In the past, I've had Time Machine decide that the sparse image it uses for a backup is corrupt and it wants to zero it out and start over. The ZFS snapshots lets you recover from that, though I haven't had it occur under macOS Monterey and/or since I switched to SMB from AFP. On the parent dataset I disabled sync. If you google for "TrueNAS Time Machine" you'll find some discussions of all this in the TrueNAS forums.
- mattalex 4y agoHe also has a hard drive based NAS, this is an additional local buffer for current work.
- CharlesW 4y agoIn this case "on a budget" means $4,329. That's reasonable if it speeds up billable work, but sadly the cost puts it a bit out of reach for my home office.
- PaywallBuster 4y agothe budget option is < 800$ ?
- magicalhippo 4y agoStill quite a lot. All you really need is an old i7 and a 10GbE Mellanox Connect-X2 or Connect-X3 card from eBay for $10-20.
- CTDOCodebases 4y agoI agree with your message but it's hard to find an i7 that supports ECC RAM and the Mellanox Connect-X 2 has lost support in modern distros. Best bet is to pick up an old HP Z620 of find someone who is upgrading their old Xeon homelab. Generally its a choice of cheap, quiet, energy efficient and you can only pick two of these options.
- geerlingguy 4y agoIn my case, one of the main goals was quiet, which ruled out some decent used 1U servers I found at the recycler :( Someday I'll have a proper closet but right now my rack is under the kitchen and my wife complains when the servers get above about 60 dB.
- CTDOCodebases 4y agoYeah I live in a studio and quiet is hard. I am also fussy and don’t want anything that looks out of place. I actually built my desktop as a near silent PC with the goal of it becoming my NAS when I upgrade but I didn’t understand the implications of running ZFS without ECC at the time.
- aetherspawn 4y agoAt this speed I’m thinking you’re probably going to bottleneck on the network/switch. Ubiquiti have a cheap fiber optic switch you could try. You could also try a 2x 10G SFP+ configuration, which would give you 20 Gbps (but only 10Gbps per client).
- neilv 4y agoThat's a neat 2U case design, and will fit in some very shallow wall-mount network switch cabinets. For installing outside of a machine room/closet/center, if you're using 2U of height, you might also fit a PSU with a larger and quieter fan, since all the Flex PSUs I've had come with noticeably loud fans. (I replace them with Noctuas, but it isn't a fun kind of soldering, IMHO.) The components from the build would also fit in a Supermicro 1U short-depth chassis, especially if you can go a little deeper in your cabinet. (My new K8s server got a used Supermicro 1U chassis for ~$60 shipped, including a PSU. In the photo on https://www.neilvandyke.org/kubernetes/ https://www.neilvandyke.org/kubernetes/ , it's the 1U immediately below the 4U.)
- jeffbee 4y agoI would have been pretty tempted to build this with a W480 Xeon platform having 2x thunderbolt ports. Conceivably that could have broken through the 1GB/s ceiling the article is seeing with 10g ethernet.
- mbreese 4y ago1.1GB/sec throughput is pretty good over a 10Gb/sec network. That’s 88% saturation. Right?
- jeffbee 4y agoThat's pretty much as fast as it can possibly go, yes. That's why I would try thunderbolt instead. It's faster.
- geerlingguy 4y agoSomeday I plan on upgrading to 25 Gbps or greater, and this build may still provide headroom since it has the free x16 PCIe 3.0 slot. But storage bandwidth would probably fall off before network bandwidth at that point.
- technofiend 4y agoJeff, if you do end up with a noise-friendly place to rack a server then options open up. The last dl360 I bought was less than $1k and came with a Mellonox dual port 40 gigabit NIC. Equipment is still cheap in the used market because those who haven't just moved to the cloud are still upgrading to newer and faster servers and networks as older servers and switches are fully depreciated and scrapped. Eventually that stream will diminish but the cycle will repeat every few years.
- mbreese 4y agoThat’s what I didn’t understand. If you want this accessible over a network, you wouldn’t want to use a direct thunderbolt connection. As mentioned below, you could try 25Gbps Ethernet, but you’d also need to upgrade the clients. I assume the clients are currently 10GigE (copper?). As it is, you’re pretty much saturating the network bandwidth. Really, he could have also saturated the bandwidth with spinning HDDs too, given enough of them.
- alexk307 4y agoHuge fan of Jeff’s work on YouTube! Highly recommend checking it out if this blog interests you
- walrus01 4y agoI would be extremely cautious about using any consumer grade TLC or quad-level-cell SSD in a "NAS" for serious purposes because of well known write lifespan issues. There's a reason that a big difference in price exists between a quad-level-cell 2TB SSD and an expensive enterprise grade one with a much higher TB-write-before-dead rating. This might look cool but check back in a few years and see how much of the drives' cumulative write lifespan is worn out. I also cannot even imagine spending $4000+ on a home file server/NAS with copper only 10GbE NIC and it not having at least one 10G SFP+ interface network card. Okay, so he wants it to be tiny? But in a home environment the major problem is more power consumption and noise, so you can often go with a well ventilated 4U height rackmount case for full size ATX motherboard, which is roughly the size of a midtower PC case turned on its side. This lets you use motherboards that will have enough PCI-E 3.0 x8 slots for at least one dual-port Intel SFP+ 10G NIC which are very, very cheap on ebay these days.
- hatware 4y agoThis is definitely an engineering disaster. Sometimes we get too caught up in how to do something that we never ask ourselves if we should.
- nichch 4y agoI was thinking the same thing, but wouldn’t these be okay if his workload is mainly WORM?
- mbreese 4y agoTrueNAS used to be designed to boot off of smaller SataDOMs that were used only for boot. They were effectively WORM. At least, it used to be a few years ago. Everything that was written for the server was either a RAM disk or spread out amongst the RAID drives (as a separate partition, which has its own issues, but still). I had assumed this is what he was using the TLC SSD for. If that’s the case, so long as there isn’t much writing to it, it should be fine.
- walrus01 4y agoThe 8TB Samsung QVO drives are quad level cell consumer grade drives, that's his main storage.
- rr888 4y agoHas anyone tried to replace NAS with a cloud service? If you have gig internet it should but I'm not sure if dropbox etc can keep up.
- karmicthreat 4y agoI just went through getting the parts for my own NAS. All SSD was way overkill for my needs, I ended up going with spinning disks and a SLOG cache. I kept waffling about the motherboard to use but I ended up with a x470d4u motherboard with a Ryzen 7 4700GE which brings the TDP down to 35W. I wanted this to be kind of quiet. I will put a 10Gb network card on it eventually. Maybe not THE BEST (tm) choices. But I was getting bad decision paralysis choosing parts.
- sorenjan 4y ago> Every minute of 4K ProRes LT footage (which is a very lightweight format, compared to RAW) is 3 GB of space Do you really need to save all of that footage? I would think keeping the pro res footage for the current projects on the workstation and reencoded archive video on a NAS would be sufficient. I'm not a video professional, but I suspect it's easy to fall in the trap of thinking that you need to save everything in highest possible quality in case you need it later, but what are the realistic chances of that? If you end up needing some old footage again, AV1 coded 4K or even HEVC 1080p would probably be just fine. The final result are Youtube videos after all. I know he mentions editing from it, but that's enough space for more than a week of pro res video.
- rektide 4y agoPersonally I kind of fright from the "have you considered throwing away your best copies" as a sales strategy. On the flip side, I think it would help a lot a lot a lot if there was more discourse out there about compressing high-dynamic-range content. AV1 supposedly has some capability to do a good-ish job with HDR. The idea of taking reels of raw video and spending a couple days squishing it to 1/10th the sizes but preserving the quality/flexibility very very well is a value proposition that I dont think is clearly attainable, even though it seems technically perhaps within reach. Right now, I think the general feeling is, raw is raw & everything else bakes in a vast amount of assumptions & constrictions. Some advocacy that compressed video can be as flexible, as dynamic, as capable need to be more present, elaborated, & proven before anyone's going to be comfortable throwing away the bits. These are people's life's works & a couple hundred or thousand dollars a year more in storage isnt a real factor for such integral, near work.
- sorenjan 4y agoI don't have a sales strategy, I'm not selling anything. I'm questioning the need of buying needless storage for data hoarding you're never going to use. Have you seen Jeff's videos? They're mostly him talking to the camera in what I think is his basement, or closeups of various PCBs. If you're running a stock photo company I get wanting to keep footage at max quality, but once your Youtube video is published, do you really need to save all those alternate takes and cut out bits? I just don't see them being very useful in the future, and if you do end up needing some of it for B roll or whatever, would a couple of seconds of compressed video really make a drastic difference for the project as a whole? Especially since the final result ends up on Youtube and consumed on a phone screen or a TV two meters away. Much of the content that's produced today is not trying to be timeless classics with endless rewatch value, it's ephemeral and only really relevant for a short time. If you can reupload your videos to other video hosts in the future that's probably good enough. Nobody is watching reviews of three year old Raspberry Pi add on boards for the low noise in dark parts of the image, or the accurate color reproduction, or even the 4K instead of 1080p resolution. Kill your darlings!
- erulabs 4y agoJeff previously reviewed our all-SSD NAS, which is far more budget, if anyone is interested: https://www.jeffgeerling.com/blog/2021/kubesails-pibox-mini-2-16-tb-ssd-storage-on-pi https://www.jeffgeerling.com/blog/2021/kubesails-pibox-mini-... Love seeing these projects from him, but this is a rare miss in my opinion. This is the strange middle ground where it's closer to professional than "budget", by quite a bit. At some point, only a tiny tiny fraction of users need 40TB of space. I guess what I'm saying is, this isn't so much a NAS as a specialized youtuber video recording appliance. We have a number of home-hosting users on our platform that run entire racks filled to the brim - and while that's awesome - it's just not what "budget" or "NAS" implies, and it's an extremely limited audience.
- jmole 4y agoEhhhh, he said "All-SSD NAS on a budget", not a "Budget SSD NAS". Agree though that this was a bit of a miss; the cost of drives here dwarfs everything else, and it makes the choice of case, MB, etc. nearly irrelevant and needlessly cost optimized. Why not spend an extra $60 and get a 5.25->6x 2.5 hot swap cage, for example?
- dylan604 4y agoAt the end of the video, the config he settled on doesn't have 40TB of useable space. He only has 16TB of useable. Essentially, a RAID-10 with a hot swap. He's only getting the speed of striping 2 drives together. Sounds like he's playing it safe choosing a bit of redundancy rather than just straight speed.
- geerlingguy 4y agoIn this case, the majority of my income depends on the edit server being online (thus the decision to go basically RAID10 + hot spare). I have a full backup plus an offsite copy, but they are over 2.5 Gbps on spinning disk, or take 6+ hours to restore from offsite, so they're emergency use only. I'm also trying to get my homelab set up to be a bit more robust / automated / reproducible, and trying out a bunch of different ideas. I'll likely settle on something else in a year's time, but the main motivation for now was to get all the storage into my rack on my 10 Gbps network.
- retcon 4y agoVideo is nicely sequential and eminently suitable for spinning disk service. What's really confusing me is the 10G lan configuration because it's leaving little headroom above uncompressed 4K 4:4:4 10bit (12bit is broadcast and where source permits archive standard). What about multiple streams for a/b'ing a grade or first/ second camera edit roll? Ed. added "really" replaced "for" with "above"
- nope96 4y agoECC RAM, yay! I've read so many posts like this where they did NOT use ECC RAM.
- abracadaniel 4y agoIt's surprisingly hard to do on a budget, since you're essentially locked into Xeon products and while CPUs are available used, getting a motherboard to support them isn't easy. The combo in this post, or waiting for a a full system to show up on ebay are probably the only real options. I'm really looking forward to DDR5 getting wide adoption, as that's really going to expand our options.
- tambourine_man 4y agoI love Jeff’s channel. Very entertaining and I learn a ton. I have a Raspberry Pi NAS with 10TB (2TB of redundancy) but it’s in a box made out of MDF. And while I had fun making it, let’s just say I’m not a woodworker. I’m amazed at how professional the stuff he does looks all while making it seem easy.
- gfody 4y agofor about the same budget you could buy a 4-bay 2.5" nvme nas enclosure and a single 30tb ssd, and have 3 bays to expand if/when you need to
- nuker 4y agoHe should’ve better used Thunderbolt 3 DAS chassis.
- mrb 4y ago"I was able to get consistent performance using a striped RAIDZ mirror pool with four SSDs and a hot spare" That doesn't make sense. The author is not using raidz. In ZFS terminology, he's mirroring two striped vdevs (like raid10), plus using a hotspare. And that is a bad choice, as he gets only 16 TB usable, and the mirrored stripes could fail with certain combinations of two drives failures. Instead, he should have set up a raidz2 across 5 drives: 50% more usable space (24 TB), can tolerate any two drive failing, and it would give him higher performance on sequential I/O. I have a raidz2 pool across 6 spinning rust 18TB hard drives, and my server can handle 900 MB/s sequential reads, and 700 MB/s sequential writes (benchmarked locally). It it was built on 5 SAMSUNG 870 QVO SATA III SSD drives like the author, it would certainly get to 1.5+ GB/s in both sequential reads and writes. In other words the bottleneck would shift to the 10 Gb/s network (1.25 GB/s). For comparison the author discloses his writes are limited to 700 MB/s (over the network, so his bottleneck is not the network link but local I/O contention). That's right, a 6-HDD raidz2 matches the write performance of his 5-SSD raid10 setup. And that's because the write overhead of raid10 is 100%, while the write overhead of a 6-drive raidz2 is only 50%. Edit: Oh and the strange performance issue that he noticed "disappeared" after a while is most likely due to cache recovery. The drive can sustain writes of about 490 MB/s for a little while then it drops to 170 MB/s, but after idling 5 minutes it can recover the initial speed. See https://www.tomshardware.com/reviews/samsung-870-qvo-sata-ssd/2 https://www.tomshardware.com/reviews/samsung-870-qvo-sata-ss...: "As we noticed with the 1TB model, the 8TB model’s cache recovery mechanisms work similarly. After letting the drive rest at idle for 30 seconds, the 870 QVO gains back 6GB of its cache. It recovers fully with 5 minutes of idle time. "
- geerlingguy 4y agoNote that the 8TB model has about 50TB of cache (quite a bit more than the 1TB model—it doubles in size with each doubling of storage capacity). One concern Wendell had with RAIDZ2 was the potential for the older Xeon to be a bottleneck for writes. It probably wouldn't be, but I didn't do too much testing with that layout. I might still, we'll see.
- mrb 4y agoFor reference, my server CPU is an AMD Ryzen 7 PRO 5750G (65W TDP, 3.8-4.6GHz, 8-core, 16-threads) and CPU usage is around 8% when writing at 700 MB/s on my raidz2. In other words, only a little more than 1 of the 16 hardware threads is utilized to handle this workload. Your older Xeon D-1521 should be able to handle that, but it's true that being an older processor, it is pushing it closer to its limits. Handling concurrently 10 Gb/s of network traffic would likely push it to more than 50% utilization. That's one reason I like to not cheap out too much on the CPU, especially for a 10GbE ZFS NAS.
- peterbraden 4y agoWhat do you use to mount this on a mac? My bottleneck with my NAS is the atrocious speeds I get over samba or NFS, and even after spending a ton of time messing with the configs, I can’t get more than sluggish speeds.
- ahepp 4y agoI followed some guides I found online and ended up with the following: ~ cat /etc/auto_master # # Automounter master map # +auto_master # Use directory service #/net -hosts -nobrowse,hidefromfinder,nosuid /home auto_home -nobrowse,hidefromfinder /Network/Servers -fstab /- -static /- auto_nfs -nobrowse,nosuid ~ cat /etc/auto_nfs /System/Volumes/Data/Users/ahepp/nfs/public -fstype=nfs,vers=4,resvport,rw nfs://10.128.1.2:/zpa/nfs/public I haven't done a lot of performance testing, but I have no problem playing 10MB/s blurays off the NFS share (and that's via wifi) On the FreeBSD server in the closet: root@tlon:~ # cat /etc/exports V4: / /zpa/nfs/public -mapall=nobody
- gigatexal 4y agoAnyone know if any core level work on ZFS where there’s an effort to audit the code base for speed ups given the big differences in designs between spinning rust and SSDs?
- mastax 4y agoI do know that if you have very fast NVMe SSDs (>6000MB/s or so) ZFS is not currently able to give you the whole performance, due to time spent memcpying to/from the ARC[0]. Direct IO support could eventually alleviate this[1]. [0]: https://github.com/openzfs/zfs/issues/8381 https://github.com/openzfs/zfs/issues/8381 [1]: https://github.com/openzfs/zfs/pull/10018 https://github.com/openzfs/zfs/pull/10018
- jsmith99 4y agoOracle themselves have been selling all flash ZFS appliances for a long time so I imagine this is a development focus.
- mbreese 4y agoI doubt any Oracle SSD performance enhancements will make it into OpenZFS though.
- mgerdts 4y agoSATA SSDs top out at about 550 MB/s. ZFS does ok with these. NMVe SSDs top out at somewhere between 3000 MB/s (typical enterprise PCIe Gen 3) and 14000 MB/s (mythical just released PCIe Gen 5 drives). ZFS (out of the box, anyway) can’t drive PCIe Gen 4 or newer drives very hard. If your IO all goes over a 10 Gb link, that will be your bottleneck before ZFS is. Since 1 MB record size was used, this may be a candidate for a bunch of mirrored HDDs with a few TB of nvme for l2arc and ZIL. But with a bunch of SSDs already on hand, why bother?
- FullyFunctional 4y agoMy "budget" NAS can trivially saturate 10 GbE with a pair of Samsung SSD 980 PRO 2TB. It cannot saturate my 100 GbE connection. While it's true that TrueNAS perhaps can't quite get the same performance that you could get from XFS on the same hardware, that's really irrelevant for me as the features (snapshots and send) and data integrity is far more important.