6 ms·
Why are my ZFS disks so noisy?
- hamandcheese 2y agoI don't know if this exists or not, but I'd like to try something like a fuse filesystem which can transparently copy a file to a fast scratch SSD when it is first accessed. I have a somewhat large zfs array and it makes consistent noise as I stream videos from it. The streaming is basically a steady trickle compared to what the array is capable of. I'd rather incur all the noise up front, as fast as possible, then continue the stream from a silent SSD.
- toast0 2y agoIf you've got enough ram, you might be able to tune prefetching to prefetch the whole file? Although, I'm not sure how tunable that actually is.
- iforgotpassword 2y agoYes, I think nowadays you do this with blockdev --setra <num_sectors> /dev/sdX But I feel like there was a sysctl for this too in the past. I used it back in the day to make the HDD in my laptop spin down immediately after a new song started playing in rhythmbox by setting it to 16MB.
- hnlmorg 2y agoNot aware of anything that directly matches your description, however all major operating systems do cache filesystem object in RAM. So if you pre-read the file, then it should be read back from cache when you come to stream it. Additionally, ZFS supports using SSDs to supplement the cache.
- spockz 2y agoJust to test whether your OS setup etc is already up to par, try reading the whole file, eg by calculating the hash with something like `md5` (yes md5 is not secure I know.). This should put the file mostly in the os cache. But with video files being able to hit more than 50GiB in size these days, you need quite a lot of RAM to keep it all in cache. Maybe you can setting the SSD as a scratch disk for swap? I’m not sure how/if you can tweak what it is used for. As a sibling says, ZFS should support this pretty transparently.
- deleted 2y ago[deleted]
- 3np 2y agoIt may not be 100% what you're looking for and will probably not make your drives silent while streaming but putting L2ARC on that SSD and tweaking prefetch might get you a good way there. Another much simpler filesystem-agnostic alternative would be to copy it over to the SSD with a script and commence streaming from there. You'll have to wait for the entire file to copy for the stream to start, though. I think some streaming servers may actually support this natively if you mount /var and/or /var/tmp on the faster drive and configure it to utilize it as a "cache".
- Mayzie 2y ago> I don't know if this exists or not, but I'd like to try something like a fuse filesystem which can transparently copy a file to a fast scratch SSD when it is first accessed. You may be interested in checking out bcache[1] or bcachefs[2]. [1] https://www.kernel.org/doc/html/latest/admin-guide/bcache.html https://www.kernel.org/doc/html/latest/admin-guide/bcache.ht... [2] https://bcachefs.org/ https://bcachefs.org/
- theblazehen 2y agolvm-cache works as well, if you're already using LVM. https://github.com/45Drives/autotier https://github.com/45Drives/autotier is exactly what they were asking for as well
- magicalhippo 2y agoI've done some testing with using ZFS on top of LVM with dm-writecache. Worked well enough on the small scale, but sadly haven't had the time or hardware to test it in a more production-like environment. Also, it was starting to feel a bit like a Jenga tower, increasing the chances of bugs and other weird issues to strike.
- kaliszad 2y agoI wouldn't recommend combining those two. It's only begging for problems.
- magicalhippo 2y agoYeah that's my worry. Still, got 6x old 3TB disks that still work and a few spare NVMEs, so would be fun to try it for teh lulz.
- kaliszad 2y agoRather build a L2ARC or metadata special device hybrid setup using ZFS or skip ZFS and go for lvmcache/ mdadm style RAID with XFS or something.
- naming_the_user 2y agoDepending on the file-size I wonder if it'd be better to just prefetch the entire file into RAM.
- fsckboy 2y agogood idea, but that's not his problem. he needs a media player that will sprint through to the end of the file when he first opens it. They don't do that cuz they figure they might be streaming from the net so why tax that part of the sytem.
- throw73737 2y agoVLC has all sorts of prefetch settings.
- usefulcat 2y agoWith ZFS that wouldn't necessarily help. I've been using ZFS at work for many years with mostly large files. Even if I repeatedly read the same file as fast as possible, ZFS will not cache the entire thing in memory (there's no shortage of RAM so that's not it). This is unlike most linux filesystems, which are usually pretty aggressive with caching. Maybe there is some combination of settings that will get it to cache more aggressively; just saying that it's not a given that it will do so.
- kaliszad 2y agoThe ZFS ARC cache tries to keep a balance between most frequently and least recently used data. Also, by default, ZFS ARC only fills out to a maximum of about half the available RAM. You can change that at runtime (by writing the size in bytes to /sys/module/zfs/parameters/zfs_arc_max or setting the module e.g. in /etc/modporbe.d/zfs.conf to something like this options zfs zfs_arc_max=<size in bytes> ). But be careful, as the ZFS ARC does not play that nice with the OOM killer.
- hamandcheese 2y agoThank you, yes this is my problem which most commenters here are missing. I'm using Jellyfin server and Infuse on Apple TV, so I don't have a great way to force-slurp a file on first read, at least not without patching Jellyfin. And I'm not super eager to learn C#. Even if I were streaming to mpv, most of my lan is only gigabit which is much less than what my storage array can put out.
- dialup_sounds 2y agoReminded of this -- ZFS on Apple's Fusion Drive http://jolly.jinx.de/teclog/2012.10.31.02-fusion-drive-loose-ends.html http://jolly.jinx.de/teclog/2012.10.31.02-fusion-drive-loose...
- nicman23 2y agoi mean you can do that with a simple wrapper have a ssd ie mounted in /tmp/movies and create a script in .bin/ (or whatever) #!/bin/sh tmp_m="/tmp/movies/$(mktemp -d)" cp "$@" $tmp_m mpv $tmp_m rm $tmp_m please note i have not tried the script but it probably works
- anotherhue 2y agompv has --cache-size (or something) that you can set at a few GB. If you run out of ram it should swap to your ssd. Edit: demuxer-max-bytes=2147483647
- 2OEH8eoCRo0 2y agoYou could use overlayfs with the upper layer on the SSD "scratch" and trigger a write operation
- madeofpalk 2y agoThis is essentially what macOS's Fusion Drive is/was https://en.wikipedia.org/wiki/Fusion_Drive https://en.wikipedia.org/wiki/Fusion_Drive I'm unsure if they ship any macs with these anymore. I guess not since the Apple Sillicon iMacs don't have spinning hard drives?
- mystified5016 2y agoBcachefs. You can do all sorts of really neat things. You can define pools of drives at different cache levels. You can have a bunch of mechanical drives for deep storage, some for hot storage, SSD to cache recently read files, then write-through from the SSD down to mechanical drives, either immediately or after a delay. It's pretty much everything I could wish for from a filesystem, though I haven't actually taken the time to try it out yet. AFAIK it's still somewhat experimental, more or less in beta.
- cassianoleal 2y agoIs it fully merged to the kernel? I remember a few weeks ago some drama between Kent Overstreet and Linus about it but I didn't go into details. Has that been resolved? Edit: or maybe it was some drama over userspace tooling, I can't remember tbh.
- MattTheRealOne 2y agoBcachefs is fully merged into the kernel since 6.7. The drama was around Overstreet trying to merge significant code changes to a Release Candidate kernel that should only be receiving minor bug fixes at that stage of the development process. It was developer communication issues, not anything that impacts the user. The changes will just have to wait until the next kernel version.
- cassianoleal 2y agoI think this is what I was recalling. https://jonathancarter.org/2024/08/29/orphaning-bcachefs-tools-in-debian/ https://jonathancarter.org/2024/08/29/orphaning-bcachefs-too...
- ThePowerOfFuet 2y agoBcachefs has not proven that it can be trusted to not eat data, and has also recently undergone an on-disk layout change which further demonstrates its lack of maturity.
- throw0101d 2y ago> I don't know if this exists or not, but I'd like to try something like a fuse filesystem which can transparently copy a file to a fast scratch SSD when it is first accessed. ZFS has caching for writes (SLOG)[0][1] and reads (L2ARC),[2][3] which was introduced many years ago when HDDs were cheap and flash was still very, very expensive: * https://www.brendangregg.com/blog/2009-10-08/hybrid-storage-pool-top-speeds.html https://www.brendangregg.com/blog/2009-10-08/hybrid-storage-... [0] https://openzfs.github.io/openzfs-docs/man/master/7/zpoolconcepts.7.html#Intent_Log https://openzfs.github.io/openzfs-docs/man/master/7/zpoolcon... [1] https://openzfs.github.io/openzfs-docs/man/master/8/zpool-create.8.html#Example_6_:_Creating_a_ZFS_Pool_with_Mirrored_Separate_Intent_Logs https://openzfs.github.io/openzfs-docs/man/master/8/zpool-cr... [2] https://openzfs.github.io/openzfs-docs/man/master/7/zpoolconcepts.7.html#Cache_Devices https://openzfs.github.io/openzfs-docs/man/master/7/zpoolcon... [3] https://openzfs.github.io/openzfs-docs/man/master/8/zpool-add.8.html#Example_2_:_Adding_Cache_Devices_to_a_ZFS_Pool https://openzfs.github.io/openzfs-docs/man/master/8/zpool-ad...
- aidenn0 2y agoI should point out that the SLOG only caches synchronous writes, which are written twice with ZFS. Also, the L2ARC is great, but does still have RAM overhead. There are also useful tunables. I had a workload on a RAM limited machine where directory walking was common, but data reads were fairly random and a L2ARC configured for metadata only speed it up by a large amount.
- kaliszad 2y agoYou can also have a special metadata only/ small files dedicated special VDEV. ZFS can pull of many tricks if you configure it well. Of course, the L2ARC is better, if you don't trust the caching device that much (e.g. only have a single SSD).
- flemhans 2y agoYou can also configure an L2ARC to only hold metadata, which is safer than a metadata device.
- _hyn3 2y agocat filename > /dev/null Reads the entire file into the OS buffer.
- sulandor 2y agoadd this to mpv.conf cache=yes demuxer-max-bytes=5G demuxer-max-back-bytes=5G
- Maledictus 2y agoI expected a way to find out what the heck the system is sending to those disks. Like per process and what the the Kernel/ZFS is adding.
- M95D 2y agoIt can be done with CONFIG_DM_LOG_WRITES and another set of drives, but AFAIK, it needs to be set up before (or more exactly under) zfs.
- Sesse__ 2y agoblktrace is fairly useful in this regard.
- hi-v-rocknroll 2y agoI'd have a look at jobs like scrubbing.
- paol 2y agoI know the article is just using the noise thing as an excuse to deep dive into zfs and proxmox, which is cool, but if what you really care about is reducing noise I thought I'd leave some practical advice here: 1. Most hard drive noise is caused by mechanical vibrations being transmitted to the chassis the drive is mounted on. 2. Consequently the most effective way to reduce noise is to reduce the mechanical coupling in the drive mounting mechanism. Having the drives in a noise-isolating case is helpful too, but only as a secondary improvement. Optimizing the drive mounting should really be the first priority. 3. If space isn't a concern the optimal thing is to have a large case (like an ATX or larger) with a large number of HDD bays. The mounting should use soft rubber or silicon grommets. Some mounting systems can work with just the grommets, but systems that use screws are ok too as long as the screw couples to the grommet not the chassis. In a good case like this any number of hard drives can be made essentially inaudible. 4. If space is a concern, a special purpose "NAS like" case (example: the Jonsbo N line of cases) can approach the size of consumer NAS boxes. The lack of space makes optimal accoustics difficult, but it will still be a 10x improvement over typical consumers NASes. 5. Lastly what you shouldn't ever do is get one of those consumers NAS boxes. They are made with no concern for noise at all, and manufacturing cheapness constraints tend to make them literally pessimal at it. I had a QNAP I got rid of that couldn't have been more effective at amplifying drive noise if it had been designed for that on purpose.
- gymbeaux 2y agoI find NASes to be a waste of money except for the “no one ever got fired for….” aspect in an enterprise environment. $600 for a NAS with a Celeron and 8GB of RAM is absurd.
- devilbunny 2y agoValue of your time and effort maintaining it is not zero. I used to play with stuff like this. It was fun when I was single and had lots of free time. I don't play with it anymore. If I pay someone $500 over nominal value to provide me with 8-9 years of support for security updates, etc., and I just install their packages... that's worth it to me. My first Syno was a DS412+ and my second was a DS1621+. Nine years between introduction of the two. The 412+ is still running just fine at a friend's house. I gave it to him with ~12 TB total drive space, said just help me next time I need something done with car audio (he's a DJ and knows cars) and we're square. He's happy, I'm happy. I go set up his network, he installs my head unit. We both win by doing what we're good at and letting someone else use their expertise instead of learning a lot of stuff we will almost never use again.
- deleted 2y ago[deleted]
- jftuga 2y agoIs there an online calculator to help you find the optimal combination of # of drives, raid level, and block size? For example, I'm interested in setting up a new RAID-Z2 pool of disks and would like to minimize noise and number of writes. Should I use 4 drives or 6? Also, what would be the optimal block size(es) in this scenario?
- thg 2y agoNot sure if this is exactly what you're looking for, but I suppose it will be of at least some help: https://docs.google.com/spreadsheets/d/1tf4qx1aMJp8Lo_R6gpT689wTjHv6CGVElrPqTA0w_ZY/edit?pli=1&gid=1576424058#gid=1576424058 https://docs.google.com/spreadsheets/d/1tf4qx1aMJp8Lo_R6gpT6...
- mastax 2y agohttps://wintelguy.com/zfs-calc.pl https://wintelguy.com/zfs-calc.pl
- deleted 2y ago[deleted]
- nubinetwork 2y agoI don't think you can really get away from noise, zfs writes to disk every 5 seconds pretty much all the time...
- m463 2y agoI run proxmox, and ever since day 1 I noticed it hits the disk. a LOT. I dug into it and even without ANY vms or containers runnning, it writes a bunch of stuff out every second. I turned off a bunch of stuff, I think: systemctl disable pve-ha-crm systemctl disable pve-ha-lrm But stuff like /var/lib/pve-firewall and /var/lib/rrdcached was still written to every second. I think I played around with commit=n mount and also The point of this is - I tried running proxmox with zfs, and it wrote to the disk even more often. maybe ok for physical hard disks, but I didn't want to burn out my ssd immediately. for physical disks it could be noisy
- hi-v-rocknroll 2y agoMy 4U JBOD NAS box noise is dominated by the goddamn 1U-style jet engine fans. 45 helium HDDs themselves are relatively quiet. PS: I ditched non-Solaris ZFS several years ago after ZoL destroyed itself unable to mount RW and the community shrugged at glaring fragility. XFS + mdadm (raid10) are solid and work. Boring and reliable get less press, but I like working over not working. Maybe folks here run Sun Thumpers at home which would be a form of ZFS that works.
- kaliszad 2y agoNow the common code base of upstream OpenZFS is the FreeBSD/ Linux code. Things have changed a lot with OpenZFS 2.0 AFAIK. Yeah, server vendors are a bit crazy with small jet engine fans. A 4U chassis could easily house 80, 92 or even 120 mm fans which could spin much slower with a much higher air flow. That would of course also be much more efficient.