5 ms·

I don't know if this exists or not, but I'd like to try something like a fuse filesystem which can transparently copy a file to a fast scratch SSD when it is fi

I don't know if this exists or not, but I'd like to try something like a fuse filesystem which can transparently copy a file to a fast scratch SSD when it is first accessed. I have a somewhat large zfs array and it makes consistent noise as I stream videos from it. The streaming is basically a steady trickle compared to what the array is capable of. I'd rather incur all the noise up front, as fast as possible, then continue the stream from a silent SSD.

toast0 2y ago
If you've got enough ram, you might be able to tune prefetching to prefetch the whole file? Although, I'm not sure how tunable that actually is.
- iforgotpassword 2y ago
  Yes, I think nowadays you do this with blockdev --setra <num_sectors> /dev/sdX But I feel like there was a sysctl for this too in the past. I used it back in the day to make the HDD in my laptop spin down immediately after a new song started playing in rhythmbox by setting it to 16MB.
hnlmorg 2y ago
Not aware of anything that directly matches your description, however all major operating systems do cache filesystem object in RAM. So if you pre-read the file, then it should be read back from cache when you come to stream it. Additionally, ZFS supports using SSDs to supplement the cache.
spockz 2y ago
Just to test whether your OS setup etc is already up to par, try reading the whole file, eg by calculating the hash with something like `md5` (yes md5 is not secure I know.). This should put the file mostly in the os cache. But with video files being able to hit more than 50GiB in size these days, you need quite a lot of RAM to keep it all in cache. Maybe you can setting the SSD as a scratch disk for swap? I’m not sure how/if you can tweak what it is used for. As a sibling says, ZFS should support this pretty transparently.
- deleted 2y ago
  [deleted]
3np 2y ago
It may not be 100% what you're looking for and will probably not make your drives silent while streaming but putting L2ARC on that SSD and tweaking prefetch might get you a good way there. Another much simpler filesystem-agnostic alternative would be to copy it over to the SSD with a script and commence streaming from there. You'll have to wait for the entire file to copy for the stream to start, though. I think some streaming servers may actually support this natively if you mount /var and/or /var/tmp on the faster drive and configure it to utilize it as a "cache".
Mayzie 2y ago
> I don't know if this exists or not, but I'd like to try something like a fuse filesystem which can transparently copy a file to a fast scratch SSD when it is first accessed. You may be interested in checking out bcache[1] or bcachefs[2]. [1] https://www.kernel.org/doc/html/latest/admin-guide/bcache.html https://www.kernel.org/doc/html/latest/admin-guide/bcache.ht... [2] https://bcachefs.org/ https://bcachefs.org/
- theblazehen 2y ago
  lvm-cache works as well, if you're already using LVM. https://github.com/45Drives/autotier https://github.com/45Drives/autotier is exactly what they were asking for as well
  - magicalhippo 2y ago
    I've done some testing with using ZFS on top of LVM with dm-writecache. Worked well enough on the small scale, but sadly haven't had the time or hardware to test it in a more production-like environment. Also, it was starting to feel a bit like a Jenga tower, increasing the chances of bugs and other weird issues to strike.
    - kaliszad 2y ago
      I wouldn't recommend combining those two. It's only begging for problems.
      - magicalhippo 2y ago
        Yeah that's my worry. Still, got 6x old 3TB disks that still work and a few spare NVMEs, so would be fun to try it for teh lulz.
        kaliszad 2y ago
        Rather build a L2ARC or metadata special device hybrid setup using ZFS or skip ZFS and go for lvmcache/ mdadm style RAID with XFS or something.
naming_the_user 2y ago
Depending on the file-size I wonder if it'd be better to just prefetch the entire file into RAM.
- fsckboy 2y ago
  good idea, but that's not his problem. he needs a media player that will sprint through to the end of the file when he first opens it. They don't do that cuz they figure they might be streaming from the net so why tax that part of the sytem.
  - throw73737 2y ago
    VLC has all sorts of prefetch settings.
  - usefulcat 2y ago
    With ZFS that wouldn't necessarily help. I've been using ZFS at work for many years with mostly large files. Even if I repeatedly read the same file as fast as possible, ZFS will not cache the entire thing in memory (there's no shortage of RAM so that's not it). This is unlike most linux filesystems, which are usually pretty aggressive with caching. Maybe there is some combination of settings that will get it to cache more aggressively; just saying that it's not a given that it will do so.
    - kaliszad 2y ago
      The ZFS ARC cache tries to keep a balance between most frequently and least recently used data. Also, by default, ZFS ARC only fills out to a maximum of about half the available RAM. You can change that at runtime (by writing the size in bytes to /sys/module/zfs/parameters/zfs_arc_max or setting the module e.g. in /etc/modporbe.d/zfs.conf to something like this options zfs zfs_arc_max=<size in bytes> ). But be careful, as the ZFS ARC does not play that nice with the OOM killer.
  - hamandcheese 2y ago
    Thank you, yes this is my problem which most commenters here are missing. I'm using Jellyfin server and Infuse on Apple TV, so I don't have a great way to force-slurp a file on first read, at least not without patching Jellyfin. And I'm not super eager to learn C#. Even if I were streaming to mpv, most of my lan is only gigabit which is much less than what my storage array can put out.
dialup_sounds 2y ago
Reminded of this -- ZFS on Apple's Fusion Drive http://jolly.jinx.de/teclog/2012.10.31.02-fusion-drive-loose-ends.html http://jolly.jinx.de/teclog/2012.10.31.02-fusion-drive-loose...
nicman23 2y ago
i mean you can do that with a simple wrapper have a ssd ie mounted in /tmp/movies and create a script in .bin/ (or whatever) #!/bin/sh tmp_m="/tmp/movies/$(mktemp -d)" cp "$@" $tmp_m mpv $tmp_m rm $tmp_m please note i have not tried the script but it probably works
anotherhue 2y ago
mpv has --cache-size (or something) that you can set at a few GB. If you run out of ram it should swap to your ssd. Edit: demuxer-max-bytes=2147483647
2OEH8eoCRo0 2y ago
You could use overlayfs with the upper layer on the SSD "scratch" and trigger a write operation
madeofpalk 2y ago
This is essentially what macOS's Fusion Drive is/was https://en.wikipedia.org/wiki/Fusion_Drive https://en.wikipedia.org/wiki/Fusion_Drive I'm unsure if they ship any macs with these anymore. I guess not since the Apple Sillicon iMacs don't have spinning hard drives?
mystified5016 2y ago
Bcachefs. You can do all sorts of really neat things. You can define pools of drives at different cache levels. You can have a bunch of mechanical drives for deep storage, some for hot storage, SSD to cache recently read files, then write-through from the SSD down to mechanical drives, either immediately or after a delay. It's pretty much everything I could wish for from a filesystem, though I haven't actually taken the time to try it out yet. AFAIK it's still somewhat experimental, more or less in beta.
- cassianoleal 2y ago
  Is it fully merged to the kernel? I remember a few weeks ago some drama between Kent Overstreet and Linus about it but I didn't go into details. Has that been resolved? Edit: or maybe it was some drama over userspace tooling, I can't remember tbh.
  - MattTheRealOne 2y ago
    Bcachefs is fully merged into the kernel since 6.7. The drama was around Overstreet trying to merge significant code changes to a Release Candidate kernel that should only be receiving minor bug fixes at that stage of the development process. It was developer communication issues, not anything that impacts the user. The changes will just have to wait until the next kernel version.
    - cassianoleal 2y ago
      I think this is what I was recalling. https://jonathancarter.org/2024/08/29/orphaning-bcachefs-tools-in-debian/ https://jonathancarter.org/2024/08/29/orphaning-bcachefs-too...
- ThePowerOfFuet 2y ago
  Bcachefs has not proven that it can be trusted to not eat data, and has also recently undergone an on-disk layout change which further demonstrates its lack of maturity.
throw0101d 2y ago
> I don't know if this exists or not, but I'd like to try something like a fuse filesystem which can transparently copy a file to a fast scratch SSD when it is first accessed. ZFS has caching for writes (SLOG)[0][1] and reads (L2ARC),[2][3] which was introduced many years ago when HDDs were cheap and flash was still very, very expensive: * https://www.brendangregg.com/blog/2009-10-08/hybrid-storage-pool-top-speeds.html https://www.brendangregg.com/blog/2009-10-08/hybrid-storage-... [0] https://openzfs.github.io/openzfs-docs/man/master/7/zpoolconcepts.7.html#Intent_Log https://openzfs.github.io/openzfs-docs/man/master/7/zpoolcon... [1] https://openzfs.github.io/openzfs-docs/man/master/8/zpool-create.8.html#Example_6_:_Creating_a_ZFS_Pool_with_Mirrored_Separate_Intent_Logs https://openzfs.github.io/openzfs-docs/man/master/8/zpool-cr... [2] https://openzfs.github.io/openzfs-docs/man/master/7/zpoolconcepts.7.html#Cache_Devices https://openzfs.github.io/openzfs-docs/man/master/7/zpoolcon... [3] https://openzfs.github.io/openzfs-docs/man/master/8/zpool-add.8.html#Example_2_:_Adding_Cache_Devices_to_a_ZFS_Pool https://openzfs.github.io/openzfs-docs/man/master/8/zpool-ad...
- aidenn0 2y ago
  I should point out that the SLOG only caches synchronous writes, which are written twice with ZFS. Also, the L2ARC is great, but does still have RAM overhead. There are also useful tunables. I had a workload on a RAM limited machine where directory walking was common, but data reads were fairly random and a L2ARC configured for metadata only speed it up by a large amount.
  - kaliszad 2y ago
    You can also have a special metadata only/ small files dedicated special VDEV. ZFS can pull of many tricks if you configure it well. Of course, the L2ARC is better, if you don't trust the caching device that much (e.g. only have a single SSD).
    - flemhans 2y ago
      You can also configure an L2ARC to only hold metadata, which is safer than a metadata device.
_hyn3 2y ago
cat filename > /dev/null Reads the entire file into the OS buffer.
sulandor 2y ago
add this to mpv.conf cache=yes demuxer-max-bytes=5G demuxer-max-back-bytes=5G

toast0 2y ago

If you've got enough ram, you might be able to tune prefetching to prefetch the whole file? Although, I'm not sure how tunable that actually is.

iforgotpassword 2y ago
Yes, I think nowadays you do this with blockdev --setra <num_sectors> /dev/sdX But I feel like there was a sysctl for this too in the past. I used it back in the day to make the HDD in my laptop spin down immediately after a new song started playing in rhythmbox by setting it to 16MB.

hnlmorg 2y ago

Not aware of anything that directly matches your description, however all major operating systems do cache filesystem object in RAM. So if you pre-read the file, then it should be read back from cache when you come to stream it. Additionally, ZFS supports using SSDs to supplement the cache.

spockz 2y ago

Just to test whether your OS setup etc is already up to par, try reading the whole file, eg by calculating the hash with something like `md5` (yes md5 is not secure I know.). This should put the file mostly in the os cache. But with video files being able to hit more than 50GiB in size these days, you need quite a lot of RAM to keep it all in cache. Maybe you can setting the SSD as a scratch disk for swap? I’m not sure how/if you can tweak what it is used for. As a sibling says, ZFS should support this pretty transparently.

deleted 2y ago
[deleted]

3np 2y ago

It may not be 100% what you're looking for and will probably not make your drives silent while streaming but putting L2ARC on that SSD and tweaking prefetch might get you a good way there. Another much simpler filesystem-agnostic alternative would be to copy it over to the SSD with a script and commence streaming from there. You'll have to wait for the entire file to copy for the stream to start, though. I think some streaming servers may actually support this natively if you mount /var and/or /var/tmp on the faster drive and configure it to utilize it as a "cache".

Mayzie 2y ago

> I don't know if this exists or not, but I'd like to try something like a fuse filesystem which can transparently copy a file to a fast scratch SSD when it is first accessed. You may be interested in checking out bcache[1] or bcachefs[2]. [1] https://www.kernel.org/doc/html/latest/admin-guide/bcache.html https://www.kernel.org/doc/html/latest/admin-guide/bcache.ht... [2] https://bcachefs.org/ https://bcachefs.org/

theblazehen 2y ago
lvm-cache works as well, if you're already using LVM. https://github.com/45Drives/autotier https://github.com/45Drives/autotier is exactly what they were asking for as well
- magicalhippo 2y ago
  I've done some testing with using ZFS on top of LVM with dm-writecache. Worked well enough on the small scale, but sadly haven't had the time or hardware to test it in a more production-like environment. Also, it was starting to feel a bit like a Jenga tower, increasing the chances of bugs and other weird issues to strike.
  - kaliszad 2y ago
    I wouldn't recommend combining those two. It's only begging for problems.
    - magicalhippo 2y ago
      Yeah that's my worry. Still, got 6x old 3TB disks that still work and a few spare NVMEs, so would be fun to try it for teh lulz.
      - kaliszad 2y ago
        Rather build a L2ARC or metadata special device hybrid setup using ZFS or skip ZFS and go for lvmcache/ mdadm style RAID with XFS or something.

naming_the_user 2y ago

Depending on the file-size I wonder if it'd be better to just prefetch the entire file into RAM.

fsckboy 2y ago
good idea, but that's not his problem. he needs a media player that will sprint through to the end of the file when he first opens it. They don't do that cuz they figure they might be streaming from the net so why tax that part of the sytem.
- throw73737 2y ago
  VLC has all sorts of prefetch settings.
- usefulcat 2y ago
  With ZFS that wouldn't necessarily help. I've been using ZFS at work for many years with mostly large files. Even if I repeatedly read the same file as fast as possible, ZFS will not cache the entire thing in memory (there's no shortage of RAM so that's not it). This is unlike most linux filesystems, which are usually pretty aggressive with caching. Maybe there is some combination of settings that will get it to cache more aggressively; just saying that it's not a given that it will do so.
  - kaliszad 2y ago
    The ZFS ARC cache tries to keep a balance between most frequently and least recently used data. Also, by default, ZFS ARC only fills out to a maximum of about half the available RAM. You can change that at runtime (by writing the size in bytes to /sys/module/zfs/parameters/zfs_arc_max or setting the module e.g. in /etc/modporbe.d/zfs.conf to something like this options zfs zfs_arc_max=<size in bytes> ). But be careful, as the ZFS ARC does not play that nice with the OOM killer.
- hamandcheese 2y ago
  Thank you, yes this is my problem which most commenters here are missing. I'm using Jellyfin server and Infuse on Apple TV, so I don't have a great way to force-slurp a file on first read, at least not without patching Jellyfin. And I'm not super eager to learn C#. Even if I were streaming to mpv, most of my lan is only gigabit which is much less than what my storage array can put out.

dialup_sounds 2y ago

Reminded of this -- ZFS on Apple's Fusion Drive http://jolly.jinx.de/teclog/2012.10.31.02-fusion-drive-loose-ends.html http://jolly.jinx.de/teclog/2012.10.31.02-fusion-drive-loose...

nicman23 2y ago

i mean you can do that with a simple wrapper have a ssd ie mounted in /tmp/movies and create a script in .bin/ (or whatever) #!/bin/sh tmp_m="/tmp/movies/$(mktemp -d)" cp "$@" $tmp_m mpv $tmp_m rm $tmp_m please note i have not tried the script but it probably works

anotherhue 2y ago

mpv has --cache-size (or something) that you can set at a few GB. If you run out of ram it should swap to your ssd. Edit: demuxer-max-bytes=2147483647

2OEH8eoCRo0 2y ago

You could use overlayfs with the upper layer on the SSD "scratch" and trigger a write operation

madeofpalk 2y ago

This is essentially what macOS's Fusion Drive is/was https://en.wikipedia.org/wiki/Fusion_Drive https://en.wikipedia.org/wiki/Fusion_Drive I'm unsure if they ship any macs with these anymore. I guess not since the Apple Sillicon iMacs don't have spinning hard drives?

mystified5016 2y ago

Bcachefs. You can do all sorts of really neat things. You can define pools of drives at different cache levels. You can have a bunch of mechanical drives for deep storage, some for hot storage, SSD to cache recently read files, then write-through from the SSD down to mechanical drives, either immediately or after a delay. It's pretty much everything I could wish for from a filesystem, though I haven't actually taken the time to try it out yet. AFAIK it's still somewhat experimental, more or less in beta.

cassianoleal 2y ago
Is it fully merged to the kernel? I remember a few weeks ago some drama between Kent Overstreet and Linus about it but I didn't go into details. Has that been resolved? Edit: or maybe it was some drama over userspace tooling, I can't remember tbh.
- MattTheRealOne 2y ago
  Bcachefs is fully merged into the kernel since 6.7. The drama was around Overstreet trying to merge significant code changes to a Release Candidate kernel that should only be receiving minor bug fixes at that stage of the development process. It was developer communication issues, not anything that impacts the user. The changes will just have to wait until the next kernel version.
  - cassianoleal 2y ago
    I think this is what I was recalling. https://jonathancarter.org/2024/08/29/orphaning-bcachefs-tools-in-debian/ https://jonathancarter.org/2024/08/29/orphaning-bcachefs-too...
ThePowerOfFuet 2y ago
Bcachefs has not proven that it can be trusted to not eat data, and has also recently undergone an on-disk layout change which further demonstrates its lack of maturity.

throw0101d 2y ago

> I don't know if this exists or not, but I'd like to try something like a fuse filesystem which can transparently copy a file to a fast scratch SSD when it is first accessed. ZFS has caching for writes (SLOG)[0][1] and reads (L2ARC),[2][3] which was introduced many years ago when HDDs were cheap and flash was still very, very expensive: * https://www.brendangregg.com/blog/2009-10-08/hybrid-storage-pool-top-speeds.html https://www.brendangregg.com/blog/2009-10-08/hybrid-storage-... [0] https://openzfs.github.io/openzfs-docs/man/master/7/zpoolconcepts.7.html#Intent_Log https://openzfs.github.io/openzfs-docs/man/master/7/zpoolcon... [1] https://openzfs.github.io/openzfs-docs/man/master/8/zpool-create.8.html#Example_6_:_Creating_a_ZFS_Pool_with_Mirrored_Separate_Intent_Logs https://openzfs.github.io/openzfs-docs/man/master/8/zpool-cr... [2] https://openzfs.github.io/openzfs-docs/man/master/7/zpoolconcepts.7.html#Cache_Devices https://openzfs.github.io/openzfs-docs/man/master/7/zpoolcon... [3] https://openzfs.github.io/openzfs-docs/man/master/8/zpool-add.8.html#Example_2_:_Adding_Cache_Devices_to_a_ZFS_Pool https://openzfs.github.io/openzfs-docs/man/master/8/zpool-ad...

aidenn0 2y ago
I should point out that the SLOG only caches synchronous writes, which are written twice with ZFS. Also, the L2ARC is great, but does still have RAM overhead. There are also useful tunables. I had a workload on a RAM limited machine where directory walking was common, but data reads were fairly random and a L2ARC configured for metadata only speed it up by a large amount.
- kaliszad 2y ago
  You can also have a special metadata only/ small files dedicated special VDEV. ZFS can pull of many tricks if you configure it well. Of course, the L2ARC is better, if you don't trust the caching device that much (e.g. only have a single SSD).
  - flemhans 2y ago
    You can also configure an L2ARC to only hold metadata, which is safer than a metadata device.

_hyn3 2y ago

cat filename > /dev/null Reads the entire file into the OS buffer.

sulandor 2y ago

add this to mpv.conf cache=yes demuxer-max-bytes=5G demuxer-max-back-bytes=5G