6 ms·

I think preventing alarm fatigue is a very good reason to fix issues. But 5% free is very low. You may want to use every single byte you feel you paid for, but

I think preventing alarm fatigue is a very good reason to fix issues. But 5% free is very low. You may want to use every single byte you feel you paid for, but allocation algorithms really break down when free space gets so low. Remember that there's not just a solid chunk of that 5% sitting around at the end of the space. That's added up over all the holes across the volume. At 20-25% free, you should already be looking at whether to get more disks and/or deciding what stuff you don't actually need to store on this volume. So a hard alarm at 5% is not unreasonable, though there should also be a way to set a soft alarm before then.

sitkack 2y ago
[flagged]
kimixa 2y ago
Also Synology use btrfs, a copy-on-write filesystem - that means there are operations that you might not expect that require allocation of new blocks - like any write, even if overwriting an existing file's data. And "unexpected" failure paths like that are often poorly tested in apps.
leptons 2y ago
No matter how many TB of online HD storage I have, hard disks are just a temporary buffer for my tape drives.
- worthless-trash 2y ago
  This is an area that i'm quickly growing into, what are you curently using and what should I stay away from ?
  - leptons 2y ago
    I got a used internal LTO5 tape drive on eBay for about $150, and then an HBA card to connect it to for about $25 or $30. I bought some LTO5 tapes, and typically I pay about $3.50/TB on eBay for new/used tapes. Many sellers charge far more for tapes, but occasionally I find a good deal. Most tapes are not used very much and have lots of life left in them (they have a chip inside the tape that tracks usage). Then I scored another 3 used LTO5 tape drives on eBay for about $100, they all worked. I mainly use 1 tape drive. I have it running on an Intel i5 system with an 8-drive RAID10 array (cheap used drives, with a $50 9260-8i hardware RAID card), which acts as my "offsite" backup out in my detached garage - it's off most of the time (cold storage?) unless I'm running a backup. I can loose up to 2 drives without losing any data, and it's been running really well for years. I have 3 of these RAID setups in 3 different systems, they work great with the cheapest used drives from Amazon. I'm not looking for high performance, I just need redundancy. I've had to replace maybe 3 drives across all 3 systems due to failure over the last 7 years. On Windows the tape drive with LTFS was not working well, I think due to Windows Defender trying to test the files as it was writing them, causing a lot of "shoeshining" of the tape, but I think Windows Defender can be disabled. But I bought tape backup software from https://www.iperiusbackup.com https://www.iperiusbackup.com - it just works and makes backups simple to set up and run. I always verify the backup. If something is really important I'll back up to at least 2 tapes. Some really important stuff I will generate parity files (win WinPar) and put those on tape too. Non-encrypted the drive runs at the full 140MB/s, but with encryption it runs at about 60MB/s, because I guess the tape drive is doing the encryption. I love it, it has changed my data-hoarding life. At $3.50/TB and 140MB/s and 1.5TB per tape, it can't be beat by DVD-R or hard drives for backup. Used LTO5 is really in a sweet spot right now on eBay, but LTO6 is looking good too recently (2.5TB/tape). LTO6 drives can read LTO5 tapes, so there's a pretty easy upgrade path. I also love that there is a physical write-protect switch on the tapes, which hard drives don't have. If you plug in a hard drive to an infected system, that hard drive could easily be compromised if you don't know your system is infected.
    - worthless-trash 2y ago
      Thank you for the detailed write up, you've started me on the path!
- buserror 2y ago
  At home I have a 48xLTO5 changer with 4 drives (I picked for a song a while back! I actually don't need it but heck, it has a ROBOT ARM), and at work I'm currently provisioning a 96 LTO 9 tape drive dual-rack. With 640 tapes available :-) I'm a STRONG believer in tapes! Even LTO 5 gives you a very cheap 1.5TB of clean, pretty much bulletproof storage.. You can pick a drive (with a SAS HBA card) for less than $200, there is zero driver issue (SCSI, baby); the linux tape changer code is stable since 1997 (with a port to VMS!). Tape FTW :-)
  - leptons 2y ago
    I don't have one but I'd definitely take a tape changer if it weren't too expensive. It would be amazing to have 72TB of storage just waiting to be filled, without needing to go out into my garage to load a tape up. LTO tapes have really changed my life, or at least my mental health. Easy and robust backup has been elusive. DVD-R was just not doing it for me. Hard drives are too expensive and lacked robustness. My wife is a pro photographer so the never-ending data dumps had filled up all our hard drives, and spending hundreds of dollars more on another 2-disk mirror RAID, and then another, and another was just stupid. Most of the data will only need to be accessed rarely, but we still want to keep it. I lost sleep over the mountains of data we were hoarding on hard drives. I've had too many hard drives just die, including RAIDs being corrupted. LTO tape changed all of that. It's relatively cheap, and pretty easy and fast compared to all the other solutions. It's no wonder it's still being used in data centers. I love all the data center hand-me-downs that flood eBay. And I do love hearing the tapes whir, it makes me smile.
j1elo 2y ago
5% of my 500 GB is 25 GB, which is already a lot of space but understandable. Not many things would fit in there nowadays. But 5% of a 5 TB volume is 250 GB, that's the size of my whole system disk! Probably not so understandable by the lay person.
- kbolino 2y ago
  This is partly why SSDs just lie nowadays and tell you they only have 75-90% of the capacity that is actually built into them. You can't directly access that excess capacity but the drive controller can when it needs to (primarily to extend the life of the drive). Some filesystems do stake out a reservation but I don't think any claim one as large as 5% (not counting the effect of fixed-size reservations on very small volumes). Maybe they ought to, as a way of managing expectations better. For people who used computers when the disks were a lot smaller, or who primarily deal in files much much smaller than the volumes they're stored on, the absolute size of a percentage reservation can seem quite large. And, in certain cases, for certain workloads, the absolute size may actually be more important than the relative size. But most file systems are designed for general use and, across a variety of different workloads, spare capacity and the impact of (not) keeping it open is more about relative than absolute sizes. Besides fragmentation, there's also bookkeeping issues, like adding one more file to a directory cascading into a complete rearrangement of the internal data structures.
  - fc417fc802 2y ago
    > sare capacity and the impact of (not) keeping it open is more about relative than absolute sizes I don't think this is correct. At least btrfs works with slabs in the 1 GB range IIRC. One of my current filesystmes is upwards of 20 TB. Reserving 5% of that would mean reserving 1 TB. I'll likely double it in the near future, at which point it would mean reserving 2 TB. At least for my use case those numbers are completely absurd.
    - kbolino 2y ago
      We're not talking about optical discs or backup tapes which usually get written in full in a single session. Hard drive storage in general use is constantly changing. As such, fragmentation is always there; absolute disk sizes don't change the propensity for typical workloads to produce fragmentation. A modern file system is not merely a bucket of files, it is a database that manages directories, metadata, files, and free space. If you mix small and large directories, small and large files, creation and deletion of files, appending to or truncating from existing files, etc., you will get fragmentation. When you get close to full, everything gets slower. Files written early in the volume's life and which haven't been altered may remain fast to access, but creating new files will be slower, and reading those files afterward will be slower too. Large directories follow the same rules as larger files, they can easily get fragmented (or, if they must be kept compact, then there will be time spent on defragmentation). If your free space is spread across the volume in small chunks, and at 95% full it almost certainly will be, then the fact that the sum of it is 1 TB confers no benefit by dint of absolute size. Even if you had SSDs accessed with NVMe, fragmentation would still be an issue, since the file system must still store lists or trees of all the fragments, and accessing those data structures still takes more time as they grow. But most NAS setups are still using conventional spinning-platter hard drives, where the effects of fragmentation are massively amplified. A 7200 RPM drive takes 8.33 ms to complete one rotation. No improvements in technology have any effect on this number (though there used to be faster-spinning drives on the market). The denser storage of modern drives improves throughput when reading sequential data, but not random seek times. Fragmentation increases the frequency of random seeks relative to sequential access. Capacity issues tend to manifest as performance cliffs, whereby operations which used to take e.g. 5 ms suddenly take 500 or 5000. Everything can seem fine one day and then not the next, or fine on some operations but terrible on others. Of course, you should be free to (ab)use the things you own as much as you wish. But make no mistake, 5% free is deep into abuse territory. Also, as a bit of an aside, a 20 TB volume split into 1 GB slabs means there's 20,000 slabs. That's about the same as the number of 512-byte sectors in a 10 MB hard drive, which was the size of the first commercially available consumer hard drive for the IBM PC in the late 1980s. That's just a coincidence of course, but I find it funny that the numbers are so close. Now, I assume the slabs are allocated from the start of the volume forward, which means external slab fragmentation is nonexistent (unless slabs can also be freed). But unless you plan to create no more than 20,000 files, each exactly 1 GB in size, in the root directory only, and never change anything on the volume ever again, then internal slab fragmentation will occur all the same.
runamok 2y ago
100%. Those disks are likely working much harder moving the head all over the place to find those empty spaces when it writes.
- gambiting 2y ago
  ....do you think the drive doesn't know where the empty space actually is?
  - genewitch 2y ago
    I think you'll find that the word "find" doesn't mean "has to search", like one can find their nose in the middle of their face, if one desires. Change the word to "seek" and it may make more sense.
    - fc417fc802 2y ago
      It makes more sense but it's not true for the modern CoW filesystems that I'm familiar with. Those allocate free space in slabs that they write to sequentially.
      - kbolino 2y ago
        If you delete files, or append to existing files, then the promises of the initial allocation strategy go out the window.
      - kbolino 2y ago
        Also, CoW isn't some kind of magic. There are two meanings I can think of here: A) When you modify a file, everything including the parts you didn't change is copied to a new location. I don't think this is how btrfs works. B) Allocated storage is never overwritten, but modifying parts of a file won't copy the unchanged parts. A file's content is composed of a sequence (list or tree) of extents (contiguous, variable-length runs of 1 or more blocks) and if you change part of the file, you first create a new disconnected extent somewhere and write to that. Then, when you're done writing, the file's existing extent limits are resized so that the portion you changed is carved out, and finally the sequence of extents is set to {old part before your change}, {your change}, {old part after your change}. This leaves behind an orphaned extent, containing the old content of the part you changed, which is now free. From what evidence I can quickly gather, this is how btrfs works. Compared to an ordinary file system, where changes that don't increase the size of a file are written directly to the original blocks, it should be fairly obvious that strategy (B) results in more fragmentation, since both appending to and simply modifying a file causes a new allocation, and the latter leaves a new hole behind. While strategy (A) with contiguous allocation could eliminate internal (file) fragmentation, it would also be much more sensitive to external (free space) fragmentation, requiring lots of spare capacity and/or frequent defrag. Either way, the use of CoW means you need more spare capacity, not less. It's designed to allow more work to be done in parallel, as fits modern hardware and software better, under the assumption that there's also ample amounts of extra space to work with. Denying it that extra space is going to make it suffer worse than a non-CoW file system would.

sitkack 2y ago

[flagged]

kimixa 2y ago

Also Synology use btrfs, a copy-on-write filesystem - that means there are operations that you might not expect that require allocation of new blocks - like any write, even if overwriting an existing file's data. And "unexpected" failure paths like that are often poorly tested in apps.

leptons 2y ago

No matter how many TB of online HD storage I have, hard disks are just a temporary buffer for my tape drives.

worthless-trash 2y ago
This is an area that i'm quickly growing into, what are you curently using and what should I stay away from ?
- leptons 2y ago
  I got a used internal LTO5 tape drive on eBay for about $150, and then an HBA card to connect it to for about $25 or $30. I bought some LTO5 tapes, and typically I pay about $3.50/TB on eBay for new/used tapes. Many sellers charge far more for tapes, but occasionally I find a good deal. Most tapes are not used very much and have lots of life left in them (they have a chip inside the tape that tracks usage). Then I scored another 3 used LTO5 tape drives on eBay for about $100, they all worked. I mainly use 1 tape drive. I have it running on an Intel i5 system with an 8-drive RAID10 array (cheap used drives, with a $50 9260-8i hardware RAID card), which acts as my "offsite" backup out in my detached garage - it's off most of the time (cold storage?) unless I'm running a backup. I can loose up to 2 drives without losing any data, and it's been running really well for years. I have 3 of these RAID setups in 3 different systems, they work great with the cheapest used drives from Amazon. I'm not looking for high performance, I just need redundancy. I've had to replace maybe 3 drives across all 3 systems due to failure over the last 7 years. On Windows the tape drive with LTFS was not working well, I think due to Windows Defender trying to test the files as it was writing them, causing a lot of "shoeshining" of the tape, but I think Windows Defender can be disabled. But I bought tape backup software from https://www.iperiusbackup.com https://www.iperiusbackup.com - it just works and makes backups simple to set up and run. I always verify the backup. If something is really important I'll back up to at least 2 tapes. Some really important stuff I will generate parity files (win WinPar) and put those on tape too. Non-encrypted the drive runs at the full 140MB/s, but with encryption it runs at about 60MB/s, because I guess the tape drive is doing the encryption. I love it, it has changed my data-hoarding life. At $3.50/TB and 140MB/s and 1.5TB per tape, it can't be beat by DVD-R or hard drives for backup. Used LTO5 is really in a sweet spot right now on eBay, but LTO6 is looking good too recently (2.5TB/tape). LTO6 drives can read LTO5 tapes, so there's a pretty easy upgrade path. I also love that there is a physical write-protect switch on the tapes, which hard drives don't have. If you plug in a hard drive to an infected system, that hard drive could easily be compromised if you don't know your system is infected.
  - worthless-trash 2y ago
    Thank you for the detailed write up, you've started me on the path!
buserror 2y ago
At home I have a 48xLTO5 changer with 4 drives (I picked for a song a while back! I actually don't need it but heck, it has a ROBOT ARM), and at work I'm currently provisioning a 96 LTO 9 tape drive dual-rack. With 640 tapes available :-) I'm a STRONG believer in tapes! Even LTO 5 gives you a very cheap 1.5TB of clean, pretty much bulletproof storage.. You can pick a drive (with a SAS HBA card) for less than $200, there is zero driver issue (SCSI, baby); the linux tape changer code is stable since 1997 (with a port to VMS!). Tape FTW :-)
- leptons 2y ago
  I don't have one but I'd definitely take a tape changer if it weren't too expensive. It would be amazing to have 72TB of storage just waiting to be filled, without needing to go out into my garage to load a tape up. LTO tapes have really changed my life, or at least my mental health. Easy and robust backup has been elusive. DVD-R was just not doing it for me. Hard drives are too expensive and lacked robustness. My wife is a pro photographer so the never-ending data dumps had filled up all our hard drives, and spending hundreds of dollars more on another 2-disk mirror RAID, and then another, and another was just stupid. Most of the data will only need to be accessed rarely, but we still want to keep it. I lost sleep over the mountains of data we were hoarding on hard drives. I've had too many hard drives just die, including RAIDs being corrupted. LTO tape changed all of that. It's relatively cheap, and pretty easy and fast compared to all the other solutions. It's no wonder it's still being used in data centers. I love all the data center hand-me-downs that flood eBay. And I do love hearing the tapes whir, it makes me smile.

j1elo 2y ago

5% of my 500 GB is 25 GB, which is already a lot of space but understandable. Not many things would fit in there nowadays. But 5% of a 5 TB volume is 250 GB, that's the size of my whole system disk! Probably not so understandable by the lay person.

kbolino 2y ago
This is partly why SSDs just lie nowadays and tell you they only have 75-90% of the capacity that is actually built into them. You can't directly access that excess capacity but the drive controller can when it needs to (primarily to extend the life of the drive). Some filesystems do stake out a reservation but I don't think any claim one as large as 5% (not counting the effect of fixed-size reservations on very small volumes). Maybe they ought to, as a way of managing expectations better. For people who used computers when the disks were a lot smaller, or who primarily deal in files much much smaller than the volumes they're stored on, the absolute size of a percentage reservation can seem quite large. And, in certain cases, for certain workloads, the absolute size may actually be more important than the relative size. But most file systems are designed for general use and, across a variety of different workloads, spare capacity and the impact of (not) keeping it open is more about relative than absolute sizes. Besides fragmentation, there's also bookkeeping issues, like adding one more file to a directory cascading into a complete rearrangement of the internal data structures.
- fc417fc802 2y ago
  > sare capacity and the impact of (not) keeping it open is more about relative than absolute sizes I don't think this is correct. At least btrfs works with slabs in the 1 GB range IIRC. One of my current filesystmes is upwards of 20 TB. Reserving 5% of that would mean reserving 1 TB. I'll likely double it in the near future, at which point it would mean reserving 2 TB. At least for my use case those numbers are completely absurd.
  - kbolino 2y ago
    We're not talking about optical discs or backup tapes which usually get written in full in a single session. Hard drive storage in general use is constantly changing. As such, fragmentation is always there; absolute disk sizes don't change the propensity for typical workloads to produce fragmentation. A modern file system is not merely a bucket of files, it is a database that manages directories, metadata, files, and free space. If you mix small and large directories, small and large files, creation and deletion of files, appending to or truncating from existing files, etc., you will get fragmentation. When you get close to full, everything gets slower. Files written early in the volume's life and which haven't been altered may remain fast to access, but creating new files will be slower, and reading those files afterward will be slower too. Large directories follow the same rules as larger files, they can easily get fragmented (or, if they must be kept compact, then there will be time spent on defragmentation). If your free space is spread across the volume in small chunks, and at 95% full it almost certainly will be, then the fact that the sum of it is 1 TB confers no benefit by dint of absolute size. Even if you had SSDs accessed with NVMe, fragmentation would still be an issue, since the file system must still store lists or trees of all the fragments, and accessing those data structures still takes more time as they grow. But most NAS setups are still using conventional spinning-platter hard drives, where the effects of fragmentation are massively amplified. A 7200 RPM drive takes 8.33 ms to complete one rotation. No improvements in technology have any effect on this number (though there used to be faster-spinning drives on the market). The denser storage of modern drives improves throughput when reading sequential data, but not random seek times. Fragmentation increases the frequency of random seeks relative to sequential access. Capacity issues tend to manifest as performance cliffs, whereby operations which used to take e.g. 5 ms suddenly take 500 or 5000. Everything can seem fine one day and then not the next, or fine on some operations but terrible on others. Of course, you should be free to (ab)use the things you own as much as you wish. But make no mistake, 5% free is deep into abuse territory. Also, as a bit of an aside, a 20 TB volume split into 1 GB slabs means there's 20,000 slabs. That's about the same as the number of 512-byte sectors in a 10 MB hard drive, which was the size of the first commercially available consumer hard drive for the IBM PC in the late 1980s. That's just a coincidence of course, but I find it funny that the numbers are so close. Now, I assume the slabs are allocated from the start of the volume forward, which means external slab fragmentation is nonexistent (unless slabs can also be freed). But unless you plan to create no more than 20,000 files, each exactly 1 GB in size, in the root directory only, and never change anything on the volume ever again, then internal slab fragmentation will occur all the same.

runamok 2y ago

100%. Those disks are likely working much harder moving the head all over the place to find those empty spaces when it writes.

gambiting 2y ago
....do you think the drive doesn't know where the empty space actually is?
- genewitch 2y ago
  I think you'll find that the word "find" doesn't mean "has to search", like one can find their nose in the middle of their face, if one desires. Change the word to "seek" and it may make more sense.
  - fc417fc802 2y ago
    It makes more sense but it's not true for the modern CoW filesystems that I'm familiar with. Those allocate free space in slabs that they write to sequentially.
    - kbolino 2y ago
      If you delete files, or append to existing files, then the promises of the initial allocation strategy go out the window.
    - kbolino 2y ago
      Also, CoW isn't some kind of magic. There are two meanings I can think of here: A) When you modify a file, everything including the parts you didn't change is copied to a new location. I don't think this is how btrfs works. B) Allocated storage is never overwritten, but modifying parts of a file won't copy the unchanged parts. A file's content is composed of a sequence (list or tree) of extents (contiguous, variable-length runs of 1 or more blocks) and if you change part of the file, you first create a new disconnected extent somewhere and write to that. Then, when you're done writing, the file's existing extent limits are resized so that the portion you changed is carved out, and finally the sequence of extents is set to {old part before your change}, {your change}, {old part after your change}. This leaves behind an orphaned extent, containing the old content of the part you changed, which is now free. From what evidence I can quickly gather, this is how btrfs works. Compared to an ordinary file system, where changes that don't increase the size of a file are written directly to the original blocks, it should be fairly obvious that strategy (B) results in more fragmentation, since both appending to and simply modifying a file causes a new allocation, and the latter leaves a new hole behind. While strategy (A) with contiguous allocation could eliminate internal (file) fragmentation, it would also be much more sensitive to external (free space) fragmentation, requiring lots of spare capacity and/or frequent defrag. Either way, the use of CoW means you need more spare capacity, not less. It's designed to allow more work to be done in parallel, as fits modern hardware and software better, under the assumption that there's also ample amounts of extra space to work with. Denying it that extra space is going to make it suffer worse than a non-CoW file system would.