5 ms·

LVM is a very cool thing that I feel is a bit underappreciated. I've always built my systems like this: two hard disks => RAID1 => LVM VG1. nvme => LVM VG2 B

LVM is a very cool thing that I feel is a bit underappreciated. I've always built my systems like this: two hard disks => RAID1 => LVM VG1. nvme => LVM VG2 Because at some point I decided LVM is just like flexible partitions and didn't pay much attention to the huge amounts of other stuff in the manpages. Turns out that's a pretty crappy way of doing things. LVM can do RAID, can deal with moving things across devices, and has many other fancy features. So instead of the above, stick it all into the same VG without RAID on top. You can RAID individual LVs. Some RAID1, some RAID5, some RAID0, some nothing, and pick on which disks stuff goes. You can add integrity checking which is not something MD RAID does. You can do caching. You can do writable snapshots. You can do thin provisioning. It's extremely flexible tech. The only thing I'd recommend staying away from is thin provisioning unless it's necessary. The performance impact is significant, virt-manager for some reason doesn't see thin LVs as legitimate disks (can be worked around easily though) and thin provisioning seems to be able to develop faults of its own. I've had one old setup where one block was somehow not readable and it seems to be a thin provisioning thing -- it doesn't say the underlying disk is bad, it's something more along the lines of the metadata being broken. To upgrade you can do things like moving LVs from one disk to another to free it up and remove it. Disks of different sizes are much less of a problem than with standard RAID, too.

linsomniac 1y ago
~20 years ago I was helping someone with their storage server for their mail server, it had 8x 10K discs in a RAID-1. It was always struggling to keep up and often couldn't keep up. Intel had just come out with their SSDs, and I had a friend with access to one that was larger in size than the array. I shipped this little laptop drive off to get installed in the storage server. Once the SSD was on the server, I did a "pvmove" to move the data from spinning array to SSD, while the system was up and running. Over the next several hours the load on the server dropped. It was comical that these 8 hard drives could be replaced by something smaller than my wallet. But pvmove was the other star of this show.
- MrDrMcCoy 1y ago
  If those drives were short-stroked, they probably could have kept up with that SSD, though at a reduced capacity. SSD would probably have a lower power bill over its life, though. I did some calculations for an array of short-stroked 15k SAS disks to replace a consumer 4GB SSD for a write-intensive app that chews through SSDs, and its performance would be in spitting distance of the SSD. Ended up not doing it due to likely not having any parts availability for 15k SAS drives in the not too distant future.
  - linsomniac 1y ago
    Except that we would have needed a lot more than the 8 drives then, to keep the same capacity. I think it was 1TB of storage around 1999. For mail storage it unfortunately needed both capacity and seek latency.
    - MrDrMcCoy 1y ago
      More drives would be acceptable if it meant not having to replace all the storage every few months.
creshal 1y ago
LVM isn't underrated, it's obsolete. ZFS in particular has none of the many, maaany problems you run into trying to make LVM work reliably in practice (no self-corrupting snapshots, e.g.).
- SoftTalker 1y ago
  My personal experience is the opposite, but to be fair I gave up on ZFS years ago, it may be (probably is) better now.
- QuantumGood 1y ago
  I appreciate these points of view, but it is of course not optimal when you see people simply making polar opposite statements with neither making an attempt at explanation.
  - creshal 1y ago
    LVM was a decent idea at the time, which was the early 1980s. But that locked in many architectural decisions that make it incredibly painful to use in practise, and its devs follow the "it was hard to write, so it should be hard to use" mentality, which make it even more painful. Self-corrupting snapshots are considered a skill issue on the user's part, e.g., while ZFS simply doesn't allow users to shoot themselves in the foot like that. (And you rarely even need block-level snapshots in the first place!) Encryption, data integrity, redundancy, are all out scope for LVM, so now you need a RAID layer and a verification layer and an encryption layer and if you stack them wrong, everything breaks. Skill issue! And not LVM's problem in the first place. ZFS doesn't make you jump through hoops to manually calculate sector offsets to figure out how you can make an SSD cache that doesn't blow up in your face either. So, no, LVM isn't underrated. It's a painful relic of a bygone age andif anything, overrated.
    - QuantumGood 1y ago
      I see you were downvoted. Interestingly, on HN downvotes are considered as appropriate as explanations, from what I've read. Sure, you should be allowed to downvote, and can't be required to explain, but in a thread about lack of explanation, downvoting without explanation is hard to understand.
SoftTalker 1y ago
AFAIK LVM RAID is mdraid under the covers, I guess the advantage of using LVM to configure and manage RAID is doing it all with one set of tools, assuming that the LVM utilities offer everything you need. I've always used mdraid for that, and (sometimes) LVM on top of the mdraid devices.
- raffraffraff 1y ago
  Oh so it doesn't avoid bit rot the way ZFS does (by checksumming)? Mdadm, not being a filesystem, doesn't concern itself with file integrity. I don't suppose LVM would either :(
  - SoftTalker 1y ago
    I think that's right. Even with RAID 4/5 I think the parity is used to reconstuct missing data if a device fails, not to verify data integrity or detect bitrot.
  - dale_glass 1y ago
    unlike mdadm, LVM does have checksumming. https://docs.redhat.com/fr/documentation/red_hat_enterprise_linux/9/html/configuring_and_managing_logical_volumes/creating-a-raid-lv-with-dm-integrity_configuring-raid-logical-volumes https://docs.redhat.com/fr/documentation/red_hat_enterprise_...
    - SoftTalker 1y ago
      Wow, didn't know about that. Though it seems to require additional metadata, not just the parity data for the RAID level. From the Ubuntu 22 lvmraid(7) man page: When creating a RAID LV with integrity, or adding integrity, space is required for integrity metadata. Every 500MB of LV data requires an additional 4MB to be allocated for integrity metadata, for each RAID image. Also: The following are not yet permitted on RAID LVs with integrity: lvreduce, pvmove, snapshots, splitmirror, raid syncaction commands, raid rebuild. The typical workaround for these seems to be remove the integrity, make the change, then add/reinitialize the integrity metadata.
raffraffraff 1y ago
It's a bit pain in three neck for me to rebuild my NAS using LVM or ZFS (to handle RAID in place of mdadm) but if I was starting over, I'd use either of them to avoid bit rot. As it is though, I'm happy with RAID 10 on SATA HDDs a and RAID 1 on NVME SDDs, using bcache to create a single volume using both, and then ext4 on top. With sufficient SMART monitoring and backups, it'll do.
RulerOf 1y ago
> Because at some point I decided LVM is just like flexible partitions and didn't pay much attention to the huge amounts of other stuff in the manpages. I remember when I was first learning to use Linux, I was mystified by the convention of LVM on top of md. I was reading the LVM man pages and it was clear I could just do all that stuff with LVM without having to involve another abstraction layer. But I had a hardware raid controller and just used that instead.
- dale_glass 1y ago
  It depends on when you started, LVM changed a fair amount over time and the earlier versions had less functionality. So a lot of people would do like I did -- make an initial decision and then stick with it for two decades, way past of it making any sense. That kind of thing also happens with documentation.

linsomniac 1y ago

~20 years ago I was helping someone with their storage server for their mail server, it had 8x 10K discs in a RAID-1. It was always struggling to keep up and often couldn't keep up. Intel had just come out with their SSDs, and I had a friend with access to one that was larger in size than the array. I shipped this little laptop drive off to get installed in the storage server. Once the SSD was on the server, I did a "pvmove" to move the data from spinning array to SSD, while the system was up and running. Over the next several hours the load on the server dropped. It was comical that these 8 hard drives could be replaced by something smaller than my wallet. But pvmove was the other star of this show.

MrDrMcCoy 1y ago
If those drives were short-stroked, they probably could have kept up with that SSD, though at a reduced capacity. SSD would probably have a lower power bill over its life, though. I did some calculations for an array of short-stroked 15k SAS disks to replace a consumer 4GB SSD for a write-intensive app that chews through SSDs, and its performance would be in spitting distance of the SSD. Ended up not doing it due to likely not having any parts availability for 15k SAS drives in the not too distant future.
- linsomniac 1y ago
  Except that we would have needed a lot more than the 8 drives then, to keep the same capacity. I think it was 1TB of storage around 1999. For mail storage it unfortunately needed both capacity and seek latency.
  - MrDrMcCoy 1y ago
    More drives would be acceptable if it meant not having to replace all the storage every few months.

creshal 1y ago

LVM isn't underrated, it's obsolete. ZFS in particular has none of the many, maaany problems you run into trying to make LVM work reliably in practice (no self-corrupting snapshots, e.g.).

SoftTalker 1y ago
My personal experience is the opposite, but to be fair I gave up on ZFS years ago, it may be (probably is) better now.
QuantumGood 1y ago
I appreciate these points of view, but it is of course not optimal when you see people simply making polar opposite statements with neither making an attempt at explanation.
- creshal 1y ago
  LVM was a decent idea at the time, which was the early 1980s. But that locked in many architectural decisions that make it incredibly painful to use in practise, and its devs follow the "it was hard to write, so it should be hard to use" mentality, which make it even more painful. Self-corrupting snapshots are considered a skill issue on the user's part, e.g., while ZFS simply doesn't allow users to shoot themselves in the foot like that. (And you rarely even need block-level snapshots in the first place!) Encryption, data integrity, redundancy, are all out scope for LVM, so now you need a RAID layer and a verification layer and an encryption layer and if you stack them wrong, everything breaks. Skill issue! And not LVM's problem in the first place. ZFS doesn't make you jump through hoops to manually calculate sector offsets to figure out how you can make an SSD cache that doesn't blow up in your face either. So, no, LVM isn't underrated. It's a painful relic of a bygone age andif anything, overrated.
  - QuantumGood 1y ago
    I see you were downvoted. Interestingly, on HN downvotes are considered as appropriate as explanations, from what I've read. Sure, you should be allowed to downvote, and can't be required to explain, but in a thread about lack of explanation, downvoting without explanation is hard to understand.

SoftTalker 1y ago

AFAIK LVM RAID is mdraid under the covers, I guess the advantage of using LVM to configure and manage RAID is doing it all with one set of tools, assuming that the LVM utilities offer everything you need. I've always used mdraid for that, and (sometimes) LVM on top of the mdraid devices.

raffraffraff 1y ago
Oh so it doesn't avoid bit rot the way ZFS does (by checksumming)? Mdadm, not being a filesystem, doesn't concern itself with file integrity. I don't suppose LVM would either :(
- SoftTalker 1y ago
  I think that's right. Even with RAID 4/5 I think the parity is used to reconstuct missing data if a device fails, not to verify data integrity or detect bitrot.
- dale_glass 1y ago
  unlike mdadm, LVM does have checksumming. https://docs.redhat.com/fr/documentation/red_hat_enterprise_linux/9/html/configuring_and_managing_logical_volumes/creating-a-raid-lv-with-dm-integrity_configuring-raid-logical-volumes https://docs.redhat.com/fr/documentation/red_hat_enterprise_...
  - SoftTalker 1y ago
    Wow, didn't know about that. Though it seems to require additional metadata, not just the parity data for the RAID level. From the Ubuntu 22 lvmraid(7) man page: When creating a RAID LV with integrity, or adding integrity, space is required for integrity metadata. Every 500MB of LV data requires an additional 4MB to be allocated for integrity metadata, for each RAID image. Also: The following are not yet permitted on RAID LVs with integrity: lvreduce, pvmove, snapshots, splitmirror, raid syncaction commands, raid rebuild. The typical workaround for these seems to be remove the integrity, make the change, then add/reinitialize the integrity metadata.

raffraffraff 1y ago

It's a bit pain in three neck for me to rebuild my NAS using LVM or ZFS (to handle RAID in place of mdadm) but if I was starting over, I'd use either of them to avoid bit rot. As it is though, I'm happy with RAID 10 on SATA HDDs a and RAID 1 on NVME SDDs, using bcache to create a single volume using both, and then ext4 on top. With sufficient SMART monitoring and backups, it'll do.

RulerOf 1y ago

> Because at some point I decided LVM is just like flexible partitions and didn't pay much attention to the huge amounts of other stuff in the manpages. I remember when I was first learning to use Linux, I was mystified by the convention of LVM on top of md. I was reading the LVM man pages and it was clear I could just do all that stuff with LVM without having to involve another abstraction layer. But I had a hardware raid controller and just used that instead.

dale_glass 1y ago
It depends on when you started, LVM changed a fair amount over time and the earlier versions had less functionality. So a lot of people would do like I did -- make an initial decision and then stick with it for two decades, way past of it making any sense. That kind of thing also happens with documentation.