8 ms·

Well I used btrfs as a root filesystem for quite a while, until I realized it was pig slow for sync() -- I mean, it would take AGES to do and apt-get upgrade fo

Well I used btrfs as a root filesystem for quite a while, until I realized it was pig slow for sync() -- I mean, it would take AGES to do and apt-get upgrade for example. I ended up having to do some tasks using 'eatmydata' [0] to make it all better, risking filesystem corruption in trade for speed. Also, at the time, there was no functioning fsck. So I moved back safely to ext4 and never looked back! [0]: https://www.flamingspork.com/projects/libeatmydata/ https://www.flamingspork.com/projects/libeatmydata/

buster 9y ago
Over the recent years on every new laptop install i switched between filesystems, so i had ext3/4, btrfs and (currently) xfs on my system. I have to say, btrfs had the most glitches (a few years back), although it worked ok'ish (no data loss). Nowadays, i must say that i very much prefer a stable filesystem with as little complicated logic as possible. I actually never use snapshots or subtrees! I never put another disk in my laptop (where would that go?!) so i don't need to do dynamic resizing (while online of course!). All this makes the filesystems a lot more complex then it has to be. I've also run into problems using ZFS on Solaris some years back which took ~2 weeks in dtracing what the hell is going on. Of course it was related to CoW. My lessons learned: Check your requirements. Will you really need and use subtrees/snapshots/XYZ on your system? Will you really need to do online-resizing? If not, just use a stable, simple filesystem. There are perfect usecases for ZFS or btrfs. But not everyone needs the advanced features.
- viraptor 9y ago
  > Will you really need and use subtrees/snapshots/XYZ on your system? It's a valid question, but not the best one. Almost nobody needs snapshots. But they make things easier. You most likely don't need a journaled fs in your laptop either (battery level notification should take care of the issues). But it does make life better. "Need" is not the threshold I'm interested in. Most features, I'd like. One feature I think I do need most is scrubbing, which is still absent from most filesystems :(
  - buster 9y ago
    "Need" as in "will you use it?". I played around with snapshots once and never really used them. So i clearly don't have a need for them on my laptop. Journaling on the other hand helps data safety a lot and i think it's not overly complex. I've had data loss happing in the past before journaling, but never again since then. So, wouldn't i need CoW for even better "data safety"? Maybe, but since i've never experienced data loss for so many years, i don't feel like the added complexity is worth it. On my laptop, for my usecase. But that's only me. Your experience may differ very much :)
    - lomnakkus 9y ago
      > "Need" as in "will you use it?". Well, many of us have experienced a botched system package upgrade or two. If the file system supports snapshots, then the package manager could automatically ensure fully atomic package upgrades. That should be reason enough, I should think. Re: The data loss issue: Yes, I've actually have XFS completely throw away a file system upon a hard power-off + boot-up cycle. (This was ages ago, I'm sure it's improved heaps since then.)
      - cyphar 9y ago
        > If the file system supports snapshots, then the package manager could automatically ensure fully atomic package upgrades. That's exactly what openSUSE / SLE do with snapper. Every upgrade or package install with YaST/zypper creates two snapshots (before/after) and you can easily rollback to an older snapshot (even doing so from GRUB). This has been enabled by default for years.
      - buster 9y ago
        Good point. I'm using Debian as my Desktop for many years. I don't remember a "botched" system package upgrade in the last 5 years, but i've probably learned over years how to handle dpkg/apt. The atomic updating is a very interesting topic and the reason why i find ostree/guix/nixos very appealing. Note that neither ostree nor guix or nixos make use of filesystem snapshots, afaik. OSTree even documents why it won't use filesystem snapshots: https://ostree.readthedocs.io/en/latest/manual/related-projects/#combining-dpkgrpm-btrfslvm https://ostree.readthedocs.io/en/latest/manual/related-proje... Debians dpkg does not use snapshots as well. So, it's a definitely a nice-to-have, but not something i need, because i can handle dpkg/apt much better then i could handle filesystem internals. That's sort of the point: I don't want the complexity in the filesystem, but i am fine with it in userspace. I can use snapshots on filesystem level. Or i can use other backup tools in userspace. While it's certainly neat that the filesystem can do that, i'm perfectly fine with handling backups on another level. Another example: It's certainly neat that there are a bunch of distributed filesystems (which by the way have A LOT of complexity and often can't handle all workloads you would expect from a filesystem). But i'd rather use either an S3-like network storage or build a system that scales well without relying on Ceph/Gluster/Quobyte/etc. For example, in a hypothetical distributed system i'd rather use Cassandra and distribute data over commodity hardware then use Ceph. I'd rather handle problems with data persistence/replication on the cassandra level then debugging on file system level. Especially, when Cassandra has a problem i'll most likely be able to access all data atleast on the filesystem level. When my filesystem is borked, i'm in a much worse situation.
- eternauta3k 9y ago
  It's ok to play with the OS in order to learn.
- lorenzfx 9y ago
  I'm using ZFS on my FreeBSD laptop. Snapshots not only make backups safer (by making sure the complete backup is taken at the same time, and by zfs sending and receiving them), the boot environments feature also make upgrading safer. I also really like that I don't need to partition my disk, if it turns out that /tmp needs > 10% of the disk for whatever reason: no problem! And as I like my data, I appreciate checksumming and copy-on-write. I haven't noticed any bad slowdowns compared to ext4 on my Debian laptop I used before.
  - rsync 9y ago
    "I'm using ZFS on my FreeBSD laptop. Snapshots not only make backups safer..." Did you know you can 'zfs send' snapshots to rsync.net ?[1][2] [1] https://arstechnica.com/information-technology/2015/12/rsync-net-zfs-replication-to-the-cloud-is-finally-here-and-its-fast/ https://arstechnica.com/information-technology/2015/12/rsync... [2] http://www.rsync.net/products/zfsintro.html http://www.rsync.net/products/zfsintro.html
- Maakuth 9y ago
  It's a good practice to use LVM there between disk partitions and volumes. It has negligible performance implications but makes things very flexible when you need to resize volumes or add space. You also gain reliable snapshotting from device mapper, although that does have some performance effects.
- pmlnr 9y ago
  My requirements: transparent compression. I'm left with: btrfs or, if possible, ZFS.
StreamBright 9y ago
We are using XFS for most of our production workloads, it turned out to be an excellent choice for most data heavy use cases. Brtfs was never an option, it is a bad idea to gamble with beta technology for data storage that a production system relies on. ext4 vs xfs is a much interesting argument, I haven't had time to follow up on this.
- jerven 9y ago
  We use XFS for sparql.uniprot.org (basically columnar database with semantic graph) there we recently retested it by accident. We use 2*4 TB consumer SSD ,ok rich consumer ;) With XFS we have 10-13% faster linear write of one big file (1.3TB) and about 20% more reads serviced per minute than EXT4. EXT4 was selected by accident instead of XFS for 1 of the 2 otherwise equal machines when upgrading these with the new 4TB SSDs instead of 1TB ones which they had before. In general I feel that the more you go into "enterprise" storage levels, the more XFS pulls ahead from the EXT family. i.e. laptops and small servers are not where the difference lies.
deleted 9y ago
[deleted]
joosters 9y ago
apt-get upgrade syncs so often that 'eatmydata' gives a noticeable speedup pretty much everywhere (I got into the habit of using it for ext3/4) I don't get why apt syncs so often - isn't the main point of log-structured file systems their ability to recover after a crash or powerloss? If so, why should you need to sync more than one every ten seconds or so?
- dsr_ 9y ago
  apt doesn't assume that you have a reliable filesystem. It assumes that you might crash at any moment, and it would be really important for you to have a consistent view of what packages are installed when you reboot.
  - joosters 9y ago
    But ext3 and more advanced filesystems have been around for almost twenty years now... it seems an odd assumption that your filesystem is unreliable on any machine that isn't completely ancient (is anyone still using ext2, for instance?)
    - yjftsjthsd-h 9y ago
      Debian calls itself "the universal operating system" and officially supports not only multiple init systems, but multiple kernels (kfreebsd); somehow I don't see it relying on specific filesystems.
    - rleigh 9y ago
      It's not about the filesystem being "unreliable". It's about having the package manager's state checkpointed so that it can recover and resume if there is e.g. power loss or any other form of interruption at any point during package installation, upgrade, removal etc. This means having all of the updated files synched on disc plus the database state which describes it. When you move to a more advanced setup such as ZFS clones, you could do the full upgrade with a cloned snapshot, and swap it with the original once the changes were complete. This would avoid the need for all intermediate syncs--if there's a problem, you can simply restart from the starting point and throw all the intermediate state away.

buster 9y ago

Over the recent years on every new laptop install i switched between filesystems, so i had ext3/4, btrfs and (currently) xfs on my system. I have to say, btrfs had the most glitches (a few years back), although it worked ok'ish (no data loss). Nowadays, i must say that i very much prefer a stable filesystem with as little complicated logic as possible. I actually never use snapshots or subtrees! I never put another disk in my laptop (where would that go?!) so i don't need to do dynamic resizing (while online of course!). All this makes the filesystems a lot more complex then it has to be. I've also run into problems using ZFS on Solaris some years back which took ~2 weeks in dtracing what the hell is going on. Of course it was related to CoW. My lessons learned: Check your requirements. Will you really need and use subtrees/snapshots/XYZ on your system? Will you really need to do online-resizing? If not, just use a stable, simple filesystem. There are perfect usecases for ZFS or btrfs. But not everyone needs the advanced features.

viraptor 9y ago
> Will you really need and use subtrees/snapshots/XYZ on your system? It's a valid question, but not the best one. Almost nobody needs snapshots. But they make things easier. You most likely don't need a journaled fs in your laptop either (battery level notification should take care of the issues). But it does make life better. "Need" is not the threshold I'm interested in. Most features, I'd like. One feature I think I do need most is scrubbing, which is still absent from most filesystems :(
- buster 9y ago
  "Need" as in "will you use it?". I played around with snapshots once and never really used them. So i clearly don't have a need for them on my laptop. Journaling on the other hand helps data safety a lot and i think it's not overly complex. I've had data loss happing in the past before journaling, but never again since then. So, wouldn't i need CoW for even better "data safety"? Maybe, but since i've never experienced data loss for so many years, i don't feel like the added complexity is worth it. On my laptop, for my usecase. But that's only me. Your experience may differ very much :)
  - lomnakkus 9y ago
    > "Need" as in "will you use it?". Well, many of us have experienced a botched system package upgrade or two. If the file system supports snapshots, then the package manager could automatically ensure fully atomic package upgrades. That should be reason enough, I should think. Re: The data loss issue: Yes, I've actually have XFS completely throw away a file system upon a hard power-off + boot-up cycle. (This was ages ago, I'm sure it's improved heaps since then.)
    - cyphar 9y ago
      > If the file system supports snapshots, then the package manager could automatically ensure fully atomic package upgrades. That's exactly what openSUSE / SLE do with snapper. Every upgrade or package install with YaST/zypper creates two snapshots (before/after) and you can easily rollback to an older snapshot (even doing so from GRUB). This has been enabled by default for years.
    - buster 9y ago
      Good point. I'm using Debian as my Desktop for many years. I don't remember a "botched" system package upgrade in the last 5 years, but i've probably learned over years how to handle dpkg/apt. The atomic updating is a very interesting topic and the reason why i find ostree/guix/nixos very appealing. Note that neither ostree nor guix or nixos make use of filesystem snapshots, afaik. OSTree even documents why it won't use filesystem snapshots: https://ostree.readthedocs.io/en/latest/manual/related-projects/#combining-dpkgrpm-btrfslvm https://ostree.readthedocs.io/en/latest/manual/related-proje... Debians dpkg does not use snapshots as well. So, it's a definitely a nice-to-have, but not something i need, because i can handle dpkg/apt much better then i could handle filesystem internals. That's sort of the point: I don't want the complexity in the filesystem, but i am fine with it in userspace. I can use snapshots on filesystem level. Or i can use other backup tools in userspace. While it's certainly neat that the filesystem can do that, i'm perfectly fine with handling backups on another level. Another example: It's certainly neat that there are a bunch of distributed filesystems (which by the way have A LOT of complexity and often can't handle all workloads you would expect from a filesystem). But i'd rather use either an S3-like network storage or build a system that scales well without relying on Ceph/Gluster/Quobyte/etc. For example, in a hypothetical distributed system i'd rather use Cassandra and distribute data over commodity hardware then use Ceph. I'd rather handle problems with data persistence/replication on the cassandra level then debugging on file system level. Especially, when Cassandra has a problem i'll most likely be able to access all data atleast on the filesystem level. When my filesystem is borked, i'm in a much worse situation.
eternauta3k 9y ago
It's ok to play with the OS in order to learn.
lorenzfx 9y ago
I'm using ZFS on my FreeBSD laptop. Snapshots not only make backups safer (by making sure the complete backup is taken at the same time, and by zfs sending and receiving them), the boot environments feature also make upgrading safer. I also really like that I don't need to partition my disk, if it turns out that /tmp needs > 10% of the disk for whatever reason: no problem! And as I like my data, I appreciate checksumming and copy-on-write. I haven't noticed any bad slowdowns compared to ext4 on my Debian laptop I used before.
- rsync 9y ago
  "I'm using ZFS on my FreeBSD laptop. Snapshots not only make backups safer..." Did you know you can 'zfs send' snapshots to rsync.net ?[1][2] [1] https://arstechnica.com/information-technology/2015/12/rsync-net-zfs-replication-to-the-cloud-is-finally-here-and-its-fast/ https://arstechnica.com/information-technology/2015/12/rsync... [2] http://www.rsync.net/products/zfsintro.html http://www.rsync.net/products/zfsintro.html
Maakuth 9y ago
It's a good practice to use LVM there between disk partitions and volumes. It has negligible performance implications but makes things very flexible when you need to resize volumes or add space. You also gain reliable snapshotting from device mapper, although that does have some performance effects.
pmlnr 9y ago
My requirements: transparent compression. I'm left with: btrfs or, if possible, ZFS.

StreamBright 9y ago

We are using XFS for most of our production workloads, it turned out to be an excellent choice for most data heavy use cases. Brtfs was never an option, it is a bad idea to gamble with beta technology for data storage that a production system relies on. ext4 vs xfs is a much interesting argument, I haven't had time to follow up on this.

jerven 9y ago
We use XFS for sparql.uniprot.org (basically columnar database with semantic graph) there we recently retested it by accident. We use 2*4 TB consumer SSD ,ok rich consumer ;) With XFS we have 10-13% faster linear write of one big file (1.3TB) and about 20% more reads serviced per minute than EXT4. EXT4 was selected by accident instead of XFS for 1 of the 2 otherwise equal machines when upgrading these with the new 4TB SSDs instead of 1TB ones which they had before. In general I feel that the more you go into "enterprise" storage levels, the more XFS pulls ahead from the EXT family. i.e. laptops and small servers are not where the difference lies.

deleted 9y ago

[deleted]

joosters 9y ago

apt-get upgrade syncs so often that 'eatmydata' gives a noticeable speedup pretty much everywhere (I got into the habit of using it for ext3/4) I don't get why apt syncs so often - isn't the main point of log-structured file systems their ability to recover after a crash or powerloss? If so, why should you need to sync more than one every ten seconds or so?

dsr_ 9y ago
apt doesn't assume that you have a reliable filesystem. It assumes that you might crash at any moment, and it would be really important for you to have a consistent view of what packages are installed when you reboot.
- joosters 9y ago
  But ext3 and more advanced filesystems have been around for almost twenty years now... it seems an odd assumption that your filesystem is unreliable on any machine that isn't completely ancient (is anyone still using ext2, for instance?)
  - yjftsjthsd-h 9y ago
    Debian calls itself "the universal operating system" and officially supports not only multiple init systems, but multiple kernels (kfreebsd); somehow I don't see it relying on specific filesystems.
  - rleigh 9y ago
    It's not about the filesystem being "unreliable". It's about having the package manager's state checkpointed so that it can recover and resume if there is e.g. power loss or any other form of interruption at any point during package installation, upgrade, removal etc. This means having all of the updated files synched on disc plus the database state which describes it. When you move to a more advanced setup such as ZFS clones, you could do the full upgrade with a cloned snapshot, and swap it with the original once the changes were complete. This would avoid the need for all intermediate syncs--if there's a problem, you can simply restart from the starting point and throw all the intermediate state away.