7 ms·

Kvm (or any overwrite workload for that matter) is the worst possible workload for btrfs because of COW. We have ideas to address this but honestly it's not hi

Kvm (or any overwrite workload for that matter) is the worst possible workload for btrfs because of COW. We have ideas to address this but honestly it's not high on the list.

loeg 10y ago
Shouldn't ZFS have similar problems with overwrite workloads?
- josefbacik 10y ago
  ZFS has a 7 year head start on BTRFS and has traditionally had about 5 times as many core contributors at any given time so I imagine they've solved this in some novel way by now.
  - deleted 10y ago
    [deleted]
  - tolle 10y ago
    Wasn't ZFS even "production ready" before BTRFS development even began? I don't remember the state of it when it was first introduced into Solaris, maybe someones memory is better then mine. Was ZFS better of in 06-07 Then BTRFS is now?
    - ryao 10y ago
      btrfs development started in 2007 while ZFS development started in 2001. If you want to do a point in time comparison, look at how ZFS was in 2010. You can get the last copy of OpenSolaris for the comparison. It would need to be done on physical hardware due to poor support for virtualization back then. By the way, ZFS was deemed production ready after 4 years of development.
      - tete 10y ago
        That's true. One quick side note though: BTRFS actually had a fairly long "draft period" (multiple years) of design. I have no clue how long that was on ZFS. I just know that the original author mentioned that somewhere.
        ryao 10y ago
        If you have time, you can watch this video where Jeff Bonwick describes the birth of ZFS: http://m.youtube.com/watch?v=dcV2PaMTAJ4 http://m.youtube.com/watch?v=dcV2PaMTAJ4 I am the guy who asked the LZJB question. To summarize my recollection, formal design work on ZFS started in 2001 when Matthew Ahrens started working at Sun. Jeff Bonwick had promised Matthew Ahrens a job at Sun making a filesystem a 6 months to a year before then when Matt was still in college. I am sure that both Jeff and Matt had some thoughts on it during that time, but there was no formal effort until Matt's employment started.
  - ryao 10y ago
    The idea that btrfs has fewer contributors ought to be a surprise for many. Users tend to assume the opposite. ZFS having 5x more seems to be on the high side to me though. How did you determine that? By the way, I was under the impression that ZFS development started in 2001 while btrfs development started in 2007. That would be a 6 year difference.
  - qaq 10y ago
    It's not just years it's the types of deployments. ZFS backs a very large number of very large critical systems.
  - rurban 10y ago
    5x more core ZFS contributors sounds to me worse and not better than BTRFS. The more the worse usually.
    - josefbacik 10y ago
      Not sure how that tracks. These file systems are huge code bases with lots of problems and areas for improvement. When you only have 2-4 people who understand most of it at a given time things move slowly. If I could tell somebody to go fix balance my life would be a whole lot easier, but in reality we end up being interrupt driven and have to prioritize differently.
- ryao 10y ago
  > Shouldn't ZFS have similar problems with overwrite workloads? ZFS does suffer from read-modify-write on partial record writes. The effect of that is apparent in the benchmarks. However, the benchmarks are being done on mechanical disks, which have low IOPS. The IOPS of a mechanical disk are roughly the same on a given sequence of IOs at different positions regardless of whether they are 4KB or 128KB in size, so it only has to pay a penalty once. If the record size were changed to 4KB, this penalty would disappear and ZFS performance should increase, provided that the VM internals are properly aligned. Also, read-modify-write overhead reduces IOPS by at most 2 and bandwidth to the smaller of the link bandwidth and the IOPS times the record size. A CoW filesystem should be able to perform roughly at that level when it does read-modify-write on records/extents. Unless btrfs' internal extents are huge, there is an issue somewhere. Of course, having huge extents by default on which read-modify-write is done could also be considered a design issue.
biokoda 10y ago
What kinds of workloads is it designed for then?
- josefbacik 10y ago
  It's a general purpose file system, it does great with metadata heavy workloads and normal streaming writes. The overwrite case is special because of COW. Eventually we'll be comparable to everybody else but we aren't now.
  - avar 10y ago
    I reinstalled my laptop with btrfs on / because "why not" about a year ago. I ran into issues with maintaining a huge ~1M file Maildir, it seemed to do very badly with huge directories. Some kernel thread would be at 99% CPU while I was trying to populate a Maildir, and the entire system ground to a halt. More importantly I would run into issues like running out of space on / with 20G left, but it was the "wrong kind of space". I.e. I had run out of metadata space but had plenty of file space left and had to run a rebalancing operation on the filesystem. I didn't need any of the COW etc. advanced features that btrfs provides, and didn't test them, but as just a normal user needing a general purpose filesystem it was a bit too much hassle, especially with needing to administer the filesystem in a manner similar to how I might administer a RDBMS. I ended up just going back to ext4.
    - tadfisher 10y ago
      A subvolume for your Maildir, mounted with "-o nodatacow", would have prevented the first issue. If btrfs could modify that behavior automatically using heuristics it would truly be general-purpose. However there are some inherent tradeoffs of that approach that users need to consider, and I consider that downside a basic constraint of the Unix filesystem abstraction.
      - avar 10y ago
        I think I did try stuff like that, including mounting / with nodatacow and other similar options. I walked away with the impression that btrfs was a powerful tool for certain use-cases, but definitely not something I'd call a "general purpose" filesystem given the need to be fairly knowledgeable about its internals to use it for common desktop use-cases. To me a general purpose filesystem is something like ext4, it doesn't have amazing performance, it's not bad either, but generally nothing unexpected happens with it and I can just leave it there and don't have to worry about it. The btrfs filesystem seemed like the opposite of that. A very powerful tool whose power and flexibility made it less general purpose by virtue of needing to keep close tabs on how you were using it.
SXX 10y ago
This is sad since even with some performance loss compression and offline deduplication for VM images sounds like nice feature. Is there any docs that explain why COW make Btrfs so slow compared to QCOW2?
- josefbacik 10y ago
  There are no docs. I'm not a qcow2 expert, what I know is very basic so anything I say about qcow2 can be very wrong. So qcow has a read only base image that gets updated when we change things. The image format just had the changes from the original image. So you update a package, it adds some metadata to point at the new stuff and adds the data in and you are done. So with btrfs you have this image on top of btrfs, so you update a file and its metadata inside the image. Say you start with a pristine image that's in nice big extents. You update a package which changes small chunks all over the file. Let's say you update 12 4K extents. So now instead of one extent you now have 36 extents. This affects everything, fsyncs take longer because there's more extents we have to write out, the space is more fragmented so cold cache reads are more expensive, the csums are no longer contiguous so they also take up a larger more fragmented area. It has this really terrible cascading effect.
  - SXX 10y ago
    I tested it with preallocated raw images on Btrfs and that setup was slow as well. If I get you right it's must be possible to somehow allign VM disk sectors to Btrfs sectors for performance boost. Or compression and deduplication usage going to kill performance anyway?
    - cmurf 10y ago
      Btrfs internally use logical addresses for all extents. The making from logical to physical is done via the chunk tree which not only indicates physical sector but also the device. So the reference for a file extent says nothing about what device the extent is on or the replication (raid profile) since that is all a function of the chunk and dev trees. I think Btrfs for a guest F's is best pointed to an LV, rather than qcow2. It's been awhile since I benchmarked that compared to 'qemu-img create -f qcow2 -o nocow=on' which will set xattr +C on the file making it nocow. The nocow xattr helps a lot with this problem.
daurnimator 10y ago
But shouldn't NOCOW mitigate that? (which the article does for some tests)
- josefbacik 10y ago
  NOCOW is horribly expensive because we still have to go check and make sure that there are no snapshots pointing at the changing extents. It only solves the fragmentation issue, and if you don't prealloc your image it doesn't even do that.
  - ryao 10y ago
    Are you sure that disabling CoW solves btrfs' fragmentation issue when you pre-allocate the image? If the volume is snapshotted regularly, CoW should be active once on each extent once until the next snapshot reactivates it. That should mean that the fragmentation would still occur, although more slowly. Is that correct or am I misunderstanding something? I am only superficially familiar with btrfs internals, but I do not see any way to implement snapshots without either doing CoW or duplicating the data in its entirety. If you are checking to see if the data is part of a snapshot, then you should be doing CoW.
    - MertsA 10y ago
      Disabling CoW disables snapshotting.
      - versteegen 10y ago
        (I'm not an expert, just a user) Snapshotting effectively disables NOCOW. Snapshotting always works, and needs COW to do so. Which means ryao is right that fragmentation will occur. btrfs doesn't copy a NOCOW file in its entirety if it's in a snapshot, or I would have run out of disk space by now. (I really need to move my VM images to a non-backed-up subvolume.)
      - josefbacik 10y ago
        No it doesn't.
zbraniecki 10y ago
Is brtfs a good choice from the power consumption standpoint? I'm on a laptop and trying to maximize my battery life, and am wondering if my choice of brtfs limits it.
- jwatte 10y ago
  You could try nilfs for that.

Kvm (or any overwrite workload for that matter) is the worst possible workload for btrfs because of COW. We have ideas to address this but honestly it's not high on the list.

loeg 10y ago

Shouldn't ZFS have similar problems with overwrite workloads?

josefbacik 10y ago
ZFS has a 7 year head start on BTRFS and has traditionally had about 5 times as many core contributors at any given time so I imagine they've solved this in some novel way by now.
- deleted 10y ago
  [deleted]
- tolle 10y ago
  Wasn't ZFS even "production ready" before BTRFS development even began? I don't remember the state of it when it was first introduced into Solaris, maybe someones memory is better then mine. Was ZFS better of in 06-07 Then BTRFS is now?
  - ryao 10y ago
    btrfs development started in 2007 while ZFS development started in 2001. If you want to do a point in time comparison, look at how ZFS was in 2010. You can get the last copy of OpenSolaris for the comparison. It would need to be done on physical hardware due to poor support for virtualization back then. By the way, ZFS was deemed production ready after 4 years of development.
    - tete 10y ago
      That's true. One quick side note though: BTRFS actually had a fairly long "draft period" (multiple years) of design. I have no clue how long that was on ZFS. I just know that the original author mentioned that somewhere.
      - ryao 10y ago
        If you have time, you can watch this video where Jeff Bonwick describes the birth of ZFS: http://m.youtube.com/watch?v=dcV2PaMTAJ4 http://m.youtube.com/watch?v=dcV2PaMTAJ4 I am the guy who asked the LZJB question. To summarize my recollection, formal design work on ZFS started in 2001 when Matthew Ahrens started working at Sun. Jeff Bonwick had promised Matthew Ahrens a job at Sun making a filesystem a 6 months to a year before then when Matt was still in college. I am sure that both Jeff and Matt had some thoughts on it during that time, but there was no formal effort until Matt's employment started.
- ryao 10y ago
  The idea that btrfs has fewer contributors ought to be a surprise for many. Users tend to assume the opposite. ZFS having 5x more seems to be on the high side to me though. How did you determine that? By the way, I was under the impression that ZFS development started in 2001 while btrfs development started in 2007. That would be a 6 year difference.
- qaq 10y ago
  It's not just years it's the types of deployments. ZFS backs a very large number of very large critical systems.
- rurban 10y ago
  5x more core ZFS contributors sounds to me worse and not better than BTRFS. The more the worse usually.
  - josefbacik 10y ago
    Not sure how that tracks. These file systems are huge code bases with lots of problems and areas for improvement. When you only have 2-4 people who understand most of it at a given time things move slowly. If I could tell somebody to go fix balance my life would be a whole lot easier, but in reality we end up being interrupt driven and have to prioritize differently.
ryao 10y ago
> Shouldn't ZFS have similar problems with overwrite workloads? ZFS does suffer from read-modify-write on partial record writes. The effect of that is apparent in the benchmarks. However, the benchmarks are being done on mechanical disks, which have low IOPS. The IOPS of a mechanical disk are roughly the same on a given sequence of IOs at different positions regardless of whether they are 4KB or 128KB in size, so it only has to pay a penalty once. If the record size were changed to 4KB, this penalty would disappear and ZFS performance should increase, provided that the VM internals are properly aligned. Also, read-modify-write overhead reduces IOPS by at most 2 and bandwidth to the smaller of the link bandwidth and the IOPS times the record size. A CoW filesystem should be able to perform roughly at that level when it does read-modify-write on records/extents. Unless btrfs' internal extents are huge, there is an issue somewhere. Of course, having huge extents by default on which read-modify-write is done could also be considered a design issue.

biokoda 10y ago

What kinds of workloads is it designed for then?

josefbacik 10y ago
It's a general purpose file system, it does great with metadata heavy workloads and normal streaming writes. The overwrite case is special because of COW. Eventually we'll be comparable to everybody else but we aren't now.
- avar 10y ago
  I reinstalled my laptop with btrfs on / because "why not" about a year ago. I ran into issues with maintaining a huge ~1M file Maildir, it seemed to do very badly with huge directories. Some kernel thread would be at 99% CPU while I was trying to populate a Maildir, and the entire system ground to a halt. More importantly I would run into issues like running out of space on / with 20G left, but it was the "wrong kind of space". I.e. I had run out of metadata space but had plenty of file space left and had to run a rebalancing operation on the filesystem. I didn't need any of the COW etc. advanced features that btrfs provides, and didn't test them, but as just a normal user needing a general purpose filesystem it was a bit too much hassle, especially with needing to administer the filesystem in a manner similar to how I might administer a RDBMS. I ended up just going back to ext4.
  - tadfisher 10y ago
    A subvolume for your Maildir, mounted with "-o nodatacow", would have prevented the first issue. If btrfs could modify that behavior automatically using heuristics it would truly be general-purpose. However there are some inherent tradeoffs of that approach that users need to consider, and I consider that downside a basic constraint of the Unix filesystem abstraction.
    - avar 10y ago
      I think I did try stuff like that, including mounting / with nodatacow and other similar options. I walked away with the impression that btrfs was a powerful tool for certain use-cases, but definitely not something I'd call a "general purpose" filesystem given the need to be fairly knowledgeable about its internals to use it for common desktop use-cases. To me a general purpose filesystem is something like ext4, it doesn't have amazing performance, it's not bad either, but generally nothing unexpected happens with it and I can just leave it there and don't have to worry about it. The btrfs filesystem seemed like the opposite of that. A very powerful tool whose power and flexibility made it less general purpose by virtue of needing to keep close tabs on how you were using it.

SXX 10y ago

This is sad since even with some performance loss compression and offline deduplication for VM images sounds like nice feature. Is there any docs that explain why COW make Btrfs so slow compared to QCOW2?

josefbacik 10y ago
There are no docs. I'm not a qcow2 expert, what I know is very basic so anything I say about qcow2 can be very wrong. So qcow has a read only base image that gets updated when we change things. The image format just had the changes from the original image. So you update a package, it adds some metadata to point at the new stuff and adds the data in and you are done. So with btrfs you have this image on top of btrfs, so you update a file and its metadata inside the image. Say you start with a pristine image that's in nice big extents. You update a package which changes small chunks all over the file. Let's say you update 12 4K extents. So now instead of one extent you now have 36 extents. This affects everything, fsyncs take longer because there's more extents we have to write out, the space is more fragmented so cold cache reads are more expensive, the csums are no longer contiguous so they also take up a larger more fragmented area. It has this really terrible cascading effect.
- SXX 10y ago
  I tested it with preallocated raw images on Btrfs and that setup was slow as well. If I get you right it's must be possible to somehow allign VM disk sectors to Btrfs sectors for performance boost. Or compression and deduplication usage going to kill performance anyway?
  - cmurf 10y ago
    Btrfs internally use logical addresses for all extents. The making from logical to physical is done via the chunk tree which not only indicates physical sector but also the device. So the reference for a file extent says nothing about what device the extent is on or the replication (raid profile) since that is all a function of the chunk and dev trees. I think Btrfs for a guest F's is best pointed to an LV, rather than qcow2. It's been awhile since I benchmarked that compared to 'qemu-img create -f qcow2 -o nocow=on' which will set xattr +C on the file making it nocow. The nocow xattr helps a lot with this problem.

daurnimator 10y ago

But shouldn't NOCOW mitigate that? (which the article does for some tests)

josefbacik 10y ago
NOCOW is horribly expensive because we still have to go check and make sure that there are no snapshots pointing at the changing extents. It only solves the fragmentation issue, and if you don't prealloc your image it doesn't even do that.
- ryao 10y ago
  Are you sure that disabling CoW solves btrfs' fragmentation issue when you pre-allocate the image? If the volume is snapshotted regularly, CoW should be active once on each extent once until the next snapshot reactivates it. That should mean that the fragmentation would still occur, although more slowly. Is that correct or am I misunderstanding something? I am only superficially familiar with btrfs internals, but I do not see any way to implement snapshots without either doing CoW or duplicating the data in its entirety. If you are checking to see if the data is part of a snapshot, then you should be doing CoW.
  - MertsA 10y ago
    Disabling CoW disables snapshotting.
    - versteegen 10y ago
      (I'm not an expert, just a user) Snapshotting effectively disables NOCOW. Snapshotting always works, and needs COW to do so. Which means ryao is right that fragmentation will occur. btrfs doesn't copy a NOCOW file in its entirety if it's in a snapshot, or I would have run out of disk space by now. (I really need to move my VM images to a non-backed-up subvolume.)
    - josefbacik 10y ago
      No it doesn't.

zbraniecki 10y ago

Is brtfs a good choice from the power consumption standpoint? I'm on a laptop and trying to maximize my battery life, and am wondering if my choice of brtfs limits it.

jwatte 10y ago
You could try nilfs for that.