3 ms·

LVM Cache Surprises

47 points by JNRowe 5y ago

mfontani 5y ago
I've been debugging a sporadic high IO wait time very recently, and we ended up tuning vm.dirty_ratio and vm.dirty_background_ratio. I wonder whether those were already tuned from the very high % of ram default, or whether it could have any impact here.
antongribok 5y ago
I have about 50-100 PB of object storage workloads sitting on top of LVM cache for the past 5 years or so in production. It works extremely well. Before going into prod, we tested LVM cache, and bcache, and LVM cache worked better for our workloads. Also, the tooling is significantly better than bcache. One thing I love about LVM cache is if you are already running with LVM, you can add the caching layer on the fly without any downtime. Regarding the note at the end where the author sees poor performance. I recommend taking a very close look at the SSD model. I've seen enterprise SSDs that perform worse with sync IO than spinning disks. When we buy hardware, we tell our vendors the exact model of SSD they can use. If we haven't qualified the SSD, we don't put it in production.
- quadrature 5y ago
  Could you expand a bit on what your use case is ?. Is this a cache in front of an s3 data warehouse or something like that ?.
  - antongribok 5y ago
    We're using LVM cache with Ceph on spinning disks for object storage workloads in a large, on-prem private cloud. In other words, this cache is behind the S3 backend, but in front of spinning disks. When we built this, we had a choice of using all-SSD, spinning disk only, or a combination of both using LVM cache. LVM cache gave us a lot better performance per $ than using all-SSD. As for what type of actual application workloads? What do you use AWS S3 for? You name it, we have it: Backups, container registry, data warehouse, pictures, PDF docs, config data...
- 10000truths 5y ago
  > I've seen enterprise SSDs that perform worse with sync IO than spinning disks. Can you elaborate on this? Specific brands to avoid? How and why they are worse?
  - antongribok 5y ago
    See this blog post for some data at the end: https://www.sebastien-han.fr/blog/2014/10/10/ceph-how-to-test-if-your-ssd-is-suitable-as-a-journal-device/ https://www.sebastien-han.fr/blog/2014/10/10/ceph-how-to-tes... The post is from 2014, but the performance data was updated for many years after that. As you can see, there is not really 1 brand to single out. That Micron P400e was a pain to swap out in production, and if you use consumer drives there is even worse out there. Note: this post is on something very specific about Ceph, but the idea translates to any other workload that does syncs (including LVM cache scenarios).
ColonelPhantom 5y ago
I use lvmcache for my desktop, which has an old and tiny SSD (most of which is given to Windows) but a large HDD. Performance can be unpredictable at times, but is generally quite good.
drdude 5y ago
I have been using lvmcache for the past 3 years on both my desktop (on which i game on linux, the hdd is for game storage and I usually load one game repetitively till I end it and move to another game) and on my NAS with 3 HDDs and one SSD partitioned to be caches for each one of them. Never faced any consistency or performance problem, it is such a nice feature and can do it on the fly and however I want to reconfigure.
gjs278 5y ago
the surprises are because you trusted magic LVM to take care of it. learn dm-cache and configure it how you want.

47 points by JNRowe 5y ago

I've been debugging a sporadic high IO wait time very recently, and we ended up tuning vm.dirty_ratio and vm.dirty_background_ratio. I wonder whether those were already tuned from the very high % of ram default, or whether it could have any impact here.

antongribok 5y ago

I have about 50-100 PB of object storage workloads sitting on top of LVM cache for the past 5 years or so in production. It works extremely well. Before going into prod, we tested LVM cache, and bcache, and LVM cache worked better for our workloads. Also, the tooling is significantly better than bcache. One thing I love about LVM cache is if you are already running with LVM, you can add the caching layer on the fly without any downtime. Regarding the note at the end where the author sees poor performance. I recommend taking a very close look at the SSD model. I've seen enterprise SSDs that perform worse with sync IO than spinning disks. When we buy hardware, we tell our vendors the exact model of SSD they can use. If we haven't qualified the SSD, we don't put it in production.

quadrature 5y ago
Could you expand a bit on what your use case is ?. Is this a cache in front of an s3 data warehouse or something like that ?.
- antongribok 5y ago
  We're using LVM cache with Ceph on spinning disks for object storage workloads in a large, on-prem private cloud. In other words, this cache is behind the S3 backend, but in front of spinning disks. When we built this, we had a choice of using all-SSD, spinning disk only, or a combination of both using LVM cache. LVM cache gave us a lot better performance per $ than using all-SSD. As for what type of actual application workloads? What do you use AWS S3 for? You name it, we have it: Backups, container registry, data warehouse, pictures, PDF docs, config data...
10000truths 5y ago
> I've seen enterprise SSDs that perform worse with sync IO than spinning disks. Can you elaborate on this? Specific brands to avoid? How and why they are worse?
- antongribok 5y ago
  See this blog post for some data at the end: https://www.sebastien-han.fr/blog/2014/10/10/ceph-how-to-test-if-your-ssd-is-suitable-as-a-journal-device/ https://www.sebastien-han.fr/blog/2014/10/10/ceph-how-to-tes... The post is from 2014, but the performance data was updated for many years after that. As you can see, there is not really 1 brand to single out. That Micron P400e was a pain to swap out in production, and if you use consumer drives there is even worse out there. Note: this post is on something very specific about Ceph, but the idea translates to any other workload that does syncs (including LVM cache scenarios).

ColonelPhantom 5y ago

I use lvmcache for my desktop, which has an old and tiny SSD (most of which is given to Windows) but a large HDD. Performance can be unpredictable at times, but is generally quite good.

drdude 5y ago

I have been using lvmcache for the past 3 years on both my desktop (on which i game on linux, the hdd is for game storage and I usually load one game repetitively till I end it and move to another game) and on my NAS with 3 HDDs and one SSD partitioned to be caches for each one of them. Never faced any consistency or performance problem, it is such a nice feature and can do it on the fly and however I want to reconfigure.

gjs278 5y ago

the surprises are because you trusted magic LVM to take care of it. learn dm-cache and configure it how you want.