4 ms·

I have about 50-100 PB of object storage workloads sitting on top of LVM cache for the past 5 years or so in production. It works extremely well. Before going

I have about 50-100 PB of object storage workloads sitting on top of LVM cache for the past 5 years or so in production. It works extremely well. Before going into prod, we tested LVM cache, and bcache, and LVM cache worked better for our workloads. Also, the tooling is significantly better than bcache. One thing I love about LVM cache is if you are already running with LVM, you can add the caching layer on the fly without any downtime. Regarding the note at the end where the author sees poor performance. I recommend taking a very close look at the SSD model. I've seen enterprise SSDs that perform worse with sync IO than spinning disks. When we buy hardware, we tell our vendors the exact model of SSD they can use. If we haven't qualified the SSD, we don't put it in production.

quadrature 5y ago
Could you expand a bit on what your use case is ?. Is this a cache in front of an s3 data warehouse or something like that ?.
- antongribok 5y ago
  We're using LVM cache with Ceph on spinning disks for object storage workloads in a large, on-prem private cloud. In other words, this cache is behind the S3 backend, but in front of spinning disks. When we built this, we had a choice of using all-SSD, spinning disk only, or a combination of both using LVM cache. LVM cache gave us a lot better performance per $ than using all-SSD. As for what type of actual application workloads? What do you use AWS S3 for? You name it, we have it: Backups, container registry, data warehouse, pictures, PDF docs, config data...
10000truths 5y ago
> I've seen enterprise SSDs that perform worse with sync IO than spinning disks. Can you elaborate on this? Specific brands to avoid? How and why they are worse?
- antongribok 5y ago
  See this blog post for some data at the end: https://www.sebastien-han.fr/blog/2014/10/10/ceph-how-to-test-if-your-ssd-is-suitable-as-a-journal-device/ https://www.sebastien-han.fr/blog/2014/10/10/ceph-how-to-tes... The post is from 2014, but the performance data was updated for many years after that. As you can see, there is not really 1 brand to single out. That Micron P400e was a pain to swap out in production, and if you use consumer drives there is even worse out there. Note: this post is on something very specific about Ceph, but the idea translates to any other workload that does syncs (including LVM cache scenarios).

quadrature 5y ago

Could you expand a bit on what your use case is ?. Is this a cache in front of an s3 data warehouse or something like that ?.

antongribok 5y ago
We're using LVM cache with Ceph on spinning disks for object storage workloads in a large, on-prem private cloud. In other words, this cache is behind the S3 backend, but in front of spinning disks. When we built this, we had a choice of using all-SSD, spinning disk only, or a combination of both using LVM cache. LVM cache gave us a lot better performance per $ than using all-SSD. As for what type of actual application workloads? What do you use AWS S3 for? You name it, we have it: Backups, container registry, data warehouse, pictures, PDF docs, config data...

10000truths 5y ago

> I've seen enterprise SSDs that perform worse with sync IO than spinning disks. Can you elaborate on this? Specific brands to avoid? How and why they are worse?

antongribok 5y ago
See this blog post for some data at the end: https://www.sebastien-han.fr/blog/2014/10/10/ceph-how-to-test-if-your-ssd-is-suitable-as-a-journal-device/ https://www.sebastien-han.fr/blog/2014/10/10/ceph-how-to-tes... The post is from 2014, but the performance data was updated for many years after that. As you can see, there is not really 1 brand to single out. That Micron P400e was a pain to swap out in production, and if you use consumer drives there is even worse out there. Note: this post is on something very specific about Ceph, but the idea translates to any other workload that does syncs (including LVM cache scenarios).