4 ms·

Can this simply be an artifact of terrible disk I/O on AWS or overall difference between ZFS/ext3? Do you think the results would have been similar if you were

Can this simply be an artifact of terrible disk I/O on AWS or overall difference between ZFS/ext3? Do you think the results would have been similar if you were to use no-compression-ZFS instead of ext3 on a proper database hardware? Basically trying to figure out if the low performance of uncompressed dataset is specific to AWS/ext3. Thanks.

mfenniak 14y ago
old-gregg makes a great observation here. The addition of a ZFS benchmark without-compression is needed to isolate the compression as a factor in the speedup. That aside, I thought this was a wonderful article with non-intuitive findings. Very interesting, CirtusDB [edit: er, CitusDB]. :-)
- codewright 14y ago
  Citus
- cwsteinbach 14y ago
  old-gregg's point is valid. Since we didn't benchmark ZFS without compression we can't say for sure how much of the performance improvement is attributable to compression vs. just ZFS. As far as AWS goes, we have noticed ephemeral disks connected to the same instance can exhibit fairly large performance differences, and attempted to control for that in our tests by reusing the same disk for each test run.
  - mattbillenstein 14y ago
    I always question benchmarks on ec2 because of "noisy neighbor" effects - maybe that's in the noise, but someone trying to replicate your results would perhaps see significantly different results depending whether their VM landed on a busy node or not... Tag this one #ymmv +1 on testing uncompressed ZFS I did see a blog post about MySQL with similar results (at least compression was a significant win) some time ago - disks are so slow compared to what throughput modern CPUs are capable of on these sorts of compression algorithms.
kevinastone 14y ago
You're also testing on a c1.xlarge that gives you excess CPU compared to I/O, so it's potentially biasing your results.
- cwsteinbach 14y ago
  While I can't claim that we logged CPU load while running these tests, I can say that I watched the output of top and iotop and that the CPU load was relatively light. It's also worth pointing out that Amazon describes the I/O performance of c1.xlarge instances as "high". We also considered using an hs1.8xlarge "High Storage" instance for these tests, but eventually decided that we were more interested in testing against conventional disks as opposed to SSDs.
  - lucian1900 14y ago
    Did you use instance storage? EBS? Provisioned IOPS? There are vast differences between those three.
bsg75 14y ago
I have seen similar advantages comparing XFS to ZFS+compression on a local server (Centos 6.3, ZFS on Linux 0.61). Using a 2 disk striped volume for PostgreSQL 9.2, I get an average of 2.5X compression (as reported by ZFS), and a 1.5 to 2X time reduction in database restores (single threaded or 8 jobs in parallel). Given this development box has relatively slow 7200 RPM disks, the tradeoff of more CPU time for less disk transfer makes sense. Edit: My use case is an OLAP server. I can't state how the tradeoffs affect OLTP performance.
mbell 14y ago
Another issue is that ZFS is extremely aggressive with caching data in ram (L1ARC). That can eat up memory you'd rather give to the database heap and also tends to skew benchmarks.
- bsg75 14y ago
  Placing limits on ARC (25% for me currently) limits that effect.
- wiredfool 14y ago
  Yes, but postgres' design in that area actually helps. Postgres relies on the OS caching the data tables for the most part. There is some caching in shared buffers, but generally that's not a huge portion of the memory of your db system. (10-25%, and not more than a few gigs)
- rscale 14y ago
  I run Postgres on ZFS, and simply limit the amount of memory dedicated to the ARC.
trotsky 14y ago
Using compression with zfs on solaris derived platforms serving as a san/nas backend for vsphere appears to speed up every workload backed on rotating storage. Well, not if vmdks use guest full disk encryption, but that's understandable.

mfenniak 14y ago

old-gregg makes a great observation here. The addition of a ZFS benchmark without-compression is needed to isolate the compression as a factor in the speedup. That aside, I thought this was a wonderful article with non-intuitive findings. Very interesting, CirtusDB [edit: er, CitusDB]. :-)

codewright 14y ago
Citus
cwsteinbach 14y ago
old-gregg's point is valid. Since we didn't benchmark ZFS without compression we can't say for sure how much of the performance improvement is attributable to compression vs. just ZFS. As far as AWS goes, we have noticed ephemeral disks connected to the same instance can exhibit fairly large performance differences, and attempted to control for that in our tests by reusing the same disk for each test run.
- mattbillenstein 14y ago
  I always question benchmarks on ec2 because of "noisy neighbor" effects - maybe that's in the noise, but someone trying to replicate your results would perhaps see significantly different results depending whether their VM landed on a busy node or not... Tag this one #ymmv +1 on testing uncompressed ZFS I did see a blog post about MySQL with similar results (at least compression was a significant win) some time ago - disks are so slow compared to what throughput modern CPUs are capable of on these sorts of compression algorithms.

kevinastone 14y ago

You're also testing on a c1.xlarge that gives you excess CPU compared to I/O, so it's potentially biasing your results.

cwsteinbach 14y ago
While I can't claim that we logged CPU load while running these tests, I can say that I watched the output of top and iotop and that the CPU load was relatively light. It's also worth pointing out that Amazon describes the I/O performance of c1.xlarge instances as "high". We also considered using an hs1.8xlarge "High Storage" instance for these tests, but eventually decided that we were more interested in testing against conventional disks as opposed to SSDs.
- lucian1900 14y ago
  Did you use instance storage? EBS? Provisioned IOPS? There are vast differences between those three.

bsg75 14y ago

I have seen similar advantages comparing XFS to ZFS+compression on a local server (Centos 6.3, ZFS on Linux 0.61). Using a 2 disk striped volume for PostgreSQL 9.2, I get an average of 2.5X compression (as reported by ZFS), and a 1.5 to 2X time reduction in database restores (single threaded or 8 jobs in parallel). Given this development box has relatively slow 7200 RPM disks, the tradeoff of more CPU time for less disk transfer makes sense. Edit: My use case is an OLAP server. I can't state how the tradeoffs affect OLTP performance.

mbell 14y ago

Another issue is that ZFS is extremely aggressive with caching data in ram (L1ARC). That can eat up memory you'd rather give to the database heap and also tends to skew benchmarks.

bsg75 14y ago
Placing limits on ARC (25% for me currently) limits that effect.
wiredfool 14y ago
Yes, but postgres' design in that area actually helps. Postgres relies on the OS caching the data tables for the most part. There is some caching in shared buffers, but generally that's not a huge portion of the memory of your db system. (10-25%, and not more than a few gigs)
rscale 14y ago
I run Postgres on ZFS, and simply limit the amount of memory dedicated to the ARC.

trotsky 14y ago

Using compression with zfs on solaris derived platforms serving as a san/nas backend for vsphere appears to speed up every workload backed on rotating storage. Well, not if vmdks use guest full disk encryption, but that's understandable.