6 ms·

Compressed file-systems increase performance

38 points by ryandvm 17y ago

sandGorgon 17y ago
There is a google spreadsheet benchmark ( http://spreadsheets.google.com/ccc?key=0AlX1n8WSRNJWdGNQME5BMXVlMnlydS1Gd19wWXZuOEE http://spreadsheets.google.com/ccc?key=0AlX1n8WSRNJWdGNQME5B... ) released by one of the developers of ZFS-Fuse. Makes for interesting reading, since it confirms this benchmark. but on Linux and shows comparable results with Solaris ZFS benchmarks. And SSD + linux BLOWS everything else away
ryandvm 17y ago
This has interesting implications when considering SSDs versus traditional hard disks. SSD reads are faster, writes are slower, and space is at a premium.
- pmjordan 17y ago
  Indeed, I'm looking into this at the moment.
- ippisl 17y ago
  Some ssd-drives have compression on board to save space,speed and help endurance.
mmastrac 17y ago
Not a big surprise. We've hit a point in the I/O bandwidth-to-spare-CPU-cycles ratio where this now saves us time getting bits into RAM. IIRC, the latest Firefox (3.6) is now compressing files on-disk using HFS' built-in compression to cut ~20% off launch time.
- jerf 17y ago
  Actually we hit this point at least a decade ago; I remember getting improved performance by using a compressing filesystem for Windows 3.1. CPUs have been outrunning disks for a while, only the number of orders of magnitude has really changed. Part of this is that even very cheap algorithms can consistently get "some" compression; as the article observes, bzip is still a net loss for many cases, but gzip-style compression has been better-than-free for a while now, as long as your CPU isn't loaded.
e1ven 17y ago
This absolutely backs up my own findings- I've been working to decrease the mailserver backup times. Our Zimbra mailstore was taking over 8 hours to backup; I started compressing teh files before writing them to disk and it made a huge difference. I was able to bring the total time down to 3 hours. I attributed this to two factors- The first was, as the article mentions, that the machine has so much CPU when compared to disk, that the smaller data-transfer helped more than the CPU overhead hurt. I found that this is true for gzip, but bzip2 slowed it down. An implementation of parallel-bzip I found online spreads the bzip cost over multiple cores, reduced this further, however. This leads me to believe that something closer to the OS, which would adjust the compression level dynamically with CPU load, may be a wonderful product. The second factor was that the compressed files were in tarballs, so didn't create nearly as many files. Zimbra creates one file per message (a maildir like format), so this means hundreds of thousands of files need to be created and sized on the disk. In the archives, however, I am able to store just a few, large files, reducing the overhead of individually creating each message.
pmjordan 17y ago
Interesting, it seems they're using "LZJB" compression. http://en.wikipedia.org/wiki/LZJB http://en.wikipedia.org/wiki/LZJB I've been on the lookout for a fast compression algorithm that gives reasonable ratios, while having a liberal enough license. LZJB is licensed under the CDDL, which is GPL-incompatible, but it looks simple enough to be re-implemented.
- wmf 17y ago
  Have you heard about QuickLZ? It's the best thing I've seen in the LZO class. http://www.quicklz.com/ http://www.quicklz.com/ Edit: Here we go, a list of compression tools ranked by speed: http://www.maximumcompression.com/data/summary_mf4.php http://www.maximumcompression.com/data/summary_mf4.php
  - pmjordan 17y ago
    Thanks - QuickLZ does look good, unfortunately we're planning to release certain versions of the software we're developing under GPL and others under a proprietary license, and the $1200 for QuickLZ are a bit steep for an as-yet-unfunded startup. (this is the only time so far that I'm wishing I was a lone founder ;-) - the license would only be $100 then) BriefLZ on that comparison page caught my eye, however. It uses the same license as zLib while being more than twice as fast. We'll have to test that, it would be a good starting point. (and the code is short and clear enough to be understood and reviewed; can't afford to have buffer overflow bugs in it) We can upgrade to QuickLZ or LZO later when we can afford them. Somehow, BriefLZ that seems to have been missing from all the other compression comparison pages I've looked at.
ComputerGuru 17y ago
Completely off-topic, but this makes me even sadder that I still can't have ZFS on my Mac and probably never will...
- ryandvm 17y ago
  Indeed. Fortunately, btrfs is shaping up to be a fine substitute (http://www.codestrom.com/wandering/2009/03/zfs-vs-btrfs-comparison.html http://www.codestrom.com/wandering/2009/03/zfs-vs-btrfs-comp...). Including compression.
  - ComputerGuru 17y ago
    And will btrfs ever see the light of day as a kernel-mode FS on OS X? You have to realize that most Apple applications make extensive use of bit-twiddling in the HFS+ filesystem. Resource Forks are used extensively in the OS X software, particularly the iSomething programs. These would all have to be extensively modified to work w/ a different FS.
    - ryandvm 17y ago
      Doh - I missed that we were talking about the near miss between ZFS and the Mac. I was just lamenting that it's not on Linux.
    - deleted 17y ago
      [deleted]
    - blasdel 17y ago
      Most Apple applications do use bundles extensively, but that has nothing to do with HFS+. Apple has religiously avoided "use of bit-twiddling in the HFS+ filesystem" for years -- everything new since 10.4 uses standard extended attributes. The only remaining uses I can think of are Finder color labels (a special bit-field in HFS), and that a bunch of the standard fonts still use resource forks (they only have a license to redistribute).
      - 17y ago
bravura 17y ago
Is it good or bad to use ZFS for hosting a database? This article alludes to a claim that ZFS has fragmentation issues: http://antydba.blogspot.com/2010/02/mongodb-backup-with-zfs.html http://antydba.blogspot.com/2010/02/mongodb-backup-with-zfs.... Because ZFS is a copy-on-write filesystem, Eliot Horowitz says he wouldn't use it for a database: http://groups.google.com/group/mongodb-user/msg/731111475835e8ca http://groups.google.com/group/mongodb-user/msg/731111475835... However, a controverting opinion is offered by Jason J. W. Williams, who claims that ZFS works well with large MySQL installs: http://groups.google.com/group/mongodb-user/msg/e94712bc287ae3c0 http://groups.google.com/group/mongodb-user/msg/e94712bc287a... [edit: Updated links]
- deleted 17y ago
  [deleted]
nwatson 17y ago
SenSage (a log aggregation tool with many security-related features) has used compressed per-column raw data at the lowest level in its distributed storage engine since 2001. Writes are fast, and when reading raw data at the lowest level to fulfill queries in its multi-level multi-node map-reduce style query processor it wins big-time over traditional RDBMSs because (a) it reads only the columns relevant to the query, not having to pull fields out of rows/pages; (b) it's faster to pull & decompress than pull uncompressed data. Storage typically is 10x+ smaller than the original logs, even with redundant copies. RDBMSs inflate the data size instead.
jrockway 17y ago
But FWIW, drives are not that slow. On my i7 machine, a really fast machine, the disks (one SSD, one RAID-1 array with 2 normal disks) can read data much faster than the blocks can be decrypted. My read speed is limited to about 60MB/s compared to the 130MB/s that the disks can actually read. (Yeah, kcryptd is single-threaded. Ouch.) It's even worse on my Atom laptop. The SSD can do about 150MB/s (as I accidentally bought a really nice one), but the tiny CPU can only manage to decrypt about 20MB a second. And on a laptop, crypto is not exactly paranoia; it's the only sane thing to do.
- epochwolf 17y ago
  I fail to see how this is relevant. The article is about disk compression, not disk encryption.
joshu 17y ago
I keep wanting to build a qnap box with zfs somehow. anyone ever do this?
Perceval 17y ago
This makes intuitive sense. A great deal of operations on computers today are not CPU bound. In fact, the CPU (plus GPU) are sitting mostly un-utilized. Processors are fast, memory is cheap--both expand following Moore's law. But hard drive speed does not follow Moore's law. If hard drives are the most significant bottleneck, it makes sense to reduce the amount of seeking and reading the hard drive is doing, shifting the work to the usually-idle multicore CPU+GPU.
ojbyrne 17y ago
So if compression not only saves space, but increases performance, it seems like its past due for disk manufacturers to actually include compression in disk controllers.
- sparky 17y ago
  This is potentially true, but it's not a slam dunk because CPUs may be much more powerful than whatever you can build into a disk controller cost-effectively. This may not be true (a high-performance ASIC implementation may be tiny; this is not unheard of in the world of bit-banging algorithms), but it is worth considering. I'll have to look into what compression algorithm they used, and whether you can get away with something smaller/less power-hungry than a Core 2 on a disk controller.
- wmf 17y ago
  How and where you do the compression matters. ZFS compression appears to be much more efficient than ext2 or NTFS compression due to the copy-on-write nature of ZFS. It would be difficult for a disk controller to implement the same optimizations.
derefr 17y ago
This is something I've been thinking about since the iPad announcement. One of the reasons I desire access to my filesystem is that I like to keep some stuff in archives, either for compression or encryption or easy transfer or avoiding long readdir(2) operations, or any combination thereof. It'd be interesting if this whole issue can be avoided, and whole-filesystem compression is a good step toward that.
fnid2 17y ago
This has been known for well over a decade: volumes that use NTFS compression deliver performance increases as high as 50 percent over their uncompressed counterparts, depending on the type of data stored on the volumes http://technet.microsoft.com/en-us/library/cc767961.aspx http://technet.microsoft.com/en-us/library/cc767961.aspx
- Groxx 17y ago
  * ahem * From the article you linked: This performance seemed too good to be true until I monitored CPU utilization during a subsequent run of the same benchmarks on a compressed NTFS volume. The CPU utilization on the test jumped from an average of 10 to 18 percent on the uncompressed NTFS volume to a whopping 30 to 80 percent on the compressed NTFS volume. And most importantly (immediately after the above, separated for extra emphasis): In addition, performance significantly decreased when I used NTFS compression on larger volume sizes (4GB or greater) and software-based, fault-tolerant RAID volumes. Methinks we're over the 4GB mark now. The article is too old to be considered accurate any more.
  - deleted 17y ago
    [deleted]

38 points by ryandvm 17y ago

There is a google spreadsheet benchmark ( http://spreadsheets.google.com/ccc?key=0AlX1n8WSRNJWdGNQME5BMXVlMnlydS1Gd19wWXZuOEE http://spreadsheets.google.com/ccc?key=0AlX1n8WSRNJWdGNQME5B... ) released by one of the developers of ZFS-Fuse. Makes for interesting reading, since it confirms this benchmark. but on Linux and shows comparable results with Solaris ZFS benchmarks. And SSD + linux BLOWS everything else away

ryandvm 17y ago

This has interesting implications when considering SSDs versus traditional hard disks. SSD reads are faster, writes are slower, and space is at a premium.

pmjordan 17y ago
Indeed, I'm looking into this at the moment.
ippisl 17y ago
Some ssd-drives have compression on board to save space,speed and help endurance.

mmastrac 17y ago

Not a big surprise. We've hit a point in the I/O bandwidth-to-spare-CPU-cycles ratio where this now saves us time getting bits into RAM. IIRC, the latest Firefox (3.6) is now compressing files on-disk using HFS' built-in compression to cut ~20% off launch time.

jerf 17y ago
Actually we hit this point at least a decade ago; I remember getting improved performance by using a compressing filesystem for Windows 3.1. CPUs have been outrunning disks for a while, only the number of orders of magnitude has really changed. Part of this is that even very cheap algorithms can consistently get "some" compression; as the article observes, bzip is still a net loss for many cases, but gzip-style compression has been better-than-free for a while now, as long as your CPU isn't loaded.

e1ven 17y ago

This absolutely backs up my own findings- I've been working to decrease the mailserver backup times. Our Zimbra mailstore was taking over 8 hours to backup; I started compressing teh files before writing them to disk and it made a huge difference. I was able to bring the total time down to 3 hours. I attributed this to two factors- The first was, as the article mentions, that the machine has so much CPU when compared to disk, that the smaller data-transfer helped more than the CPU overhead hurt. I found that this is true for gzip, but bzip2 slowed it down. An implementation of parallel-bzip I found online spreads the bzip cost over multiple cores, reduced this further, however. This leads me to believe that something closer to the OS, which would adjust the compression level dynamically with CPU load, may be a wonderful product. The second factor was that the compressed files were in tarballs, so didn't create nearly as many files. Zimbra creates one file per message (a maildir like format), so this means hundreds of thousands of files need to be created and sized on the disk. In the archives, however, I am able to store just a few, large files, reducing the overhead of individually creating each message.

pmjordan 17y ago

Interesting, it seems they're using "LZJB" compression. http://en.wikipedia.org/wiki/LZJB http://en.wikipedia.org/wiki/LZJB I've been on the lookout for a fast compression algorithm that gives reasonable ratios, while having a liberal enough license. LZJB is licensed under the CDDL, which is GPL-incompatible, but it looks simple enough to be re-implemented.

wmf 17y ago
Have you heard about QuickLZ? It's the best thing I've seen in the LZO class. http://www.quicklz.com/ http://www.quicklz.com/ Edit: Here we go, a list of compression tools ranked by speed: http://www.maximumcompression.com/data/summary_mf4.php http://www.maximumcompression.com/data/summary_mf4.php
- pmjordan 17y ago
  Thanks - QuickLZ does look good, unfortunately we're planning to release certain versions of the software we're developing under GPL and others under a proprietary license, and the $1200 for QuickLZ are a bit steep for an as-yet-unfunded startup. (this is the only time so far that I'm wishing I was a lone founder ;-) - the license would only be $100 then) BriefLZ on that comparison page caught my eye, however. It uses the same license as zLib while being more than twice as fast. We'll have to test that, it would be a good starting point. (and the code is short and clear enough to be understood and reviewed; can't afford to have buffer overflow bugs in it) We can upgrade to QuickLZ or LZO later when we can afford them. Somehow, BriefLZ that seems to have been missing from all the other compression comparison pages I've looked at.

ComputerGuru 17y ago

Completely off-topic, but this makes me even sadder that I still can't have ZFS on my Mac and probably never will...

ryandvm 17y ago
Indeed. Fortunately, btrfs is shaping up to be a fine substitute (http://www.codestrom.com/wandering/2009/03/zfs-vs-btrfs-comparison.html http://www.codestrom.com/wandering/2009/03/zfs-vs-btrfs-comp...). Including compression.
- ComputerGuru 17y ago
  And will btrfs ever see the light of day as a kernel-mode FS on OS X? You have to realize that most Apple applications make extensive use of bit-twiddling in the HFS+ filesystem. Resource Forks are used extensively in the OS X software, particularly the iSomething programs. These would all have to be extensively modified to work w/ a different FS.
  - ryandvm 17y ago
    Doh - I missed that we were talking about the near miss between ZFS and the Mac. I was just lamenting that it's not on Linux.
  - deleted 17y ago
    [deleted]
  - blasdel 17y ago
    Most Apple applications do use bundles extensively, but that has nothing to do with HFS+. Apple has religiously avoided "use of bit-twiddling in the HFS+ filesystem" for years -- everything new since 10.4 uses standard extended attributes. The only remaining uses I can think of are Finder color labels (a special bit-field in HFS), and that a bunch of the standard fonts still use resource forks (they only have a license to redistribute).
    - 17y ago

bravura 17y ago

Is it good or bad to use ZFS for hosting a database? This article alludes to a claim that ZFS has fragmentation issues: http://antydba.blogspot.com/2010/02/mongodb-backup-with-zfs.html http://antydba.blogspot.com/2010/02/mongodb-backup-with-zfs.... Because ZFS is a copy-on-write filesystem, Eliot Horowitz says he wouldn't use it for a database: http://groups.google.com/group/mongodb-user/msg/731111475835e8ca http://groups.google.com/group/mongodb-user/msg/731111475835... However, a controverting opinion is offered by Jason J. W. Williams, who claims that ZFS works well with large MySQL installs: http://groups.google.com/group/mongodb-user/msg/e94712bc287ae3c0 http://groups.google.com/group/mongodb-user/msg/e94712bc287a... [edit: Updated links]

deleted 17y ago
[deleted]

nwatson 17y ago

SenSage (a log aggregation tool with many security-related features) has used compressed per-column raw data at the lowest level in its distributed storage engine since 2001. Writes are fast, and when reading raw data at the lowest level to fulfill queries in its multi-level multi-node map-reduce style query processor it wins big-time over traditional RDBMSs because (a) it reads only the columns relevant to the query, not having to pull fields out of rows/pages; (b) it's faster to pull & decompress than pull uncompressed data. Storage typically is 10x+ smaller than the original logs, even with redundant copies. RDBMSs inflate the data size instead.

jrockway 17y ago

But FWIW, drives are not that slow. On my i7 machine, a really fast machine, the disks (one SSD, one RAID-1 array with 2 normal disks) can read data much faster than the blocks can be decrypted. My read speed is limited to about 60MB/s compared to the 130MB/s that the disks can actually read. (Yeah, kcryptd is single-threaded. Ouch.) It's even worse on my Atom laptop. The SSD can do about 150MB/s (as I accidentally bought a really nice one), but the tiny CPU can only manage to decrypt about 20MB a second. And on a laptop, crypto is not exactly paranoia; it's the only sane thing to do.

epochwolf 17y ago
I fail to see how this is relevant. The article is about disk compression, not disk encryption.

joshu 17y ago

I keep wanting to build a qnap box with zfs somehow. anyone ever do this?

Perceval 17y ago

This makes intuitive sense. A great deal of operations on computers today are not CPU bound. In fact, the CPU (plus GPU) are sitting mostly un-utilized. Processors are fast, memory is cheap--both expand following Moore's law. But hard drive speed does not follow Moore's law. If hard drives are the most significant bottleneck, it makes sense to reduce the amount of seeking and reading the hard drive is doing, shifting the work to the usually-idle multicore CPU+GPU.

ojbyrne 17y ago

So if compression not only saves space, but increases performance, it seems like its past due for disk manufacturers to actually include compression in disk controllers.

sparky 17y ago
This is potentially true, but it's not a slam dunk because CPUs may be much more powerful than whatever you can build into a disk controller cost-effectively. This may not be true (a high-performance ASIC implementation may be tiny; this is not unheard of in the world of bit-banging algorithms), but it is worth considering. I'll have to look into what compression algorithm they used, and whether you can get away with something smaller/less power-hungry than a Core 2 on a disk controller.
wmf 17y ago
How and where you do the compression matters. ZFS compression appears to be much more efficient than ext2 or NTFS compression due to the copy-on-write nature of ZFS. It would be difficult for a disk controller to implement the same optimizations.

derefr 17y ago

This is something I've been thinking about since the iPad announcement. One of the reasons I desire access to my filesystem is that I like to keep some stuff in archives, either for compression or encryption or easy transfer or avoiding long readdir(2) operations, or any combination thereof. It'd be interesting if this whole issue can be avoided, and whole-filesystem compression is a good step toward that.

fnid2 17y ago

This has been known for well over a decade: volumes that use NTFS compression deliver performance increases as high as 50 percent over their uncompressed counterparts, depending on the type of data stored on the volumes http://technet.microsoft.com/en-us/library/cc767961.aspx http://technet.microsoft.com/en-us/library/cc767961.aspx

Groxx 17y ago
* ahem * From the article you linked: This performance seemed too good to be true until I monitored CPU utilization during a subsequent run of the same benchmarks on a compressed NTFS volume. The CPU utilization on the test jumped from an average of 10 to 18 percent on the uncompressed NTFS volume to a whopping 30 to 80 percent on the compressed NTFS volume. And most importantly (immediately after the above, separated for extra emphasis): In addition, performance significantly decreased when I used NTFS compression on larger volume sizes (4GB or greater) and software-based, fault-tolerant RAID volumes. Methinks we're over the 4GB mark now. The article is too old to be considered accurate any more.
- deleted 17y ago
  [deleted]