20 ms·

Google releases snappy, the compression library used in Bigtable

235 points by tonfa 16y ago

ot 16y ago
I wonder if they had evaluated LZO (http://www.oberhumer.com/opensource/lzo/ http://www.oberhumer.com/opensource/lzo/) before writing this. It is quite well-tested (a variant on it runs in the Mars Rovers) and very very fast: the author reports 16MB/sec on a Pentium 133, on modern architectures it should easily get to the 500MB/sec claimed by snappy.
- lemming 16y ago
  "The LZO algorithms and implementations are copyrighted OpenSource distributed under the GNU General Public License." Does this even make sense? Can you apply the GPL to an algorithm? As I understand it, if there's no patent I should be able to implement it with no problems.
  - wmf 16y ago
    People who want to upsell you a commercial version tend to say such things.
    - bhickey 16y ago
      Be warned: the main source code in the 'src' directory is a real pain to understand as I've experimented with hundreds of slightly different versions. It contains many #if and some gotos, and is *completely optimized for speed* and not for readability. Code sharing of the different algorithms is implemented by stressing the preprocessor - this can be really confusing. Lots of marcos and assertions don't make things better. Given the author's statements, I don't know that I'd feel comfortable using LZO in a production environment.
      - malkia 16y ago
        We used it in our games since Ultimate Spiderman. I've got assembly version working on the PS2 IOP chip (33Mhz, the chip used to run PS1 games, otherwise I/O and sound for PS2). I've got 5mb/s decompressed data speed, so that was speeding our disk access. I had to tweak just a little bit the source code and make sure unaligned writes (4 bytes) were used and that made it x2 or x3 faster.
- drv 16y ago
  The README says: "In our tests, Snappy usually is faster than algorithms in the same class (e.g. LZO, LZF, FastLZ, QuickLZ, etc.) while achieving comparable compression ratios."
  - ot 16y ago
    Interesting, I had missed that. It also says > Finally, snappy can benchmark Snappy against a few other compression libraries (zlib, LZO, LZF, FastLZ and QuickLZ), if they were detected at configure time. I'll do some tests, I'm curious about the results
    - tonfa 16y ago
      Please post the results :)
- wolf550e 16y ago
  ~/snappy-read-only $ ./snappy_unittest -norun_microbenchmarks -lzo testdata/* testdata/alice29.txt : LZO: [b 1M] bytes 152089 -> 82691 54.4% comp 64.5 MB/s uncomp 206.4 MB/s SNAPPY: [b 4M] bytes 152089 -> 90965 59.8% comp 171.5 MB/s uncomp 375.8 MB/s testdata/asyoulik.txt : LZO: [b 1M] bytes 125179 -> 73217 58.5% comp 52.5 MB/s uncomp 173.2 MB/s SNAPPY: [b 4M] bytes 125179 -> 80207 64.1% comp 137.8 MB/s uncomp 301.7 MB/s testdata/baddata1.snappy : LZO: [b 1M] bytes 27512 -> 26487 96.3% comp 24.4 MB/s uncomp 491.6 MB/s SNAPPY: [b 4M] bytes 27512 -> 26675 97.0% comp 305.2 MB/s uncomp 1465.7 MB/s testdata/baddata2.snappy : LZO: [b 1M] bytes 27483 -> 26528 96.5% comp 24.3 MB/s uncomp 499.1 MB/s SNAPPY: [b 4M] bytes 27483 -> 26724 97.2% comp 331.8 MB/s uncomp 1660.4 MB/s testdata/baddata3.snappy : LZO: [b 1M] bytes 28384 -> 27380 96.5% comp 24.1 MB/s uncomp 488.2 MB/s SNAPPY: [b 4M] bytes 28384 -> 27476 96.8% comp 275.7 MB/s uncomp 1346.5 MB/s testdata/cp.html : LZO: [b 1M] bytes 24603 -> 11621 47.2% comp 57.2 MB/s uncomp 258.5 MB/s SNAPPY: [b 4M] bytes 24603 -> 11838 48.1% comp 190.0 MB/s uncomp 443.5 MB/s testdata/fields.c : LZO: [b 1M] bytes 11150 -> 4663 41.8% comp 73.8 MB/s uncomp 259.8 MB/s SNAPPY: [b 4M] bytes 11150 -> 4728 42.4% comp 207.3 MB/s uncomp 431.7 MB/s testdata/geo.protodata : LZO: [b 1M] bytes 100000 -> 20423 20.4% comp 115.8 MB/s uncomp 429.9 MB/s SNAPPY: [b 4M] bytes 100000 -> 23488 23.5% comp 359.0 MB/s uncomp 625.8 MB/s testdata/grammar.lsp : LZO: [b 1M] bytes 3721 -> 1781 47.9% comp 66.8 MB/s uncomp 311.4 MB/s SNAPPY: [b 4M] bytes 3721 -> 1800 48.4% comp 214.9 MB/s uncomp 461.2 MB/s testdata/house.jpg : LZO: [b 1M] bytes 126958 -> 127173 100.2% comp 20.2 MB/s uncomp 1420.8 MB/s SNAPPY: [b 4M] bytes 126958 -> 126803 99.9% comp 2037.6 MB/s uncomp 7578.7 MB/s testdata/html : LZO: [b 1M] bytes 102400 -> 21027 20.5% comp 115.1 MB/s uncomp 423.8 MB/s SNAPPY: [b 4M] bytes 102400 -> 24140 23.6% comp 362.4 MB/s uncomp 703.5 MB/s testdata/html_x_4 : LZO: [b 1M] bytes 409600 -> 82980 20.3% comp 120.9 MB/s uncomp 416.4 MB/s SNAPPY: [b 4M] bytes 409600 -> 96472 23.6% comp 359.0 MB/s uncomp 698.0 MB/s testdata/kennedy.xls : LZO: [b 1M] bytes 1029744 -> 357315 34.7% comp 133.5 MB/s uncomp 533.6 MB/s SNAPPY: [b 4M] bytes 1029744 -> 425735 41.3% comp 294.1 MB/s uncomp 432.6 MB/s testdata/kppkn.gtb : LZO: [b 1M] bytes 184320 -> 71671 38.9% comp 84.3 MB/s uncomp 235.9 MB/s SNAPPY: [b 4M] bytes 184320 -> 70535 38.3% comp 231.7 MB/s uncomp 373.8 MB/s testdata/lcet10.txt : LZO: [b 1M] bytes 426754 -> 221290 51.9% comp 57.6 MB/s uncomp 181.9 MB/s SNAPPY: [b 4M] bytes 426754 -> 243710 57.1% comp 153.7 MB/s uncomp 341.4 MB/s testdata/mapreduce-osdi-1.pdf : LZO: [b 1M] bytes 94330 -> 76999 81.6% comp 24.9 MB/s uncomp 810.9 MB/s SNAPPY: [b 4M] bytes 94330 -> 77477 82.1% comp 709.0 MB/s uncomp 1669.5 MB/s testdata/plrabn12.txt : LZO: [b 1M] bytes 481861 -> 294610 61.1% comp 51.4 MB/s uncomp 164.8 MB/s SNAPPY: [b 4M] bytes 481861 -> 329339 68.3% comp 129.8 MB/s uncomp 281.9 MB/s testdata/ptt5 : LZO: [b 1M] bytes 513216 -> 86232 16.8% comp 119.0 MB/s uncomp 506.1 MB/s SNAPPY: [b 4M] bytes 513216 -> 93455 18.2% comp 479.8 MB/s uncomp 670.8 MB/s testdata/sum : LZO: [b 1M] bytes 38240 -> 17686 46.2% comp 57.8 MB/s uncomp 267.0 MB/s SNAPPY: [b 4M] bytes 38240 -> 19837 51.9% comp 194.5 MB/s uncomp 401.0 MB/s testdata/urls.10K : LZO: [b 1M] bytes 702087 -> 309320 44.1% comp 55.7 MB/s uncomp 265.1 MB/s SNAPPY: [b 4M] bytes 702087 -> 357267 50.9% comp 216.9 MB/s uncomp 499.3 MB/s testdata/xargs.1 : LZO: [b 1M] bytes 4227 -> 2450 58.0% comp 56.3 MB/s uncomp 286.2 MB/s SNAPPY: [b 4M] bytes 4227 -> 2509 59.4% comp 173.8 MB/s uncomp 397.3 MB/s This is on a Core2, 64bit, GCC 4.5.2. Please someone port it to C and linux-kernel code so it could be used with zram instead of LZO.
chubs 16y ago
Had a look at the code, it's quite neat and tidy, i'm really impressed and surprised considering the need for speed/optimisation in libraries like this tends to make the code unreadable...
tptacek 16y ago
When you can measure efficiency improvements like this in millions of dollars, I'm sure this makes a whole hell of a lot of sense. But for anyone below, say, Twitter's scale: is this ever an engineering win over zlib?
- Andys 16y ago
  Perhaps a more pertinent question: Is it any better than LZJB or LZO?
  - wmf 16y ago
    It's got a better license than LZJB (and maybe LZO depending on your views).
    - lemming 16y ago
      And LZO.
    - ot 16y ago
      Doesn't Google use GPL2 all over the place?
      - wmf 16y ago
        Just to use one example, I would assume that the crawler/indexer/ranker "secret sauce" in the appliance can't link against any GPL libraries.
        tptacek 16y ago
        If you don't distribute your GPL-tainted code, I don't believe you need to do anything to comply with the GPL. This is why the Affero GPL exists.
        tonfa 16y ago
        The keyword in the parent comment was "appliance" :)
endgame 16y ago
IMHO, the build system could do with a little work: * The various bits generated from and added by the autotools shouldn't be committed. autoreconf -i works really well these days. That's INSTALL Makefile.in aclocal.m4 compile config.guess config.h.in config.sub configure depcomp install-sh ltmain.sh missing mkinstalldirs. configure.ac: * Needs to call AC_SUBST([LIBTOOL_DEPS]) or else the rule to rebuild libtool in Makefile.am won't work. * A lot of macro calls are underquoted. It'll probably work fine, but it's poor style. * The dance with EXTRA_LIBSNAPPY_LDFLAGS seems odd. It'd be more conventional to do something like: SNAPPY_LTVERSION=snappy_version AC_SUBST([SNAPPY_LTVERSION]) and set the -version-info flag directly in Makefile.am. If it's to allow the user to provide custom LDFLAGS, it's unnecessary: LDFLAGS is part of libsnappy_la_LINK. Here's the snippet from Makefile.in: libsnappy_la_LINK = $(LIBTOOL) --tag=CXX $(AM_LIBTOOLFLAGS) \ $(LIBTOOLFLAGS) --mode=link $(CXXLD) $(AM_CXXFLAGS) \ $(CXXFLAGS) $(libsnappy_la_LDFLAGS) $(LDFLAGS) -o $@ * There should be an AC_ARG_WITH for gflags, because automagic dependencies aren't cool: http://www.gentoo.org/proj/en/qa/automagic.xml http://www.gentoo.org/proj/en/qa/automagic.xml * Shell variables starting with ac_ are in autoconf's namespace. Setting things like ac_have_builtin_ctz is therefore equally uncool. See http://www.gnu.org/s/hello/manual/autoconf/Macro-Names.html http://www.gnu.org/s/hello/manual/autoconf/Macro-Names.html : > To ensure that your macros don't conflict with present or future Autoconf macros, you should prefix your own macro names and any shell variables they use with some other sequence. Possibilities include your initials, or an abbreviation for the name of your organization or software package. * Use AS_IF instead of directly using the shell's `if`: http://www.gnu.org/software/hello/manual/autoconf/Limitations-of-Builtins.html#Limitations-of-Builtins http://www.gnu.org/software/hello/manual/autoconf/Limitation... and http://www.gnu.org/s/hello/manual/autoconf/Common-Shell-Constructs.html http://www.gnu.org/s/hello/manual/autoconf/Common-Shell-Cons... . * Consider adding -Wall to either AUTOMAKE_OPTIONS in Makefile.am or as an argument to AM_INIT_AUTOMAKE. If you don't mind using a modern automake (1.11 or later), also call AM_SILENT_RULES([yes]). Even MSYS has automake-1.11 these days. Makefile.am: * Adding $(GTEST_CPPFLAGS) to both snappy_unittest_CPPFLAGS and snappy_unittest_CXXFLAGS is redundant. See this part of Makefile.in: snappy_unittest-snappy-test.o: snappy-test.cc @am__fastdepCXX_TRUE@ $(CXX) $(DEFS) $(DEFAULT_INCLUDES) $(INCLUDES) $(snappy_unittest_CPPFLAGS) $(CPPFLAGS) $(snappy_unittest_CXXFLAGS) $(CXXFLAGS) -MT snappy_unittest-snappy-test.o -MD -MP -MF $(DEPDIR)/snappy_unittest-snappy-test.Tpo -c -o snappy_unittest-snappy-test.o `test -f 'snappy-test.cc' || echo '$(srcdir)/'`snappy-test.cc @am__fastdepCXX_TRUE@ $(am__mv) $(DEPDIR)/snappy_unittest-snappy-test.Tpo $(DEPDIR)/snappy_unittest-snappy-test.Po @AMDEP_TRUE@@am__fastdepCXX_FALSE@ source='snappy-test.cc' object='snappy_unittest-snappy-test.o' libtool=no @AMDEPBACKSLASH@ @AMDEP_TRUE@@am__fastdepCXX_FALSE@ DEPDIR=$(DEPDIR) $(CXXDEPMODE) $(depcomp) @AMDEPBACKSLASH@ @am__fastdepCXX_FALSE@ $(CXX) $(DEFS) $(DEFAULT_INCLUDES) $(INCLUDES) $(snappy_unittest_CPPFLAGS) $(CPPFLAGS) $(snappy_unittest_CXXFLAGS) $(CXXFLAGS) -c -o snappy_unittest-snappy-test.o `test -f 'snappy-test.cc' || echo '$(srcdir)/'`snappy-test.cc * snappy_unittest should be in check_PROGRAMS, not noinst_PROGRAMS. That way, it's built as part of `make check`, not `make all`.
patrickaljord 16y ago
Anyone feels like writing a tiny C client? Looks like this comes only as a lib.
- tonfa 16y ago
  I think it is already a command line application (if you use the google cmdline flag lib, see the README).
  - endgame 16y ago
    I don't believe so. Look at the Makefile.am. It only declares a libtool library (lib_LTLIBRARIES = libsnappy.la, which causes libsnappy.so.x.y.z and libsnappy.a to be built), and a unittest (TESTS = snappy_unittest,noinst_PROGRAMS = $(TESTS)). EDIT: I'm wrong: > Actually, snappy_unittest is more than just a test, despite the name; in particular, if you have the gflags package installed, it can be used for benchmarking. Thus, it's useful to have it built as part of make all. I'm closing this one (but thanks for all the others!). http://code.google.com/p/snappy/issues/detail?id=11&can=1 http://code.google.com/p/snappy/issues/detail?id=11&can=...
deleted 16y ago
[deleted]
haberman 16y ago
I did a double-take when I saw this -- the library is called "zippy" internally, but there must have been some kind of trademark issue with that. This is used in more than just BigTable; it's used extensively inside Google, in particular for the RPC system. It's great to see it open-sourced!
- stevenbedrick 16y ago
  Maybe the issue was regarding the comic strip, "Zippy the pinhead"?
  - pmjordan 16y ago
    My first association was with the character from the UK children's TV show Rainbow: http://en.wikipedia.org/wiki/Zippy_(Rainbow) http://en.wikipedia.org/wiki/Zippy_(Rainbow) I doubt it's a trademark issue, as trademarks tend to be usage-specific. (e.g. Apple Computer vs. Apple Music) It's probably just a matter of Google feeling there are too many different things already associated with the name 'Zippy.' (I also now have the "Rainbow" theme tune stuck in my head.)
- tonfa 16y ago
  There are quite a lot of hits about random projects being called zippy.
- fryguy 16y ago
  Do you remember go?
  - cdibona 16y ago
    Go is common enough that it is untrademarkable. Zippy? Not so much, hence the name change.
MichaelGG 16y ago
For pure speed, check out QuickLZ[1]. While it probably doesn't compress well as Snappy, it does hit 300MB+/core. But, it's GPL instead of Apache. 1: http://www.quicklz.com/ http://www.quicklz.com/
- _delirium 16y ago
  They claim in the README that Snappy is usually faster than QuickLZ. Haven't tested myself, though.
  - deleted 16y ago
    [deleted]
  - Lazze 16y ago
    They compare with QuickLZ 1.0 from 2006 (you can see that from snappy_unittest.cc). I've done a quick benchmark with the files on quicklz.com on the same machine with QuickLZ 1.5.0: test file; library; compressed size; compression speed in mb/s average QuickLZ 47.9% 308 snappy 53.0% 261 proteins.txt QuickLZ 1.5.0 35.6% 331 snappy 40.5% 232 plaintext.txt QuickLZ 1.5.0 48.1% 245 snappy 55.5% 193 gdb.exe QuickLZ 1.5.0 45.8% 270 snappy 51.1% 214 flower.bmp QuickLZ 1.5.0 86.7% snappy 91.5% 208 northwind.mdf QuickLZ 1.5.0 23.2% snappy 26.4% 456
    - Sesse_ 16y ago
      Hi, I work at Google, and I've been working on the Snappy open-sourcing. First of all, let me state that when doing our benchmarks, we were definitely comparing against the latest public version (1.5.0) of QuickLZ. I'm having problems reproducing your benchmark results; could you provide some more details about the exact settings, compiler version, architecture etc.? I can't even get all of the compression ratios to match up exactly with the ones I'm seeing, so there must be some sort of difference between the setups. Are you perchance running Snappy with assertions enabled? (The microbenchmark will complain if you do, so it's easy to check.) For what it's worth, I ran the QuickLZ test suite on my own machine (you can see the exact testing environment below), and got quite different results: qlz-testsuite/flower.bmp : QUICKLZ: [b 1M] bytes 5922816 -> 5087471 85.9% comp 132.4 MB/s uncomp 140.8 MB/s SNAPPY: [b 4M] bytes 5922816 -> 5369439 90.7% comp 180.6 MB/s uncomp 484.0 MB/s qlz-testsuite/gdb.exe : QUICKLZ: [b 1M] bytes 8872609 -> 4074990 45.9% comp 177.4 MB/s uncomp 213.3 MB/s SNAPPY: [b 4M] bytes 8872609 -> 4530851 51.1% comp 224.7 MB/s uncomp 468.7 MB/s qlz-testsuite/northwind.mdf : QUICKLZ: [b 1M] bytes 2752512 -> 628607 22.8% comp 291.3 MB/s uncomp 409.6 MB/s SNAPPY: [b 4M] bytes 2752512 -> 726437 26.4% comp 456.6 MB/s uncomp 790.1 MB/s qlz-testsuite/plaintext.txt : QUICKLZ: [b 1M] bytes 2899483 -> 1436466 49.5% comp 155.4 MB/s uncomp 182.7 MB/s SNAPPY: [b 4M] bytes 2899483 -> 1629408 56.2% comp 195.2 MB/s uncomp 487.5 MB/s qlz-testsuite/proteins.txt : QUICKLZ: [b 1M] bytes 7254685 -> 2600129 35.8% comp 218.2 MB/s uncomp 261.2 MB/s SNAPPY: [b 4M] bytes 7254685 -> 2934406 40.4% comp 269.4 MB/s uncomp 556.9 MB/s Average compression ratio (geometric mean, lower is better): QuickLZ 43.7%, Snappy 48.9%. Average compression speed (harmonic mean): QuickLZ 180.9 MB/s, Snappy 238.0 MB/s. Average decompresion speed (harmonic mean): QuickLZ 212.5 MB/s, Snappy 536.9 MB/s. In addition, there's one nearly-incompressible file listed on the same page, which I also ran on (it's not included in the averages above): qlz-testsuite/NotTheMusic.mp4 : QUICKLZ: [b 1M] bytes 9832475 -> 9832565 100.0% comp 289.3 MB/s uncomp 2815.9 MB/s SNAPPY: [b 4M] bytes 9832475 -> 9485433 96.5% comp 1308.4 MB/s uncomp 2477.1 MB/s This is on a Nehalem 2.27GHz, running Debian GNU/Linux 6.0 in 64-bit mode, GCC version 4.4.5, with flags -O2 -g -DNDEBUG for both Snappy (1.0.0) and QuickLZ (1.5.0), running snappy_unittest to compare. QuickLZ was left at the default settings, ie. the files were exactly as downloaded from http://www.quicklz.com/quicklz.[ch] http://www.quicklz.com/quicklz.[ch] as of today. In particular, this means QuickLZ was running in unsafe mode, which means it could crash on corrupted compressed input. (Safe mode, which is comparable to how Snappy runs, is, according to http://www.quicklz.com/download.html http://www.quicklz.com/download.html, 15–20% slower at decompression, so the difference would be bigger.) Exact benchmark results will, as always, vary with differing compiler, set of flags, CPU, and data set. The average numbers from this benchmark, however, show QuickLZ 1.5.0 (in unsafe mode) compressing about 11% more densely than Snappy 1.0.0 does, but Snappy compressing about 32% faster and decompressing about 153% faster (ie., about 2.53 times as fast). /* Steinar */
swah 16y ago
Sometimes its odd to think a thousand lines of C++ is something folks were waiting to be released for years.
delineal 16y ago
I may convert my monthly archive of websites over to snappy; the speed of the compression / decompression will allow me to implement a more consolidated storage scheme than I'm using now.
nonameman 16y ago
You should probably mention why someone would choose this library over another compression library. I think good advice would be to use Snappy to compress data that is meant to be kept in memory, as Bigtable does with the underlying SSTables. If you are reading from disk, a slower algorithm with a better compression ratio is probably a better choice because the cost of the disk seek will dominate the cost of the compression algorithm.
sanxiyn 16y ago
A note from a developer: http://blog.sesse.net/blog/tech/2011-03-22-19-24_snappy.html http://blog.sesse.net/blog/tech/2011-03-22-19-24_snappy.html
tezza 16y ago
Curious, it's written in C++ . IMHO I think straight C would have been easier for World + Dog to link against.
- bajsejohannes 16y ago
  If exports are within `extern "C"`, it shouldn't be any more difficult.
wladimir 16y ago
Another incredible internal project open-sourced by Google. I really respect Google's dedication to improving the speed of the internet in general, and to open source. Of course this benefits them as well, but it's a form of enlightened self-interest that, to me, is very refreshing compared to for example Microsoft, and other companies that only care about their own software/platforms and only release stuff on need-to-know basis.
philf 16y ago
I'm puzzled by snappy-stubs-internal.h l105-118 Why would one log by instantiating a class, not using the result, therefore leading to the destructor being called which writes the log message? Can anyone come up with a reason for this?
- daniel02216 16y ago
  It uses the same construct as VLOG above does, which allows you to << an error string into CRASH_UNLESS and VLOG. The weirdness comes in when you want to conditionalize the log stream, which isn't implemented in the stubs here. The real Google logging classes are probably more sophisticated, and this is just a shim to get the snappy code to work unchanged. Specifically the answer to your question is that printing in the destructor is there to print a newline right before calling abort but after having printed whatever was <<'ed to CRASH_UNLESS. Check out libjingle's logging implementation which has some similar weirdness in it: http://code.google.com/p/libjingle/source/browse/trunk/talk/base/logging.h http://code.google.com/p/libjingle/source/browse/trunk/talk/...

235 points by tonfa 16y ago

ot 16y ago

I wonder if they had evaluated LZO (http://www.oberhumer.com/opensource/lzo/ http://www.oberhumer.com/opensource/lzo/) before writing this. It is quite well-tested (a variant on it runs in the Mars Rovers) and very very fast: the author reports 16MB/sec on a Pentium 133, on modern architectures it should easily get to the 500MB/sec claimed by snappy.

lemming 16y ago
"The LZO algorithms and implementations are copyrighted OpenSource distributed under the GNU General Public License." Does this even make sense? Can you apply the GPL to an algorithm? As I understand it, if there's no patent I should be able to implement it with no problems.
- wmf 16y ago
  People who want to upsell you a commercial version tend to say such things.
  - bhickey 16y ago
    Be warned: the main source code in the 'src' directory is a real pain to understand as I've experimented with hundreds of slightly different versions. It contains many #if and some gotos, and is *completely optimized for speed* and not for readability. Code sharing of the different algorithms is implemented by stressing the preprocessor - this can be really confusing. Lots of marcos and assertions don't make things better. Given the author's statements, I don't know that I'd feel comfortable using LZO in a production environment.
    - malkia 16y ago
      We used it in our games since Ultimate Spiderman. I've got assembly version working on the PS2 IOP chip (33Mhz, the chip used to run PS1 games, otherwise I/O and sound for PS2). I've got 5mb/s decompressed data speed, so that was speeding our disk access. I had to tweak just a little bit the source code and make sure unaligned writes (4 bytes) were used and that made it x2 or x3 faster.
drv 16y ago
The README says: "In our tests, Snappy usually is faster than algorithms in the same class (e.g. LZO, LZF, FastLZ, QuickLZ, etc.) while achieving comparable compression ratios."
- ot 16y ago
  Interesting, I had missed that. It also says > Finally, snappy can benchmark Snappy against a few other compression libraries (zlib, LZO, LZF, FastLZ and QuickLZ), if they were detected at configure time. I'll do some tests, I'm curious about the results
  - tonfa 16y ago
    Please post the results :)
wolf550e 16y ago
~/snappy-read-only $ ./snappy_unittest -norun_microbenchmarks -lzo testdata/* testdata/alice29.txt : LZO: [b 1M] bytes 152089 -> 82691 54.4% comp 64.5 MB/s uncomp 206.4 MB/s SNAPPY: [b 4M] bytes 152089 -> 90965 59.8% comp 171.5 MB/s uncomp 375.8 MB/s testdata/asyoulik.txt : LZO: [b 1M] bytes 125179 -> 73217 58.5% comp 52.5 MB/s uncomp 173.2 MB/s SNAPPY: [b 4M] bytes 125179 -> 80207 64.1% comp 137.8 MB/s uncomp 301.7 MB/s testdata/baddata1.snappy : LZO: [b 1M] bytes 27512 -> 26487 96.3% comp 24.4 MB/s uncomp 491.6 MB/s SNAPPY: [b 4M] bytes 27512 -> 26675 97.0% comp 305.2 MB/s uncomp 1465.7 MB/s testdata/baddata2.snappy : LZO: [b 1M] bytes 27483 -> 26528 96.5% comp 24.3 MB/s uncomp 499.1 MB/s SNAPPY: [b 4M] bytes 27483 -> 26724 97.2% comp 331.8 MB/s uncomp 1660.4 MB/s testdata/baddata3.snappy : LZO: [b 1M] bytes 28384 -> 27380 96.5% comp 24.1 MB/s uncomp 488.2 MB/s SNAPPY: [b 4M] bytes 28384 -> 27476 96.8% comp 275.7 MB/s uncomp 1346.5 MB/s testdata/cp.html : LZO: [b 1M] bytes 24603 -> 11621 47.2% comp 57.2 MB/s uncomp 258.5 MB/s SNAPPY: [b 4M] bytes 24603 -> 11838 48.1% comp 190.0 MB/s uncomp 443.5 MB/s testdata/fields.c : LZO: [b 1M] bytes 11150 -> 4663 41.8% comp 73.8 MB/s uncomp 259.8 MB/s SNAPPY: [b 4M] bytes 11150 -> 4728 42.4% comp 207.3 MB/s uncomp 431.7 MB/s testdata/geo.protodata : LZO: [b 1M] bytes 100000 -> 20423 20.4% comp 115.8 MB/s uncomp 429.9 MB/s SNAPPY: [b 4M] bytes 100000 -> 23488 23.5% comp 359.0 MB/s uncomp 625.8 MB/s testdata/grammar.lsp : LZO: [b 1M] bytes 3721 -> 1781 47.9% comp 66.8 MB/s uncomp 311.4 MB/s SNAPPY: [b 4M] bytes 3721 -> 1800 48.4% comp 214.9 MB/s uncomp 461.2 MB/s testdata/house.jpg : LZO: [b 1M] bytes 126958 -> 127173 100.2% comp 20.2 MB/s uncomp 1420.8 MB/s SNAPPY: [b 4M] bytes 126958 -> 126803 99.9% comp 2037.6 MB/s uncomp 7578.7 MB/s testdata/html : LZO: [b 1M] bytes 102400 -> 21027 20.5% comp 115.1 MB/s uncomp 423.8 MB/s SNAPPY: [b 4M] bytes 102400 -> 24140 23.6% comp 362.4 MB/s uncomp 703.5 MB/s testdata/html_x_4 : LZO: [b 1M] bytes 409600 -> 82980 20.3% comp 120.9 MB/s uncomp 416.4 MB/s SNAPPY: [b 4M] bytes 409600 -> 96472 23.6% comp 359.0 MB/s uncomp 698.0 MB/s testdata/kennedy.xls : LZO: [b 1M] bytes 1029744 -> 357315 34.7% comp 133.5 MB/s uncomp 533.6 MB/s SNAPPY: [b 4M] bytes 1029744 -> 425735 41.3% comp 294.1 MB/s uncomp 432.6 MB/s testdata/kppkn.gtb : LZO: [b 1M] bytes 184320 -> 71671 38.9% comp 84.3 MB/s uncomp 235.9 MB/s SNAPPY: [b 4M] bytes 184320 -> 70535 38.3% comp 231.7 MB/s uncomp 373.8 MB/s testdata/lcet10.txt : LZO: [b 1M] bytes 426754 -> 221290 51.9% comp 57.6 MB/s uncomp 181.9 MB/s SNAPPY: [b 4M] bytes 426754 -> 243710 57.1% comp 153.7 MB/s uncomp 341.4 MB/s testdata/mapreduce-osdi-1.pdf : LZO: [b 1M] bytes 94330 -> 76999 81.6% comp 24.9 MB/s uncomp 810.9 MB/s SNAPPY: [b 4M] bytes 94330 -> 77477 82.1% comp 709.0 MB/s uncomp 1669.5 MB/s testdata/plrabn12.txt : LZO: [b 1M] bytes 481861 -> 294610 61.1% comp 51.4 MB/s uncomp 164.8 MB/s SNAPPY: [b 4M] bytes 481861 -> 329339 68.3% comp 129.8 MB/s uncomp 281.9 MB/s testdata/ptt5 : LZO: [b 1M] bytes 513216 -> 86232 16.8% comp 119.0 MB/s uncomp 506.1 MB/s SNAPPY: [b 4M] bytes 513216 -> 93455 18.2% comp 479.8 MB/s uncomp 670.8 MB/s testdata/sum : LZO: [b 1M] bytes 38240 -> 17686 46.2% comp 57.8 MB/s uncomp 267.0 MB/s SNAPPY: [b 4M] bytes 38240 -> 19837 51.9% comp 194.5 MB/s uncomp 401.0 MB/s testdata/urls.10K : LZO: [b 1M] bytes 702087 -> 309320 44.1% comp 55.7 MB/s uncomp 265.1 MB/s SNAPPY: [b 4M] bytes 702087 -> 357267 50.9% comp 216.9 MB/s uncomp 499.3 MB/s testdata/xargs.1 : LZO: [b 1M] bytes 4227 -> 2450 58.0% comp 56.3 MB/s uncomp 286.2 MB/s SNAPPY: [b 4M] bytes 4227 -> 2509 59.4% comp 173.8 MB/s uncomp 397.3 MB/s This is on a Core2, 64bit, GCC 4.5.2. Please someone port it to C and linux-kernel code so it could be used with zram instead of LZO.

chubs 16y ago

Had a look at the code, it's quite neat and tidy, i'm really impressed and surprised considering the need for speed/optimisation in libraries like this tends to make the code unreadable...

tptacek 16y ago

When you can measure efficiency improvements like this in millions of dollars, I'm sure this makes a whole hell of a lot of sense. But for anyone below, say, Twitter's scale: is this ever an engineering win over zlib?

Andys 16y ago
Perhaps a more pertinent question: Is it any better than LZJB or LZO?
- wmf 16y ago
  It's got a better license than LZJB (and maybe LZO depending on your views).
  - lemming 16y ago
    And LZO.
  - ot 16y ago
    Doesn't Google use GPL2 all over the place?
    - wmf 16y ago
      Just to use one example, I would assume that the crawler/indexer/ranker "secret sauce" in the appliance can't link against any GPL libraries.
      - tptacek 16y ago
        If you don't distribute your GPL-tainted code, I don't believe you need to do anything to comply with the GPL. This is why the Affero GPL exists.
        tonfa 16y ago
        The keyword in the parent comment was "appliance" :)

endgame 16y ago

IMHO, the build system could do with a little work: * The various bits generated from and added by the autotools shouldn't be committed. autoreconf -i works really well these days. That's INSTALL Makefile.in aclocal.m4 compile config.guess config.h.in config.sub configure depcomp install-sh ltmain.sh missing mkinstalldirs. configure.ac: * Needs to call AC_SUBST([LIBTOOL_DEPS]) or else the rule to rebuild libtool in Makefile.am won't work. * A lot of macro calls are underquoted. It'll probably work fine, but it's poor style. * The dance with EXTRA_LIBSNAPPY_LDFLAGS seems odd. It'd be more conventional to do something like: SNAPPY_LTVERSION=snappy_version AC_SUBST([SNAPPY_LTVERSION]) and set the -version-info flag directly in Makefile.am. If it's to allow the user to provide custom LDFLAGS, it's unnecessary: LDFLAGS is part of libsnappy_la_LINK. Here's the snippet from Makefile.in: libsnappy_la_LINK = $(LIBTOOL) --tag=CXX $(AM_LIBTOOLFLAGS) \ $(LIBTOOLFLAGS) --mode=link $(CXXLD) $(AM_CXXFLAGS) \ $(CXXFLAGS) $(libsnappy_la_LDFLAGS) $(LDFLAGS) -o $@ * There should be an AC_ARG_WITH for gflags, because automagic dependencies aren't cool: http://www.gentoo.org/proj/en/qa/automagic.xml http://www.gentoo.org/proj/en/qa/automagic.xml * Shell variables starting with ac_ are in autoconf's namespace. Setting things like ac_have_builtin_ctz is therefore equally uncool. See http://www.gnu.org/s/hello/manual/autoconf/Macro-Names.html http://www.gnu.org/s/hello/manual/autoconf/Macro-Names.html : > To ensure that your macros don't conflict with present or future Autoconf macros, you should prefix your own macro names and any shell variables they use with some other sequence. Possibilities include your initials, or an abbreviation for the name of your organization or software package. * Use AS_IF instead of directly using the shell's `if`: http://www.gnu.org/software/hello/manual/autoconf/Limitations-of-Builtins.html#Limitations-of-Builtins http://www.gnu.org/software/hello/manual/autoconf/Limitation... and http://www.gnu.org/s/hello/manual/autoconf/Common-Shell-Constructs.html http://www.gnu.org/s/hello/manual/autoconf/Common-Shell-Cons... . * Consider adding -Wall to either AUTOMAKE_OPTIONS in Makefile.am or as an argument to AM_INIT_AUTOMAKE. If you don't mind using a modern automake (1.11 or later), also call AM_SILENT_RULES([yes]). Even MSYS has automake-1.11 these days. Makefile.am: * Adding $(GTEST_CPPFLAGS) to both snappy_unittest_CPPFLAGS and snappy_unittest_CXXFLAGS is redundant. See this part of Makefile.in: snappy_unittest-snappy-test.o: snappy-test.cc @am__fastdepCXX_TRUE@ $(CXX) $(DEFS) $(DEFAULT_INCLUDES) $(INCLUDES) $(snappy_unittest_CPPFLAGS) $(CPPFLAGS) $(snappy_unittest_CXXFLAGS) $(CXXFLAGS) -MT snappy_unittest-snappy-test.o -MD -MP -MF $(DEPDIR)/snappy_unittest-snappy-test.Tpo -c -o snappy_unittest-snappy-test.o `test -f 'snappy-test.cc' || echo '$(srcdir)/'`snappy-test.cc @am__fastdepCXX_TRUE@ $(am__mv) $(DEPDIR)/snappy_unittest-snappy-test.Tpo $(DEPDIR)/snappy_unittest-snappy-test.Po @AMDEP_TRUE@@am__fastdepCXX_FALSE@ source='snappy-test.cc' object='snappy_unittest-snappy-test.o' libtool=no @AMDEPBACKSLASH@ @AMDEP_TRUE@@am__fastdepCXX_FALSE@ DEPDIR=$(DEPDIR) $(CXXDEPMODE) $(depcomp) @AMDEPBACKSLASH@ @am__fastdepCXX_FALSE@ $(CXX) $(DEFS) $(DEFAULT_INCLUDES) $(INCLUDES) $(snappy_unittest_CPPFLAGS) $(CPPFLAGS) $(snappy_unittest_CXXFLAGS) $(CXXFLAGS) -c -o snappy_unittest-snappy-test.o `test -f 'snappy-test.cc' || echo '$(srcdir)/'`snappy-test.cc * snappy_unittest should be in check_PROGRAMS, not noinst_PROGRAMS. That way, it's built as part of `make check`, not `make all`.

patrickaljord 16y ago

Anyone feels like writing a tiny C client? Looks like this comes only as a lib.

tonfa 16y ago
I think it is already a command line application (if you use the google cmdline flag lib, see the README).
- endgame 16y ago
  I don't believe so. Look at the Makefile.am. It only declares a libtool library (lib_LTLIBRARIES = libsnappy.la, which causes libsnappy.so.x.y.z and libsnappy.a to be built), and a unittest (TESTS = snappy_unittest,noinst_PROGRAMS = $(TESTS)). EDIT: I'm wrong: > Actually, snappy_unittest is more than just a test, despite the name; in particular, if you have the gflags package installed, it can be used for benchmarking. Thus, it's useful to have it built as part of make all. I'm closing this one (but thanks for all the others!). http://code.google.com/p/snappy/issues/detail?id=11&can=1 http://code.google.com/p/snappy/issues/detail?id=11&can=...

deleted 16y ago

[deleted]

haberman 16y ago

I did a double-take when I saw this -- the library is called "zippy" internally, but there must have been some kind of trademark issue with that. This is used in more than just BigTable; it's used extensively inside Google, in particular for the RPC system. It's great to see it open-sourced!

stevenbedrick 16y ago
Maybe the issue was regarding the comic strip, "Zippy the pinhead"?
- pmjordan 16y ago
  My first association was with the character from the UK children's TV show Rainbow: http://en.wikipedia.org/wiki/Zippy_(Rainbow) http://en.wikipedia.org/wiki/Zippy_(Rainbow) I doubt it's a trademark issue, as trademarks tend to be usage-specific. (e.g. Apple Computer vs. Apple Music) It's probably just a matter of Google feeling there are too many different things already associated with the name 'Zippy.' (I also now have the "Rainbow" theme tune stuck in my head.)
tonfa 16y ago
There are quite a lot of hits about random projects being called zippy.
fryguy 16y ago
Do you remember go?
- cdibona 16y ago
  Go is common enough that it is untrademarkable. Zippy? Not so much, hence the name change.

MichaelGG 16y ago

For pure speed, check out QuickLZ[1]. While it probably doesn't compress well as Snappy, it does hit 300MB+/core. But, it's GPL instead of Apache. 1: http://www.quicklz.com/ http://www.quicklz.com/

_delirium 16y ago
They claim in the README that Snappy is usually faster than QuickLZ. Haven't tested myself, though.
- deleted 16y ago
  [deleted]
- Lazze 16y ago
  They compare with QuickLZ 1.0 from 2006 (you can see that from snappy_unittest.cc). I've done a quick benchmark with the files on quicklz.com on the same machine with QuickLZ 1.5.0: test file; library; compressed size; compression speed in mb/s average QuickLZ 47.9% 308 snappy 53.0% 261 proteins.txt QuickLZ 1.5.0 35.6% 331 snappy 40.5% 232 plaintext.txt QuickLZ 1.5.0 48.1% 245 snappy 55.5% 193 gdb.exe QuickLZ 1.5.0 45.8% 270 snappy 51.1% 214 flower.bmp QuickLZ 1.5.0 86.7% snappy 91.5% 208 northwind.mdf QuickLZ 1.5.0 23.2% snappy 26.4% 456
  - Sesse_ 16y ago
    Hi, I work at Google, and I've been working on the Snappy open-sourcing. First of all, let me state that when doing our benchmarks, we were definitely comparing against the latest public version (1.5.0) of QuickLZ. I'm having problems reproducing your benchmark results; could you provide some more details about the exact settings, compiler version, architecture etc.? I can't even get all of the compression ratios to match up exactly with the ones I'm seeing, so there must be some sort of difference between the setups. Are you perchance running Snappy with assertions enabled? (The microbenchmark will complain if you do, so it's easy to check.) For what it's worth, I ran the QuickLZ test suite on my own machine (you can see the exact testing environment below), and got quite different results: qlz-testsuite/flower.bmp : QUICKLZ: [b 1M] bytes 5922816 -> 5087471 85.9% comp 132.4 MB/s uncomp 140.8 MB/s SNAPPY: [b 4M] bytes 5922816 -> 5369439 90.7% comp 180.6 MB/s uncomp 484.0 MB/s qlz-testsuite/gdb.exe : QUICKLZ: [b 1M] bytes 8872609 -> 4074990 45.9% comp 177.4 MB/s uncomp 213.3 MB/s SNAPPY: [b 4M] bytes 8872609 -> 4530851 51.1% comp 224.7 MB/s uncomp 468.7 MB/s qlz-testsuite/northwind.mdf : QUICKLZ: [b 1M] bytes 2752512 -> 628607 22.8% comp 291.3 MB/s uncomp 409.6 MB/s SNAPPY: [b 4M] bytes 2752512 -> 726437 26.4% comp 456.6 MB/s uncomp 790.1 MB/s qlz-testsuite/plaintext.txt : QUICKLZ: [b 1M] bytes 2899483 -> 1436466 49.5% comp 155.4 MB/s uncomp 182.7 MB/s SNAPPY: [b 4M] bytes 2899483 -> 1629408 56.2% comp 195.2 MB/s uncomp 487.5 MB/s qlz-testsuite/proteins.txt : QUICKLZ: [b 1M] bytes 7254685 -> 2600129 35.8% comp 218.2 MB/s uncomp 261.2 MB/s SNAPPY: [b 4M] bytes 7254685 -> 2934406 40.4% comp 269.4 MB/s uncomp 556.9 MB/s Average compression ratio (geometric mean, lower is better): QuickLZ 43.7%, Snappy 48.9%. Average compression speed (harmonic mean): QuickLZ 180.9 MB/s, Snappy 238.0 MB/s. Average decompresion speed (harmonic mean): QuickLZ 212.5 MB/s, Snappy 536.9 MB/s. In addition, there's one nearly-incompressible file listed on the same page, which I also ran on (it's not included in the averages above): qlz-testsuite/NotTheMusic.mp4 : QUICKLZ: [b 1M] bytes 9832475 -> 9832565 100.0% comp 289.3 MB/s uncomp 2815.9 MB/s SNAPPY: [b 4M] bytes 9832475 -> 9485433 96.5% comp 1308.4 MB/s uncomp 2477.1 MB/s This is on a Nehalem 2.27GHz, running Debian GNU/Linux 6.0 in 64-bit mode, GCC version 4.4.5, with flags -O2 -g -DNDEBUG for both Snappy (1.0.0) and QuickLZ (1.5.0), running snappy_unittest to compare. QuickLZ was left at the default settings, ie. the files were exactly as downloaded from http://www.quicklz.com/quicklz.[ch] http://www.quicklz.com/quicklz.[ch] as of today. In particular, this means QuickLZ was running in unsafe mode, which means it could crash on corrupted compressed input. (Safe mode, which is comparable to how Snappy runs, is, according to http://www.quicklz.com/download.html http://www.quicklz.com/download.html, 15–20% slower at decompression, so the difference would be bigger.) Exact benchmark results will, as always, vary with differing compiler, set of flags, CPU, and data set. The average numbers from this benchmark, however, show QuickLZ 1.5.0 (in unsafe mode) compressing about 11% more densely than Snappy 1.0.0 does, but Snappy compressing about 32% faster and decompressing about 153% faster (ie., about 2.53 times as fast). /* Steinar */

swah 16y ago

Sometimes its odd to think a thousand lines of C++ is something folks were waiting to be released for years.

delineal 16y ago

I may convert my monthly archive of websites over to snappy; the speed of the compression / decompression will allow me to implement a more consolidated storage scheme than I'm using now.

nonameman 16y ago

You should probably mention why someone would choose this library over another compression library. I think good advice would be to use Snappy to compress data that is meant to be kept in memory, as Bigtable does with the underlying SSTables. If you are reading from disk, a slower algorithm with a better compression ratio is probably a better choice because the cost of the disk seek will dominate the cost of the compression algorithm.

sanxiyn 16y ago

A note from a developer: http://blog.sesse.net/blog/tech/2011-03-22-19-24_snappy.html http://blog.sesse.net/blog/tech/2011-03-22-19-24_snappy.html

tezza 16y ago

Curious, it's written in C++ . IMHO I think straight C would have been easier for World + Dog to link against.

bajsejohannes 16y ago
If exports are within `extern "C"`, it shouldn't be any more difficult.

wladimir 16y ago

Another incredible internal project open-sourced by Google. I really respect Google's dedication to improving the speed of the internet in general, and to open source. Of course this benefits them as well, but it's a form of enlightened self-interest that, to me, is very refreshing compared to for example Microsoft, and other companies that only care about their own software/platforms and only release stuff on need-to-know basis.

philf 16y ago

I'm puzzled by snappy-stubs-internal.h l105-118 Why would one log by instantiating a class, not using the result, therefore leading to the destructor being called which writes the log message? Can anyone come up with a reason for this?

daniel02216 16y ago
It uses the same construct as VLOG above does, which allows you to << an error string into CRASH_UNLESS and VLOG. The weirdness comes in when you want to conditionalize the log stream, which isn't implemented in the stubs here. The real Google logging classes are probably more sophisticated, and this is just a shim to get the snappy code to work unchanged. Specifically the answer to your question is that printing in the destructor is there to print a newline right before calling abort but after having printed whatever was <<'ed to CRASH_UNLESS. Check out libjingle's logging implementation which has some similar weirdness in it: http://code.google.com/p/libjingle/source/browse/trunk/talk/base/logging.h http://code.google.com/p/libjingle/source/browse/trunk/talk/...