3 ms·
SenSage (a log aggregation tool with many security-related features) has used compressed per-column raw data at the lowest level in its distributed storage engi
by nwatson 17y ago
SenSage (a log aggregation tool with many security-related features) has used compressed per-column raw data at the lowest level in its distributed storage engine since 2001. Writes are fast, and when reading raw data at the lowest level to fulfill queries in its multi-level multi-node map-reduce style query processor it wins big-time over traditional RDBMSs because (a) it reads only the columns relevant to the query, not having to pull fields out of rows/pages; (b) it's faster to pull & decompress than pull uncompressed data. Storage typically is 10x+ smaller than the original logs, even with redundant copies. RDBMSs inflate the data size instead.