3 ms·
I'm no expert, but here's my take ... + "Appending cells to a column" ... when writing data to a "table" the columnar database goes through a lot of effort to
by nwatson 8y ago
I'm no expert, but here's my take ...
+ "Appending cells to a column" ... when writing data to a "table" the columnar database goes through a lot of effort to compress each column of data into separate files on disk -- the benefits of compression come from regarding many "rows" of data, per column, at once, so appends probably go to transient areas, and then in the background recent appends are merged-and-compressed into files on disk ... data in all columns in the columnar DBs usually are stored sorted per some timestamp key, and that data doesn't always come in perfect order (e.g., there are different latencies for log records from the same timestamp arriving from different hosts / devices / applications in an SIEM system) -- so appends will be fast to keep up with the huge ingestion flow, but a lot of painstaking expensive compression is done while merging, and re-merging data -- most columnar DBs systems will segment data based on buckets of the timestamp range (and perhaps the more dense the table time-wise, the narrower are the buckets)
+ "Removing a range of cells from a column" ... removing a range of timestamp-ordered or even worse a filtered set of cells will be very expensive -- for all columns in the range to be removed, all deleted items will eventually need to be removed (e.g., 100 and 200 rows in two different timestamp buckets, per column, of 1,000,000 rows of data per bucket) -- reading, uncompressing, and filtering out removed columns will be fast -- recompression and re-writes will be slow ... there may be ways these DBs use side-data to instead, for a period before "vacuum" or in perpetuity, mark rows as deleted rather actually removing and recompressing the data -- these DBs are usually meant for write-once-and-never-delete, read-many-times situations
+ "Searching for data in a column" -- the main benefit from columnar DBs is that, once the data is available in compressed form on disk ... a query needs to read from disk only the specific columns involved in the search filter/terms, plus those needed to generate the output payload -- a typical query might surface just 5 of 80 columns, so compared to a traditional DB that reads all the data for each row (if data strays outside indexed columns), a columnar search will quickly fetch only the already-compressed data for a small number of columns off disk. queries usually involve scanning a huge number of rows in a timerange often expanding multiple timestamp buckets -- for each column, if the columns were stored uncompressed, the disk latency and bandwidth for reading the huge number of row values per column would dwarf the time taken in CPU/memory for processing the data -- with a columnar-DB with compressed columns, each disk read (the slow part) yields a huge number of easily-and-quickly-decompressed row values from that column. the compression is slow, but decompression is very fast ... so many rows of data, per column, can be pumped out to filtering/aggregation portions of the engine, much faster than in a traditional DB (I worked at a place that got 40x compression over traditional DBs)
+ "Performing a map over a column" -- the DB engine can process many rows across the columns of data involved quickly enough -- filters based on value are easy, functions can be applied to columns of data, or data from multiple columns can be joined -- as long as computations involve only data per-row, is usually fast -- it's additional aggregation, and multi-level query processing (results of one query, perhaps aggregated, feed into another), that involve careful planning of a query, especially in a distributed cluster where aggregation "keys" may require retrieval of rows from shards on different hosts
+ "Filtering a column using a user-defined function" ... fast, just like prior point
+ "Combining two or more columns through some user-defined function" ... fairly fast
+ "Sorting" ... here it would get tricky, because to sort a bunch of raw records without filtering takes a huge number of resources ... unless your sort is already on the timestamp field that's the organizing sort field for the columnar DB ... most queries won't involve retrieving 5 hours of data, sorting the raw data by some field other than "main timestamp", and returning results ... usually data is fetched, some aggregation is done in the query (perhaps multiple levels), and sorting is done on the vastly reduced (several orders of magnitude) data -- this kind of sorting is much easier
+ ...
+ "Joins" ... joins between two huge datasets on the raw contents of each table will be real expensive ... so expensive that the columnar DB vendor I worked at (starting in 2000) chose to not implement joins other than joining massive-DB-table-data with vastly smaller lookup-tables -- the value proposition for us was not in doing the joins, it was vastly reduced, reliable storage of huge numbers of log records, resulting in cheaper compression and very fast querying in a distributed cluster