This looked interesting at first sight (considering I work on ORC & Hive) - the single machine configuration with TinyLog and the aggregate indexes look good[1]

This looked interesting at first sight (considering I work on ORC & Hive) - the single machine configuration with TinyLog and the aggregate indexes look good[1]. But the more I look into the distributed storage section, the more corners I see cut in the data consistency section. "There are no quorum writes. You can't write data with confirmation that it was received by more than one replica. If you write a batch of data to one replica and the server with this data ceases to exist before the data has time to get to the other replicas, this data will be lost." "The reason for this is in case of network failures when the client application doesn't know if the data was written to the DB, so the INSERT query can simply be repeated. It doesn't matter which replica INSERTs were sent to with identical data - INSERTs are idempotent. This only works for the last 100 blocks inserted in a table." "Failover is automatic (for small differences in data) or semi-automatic (when data differs too much, which may indicate a configuration error)." "When the server starts (or establishes a new session with ZooKeeper), it only checks the quantity and sizes of all files. If the file sizes match but bytes have been changed somewhere in the middle, this is not detected immediately, but only when attempting to read the data for a SELECT query." These sort of network blips and bad disk problems happen every week at least once on a big cluster and something like hadoop wastes a lot of IO doing bit-rot checks on cold data all the time. The nature of clickstream data makes it somewhat okay to lose a few chunks in transit - I can imagine at least a few of the beacons will get dropped purely over the HTTP mechanism which pumps data into the system. At some point, the data consistency costs money, slows down inserts and creates all sorts of limitations on how recovery of data would play out. But as a general purpose replicated DBMS which serves as a system of record against fraud allegations (for instance), I can't see this comparing well. [1] - 200 million rows/sec for join+aggregation is ~2x as good as Hive 2.0 single node LLAP on a hot run (https://people.apache.org/~gopalv/LLAP.gif https://people.apache.org/~gopalv/LLAP.gif)