3 ms·
The performance of different queries is different across a typical RDBMS and something like Spark/BigQuery. A traditional database is efficient (i.e. cost effe
by Eridrus 4y ago
The performance of different queries is different across a typical RDBMS and something like Spark/BigQuery.
A traditional database is efficient (i.e. cost effective) at doing a lot of the same query repeatedly, e.g. looking up a customer's account balance, whereas these query engines are good at doing infrequent queries with lots of complicated, expensive logic on very large datasets.
You could run simple account lookup queries in a CRUD app with Spark, but you'd be setting a lot of compute/money on fire, and your latency would probably be terrible.
- alexott 4y agoThere are a lot of optimizations in query engines as well - results cache, local caches of files stored in cloud, data layout optimization and skipping (to avoid reading not necessary files), etc.