4 ms·
Increasingly databases can get do-overs on bad plans. A micro-level example might be dataflow splitting particularly expensive tasks so as to better parallelize
by CHY872 4y ago
Increasingly databases can get do-overs on bad plans. A micro-level example might be dataflow splitting particularly expensive tasks so as to better parallelize them: https://cloud.google.com/blog/products/gcp/no-shard-left-behind-dynamic-work-rebalancing-in-google-cloud-dataflow https://cloud.google.com/blog/products/gcp/no-shard-left-beh...
Spark will re-run the query planner after doing the map stage of a map-reduce operation so as to leverage the information collected: https://databricks.com/blog/2020/05/29/adaptive-query-execution-speeding-up-spark-sql-at-runtime.html https://databricks.com/blog/2020/05/29/adaptive-query-execut...
Both examples represent changing parameters about a lower level plan than simply join operators, but the industry is certainly moving in this direction.
- sitkack 4y agoMost systems that make expensive scans should be updating or creating a count-min-sketch histogram of the data to allow schedulers and query planners to do a better job. https://en.wikipedia.org/wiki/Count%E2%80%93min_sketch https://en.wikipedia.org/wiki/Count%E2%80%93min_sketch https://duckduckgo.com/?t=ffab&q=count-min+sketch+site%3Avldb.org&ia=web https://duckduckgo.com/?t=ffab&q=count-min+sketch+site%3Avld... https://www.semanticscholar.org/search?q=count-min%20sketch%20database&sort=relevance https://www.semanticscholar.org/search?q=count-min%20sketch%...