You can accomplish this using time-travel queries in frameworks like Apache Hudi and Databricks Delta that i mentioned in more detail in an earlier comment. The

You can accomplish this using time-travel queries in frameworks like Apache Hudi and Databricks Delta that i mentioned in more detail in an earlier comment. They only work for Spark-based data pipelines.