4 ms·
If you are working within the Apache Spark ecosystem you can us DeltaLake https://delta.io/ https://delta.io/ to create 'merge' datasets which are transactional
by seddonm1 6y ago
If you are working within the Apache Spark ecosystem you can us DeltaLake https://delta.io/ https://delta.io/ to create 'merge' datasets which are transactional, versioned and allow time travel by both version number and timestamp.
- jamesblonde 6y agoAnother alternative to Deltalake is Apache Hudi, which also includes bloom filters for indexing time-travel queries (efficiently exclude any files given the supplied time constraint). Z-ordered indexing in Deltalake is not available yet in open-source deltalake, only in Databricks version.