3 ms·
It's ever discussed in https://github.com/apache/arrow-rs/issues/3888 https://github.com/apache/arrow-rs/issues/3888 for comparing object_store in Apache Arrow
by tison 3y ago
It's ever discussed in https://github.com/apache/arrow-rs/issues/3888 https://github.com/apache/arrow-rs/issues/3888 for comparing object_store in Apache Arrow to the APIs provided by Apache OpenDAL.
Briefly, Apache OpenDAL is a library providing FS-like APIs over multiple storage backends, including S3 and many other cloud storage.
A few database systems, such as GreptimeDB and Databend, use OpenDAL as a better S3 SDK to access data on cloud storage.
Other solutions exist to manage filesystem-like interfaces over S3, including Alluxio and JuiceFS. Unlike Apache OpenDAL, Alluxio and JuiceFS need to be deployed standalone and have a dedicated internal metadata service.
- Lucasoato 3y agoI'm not sure if Alluxio could be substituted by OpenDAL as a local cache layer for TrinoDB.
- tison 3y agoIf I get "local cache layer" correctly, it's possible. And it's even desired if you want to reduce the deployment burden. Here are some related codes on how we implement such a layer in GreptimeDB: * https://github.com/GreptimeTeam/greptimedb/blob/v0.7.0/src/object-store/src/layers/lru_cache.rs https://github.com/GreptimeTeam/greptimedb/blob/v0.7.0/src/o... * https://github.com/GreptimeTeam/greptimedb/blob/v0.7.0/src/mito2/src/sst/parquet/helper.rs https://github.com/GreptimeTeam/greptimedb/blob/v0.7.0/src/m...