5 ms·
Stavros from TileDB, Inc. here: Arrow employs a columnar format to store objects like data frames. TileDB is also columnar (so you can sub-select attributes and
by stavrospap 9y ago
Stavros from TileDB, Inc. here: Arrow employs a columnar format to store objects like data frames. TileDB is also columnar (so you can sub-select attributes and perform analytics with very similar optimizations to Arrow), but the first-class citizens are (the more general) dense and sparse multi-dimensional arrays. Moreover, TileDB focuses on optimizing for the persistent storage backend, so that it can handle out-of-core analytics on massive datasets that cannot fit in main memory (e.g., genomics), or offer the same performance using less RAM (leading to cost savings in the cloud). Nevertheless, we are quite fond of Arrow, so we hope to work together at some point and integrate as seamlessly as possible.
- polskibus 9y agoWhat do you mean by out-of-core?
- stavrospap 9y agoIn-core algorithms require the entire array(s) in main memory to perform some computation. Out-of-core algorithms are typically block-based and stream the array blocks from persistent storage to main memory on demand, working on parts of the array(s) at a time, thus minimizing the memory requirements. If this is done asynchronously and carefully, for some CPU-bound algorithms you may be able to completely hide the storage-to-memory cost, thus saving memory without losing performance.