3 ms·
I was under the impression that Apache Arrow was supposed to be the unified storage representation of data that can be used across Apache projects. i.e HBase s
by filereaper 9y ago
I was under the impression that Apache Arrow was supposed to be the unified storage representation of data that can be used across Apache projects.
i.e HBase stores data as Apache Arrow, which can be directly queried by Spark or Hive without the need for serialization. As in eliminating the current overhead of going from HBase HFiles to Spark's internal RDD/DataFrame representation with lots of ser/deser going on on both fronts.
I think the comparisons to Parquet and ORC are surface level, I expected Arrow to become the "One storage to rule-them-all".
Did I get this wrong? Kindly correct if so.
- jacquesnadeau 9y agoArrow is all about in-memory, not long-term persistence. Systems can write it to disk but it is the one in-memory representation to rule them all, not storage/disk. Disk has its own requirements and challenges outside the scope of Arrow.