3 ms·
Arrow has union types (as well as structs and dictionary types). Parquet doesn't but I think it has an intentionally shallow types system to allow flexibility i
by thinkharderdev 4y ago
Arrow has union types (as well as structs and dictionary types). Parquet doesn't but I think it has an intentionally shallow types system to allow flexibility in encoding. Basically everything is either a numeric or binary and the logical type for binary columns is defined in metadata. So you can use, for instance, Arrow as the encoding.
- Kinrany 4y agoYes, the comparison with Arrow ecosystem should really be more in depth since that's the closest thing that exists.