3 ms·
Is it possible to apply these optimizations to Arrow? Arrow's struct is called StructArray. Fields of StructArray have a StructType. StructType has .bit_width
by westurner 1y ago
Is it possible to apply these optimizations to Arrow?
Arrow's struct is called StructArray. Fields of StructArray have a StructType.
StructType has .bit_width and .byte_width attrs in Python and probably the other implementations too: https://arrow.apache.org/docs/python/generated/pyarrow.StructType.html#pyarrow.StructType.bit_width https://arrow.apache.org/docs/python/generated/pyarrow.Struc...
Arrow supports bitfields with BooleanArray, and enums with categoricals but
"BUG: Categorical columns using the PyArrow backend requires 4x more memory" (open)
https://github.com/pandas-dev/pandas/issues/58062 https://github.com/pandas-dev/pandas/issues/58062 :
> On disk Parquet appears to store the category data as logical type String which is compressed with snappy and encoded
Arrow Flight RPC handles nested structs with enums over the wire somehow too FWIU