Arrow has two possible formats - a stream based one where the first block is the schema/metadata followed by data blocks (ie record batches), and a file based o

Arrow has two possible formats - a stream based one where the first block is the schema/metadata followed by data blocks (ie record batches), and a file based one where the schema is also stored at the end, along with the index locations of each batch (this also allows random access): https://arrow.apache.org/docs/format/Columnar.html#ipc-streaming-format https://arrow.apache.org/docs/format/Columnar.html#ipc-strea... vs https://arrow.apache.org/docs/format/Columnar.html#ipc-file-format https://arrow.apache.org/docs/format/Columnar.html#ipc-file-...