Avro is a strong contender for this use case: https://avro.apache.org/docs/current/ https://avro.apache.org/docs/current/ https://medium.com/ssense-tech/csv-vs

Avro is a strong contender for this use case: https://avro.apache.org/docs/current/ https://avro.apache.org/docs/current/ https://medium.com/ssense-tech/csv-vs-parquet-vs-avro-choosing-the-right-tool-for-the-right-job-79c9f56914a8 https://medium.com/ssense-tech/csv-vs-parquet-vs-avro-choosi... It's well-supported in Pandas and Kafka, has good schema support, and reasonably small compressed file sizes.