pyarrow and parquet may fit your requirements. There are a number of analysis engines (spark, drill, hive, etc) that support parquet. For more detail see: http

pyarrow and parquet may fit your requirements. There are a number of analysis engines (spark, drill, hive, etc) that support parquet. For more detail see: https://tech.blue-yonder.com/efficient-dataframe-storage-with-apache-parquet/ https://tech.blue-yonder.com/efficient-dataframe-storage-wit... https://www.slideshare.net/julienledem/strata-ny-2017-parquet-arrow-roadmap https://www.slideshare.net/julienledem/strata-ny-2017-parque... If ACID guarantees are a must then use Kafka as a message broker between memory and file, but this has the cost of added complexity. For more info: https://eng-staging.verizondigitalmedia.com/2017/04/28/Kafka-to-Hdfs-ParquetSerializer/ https://eng-staging.verizondigitalmedia.com/2017/04/28/Kafka... http://activisiongamescience.github.io/2016/06/15/Kafka-Client-Benchmarking/ http://activisiongamescience.github.io/2016/06/15/Kafka-Clie...