I was looking for this exact type of project to back some of my ML projects datasets. These datasets - Are less than 1 TB at maximum, and are typically less t

I was looking for this exact type of project to back some of my ML projects datasets. These datasets - Are less than 1 TB at maximum, and are typically less than 50GB - Written once and read many times - Require heavy and variable post-processing prior to training - Usually run on an ephemeral large instance which reads from s3, does it's thing, and then shuts down. This last task is incredibly slow in python, and limited on a standard sqllite/postgres DB to a single thread. Having access to an embeddable columnar sql engine could greatly simplify and accelerate this final transformation stage before training.