csv is notoriously bad with spark - in fact you couldn't write to csv for the longest time without external libraries like the databricks one, so I don't contes

csv is notoriously bad with spark - in fact you couldn't write to csv for the longest time without external libraries like the databricks one, so I don't contest your last point. SQlite is usually a single file representing a database though, I don't know how it would work with partitioning and stuff, and then how to handle the schema evolving across sqlite files.