I use pandas or pyspark when I am transforming the data. When I am ready to distribute the data or serve the data behind an api, script or web app I write it to

I use pandas or pyspark when I am transforming the data. When I am ready to distribute the data or serve the data behind an api, script or web app I write it to SQLite. The benefit is every language can read sqlite, theres a rich ecosystem of sqlite tools like datasette and peewee to build with and sqlite can act as a serverless database keeping infrastructure costs low for serving the data. I also like that sqlite has built in full text search and fuzzy matching capabilities making it quick to cook up a nice webapp for static data. Also depending on your use case its nice to not have a dependency on pandas or take up a lot of room in your lambda package size for instance.