6 ms·
Side note this is a useful pattern I use in general. Use python to get some data into SQLite then I either use fast api, datasette, or Streamlit depending on wh
by faizshah 5y ago
Side note this is a useful pattern I use in general. Use python to get some data into SQLite then I either use fast api, datasette, or Streamlit depending on what I want to do with the data. SQLite is great and a lot of the power of SQLite I feel hasn’t even been fully tapped yet by the community, especially virtual tables.
- simonw 5y agoYeah SQLite is such a powerful intermediary file format, I use it like that all the time.
- omegalulw 5y agoWhy not use Pandas?
- simonw 5y agoYou can use Pandas with SQLite - Pandas saves to and loads from SQLite databases out of the box.
- bfung 5y agoWhere would pandas fit in here? For what purpose?
- faizshah 5y agoI use pandas or pyspark when I am transforming the data. When I am ready to distribute the data or serve the data behind an api, script or web app I write it to SQLite. The benefit is every language can read sqlite, theres a rich ecosystem of sqlite tools like datasette and peewee to build with and sqlite can act as a serverless database keeping infrastructure costs low for serving the data. I also like that sqlite has built in full text search and fuzzy matching capabilities making it quick to cook up a nice webapp for static data. Also depending on your use case its nice to not have a dependency on pandas or take up a lot of room in your lambda package size for instance.