3 ms·
Not "in lieu", but I have found SQLite to be a good companion to pandas. pandas is good when you want to lend some easily navigable structure to your data witho
by abhgh 6y ago
Not "in lieu", but I have found SQLite to be a good companion to pandas. pandas is good when you want to lend some easily navigable structure to your data without leaving python, but becomes cumbersome (it feels cumbersome esp if you know SQL) when you want to do some serious processing. At such times, I just dump my data into a SQLite DB, and from then on my code has SQL queries to it, which I find are far more readable than corresponding pandas statements (also easier for the reviewer).
Another advantage you get is if you wanted to look at these intermediate data, you don't need to run your code in debug mode and view the dataframe at a breakpoint - you can use something like datasette[1] or a standard SQLite DB viewer.
So a function that does complex data processing has approximately this structure my code:
(1) <non-pandas python code to fetch data, other "normal" stuff>
(2) <pandas code to do basic navigation, filtering, grouping etc>
(3) <dump to SQLite, followed by SQL queries for the really heavy stuff>
(4) <back to python, possibly pandas, to return results, write to a file, plot etc>
Step (3) used to be pandas for me before, but depending on how complex your operations are this can become hard to read and/or review.
[1] https://github.com/simonw/datasette https://github.com/simonw/datasette - datasette doesnt replace a standard DB IDE, but is a very good lightweight alternative to one if don't intend to perform updates/inserts directly on a table.
- steve_gh 6y agoI have exactly the opposite workflow. Import data into salute, perform as much heavy lifting as possible in salute, and export into a python data science workflow via led to if necessary.