5 ms·
> (Yes, my dear aspring data scientist, do not load everything in a huge DataFrame, go learn yourself some SQL :-) ) This hits a little bit too close to home f
by stuxnet79 8y ago
> (Yes, my dear aspring data scientist, do not load everything in a huge DataFrame, go learn yourself some SQL :-) )
This hits a little bit too close to home for me. I am quite proficient at writing performant SQL queries and recently started using Pandas. I find the data frame abstraction better for certain data manipulation tasks. Assuming there is enough RAM available is it still better to offload everything to the database engine?
- fforflo 8y agoAs usual: it depends... If you're doing prototyping and are working in a Jupyter notebook, sure, go ahead and work on the Pandas-level. Once however you're done with prototyping and have settled to a "final_df" (I bet you have something like that in your last notebook cells), maybe you should think transforming some of the "columns" to sql queries (which are VCS-able, sometimes are faster, and most importantly other people can use them too. And instead of 10 people loading 10 different DFs, you can have 10 people querying the same table/view.
- pletnes 8y agoDask has an out-of-memory dataframe implementation. Works great! I think it might support sql queries for that matter.
- paulddraper 8y ago> Assuming there is enough RAM available You will also have to assume you don't care much about the latency introduced by transferring everything over to your working process. (And in batch data processing situations you typically don't care much.)
- perturbation 8y agoDplyr works great with SQL (both SQLite and others).