3 ms·
I think the big contender has been dataframe processing using Pandas, R and others. These have their own methods for doing the type of data munging and analysis
by reilly3000 7y ago
I think the big contender has been dataframe processing using Pandas, R and others. These have their own methods for doing the type of data munging and analysis that has been the traditional territory of SQL. I won’t soon forget the half day I plowed into getting the right Pandas pipeline dialed in only to realize I could have written a better SQL query to fetch the correct data to begin with.
- prepend 7y agoI’m sure there will be an xkcd about this soon. I’ve also tried to do things in sql only to realize that I could have done it easier in pandas. I think it’s usually a mix of both for me, but I try to at least get a decent “tidy” dataset out of sql and then do all the manipulation in pandas because a python pipeline is more portable than a sql pipeline, I think.