4 ms·
> Instead of loading data into Pandas and doing a group by SQL can do `group by`. No need for overkill with "data science tools".
by classified 2y ago
> Instead of loading data into Pandas and doing a group by
SQL can do `group by`. No need for overkill with "data science tools".
- sanderjd 2y agoIf you have a csv, pandas is the "less overkill" route than loading it into a database to use sql.
- chrisandchris 2y agoEach their own tool, but that's nothing awk can't do [1]. I prefer a shell script, because shell is everywhere. [1] https://stackoverflow.com/a/75073649 https://stackoverflow.com/a/75073649
- sanderjd 2y agoSorry, but no, awk and shell scripts are not a good choice for this kind of work. I'm sorry, but they just aren't! Sure, go for it if you're just doing stuff for your own personal interest. But if you're doing serious data-driven work in a professional environment, this is going to be an awful choice.
- infecto 2y agoWhile definitely doable, using a SQL database to do data discovery is obtuse.
- orthoxerox 2y agoNot really. DuckDB can operate on raw csv without ingesting them. This is as easy if not easier than setting up python and pandas.
- infecto 2y agoTotally agree that it depends on the tools available but given a typical toolset (sql and python). I would lean towards python because some types of analysis are easier to express in code, especially when working on top of the multiple data sets.
- zerkten 2y agoIt's obtuse, but it's effective for some people. I know someone who hires devs from the .NET and Python space for finance. When asked to suggest a solution to any kind of data problem like this the interviewees split down the middle with the .NET ones almost entirely using a relational database and the Python ones using either the library du jour or suggesting something on the command line (since they often have Linux experience.) In both parts of their org, the .NET area tends to use SQL exclusively for analysis and the Python folks use Pandas and bunch of other stuff. These departments are also in significantly different parts of their organization with their own mandates and culture.
- hermitdev 2y agoAs someone with +20 years in finance (hedge funds/trading) and knows .Net, Java, Python, C++, shell, etc, the first question I'd ask is: where's the data? If I'm being asked to do data analysis, it's because we need the answer yesterday. So, the tool I choose always going to be a matter of which gets me the answer fastest and with the least amount of friction. That's almost always dictated by where the data is _now_. Not where it'd ideally be.
- ryanjshaw 2y agoIt's in this email attachment right here. Now what?
- dotancohen 2y agoTo me that sounds like a csv file and it'll likely be a python script from me.
- pjmlp 2y agoDepends, that is what OLAP is all about, naturally most of the good tools happen to be commercial for the enterprise space.