3 ms·

While definitely doable, using a SQL database to do data discovery is obtuse.

orthoxerox 2y ago
Not really. DuckDB can operate on raw csv without ingesting them. This is as easy if not easier than setting up python and pandas.
- infecto 2y ago
  Totally agree that it depends on the tools available but given a typical toolset (sql and python). I would lean towards python because some types of analysis are easier to express in code, especially when working on top of the multiple data sets.
zerkten 2y ago
It's obtuse, but it's effective for some people. I know someone who hires devs from the .NET and Python space for finance. When asked to suggest a solution to any kind of data problem like this the interviewees split down the middle with the .NET ones almost entirely using a relational database and the Python ones using either the library du jour or suggesting something on the command line (since they often have Linux experience.) In both parts of their org, the .NET area tends to use SQL exclusively for analysis and the Python folks use Pandas and bunch of other stuff. These departments are also in significantly different parts of their organization with their own mandates and culture.
- hermitdev 2y ago
  As someone with +20 years in finance (hedge funds/trading) and knows .Net, Java, Python, C++, shell, etc, the first question I'd ask is: where's the data? If I'm being asked to do data analysis, it's because we need the answer yesterday. So, the tool I choose always going to be a matter of which gets me the answer fastest and with the least amount of friction. That's almost always dictated by where the data is _now_. Not where it'd ideally be.
  - ryanjshaw 2y ago
    It's in this email attachment right here. Now what?
    - dotancohen 2y ago
      To me that sounds like a csv file and it'll likely be a python script from me.
pjmlp 2y ago
Depends, that is what OLAP is all about, naturally most of the good tools happen to be commercial for the enterprise space.

Not really. DuckDB can operate on raw csv without ingesting them. This is as easy if not easier than setting up python and pandas.

infecto 2y ago
Totally agree that it depends on the tools available but given a typical toolset (sql and python). I would lean towards python because some types of analysis are easier to express in code, especially when working on top of the multiple data sets.

zerkten 2y ago

It's obtuse, but it's effective for some people. I know someone who hires devs from the .NET and Python space for finance. When asked to suggest a solution to any kind of data problem like this the interviewees split down the middle with the .NET ones almost entirely using a relational database and the Python ones using either the library du jour or suggesting something on the command line (since they often have Linux experience.) In both parts of their org, the .NET area tends to use SQL exclusively for analysis and the Python folks use Pandas and bunch of other stuff. These departments are also in significantly different parts of their organization with their own mandates and culture.

hermitdev 2y ago
As someone with +20 years in finance (hedge funds/trading) and knows .Net, Java, Python, C++, shell, etc, the first question I'd ask is: where's the data? If I'm being asked to do data analysis, it's because we need the answer yesterday. So, the tool I choose always going to be a matter of which gets me the answer fastest and with the least amount of friction. That's almost always dictated by where the data is _now_. Not where it'd ideally be.
- ryanjshaw 2y ago
  It's in this email attachment right here. Now what?
  - dotancohen 2y ago
    To me that sounds like a csv file and it'll likely be a python script from me.

pjmlp 2y ago

Depends, that is what OLAP is all about, naturally most of the good tools happen to be commercial for the enterprise space.