4 ms·
I'm pretty sure the premise here is that the many gigabytes/terabytes of data reside in an RDBMS to begin with, so we aren't introducing another tool -- we're s
by EForEndeavour 5y ago
I'm pretty sure the premise here is that the many gigabytes/terabytes of data reside in an RDBMS to begin with, so we aren't introducing another tool -- we're saying it makes sense to leverage multiple decades' worth of database optimizations by doing a bunch of data filtering, processing, and analysis in the database software and then exporting intermediate results to a pandas environment for final tweaks, visualization, etc.
- dragonwriter 5y ago> I'm pretty sure the premise here is that the many gigabytes/terabytes of data reside in an RDBMS to begin with, so we aren't introducing another tool psql is a separate scriptable client tool from the postgres database server.
- shkkmo 5y agoThe point was about doing more work using the SQL DB engine, the client library you do that with seems irrelevant to making that point.
- dragonwriter 5y ago> The point was about doing more work using the SQL DB engine The specific reference was to “psql script”, which runs in the psql client program, not the DB engine. Since I suspected that might be an error intending to reference SQL running on the server, I separately addressed, in my initial response, both what was literally said (psql script) and sql running in the DB.
- shkkmo 5y agoAny psql script would be building and running SQL queries so I fail to see how that has any impact of the point being made. Edit: The argument is that there are operations that should be done in the SQL layer and it is worth time learning enough about that layer to understand when and how to use it for computation. Once you learn that, it isn't really relevant if you are using psql or some other client library to build those queries.