3 ms·
I've grown pretty fond of the way Spark SQL queries can be represented with DataFrame operations. There is a more or less 1:1 relationship with SQL, except the
by Nihilartikel 6y ago
I've grown pretty fond of the way Spark SQL queries can be represented with DataFrame operations. There is a more or less 1:1 relationship with SQL, except the commands can be programatically generated and composed. It sure beats stitching together a SQL string when you have a bunch of query clauses that might be optional, or need a generalized way to take 30 result columns, and rename them with a prefix or something.
e.g.
result = dframe.select(*[f.col(colname).alias(f"{colname}_old") for colname in dframe.columns]).join(other_df, 'joincolumn', type='outer')
and so forth.
- Icathian 6y agoI used to use SAS macros to conditionally generate monstrous SQL queries all the time. It was a bit hacky, but man I could make a thousand lines of SAS do just about whatever I wanted however I wanted. It really feels like a powerful way to tackle messy real-life business logic.