3 ms·
SQL. Most of my ETL jobs consist of setting up the overhead pieces in pyspark then using SQL for the most important logic. It's way more portable that way.
by sbussard 8y ago
SQL. Most of my ETL jobs consist of setting up the overhead pieces in pyspark then using SQL for the most important logic. It's way more portable that way.
- sitkack 8y agoWhat I like about using SQL is I can prototype with SQLite, Postgres, etc and then adapt and move to Hive, Impala, BigQuery, etc. SQL is literally everywhere, and I don't have to learn to paradigms and frameworks.