3 ms·
Regarding the reduction in lines of SQL. Doesn't this imply a tradeoff with where compute is utilized? By reducing SQL, wouldn't you be "pushing up" the compute
by chrisjc 5y ago
Regarding the reduction in lines of SQL. Doesn't this imply a tradeoff with where compute is utilized? By reducing SQL, wouldn't you be "pushing up" the compute from the database to the JVM while also increasing the amount of data that has to be moved around?
Edit: Wait, did you imply that the SQL you replaced with Scala ultimately gets transpiled into SQL (spark or other dataframe engine) and "pushed down" to the database? Now you're just maintaining Scala and not a rats nest of SQL strings concatenated together? If so, that's pretty awesome.
- random314 5y agoMost of this was Hadoop. Which executes mapreduce jobs written in sql on the jvm. The scalding jobs executed faster, except for one job for which we had to stick with Apache pig. Spark jobs were native RDDs before the SQL Spark era.
- random314 5y agoNo. There was no transpiling to SQL. This was several years ago, before Spark SQL. The SQL was HiveQL. Spark didn't really support SQL at the time and was native JVM Scala RDDs. HiveQL with UDFs compiled to JVM bytecode typically ended up performing worse than scalding that corresponds to straight raw jvm map reduce jobs that are the real native interface of Hadoop. The real performance drag from HQL came from developers writing inefficient map reduce jobs with extra joins to avoid writing Java UDFs because they were a huge PITA. In scalding, developers wrote map reduce jobs with as few group bys as possible. Our hadoop infra team actually complimented us on how great the execution profile of our teams jobs were.