No. There was no transpiling to SQL. This was several years ago, before Spark SQL. The SQL was HiveQL. Spark didn't really support SQL at the time and was nati

No. There was no transpiling to SQL. This was several years ago, before Spark SQL. The SQL was HiveQL. Spark didn't really support SQL at the time and was native JVM Scala RDDs. HiveQL with UDFs compiled to JVM bytecode typically ended up performing worse than scalding that corresponds to straight raw jvm map reduce jobs that are the real native interface of Hadoop. The real performance drag from HQL came from developers writing inefficient map reduce jobs with extra joins to avoid writing Java UDFs because they were a huge PITA. In scalding, developers wrote map reduce jobs with as few group bys as possible. Our hadoop infra team actually complimented us on how great the execution profile of our teams jobs were.