Datafusion, and Ballista by definition, also provides a Dataframe API that let's you construct queries programmatically. It also has preliminary support for UDF

Datafusion, and Ballista by definition, also provides a Dataframe API that let's you construct queries programmatically. It also has preliminary support for UDFs. We also have community members implementing Spark native executors using Datafusion, which showed significant speed improvements in the initial PoC.