My company is working [#] on a domain-specific language with similar premises. When running a script, we load data from CSV, and use a clever caching structure

My company is working [#] on a domain-specific language with similar premises. When running a script, we load data from CSV, and use a clever caching structure to avoid any work done on a previous run. https://www.lokad.com/envision-more-technical-overview https://www.lokad.com/envision-more-technical-overview We've actually encountered two kinds of work with such data sets: exploratory work (when a data scientist is running brand new queries on a fixed data set) and production work (when a pre-written script is run on new data a daily or weekly basis, maybe with some user-provided parameters). These are separate optimization problems, the first is usually solved by caching the results of intermediate processing while the second is mostly about optimizing the script execution. In the latter case, I think one of the main difficulties with SQL is that queries are independent and a typical data analysis session requires many queries. With typical database pipelines, there's no obvious spot to perform global static analysis and optimize the query sequence as a whole (for instance, finding shared common expressions or indices). Stored procedures are a good start, but typical query optimizers just aren't designed to achieve any kind of procedure-level analysis. [#] and we're looking to hire another compiler expert, if anyone's interested.