Cool! Most of the time we run Spark as a single-node (i.e. --master local[*]) as now we can easily utilise large nodes like 128 core, 512GB and Spark does scal

Cool! Most of the time we run Spark as a single-node (i.e. --master local[*]) as now we can easily utilise large nodes like 128 core, 512GB and Spark does scale vertically well but also runs relatively well on a small node like the Docker example on the website running on a laptop. The ability to run SQL against separate storage is Spark's killer feature in my view. Arc does support the full Scala API which you can implement as a plugin (https://arc.tripl.ai/plugins/ https://arc.tripl.ai/plugins/) so for advanced teams they have full control. The reason we went for SQL-first is that we are trying to find the balance that allows Business Analysts to develop their own logic without having to learn Scala or even Python - as they probably already know SQL. Hopefully some of the ideas are relevant to what you are building.