Because if I have to use DataBricks then I need to have things as notebooks, something that I'm desperately trying to migrate the team I work in away from so th

Because if I have to use DataBricks then I need to have things as notebooks, something that I'm desperately trying to migrate the team I work in away from so that we can have actually maintainable code that gets deployed, monitored and held to the same rigour/maintainability that we hold normal dev code to. Also, being forced to use a cluster is such catastrophic overkill for so, so many tasks. I have teammates wanting to use spark/databricks just to process a handful of files in S3 totalling a few gb tops. Realistically we could do the same work, in a single container with Python/Julia/Scala/language of choice in same or less amount of time and with an order of magnitude more maintainability.