This doesn't really respond to the article. The argument here is "jupyter notebooks are clearer", but that's pretty subjective, and the article specifically add

This doesn't really respond to the article. The argument here is "jupyter notebooks are clearer", but that's pretty subjective, and the article specifically addresses this kind of point: "By using SQL, a much wider range of people can read your code, including BI developers, business analysts, data engineers and data scientists." If you really want to advocate for PySpark in this context, you've gotta contend with everything the article brings up: - More people will be able to understand your code - Future proofed, with automatic speed improvements and ‘autoscaling’ - Make data typing someone else’s problem - Simpler maintenance, with less dependency management - Compatibility with good practice software engineering - SQL is more expressive and versatile than it used to be