2 ms·
How familiar are you with Spark and the like? This is what it looks like: https://spark.apache.org/examples.html https://spark.apache.org/examples.html SQL is
by esafak 3y ago
How familiar are you with Spark and the like? This is what it looks like:
https://spark.apache.org/examples.html https://spark.apache.org/examples.html
SQL is just a DSL; it is not the only or primary API for Spark, and there's nothing magical about it. If you ditch it you can get your type safety, composability, and testability back, like so:
https://medium.com/@sergey.kotlov/unit-testing-of-spark-applications-cf0162a5cb3c https://medium.com/@sergey.kotlov/unit-testing-of-spark-appl...
See those case classes that neatly encapsulate business objects? Add to that functional transforms that concisely express typical operations like filtering, mapping, and so on, you get something that is simply superior to SQL.
- monero-xmr 3y agoAny ORM provides the same features for any SQL database. There is nothing special going on here. If perhaps the database autogenerated a bunch of classes, maybe that’s interesting? I think some projects have introspected a database and created all the boilerplate language classes before. There is nothing magical about wrapping database objects in language classes. This has been happening forever. https://docs.sqlalchemy.org/en/20/orm/quickstart.html#select-with-join https://docs.sqlalchemy.org/en/20/orm/quickstart.html#select... Nothing magical about using a function call rather than raw SQL.
- esafak 3y agoNo, it does not. Furthermore, if you're using an ORM you're not programming in SQL any more. You're using a poor man's Spark. Spark lets you drop down to SQL in all its APIs too. I don't understand your argument if you're comfortable with ORMs.
- monero-xmr 3y agoI’m not even sure what we are debating. Spark is for big data work, it isn’t something you would typically use as a general purpose database backing your application. My original point way back is that expecting customers to learn an entirely new paradigm for storing and querying data is a poor decision, and limits your work to niche use cases. Spark has a large niche but it isn’t comparable to MySQL / Postgres or other newish data stores like Dynamo. Furthermore any competent engineer knows SQL because ORMs are cumbersome and annoying for anything except basic use.