3 ms·
This is cool and in my mind super useful for migrations. It seems the main benefit of using something like that in daily life is that it's more convenient to g
by zurfer 2y ago
This is cool and in my mind super useful for migrations.
It seems the main benefit of using something like that in daily life is that it's more convenient to generate complex SQL statements (like pivoting a table with a lot of columns).
However, I never really liked the PySpark dataframe api and looking at the code examples, SQL has the same visual complexity.
Snowflake has built something similar (just for Snowflake) SnowPark [1]. Here one promoted benefit was that you could also inject native Python function and "extend" the SQL dialect. However, I don't think it really took off.
[1] https://github.com/snowflakedb/snowpark-python https://github.com/snowflakedb/snowpark-python
- dayflyer 2y agoYeah I'm familiar with Snowpark and I think their mistake is they implemented an API very similar to PySpark but not exactly the same. I can actually see that the 2nd most common engine looked at in the code of my repo is Snowflake so I think there is interest in it even with Snowpark. I will be adding Snowflake support next. In terms of not liking PySpark, I get that the API is not for everyone and if someone doesn't like PySpark then they are unlikely to like SQLFrame. I think it is great we have a variety of options available to us and ideally each team/organization can find an API that works best for their skillsets/needs but also reduces vendor lock. SQLFrame is a great fit for those teams/organizations who want to standadize on PySpark.