3 ms·
Have you seen https://ibis-project.org/ https://ibis-project.org/ ?
by anentropic 2y ago
Have you seen https://ibis-project.org/ https://ibis-project.org/ ?
- dayflyer 2y agoYes I'm familiar with Ibis and we use the same technology to do the SQL generation (SQLGlot). Like most products, I made this due to two problems I commonly faced: 1. Generate SQL that represents a PySpark DataFrame pipeline that I maintain so others who prefer SQL can easily understand it 2. I'm already proficient with the PySpark DataFrame API but not all projects I work on require Spark. In both cases Ibis would not have been a solution for my issue. I think if you haven't learned either then it would come down to if you are more interested in the learning the PySpark DataFrame API or Ibis DataFrame API.