3 ms·
Have you tried Apache Spark + pyspark + Jupyter? Spark allows SQL against all data frames bound to the current context: df1.createOrReplaceTempView("table1")
by tomnipotent 9y ago
Have you tried Apache Spark + pyspark + Jupyter? Spark allows SQL against all data frames bound to the current context:
df1.createOrReplaceTempView("table1")
df2.createOrReplaceTempView("table2")
spark.sql("SELECT * FROM table1 t1, table2 t2 WHERE t1.id=t2.fid")
- darksaints 9y agoI haven't tried it but there is a package for R that I find similarly useful, called sqldf. The back end is sqlite, which is quite fast for anything that fits on disk. I find both hobbled, however, by their rather simple SQL dialects. No window functions, common table expressions, grouping sets or similar, etc. An improvement, no doubt, but Postgres still has it beat. I honestly wish the vast corpus of effort put into statistics and machine learning in R and Python were more easily portable as Postgres foreign function packages. I've tried the ones I've seen mentioned and they were all horrible, mostly due to embedding an entire language's runtime and memory model on top of the SQL inner workings. A project on my backburner, waiting for the time to work on it :)