3 ms·
Also: df_out.coalesce(1).write.format("com.databricks.spark.csv").option("header", "true").save("filename") I'm not sure if this is the same as repartition
by nl 10y ago
Also:
df_out.coalesce(1).write.format("com.databricks.spark.csv").option("header", "true").save("filename")
I'm not sure if this is the same as repartition(1), but I haven't noticed any problems with data structures. Or the to_pandas() solution works.
The problem with all these approaches is that they bring them all into memory on the driver. The real solution should be to stream to the node and write out as it does.
- minimaxir 10y agoYea, using repartition and coalesce had the same result on my output.