4 ms·
Yes, I meant coalesce. It makes sense that output is distributed, but it caught me by surprise since I had not seen anything about that on the official site, n
by minimaxir 10y ago
Yes, I meant coalesce.
It makes sense that output is distributed, but it caught me by surprise since I had not seen anything about that on the official site, nor in any literature. It is unusual since the other Spark libraries have smart defaults.
My CSV had a columns of DenseVectors containing floats and when exported, it used the internal representation of numbers (see second tweet in reply to first tweet)
- nchammas 10y agoI think it would be dangerous to have the default be "coalesce to 1 partition before writing", but I agree this should be better documented since it takes many people by surprise. As for the DenseVectors, that looks strange and is perhaps worthy of a report on the project tracker.