4 ms·
I'm pretty sure the problem was that it was fragmented across workers and such. I had already computed the filters and done a df.show() and count prior to tryi
by chaxor 5y ago
I'm pretty sure the problem was that it was fragmented across workers and such.
I had already computed the filters and done a df.show() and count prior to trying save it.
It's just a really unfriendly system for creating packages for everyone to run on a single node unfortunately.
- disgruntledphd2 5y agoOh yeah, there's stuff you can do, but again it takes tuning and ideally it shouldn't be so annoying. I definitely get the frustration, but waiting for sklearn for hours to fit a fraction of the observations that spark can do is also frustrating, in a different way. > run on a single node yeah, Spark is pretty crap here, but it's not really one of their main use cases, to be fair.