Ummmm no I would disagree with you. If you expect all of it to be compressed into one partition each executor has to send its data back to the parent/driver and

Ummmm no I would disagree with you. If you expect all of it to be compressed into one partition each executor has to send its data back to the parent/driver and the driver has to handle the task. In most cases, like ours, we have a driver that is skimpy on RAM (around 2 gig) and each executor/slave has around 4-8 gigs. Each slave/executor thread it spawned has to do it's own job and return when it's done. I am planning to start a blog for our nlp search engine www.shoten.xyz Our pains and our aims.