I was trying to run Spark on EMR, so I was running it under YARN on EMR. The problem I found seemed to have been discovered by one or two other people on maili

I was trying to run Spark on EMR, so I was running it under YARN on EMR. The problem I found seemed to have been discovered by one or two other people on mailing lists - it seemed something to do with YARN terminating spark executors for memory usage. It was odd since I wasn't using the cluster for anything else - YARN was only used as the scheduler because it was installed in the AMI, and I didn't want to use the ec2 scripts to start a standalone cluster just on EC2s. It was difficult to debug because YARN makes things pretty opaque when tasks just die - logs weren't copied around the cluster so I had to SSH to the instance that died and try to find something to indicate what had happened, and half the time the error messages were vague. I also didn't work out how to view the job tracker UI when running under YARN. "Learning Spark" says I need to proxy through the YARN cluster somehow but doesn't give an example... So I have no progress about running jobs at the moment, only about finished jobs. In the end, I upped the instance size I was using and everything went well. I think my dataset must have been too large and spark was spilling to disk which tripped things up. So at the moment I don't have confidence that I could process huge datasets with Spark on EMR which is a bit annoying :-(