3 ms·

I notice they're using spark standalone clusters. I've had problems with executors dying when running under YARN, and it makes keeping track of running jobs an

by estefan 12y ago

I notice they're using spark standalone clusters. I've had problems with executors dying when running under YARN, and it makes keeping track of running jobs and finding log output more difficult. So unless you really need to run on the same instances as other MR tech, it seems standalone is the way to go. If only AWS provided EMR AMIs for spark standalone clusters I could switch...

sgt101 12y ago
Is this Spark on YARN or YARN or AWS that is causing the issue?
- estefan 12y ago
  I was trying to run Spark on EMR, so I was running it under YARN on EMR. The problem I found seemed to have been discovered by one or two other people on mailing lists - it seemed something to do with YARN terminating spark executors for memory usage. It was odd since I wasn't using the cluster for anything else - YARN was only used as the scheduler because it was installed in the AMI, and I didn't want to use the ec2 scripts to start a standalone cluster just on EC2s. It was difficult to debug because YARN makes things pretty opaque when tasks just die - logs weren't copied around the cluster so I had to SSH to the instance that died and try to find something to indicate what had happened, and half the time the error messages were vague. I also didn't work out how to view the job tracker UI when running under YARN. "Learning Spark" says I need to proxy through the YARN cluster somehow but doesn't give an example... So I have no progress about running jobs at the moment, only about finished jobs. In the end, I upped the instance size I was using and everything went well. I think my dataset must have been too large and spark was spilling to disk which tripped things up. So at the moment I don't have confidence that I could process huge datasets with Spark on EMR which is a bit annoying :-(

3 ms·

I notice they're using spark standalone clusters. I've had problems with executors dying when running under YARN, and it makes keeping track of running jobs an

by estefan 12y ago

sgt101 12y ago
Is this Spark on YARN or YARN or AWS that is causing the issue?
- estefan 12y ago
  I was trying to run Spark on EMR, so I was running it under YARN on EMR. The problem I found seemed to have been discovered by one or two other people on mailing lists - it seemed something to do with YARN terminating spark executors for memory usage. It was odd since I wasn't using the cluster for anything else - YARN was only used as the scheduler because it was installed in the AMI, and I didn't want to use the ec2 scripts to start a standalone cluster just on EC2s. It was difficult to debug because YARN makes things pretty opaque when tasks just die - logs weren't copied around the cluster so I had to SSH to the instance that died and try to find something to indicate what had happened, and half the time the error messages were vague. I also didn't work out how to view the job tracker UI when running under YARN. "Learning Spark" says I need to proxy through the YARN cluster somehow but doesn't give an example... So I have no progress about running jobs at the moment, only about finished jobs. In the end, I upped the instance size I was using and everything went well. I think my dataset must have been too large and spark was spilling to disk which tripped things up. So at the moment I don't have confidence that I could process huge datasets with Spark on EMR which is a bit annoying :-(