4 ms·
Very nice. I prefer using the built in command line ec2 scripts packages with all Spark versions. https://spark.apache.org/docs/latest/ec2-scripts.html https:
by eranation 11y ago
Very nice.
I prefer using the built in command line ec2 scripts packages with all Spark versions.
https://spark.apache.org/docs/latest/ec2-scripts.html https://spark.apache.org/docs/latest/ec2-scripts.html
You can even specify the spot instance price, instance type, number of nodes, etc.
e.g.
./spark-ec2 --key-pair=awskey --identity-file=awskey.pem --region=us-west-1 --zone=us-west-1a --spot-price=0.2 --instance-type=m4.4xlarge launch my-spark-cluster
- ddrum001 11y agoIndeed, the script works quite well for Spark - but we also wanted to provide a guide for those who would like to further understand the config files, especially for those who want to 'tinker' with these down the road (e.g. adding nodes to the cluster). Also, we find it helpful for learning to set up Spark on other platforms, and setting up other systems that don't have ec2 scripts yet.