3 ms·
100GB can be handled without problems by mysql or postgre on a single node (yeah maybe not a laptop). You need to tune your database and take the right design d
by matteuan 10y ago
100GB can be handled without problems by mysql or postgre on a single node (yeah maybe not a laptop).
You need to tune your database and take the right design decisions, but it's still less work than setting up a distributed system.
- EdwardDiego 10y agoDepends on how you're setting it up. aws s3 cp --recursive bunch-o-data s3://some-bucket/ spark-ec2 --region eu-west-1 --identity-file s.pem --key-pair=spark --instance-type m3.2xlarge --slaves 40 -v 1.5.2 launch my-cluster Is significantly easier than making PG work easily at the 100GB scale in my experience. spark-ec2 is a script that ships with Spark to make it easy to set up a cluster in AWS.
- nchammas 10y agoAs of Spark 2.0 spark-ec2 doesn't ship with Spark anymore [0]. It's been moved here [1]. [0] http://spark.apache.org/releases/spark-release-2-0-0.html#removals http://spark.apache.org/releases/spark-release-2-0-0.html#re... [1] https://github.com/amplab/spark-ec2 https://github.com/amplab/spark-ec2
- RBerenguel 10y agoI have a custom EMR launcher to get a cluster with Spark, Zeppelin and Hive. So yes, much faster for ad-hoc than tuning Postgres, basically it is a matter of if I need ad-hoc reporting/analysis or need it for normal operations. For 100 GB, regular, I'd just drop it into our running Redshift and let it eat it.