4 ms·
Can this be installed on AWS EMR clusters?
by Lucasoato 6y ago
Can this be installed on AWS EMR clusters?
- daviesliu 6y agoJuiceFS can used in AWS EMR cluster, we have customers using JuiceFS to address the consistency and performance issues from S3. There is a blog post [1] talking about this use case. Unfortunately, we have not publish a English version, you can read it using google translate . [1] https://juicefs.com/blog/cn/posts/globalegrow-big-data-platform-user-case/ https://juicefs.com/blog/cn/posts/globalegrow-big-data-platf... Disclaimer: Founder of JuiceFS here.
- threeseed 6y agoYou could but I would do thorough real-world benchmarks. EMRFS has dozens of optimisations for Spark/Hadoop workloads e.g. S3 select, partitioning pruning, optimised committers etc and since EMR is a core product it is continually being improved. Using HopsFS would negate all of that.
- SirOibaf 6y agoNot really, but you can try it out on https://hopsworks.ai https://hopsworks.ai It's conceptually similar to EMR in the way it works. You connect your AWS account and we'll deploy a cluster there. HopsFS will run on top a S3 bucket in your organization. You get a fully featured Spark environment (With metrics and logging included - no need for cloudwatch). UI with Jupyter notebooks, the Hopsworks feature store and ML capabilities that EMR does not provide.
- jamesblonde 6y agoYeah, https://hopsworks.ai https://hopsworks.ai is an alternative to EMR that includes HopsFS out-of-the-box. You get $4k free credits right now - it includes Spark, Flink, Python, Hive, Feature Store, GPUs (tensorflow), Jupyter, notebooks as jobs, AirFlow. And a UI for development and operations. So kind of like Databricks with more of a ML focus, but still supporting Spark.