4 ms·

Great questions! The Spark hyperparameter selection uses an exhaustive grid search approach, which can take a long time (complexity grows exponentially with nu

Great questions! The Spark hyperparameter selection uses an exhaustive grid search approach, which can take a long time (complexity grows exponentially with number of parameters to tune) and produce poor results when compared to other methods [1]. Bayesian optimization is a great way to tune time consuming and expensive functions like ML pipelines, where finding a good configuration in a small number of total attempts is the only tractable way to tune the system. ALS implementation in MLLib still requires ratings in RDD and hasn't moved over to ML/DataFrames yet. Edit: Looks like the original comment was edited, but this post does in fact use the built in MLlib ALS implementation. [1]: http://arxiv.org/abs/1603.09441 http://arxiv.org/abs/1603.09441

minimaxir 10y ago
> ALS implementation in MLLib still requires ratings in RDD and hasn't moved over to ML/DataFrames yet. ALS has been available in ML/DataFrames since 1.3.0, per the documentation. (http://spark.apache.org/docs/latest/api/scala/index.html#org.apache.spark.ml.recommendation.ALS http://spark.apache.org/docs/latest/api/scala/index.html#org...) > Edit: Looks like the original comment was edited, but this post does in fact use the built in MLlib cross validation methods. I edited the comment to correct my error that ALS was not mentioned as being native to Spark. However, for Hyperparameter cross validation, I looked at the code in the repository (https://github.com/sigopt/sigopt-examples/blob/master/spark/recommender_sys/src/main/scala/MovieLense.scala https://github.com/sigopt/sigopt-examples/blob/master/spark/...), and while the Spark CrossValidation function is imported at the beginning of the Scala file (from ML, not MLlib), it is not used in the code, in favor of SigOpt. I'm confused.
- idewanck 10y ago
  Thanks for pointing out the switch away from RDD, looking at the master branch for MLlib in github it looks like ALS is still using RDDs? https://github.com/apache/spark/blob/master/mllib/src/main/scala/org/apache/spark/ml/recommendation/ALS.scala https://github.com/apache/spark/blob/master/mllib/src/main/s... Admittedly I'm not a spark expert though, thanks for bringing this up!
  - minimaxir 10y ago
    MLlib is for operating in RDDs, ML is for operating in DataFrames. RDDs themselves, however, are obsolete to DataFrames as DataFrames are faster and cover most of the functionality of RDDs for common use cases.
    - Zephyr314 10y ago
      Thanks for the heads up! We'll make sure to use DataFrames in our next Spark example. As mentioned before, we thought this would be a cool application of Bayesian optimization (especially since the Spark docs advocate the exponentially slower grid search approach as you pointed out), but we are not Spark experts. We appreciate the feedback!
    - kod 10y ago
      RDDs are not obsolete. The reason dataframes are faster is exactly because they are more restrictive than RDDs. If you need to run arbitrary code, RDDs are stil more flexible.
      - dxbydt 10y ago
        Since every dataframe has a lazy instance of rdd, and several methods on dataframes simply call the corresponding method on rdd ( eg. foreach ), I am not sure about the faster bit of your assertion.
        nl 10y ago
        They are faster in the sense that many thing you previously had to do in a RDD lambda are now Dataframe operations (which are optimized by the Catalyst compiler). So if you want to do one of the operations in the sql.functions package[1] then Dataframes (and Datasets) are very valuable. If not, then they won't give you much benefit. However, you will get a little improvement because the Tachyon out-of-JVM-memory framework[2] which I don't think RDD version has access to. [1] http://spark.apache.org/docs/2.0.0/api/python/pyspark.sql.html#module-pyspark.sql.functions http://spark.apache.org/docs/2.0.0/api/python/pyspark.sql.ht... [2] https://dzone.com/articles/Accelerate-In-Memory-Processing-with-Spark-from-Hours-to-Seconds-With-Tachyon https://dzone.com/articles/Accelerate-In-Memory-Processing-w...

minimaxir 10y ago

> ALS implementation in MLLib still requires ratings in RDD and hasn't moved over to ML/DataFrames yet. ALS has been available in ML/DataFrames since 1.3.0, per the documentation. (http://spark.apache.org/docs/latest/api/scala/index.html#org.apache.spark.ml.recommendation.ALS http://spark.apache.org/docs/latest/api/scala/index.html#org...) > Edit: Looks like the original comment was edited, but this post does in fact use the built in MLlib cross validation methods. I edited the comment to correct my error that ALS was not mentioned as being native to Spark. However, for Hyperparameter cross validation, I looked at the code in the repository (https://github.com/sigopt/sigopt-examples/blob/master/spark/recommender_sys/src/main/scala/MovieLense.scala https://github.com/sigopt/sigopt-examples/blob/master/spark/...), and while the Spark CrossValidation function is imported at the beginning of the Scala file (from ML, not MLlib), it is not used in the code, in favor of SigOpt. I'm confused.

idewanck 10y ago
Thanks for pointing out the switch away from RDD, looking at the master branch for MLlib in github it looks like ALS is still using RDDs? https://github.com/apache/spark/blob/master/mllib/src/main/scala/org/apache/spark/ml/recommendation/ALS.scala https://github.com/apache/spark/blob/master/mllib/src/main/s... Admittedly I'm not a spark expert though, thanks for bringing this up!
- minimaxir 10y ago
  MLlib is for operating in RDDs, ML is for operating in DataFrames. RDDs themselves, however, are obsolete to DataFrames as DataFrames are faster and cover most of the functionality of RDDs for common use cases.
  - Zephyr314 10y ago
    Thanks for the heads up! We'll make sure to use DataFrames in our next Spark example. As mentioned before, we thought this would be a cool application of Bayesian optimization (especially since the Spark docs advocate the exponentially slower grid search approach as you pointed out), but we are not Spark experts. We appreciate the feedback!
  - kod 10y ago
    RDDs are not obsolete. The reason dataframes are faster is exactly because they are more restrictive than RDDs. If you need to run arbitrary code, RDDs are stil more flexible.
    - dxbydt 10y ago
      Since every dataframe has a lazy instance of rdd, and several methods on dataframes simply call the corresponding method on rdd ( eg. foreach ), I am not sure about the faster bit of your assertion.
      - nl 10y ago
        They are faster in the sense that many thing you previously had to do in a RDD lambda are now Dataframe operations (which are optimized by the Catalyst compiler). So if you want to do one of the operations in the sql.functions package[1] then Dataframes (and Datasets) are very valuable. If not, then they won't give you much benefit. However, you will get a little improvement because the Tachyon out-of-JVM-memory framework[2] which I don't think RDD version has access to. [1] http://spark.apache.org/docs/2.0.0/api/python/pyspark.sql.html#module-pyspark.sql.functions http://spark.apache.org/docs/2.0.0/api/python/pyspark.sql.ht... [2] https://dzone.com/articles/Accelerate-In-Memory-Processing-with-Spark-from-Hours-to-Seconds-With-Tachyon https://dzone.com/articles/Accelerate-In-Memory-Processing-w...