4 ms·

Generating Recommendations at Amazon Scale with Apache Spark and Amazon DSSTNE

131 points by shanxS 10y ago

minimaxir 10y ago
If you haven't kept up with Spark, and do not have Amazon-level workloads, the built-in machine learning APIs are extremely robust and well-documented (https://people.apache.org/~pwendell/spark-nightly/spark-master-docs/latest/api/python/pyspark.ml.html https://people.apache.org/~pwendell/spark-nightly/spark-mast...), even in the non-Scala languages like Python. There's even a Multilevel Perceptron model creation function, which creates artificial neural networks using the feed-forward/backpropagation everyone loves. The new Spark DataFrames also make manipulating data almost as easy as with Python Pandas / R dplyr. The new Berkeley eDX course (https://courses.edx.org/courses/course-v1:BerkeleyX+CS105x+1T2016/info https://courses.edx.org/courses/course-v1:BerkeleyX+CS105x+1...) is a very good explainer. After Spark 2.0.0 is released, it wouldn't surprise me if it really takes off. (as long as setting up a cluster is a bit easier!)
- nchammas 10y ago
  > as long as setting up a cluster is a bit easier! If you're on AWS, it's already quite easy to set up a cluster today, no? There's EMR of course, and there are tools like spark-ec2 [0] and Flintrock [1]. There are a few more tools listed on Spark Packages that target different cloud providers [2], too. Disclaimer: I am the primary author of Flintrock and am a contributor to spark-ec2. [0] https://github.com/amplab/spark-ec2 https://github.com/amplab/spark-ec2 [1] https://github.com/nchammas/flintrock https://github.com/nchammas/flintrock [2] https://spark-packages.org/?q=tags%3Adeployment https://spark-packages.org/?q=tags%3Adeployment
  - minimaxir 10y ago
    Setting up Spark clusters is easy relative to setting up clusters, but not easy enough yet to set up and configure relative to simply downloading a package in R/Python.
    - nchammas 10y ago
      Well, the absolute easiest way to run Spark is to do it locally (e.g. you can brew install it on a Mac and just go) or to pay for a proprietary service like Databricks, which makes setting up a cluster take a few clicks. That said, I think `flintrock launch my-cluster` is almost as easy as doing `pip install ...`. You do need an AWS account and you do need to set your preferences like region and key name in a config file, but I don't see how you can get out of doing even that without subscribing to some managed service like Databricks that abstracts everything away and replaces it with a nice Web UI.
    - mgummelt 10y ago
      If you're running a DC/OS cluster, it actually is! > dcos package install spark This gives you a Spark CLI, and installs the Spark Cluster Dispatcher so you can run jobs async. It also has options for installing the history server, and for SSL/Kerberos support. Disclaimer: I work at Mesosphere and contribute to the DC/OS Spark package.
vonnik 10y ago
We already built what Amazon was looking for: An extensible deep-learning framework that works on distributed CPUs and GPUs heterogeneously, integrating with Spark as an access layer to orchestrate multiple host threads. https://github.com/deeplearning4j https://github.com/deeplearning4j DL4J may be the DL library with the most sophisticated Spark integration at this point. The trick is to avoid using Spark as a computation layer, since it doesn't do that well. We're pushing a CuDNN wrapper tomorrow. http://deeplearning4j.org/quickstart http://deeplearning4j.org/quickstart Unlike DSSTNE, Tensorflow or CNTK, our deep learning library is neutral, and not designed with the intention of locking people into to a cloud service.
- Fenntrek 10y ago
  >Unlike DSSTNE, Tensorflow or CNTK, our deep learning library is neutral, and not designed with the intention of locking people into to a cloud service. How do any of these lock you into a cloud service?
  - dgacmu 10y ago
    And to reinforce Fenntrek's point: One of the core reasons for open sourcing TensorFlow is to make sure people have the ability to take their code and run it somewhere else. The exact opposite of lock-in. (I can say that with some knowledge - I'm working there right now.) In fact, the cloud service for "hosted" TF isn't even out of alpha yet, but there are many people already using TF on their own hardware. I can't speak for MSFT or AMZN, but all three of these frameworks are completely open source with permissive licenses. I seriously doubt that any of these three providers has lock-in as a goal by releasing their stuff OSS. Goodwill? Absolutely. Getting people to learn the technology they care about internally? Certainly. Hoping that people will play with it and find that they want to run their models on 100 nodes rented from a cloud service? Seems likely. For MSFT, wanting to make sure there was a high-performance DNN framework for Windows? Would make sense to me. But lock-in by giving away your stuff in a way that lets anyone run it on their own hardware or a competitor's cloud? The DSSTNE benchmarks, for example, ran TensorFlow on AWS...
    - 10y ago
Havoc 10y ago
Their suggestions aren't all that great in my experience. Its either something I looked at/bought before or its something completely arbitrary.

131 points by shanxS 10y ago

If you haven't kept up with Spark, and do not have Amazon-level workloads, the built-in machine learning APIs are extremely robust and well-documented (https://people.apache.org/~pwendell/spark-nightly/spark-master-docs/latest/api/python/pyspark.ml.html https://people.apache.org/~pwendell/spark-nightly/spark-mast...), even in the non-Scala languages like Python. There's even a Multilevel Perceptron model creation function, which creates artificial neural networks using the feed-forward/backpropagation everyone loves. The new Spark DataFrames also make manipulating data almost as easy as with Python Pandas / R dplyr. The new Berkeley eDX course (https://courses.edx.org/courses/course-v1:BerkeleyX+CS105x+1T2016/info https://courses.edx.org/courses/course-v1:BerkeleyX+CS105x+1...) is a very good explainer. After Spark 2.0.0 is released, it wouldn't surprise me if it really takes off. (as long as setting up a cluster is a bit easier!)

nchammas 10y ago
> as long as setting up a cluster is a bit easier! If you're on AWS, it's already quite easy to set up a cluster today, no? There's EMR of course, and there are tools like spark-ec2 [0] and Flintrock [1]. There are a few more tools listed on Spark Packages that target different cloud providers [2], too. Disclaimer: I am the primary author of Flintrock and am a contributor to spark-ec2. [0] https://github.com/amplab/spark-ec2 https://github.com/amplab/spark-ec2 [1] https://github.com/nchammas/flintrock https://github.com/nchammas/flintrock [2] https://spark-packages.org/?q=tags%3Adeployment https://spark-packages.org/?q=tags%3Adeployment
- minimaxir 10y ago
  Setting up Spark clusters is easy relative to setting up clusters, but not easy enough yet to set up and configure relative to simply downloading a package in R/Python.
  - nchammas 10y ago
    Well, the absolute easiest way to run Spark is to do it locally (e.g. you can brew install it on a Mac and just go) or to pay for a proprietary service like Databricks, which makes setting up a cluster take a few clicks. That said, I think `flintrock launch my-cluster` is almost as easy as doing `pip install ...`. You do need an AWS account and you do need to set your preferences like region and key name in a config file, but I don't see how you can get out of doing even that without subscribing to some managed service like Databricks that abstracts everything away and replaces it with a nice Web UI.
  - mgummelt 10y ago
    If you're running a DC/OS cluster, it actually is! > dcos package install spark This gives you a Spark CLI, and installs the Spark Cluster Dispatcher so you can run jobs async. It also has options for installing the history server, and for SSL/Kerberos support. Disclaimer: I work at Mesosphere and contribute to the DC/OS Spark package.

vonnik 10y ago

We already built what Amazon was looking for: An extensible deep-learning framework that works on distributed CPUs and GPUs heterogeneously, integrating with Spark as an access layer to orchestrate multiple host threads. https://github.com/deeplearning4j https://github.com/deeplearning4j DL4J may be the DL library with the most sophisticated Spark integration at this point. The trick is to avoid using Spark as a computation layer, since it doesn't do that well. We're pushing a CuDNN wrapper tomorrow. http://deeplearning4j.org/quickstart http://deeplearning4j.org/quickstart Unlike DSSTNE, Tensorflow or CNTK, our deep learning library is neutral, and not designed with the intention of locking people into to a cloud service.

Fenntrek 10y ago
>Unlike DSSTNE, Tensorflow or CNTK, our deep learning library is neutral, and not designed with the intention of locking people into to a cloud service. How do any of these lock you into a cloud service?
- dgacmu 10y ago
  And to reinforce Fenntrek's point: One of the core reasons for open sourcing TensorFlow is to make sure people have the ability to take their code and run it somewhere else. The exact opposite of lock-in. (I can say that with some knowledge - I'm working there right now.) In fact, the cloud service for "hosted" TF isn't even out of alpha yet, but there are many people already using TF on their own hardware. I can't speak for MSFT or AMZN, but all three of these frameworks are completely open source with permissive licenses. I seriously doubt that any of these three providers has lock-in as a goal by releasing their stuff OSS. Goodwill? Absolutely. Getting people to learn the technology they care about internally? Certainly. Hoping that people will play with it and find that they want to run their models on 100 nodes rented from a cloud service? Seems likely. For MSFT, wanting to make sure there was a high-performance DNN framework for Windows? Would make sense to me. But lock-in by giving away your stuff in a way that lets anyone run it on their own hardware or a competitor's cloud? The DSSTNE benchmarks, for example, ran TensorFlow on AWS...
  - 10y ago

Havoc 10y ago

Their suggestions aren't all that great in my experience. Its either something I looked at/bought before or its something completely arbitrary.