6 ms·
Hello, Tensorflow
- tromobne8vb 10y agoAs I've been reading about tensorflow lately I feel like I'm missing something regarding distributed processing. How can Tensorflow 'scale up' easily if you are outside of Google? We have big datasets that I want to run learning on but it seems awkward to do with tensorflow. We're big enough that the team managing our cluster is separate than development and it is a huge pain if we need them to go install tools on each node. Even with Spark support it seems like the tensorflow python libraries need to be set up on each machine in the cluster ahead of time. Am I missing something?
- jamesblonde 10y agoNo, you're not. Google did this with their build engine (Blaze, internally - Bazel is the open-source API, lacking a distributed build platform). Google are doing this with Apache Beam (the API to Google dataflow) - releasing an API for local testing but not releasing the distributed engine. If you have your data in a Hadoop cluster and are doing image recognition, Yahoo's Cafe on Spark is the only truly distributed engine out there. It uses MPI to share model state between executors.
- agibsonccc 10y agoKeep in mind there's different kinds of parallelism though. If you mean model parallel, a lot of shops are doing that via RDMA as well as MPI. It depends on how you handle state though. There's also data parallelism with parameter averaging which we've been doing in deeplearning4j for the last few years. We also support ALOT more than just images. We have the ETL pipelines (kafka etc) to go with it. Watch for a blog post from us on parallel for all (nvidia's blog) where we explain some of this. I gave a framework agnostic view of the concepts you should consider when looking at distributed deep learning as well: http://www.slideshare.net/agibsonccc/brief-introduction-to-distributed-deep-learning http://www.slideshare.net/agibsonccc/brief-introduction-to-d...
- netinstructions 10y agoHave you read through https://www.tensorflow.org/versions/master/how_tos/distributed/index.html https://www.tensorflow.org/versions/master/how_tos/distribut... ? Since version 0.8 of TensorFlow they've had a way to do distributed processing. Their blog post in April mentioned it - https://research.googleblog.com/2016/04/announcing-tensorflow-08-now-with.html https://research.googleblog.com/2016/04/announcing-tensorflo... That said, I haven't actually attempted any distributed processing, but it looks possible. If anyone has actually tried it and can speak to it I would be curious to what people with experience have to say about it.
- tromobne8vb 10y agoI have read that. I even re-read it before making my post. That implementation requires starting individual tasks on each node in your cluster. >To create a cluster, you start one TensorFlow server per task in the cluster. Each task typically runs on a different machine, but you can run multiple tasks on the same machine (e.g. to control different GPU devices). I'm used to using tools that can roll out to a cluster with more finesse than that. The Spark wrapper seems to provide some capability to do this automatically, but even the Spark wrapper requires installing python libraries on each node.
- tanlermin 10y agowould something like conda or anaconda help? https://docs.continuum.io/anaconda-cluster/index https://docs.continuum.io/anaconda-cluster/index https://www.continuum.io/blog/developer-blog/getting-most-out-anaconda-your-cluster https://www.continuum.io/blog/developer-blog/getting-most-ou...
- bhntr3 10y agoYeah, I'm trying to figure this out too. TensorFlow needs Yarn support. Ideally, Yarn would allocate resources and inform the processes of the various workers in the graph, etc. etc. I see that as the harder part. If you use mesos, then there is some preliminary support for that. https://github.com/tensorflow/tensorflow/issues/1996 https://github.com/tensorflow/tensorflow/issues/1996 Since TensorFlow has native dependencies on CUDA stuff for GPU support, I don't think there's much of a way to get around installing things on every machine. You might be able to package a python env without CUDA for spark to run using conda. Here's an interesting blog post about that: https://www.continuum.io/blog/developer-blog/conda-spark https://www.continuum.io/blog/developer-blog/conda-spark But I'm not sure I see the point in running TensorFlow without GPU support. And if you're hoping to run GPU machines on an existing spark cluster and intelligently allocate the GPU stuff to the right machine. . . that's gonna be tough. Here's an interesting talk on that from the last spark summit: https://www.youtube.com/watch?v=k6IOWblLQK8&feature=youtu.be https://www.youtube.com/watch?v=k6IOWblLQK8&feature=youtu.be Ultimately, you're probably better off just running your own gpu cluster strictly for your TensorFlow model on ephemeral AWS spot instances. Or just use Google Cloud Machine Learning. That's what Google wants and expects you to do anyway. Borg is the Borg. You will be assimilated.
- ajschumacher 10y agoWow! I was going to post this but here it is already! I wrote (with a lot of help) the article there. I also jotted down some notes on the process of writing it with O'Reilly in case anybody's interested in that side of things: http://planspace.org/20160619-writing_with_oreilly/ http://planspace.org/20160619-writing_with_oreilly/
- krisroadruck 10y agoGreat article! You did miss a GoT plug though "An Object has no name". Tisk! But seriously though thanks for the background post to go with. Cheers.
- ajschumacher 10y agoThanks! May objects not bring you any unwanted gifts! :)
- bsg75 10y agohttp://blog.revolutionanalytics.com/2016/05/an-object-has-no-name.html http://blog.revolutionanalytics.com/2016/05/an-object-has-no...
- JPKab 10y agoGood post man. Hope DC is treating you well these days.
- ajschumacher 10y agoThank you! Things are going great in DC - I happen to be in NYC for ICML this week though, where things are also going pretty great! :)
- tedmiston 10y agoIt was nice to hear a little behind the scenes. I'd be curious to hear any analytics about unique visitors, etc so far from the O'Reilly post if you can share.
- JPKab 10y agoHey Aaron! Hello from one of the people you taught in your first GADS class in DC.
- ajschumacher 10y agoHi! Good to hear from you here! :)
- tedmiston 10y ago> For more on basic techniques and coding your own machine learning algorithms, check out our O'Reilly Learning Path, "Machine Learning." This learning path is also available free for Safari Books Online subscribers. https://www.safaribooksonline.com/library/view/learning-path-machine/9781491958155/ https://www.safaribooksonline.com/library/view/learning-path...
- Fenntrek 10y agoWonder when we will see the day when an O'Reilly book is written by AI. Could be a nice little loop if part of the creators of the AI that would accomplish this learned part of their craft from O'Reilly books.
- farresito 10y agoI love this short tutorials that give you an introduction to anything in an hour. They help you get interested in stuff you wouldn't have gotten interested in otherwise.
- 50CNT 10y agoTensorFlow is admirably easier to install than some other frameworks I thought most frameworks are fairly easy to install in python, usually with a single call to pip. NLTK takes one "pip install nltk" and then "python", "import nltk", "nltk.download()" to download all the corpuses and miscellaneous data. Installing tensorflow seems complicated compared to that. # Ubuntu/Linux 64-bit, CPU only: $ sudo pip install --upgrade https://storage.googleapis.com/tensorflow/linux/cpu/tensorflow-0.8.0-cp27-none-linux_x86_64.whl # Ubuntu/Linux 64-bit, GPU enabled. Requires CUDA toolkit 7.5 and CuDNN v4. For # other versions, see "Install from sources" below. $ sudo pip install --upgrade https://storage.googleapis.com/tensorflow/linux/gpu/tensorflow-0.8.0-cp27-none-linux_x86_64.whl Not that either are particularly complicated, but saying other frameworks (assuming they're referring to python frameworks) are "a lot harder to install" seems disingenuous. That said, I haven't played around with AI frameworks too much, so I might just be missing a real stinker.
- JPKab 10y agoCompared to other frameworks which allow for deep learning, it's much easier to install.
- 50CNT 10y agoI see. The article was a bit vague for me in what it means by frameworks (does NLTK count? Django? DL frameworks in other languages?), and since I don't know the area too well, that struck out to me as odd. It honestly looks like pretty cool stuff, looking forward to having time to play around with it some day.
- ajschumacher 10y agoThanks! I didn't want to name names in the comparison to other frameworks, but I was thinking mostly of Caffe, which doesn't have any install path as simple as just a `pip install`. Other DL frameworks would be things like torch, theano, etc.
- limsup 10y agoclearly you haven't tried to install caffe
- anilshanbhag 10y agoI like the clarity of thought and structure of the article. I have used Tensorflow and had to explain it to a friend. So many times, I end up assuming things which are obvious to me but not to someone getting started. As said in the article, Tensorflow stands out for the ease of use and is to the best of my knowledge first distributed learning framework. Theano, Torch et al are faster but do not come with goodies like Tensorboard.
- gradys 10y agoThere are different notions of "fast" as well. Theano might be faster on a single GPU (not actually sure if this is still true), but TensorFlow can be distributed to multiple GPUs in one machine or even across several machines. Several groups have reported speedups that are near-linear in the number of GPUs you distribute across. This can make TensorFlow orders of magnitude faster than Theano.
- chronic80 10y agoTensorFlow is second in speed, right behind torch (fp16). Theano is quite far behind at this point.
- bduerst 10y agoThis is fantastic - thank you for doing this. When paired with the browser tool it makes a lot more sense: http://playground.tensorflow.org/ http://playground.tensorflow.org/ Is this planned to be released as an intro in a book about tensorflow?
- ajschumacher 10y agoThanks! There aren't any firm plans, but we are thinking about what might make sense.
- ajschumacher 10y agoI presented a version of this at a meetup! Presentation and supporting materials here: http://planspace.org/20160629-presenting_hello_tensorflow/ http://planspace.org/20160629-presenting_hello_tensorflow/