15 ms·

TensorFlow: open-source library for machine intelligence

1559 points by jmomarty 11y ago

Schiphol 11y ago
> The TensorFlow Python API requires Python 2.7 The whole scientific Python stack is available for Python 3. This seems like a somewhat backwards thing to do -- or perhaps the requirement is intended to mean at least 2.7? Edit: added context Edited again: More careful wording
- jdiez17 11y ago
  It's open source, so feel free to port it to Python 3.
- vrv 11y ago
  We're looking to support Python 3 -- there are a few changes we are aware of that are required, and we welcome contributions to help! Tracking here: https://github.com/tensorflow/tensorflow/issues/1 https://github.com/tensorflow/tensorflow/issues/1
  - Schiphol 11y ago
    Thanks for a constructive reply to a (well-meaning, but) not very nice comment! Congrats on the library, which looks awesome.
    - witty_username 11y ago
      Criticism is considered "not very nice"? Your initial comment wasn't impolite or anything.
  - cdnsteve 11y ago
    Python 3 is very much needed! <3 because it's issue #1 :)
  - deleted 11y ago
    [deleted]
  - sandGorgon 11y ago
    is this compatible with pypy (or plan to be). Pypy is such an awesome project, done on such a small budget that it needs some well deserved love !
- deleted 11y ago
  [deleted]
- kragen 11y ago
  This "backwards" meme doesn't make any sense. Python 2 and Python 3 are different languages that happen to share the same name, a lot of syntax, and near-source compatibility, but they've been developed concurrently for ten years now. It's kind of like C and C++. If the current Python 2 team (which is part of the Python Software Foundation) abandons it, probably somebody else will take over maintenance, because it seems unlikely that it will fall out of use. If the Python Software Foundation drops the ball, someone else will pick it up.
  - nonelement 11y ago
    Let me start by saying I understand what you're saying. I understand why what you're saying is true. However, many people see (and consequently think) this: "Short version: Python 2.x is legacy, Python 3.x is the present and future of the language" ... which was lifted from https://wiki.python.org/moin/Python2orPython3 https://wiki.python.org/moin/Python2orPython3. It follows that some people would think something along the lines of "...so this library was built for legacy Python?"
  - asdfologist 11y ago
    The Python Software Foundation has stated explicitly that Python 2.7 will EOL in 2020...
    - vegabook 11y ago
      ...but the science guys have been offered zero incentive to move, this "end of life" argument has been worked to death without success, and so it's perhaps time to shift focus to the science community's real needs (ie: GPU).
      - Nrpf 11y ago
        Check out Dynd.
  - 1ris 11y ago
    Python 2 is dead.
- vegabook 11y ago
  I am really excited about this library. Tensors are the future; matrices are going to look so old in a few years. In fairness, Theano already knew that. I cannot wait to dig into this and am very relieved that Google's best are still using Python for this endeavour...but...using Python 2.7. I have just recently been persuaded by the community that 3.5 is cost free, and here I have this enormous counterexample. For the science guys, clearly, the message is not getting through, and I'm not surprised: 3.x offers them nothing. Hence they're blissfully continuing with 2.7. So I guess I'll have to run two Pythons on my system. Not the ideal situation. Now if Python 3.(x > 5) would give us native GPU, which might require putting (a GPU-enabled) Numpy into the standard library...as opposed to, say, spending 2 years re-inventing Tornado....
  - pathsjs 11y ago
    > Tensors are the future; matrices are going to look so old in a few years. I am honestly curious about this point of view. Is there any example where actual multidimensional tensor have any relevance? What I mostly see around is just standard linear algebra operations on matrices and vectors lifted to higher-dimensional tensors point-wise (for instance applying a certain operation to all 2-dimensional subtensors of a given tensor). I never saw the need for modeling matrices in terms of tensors - the latter seem just more complex to use and usually tensor operations (like symmetrization, antisymmetrization, wedge products...) are not even available in libraries (And by the way, both matrices and tensors are now more than one century old...)
    - vegabook 11y ago
      In my domain (finance) correlations between vectors are unstable but (maybe) dependent on cross-sectional relationships in the problem space. Some of the mathematics behind elastic body deformation (car tyres in in mechanical engineering, fluid dynamics in weather forecasting) have high applicability. Tensors are required. It's true that tensors are hard to reason about - they overclock my brain most of the time - but there is no doubt that, just like moving from scalars to vectors massively increases your real-world modelling power, so does the natural extrapolation to matrices, and from there, tensors.
- pacala 11y ago
  It has a C++ API, so there is no lock-in to Python, either 2.7 or 3.x.
  - gbrown 11y ago
    It looks like the C++ API is for executing graphs only, which is kind of odd.
MoOmer 11y ago
I'm really excited to dig in and read the source, actually. Thank you for open sourcing this.
cogware 11y ago
Here's the TensorFlow whitepaper http://download.tensorflow.org/paper/whitepaper2015.pdf http://download.tensorflow.org/paper/whitepaper2015.pdf
aconz2 11y ago
> This open source release supports single machines and mobile devices. Can someone clarify if "single machines" means the Apache license only applies to single machines and not distributed systems? Sidenote: I wish every open source project had a release video.
- sudhirj 11y ago
  There's a note on the site about being able to move computation to a process on a different machine, but it does mention copying the model. If I understand correctly, I think models tend to be quite small, though. Does it actually make sense to train a model simultaneously on different machines? I can understand running the same model on lots of machines, of course.
  - aconz2 11y ago
    I was interested in whether the license only permits a single machine.
    - Sanddancer 11y ago
      Nope, nothing in the license prohibiting it. Seems more a case that they just haven't released the glue for it to run on multiple machines, but hope to have it added one way or another.
  - prajit 11y ago
    Yes, there are two types of parallelism: model and data. Data parallelism is simply training the the model on multiple computers with different minibatches, and aggregating the gradients. Model parallelism is hosting different parts of the model on different computers. Of course, these two parallelisms can be combined. A good explanation is the One Weird Trick paper: http://arxiv.org/abs/1404.5997 http://arxiv.org/abs/1404.5997
  - dr_zoidberg 11y ago
    If there is randomness in some part of the training process (and most surely there will be), then it makes sense. Different trainings may end up in different local minimums/maximums, so there is value in running multiple trainings simultaneously in different machines.
kragen 11y ago
This is really significant. At this moment in history, the growth of computer power has made a bunch of important signal-processing and statistical tasks just feasible, so we are seeing things like self-driving cars, superhuman image recognition, and so on. But it's been very difficult to take advantage of the available computational power, because it's in the form of GPUs and clusters. TensorFlow is a library designed to make it easy to do exactly these things, and to scale them with your available computing power, along with libraries of the latest tricks in neural networks, machine learning (which is pretty close to "statistics"). As a bonus, it has built-in automatic differentiation so that you can run gradient descent on any algorithm — which means that you can just write a program to evaluate the goodness of a solution and efficiently iterate it to a local maximum. If you do this enough times, hopefully you'll find a global maximum. There are a variety of other numerical optimization algorithms, but gradient descent is very simple and broadly applicable. And it runs in IPython (now Jupyter), which is a really amazingly powerful way to do exploratory software development. If you haven't tried Jupyter/IPython, google up a screencast and take a look. I'm just repeating the stuff that's on the home page in a different form, but this is a really big deal. Most of the most significant software of the next five or ten years is going to be built in TensorFlow or something like it. (Maybe Dan Amelang's Nile; see Bret Victor's amazing video on the dataflow visualization he and Dan built for Nile recently at https://youtu.be/oUaOucZRlmE?t=24m53s https://youtu.be/oUaOucZRlmE?t=24m53s, or try the live demo at http://tinlizzie.org/dbjr/high_contrast.html http://tinlizzie.org/dbjr/high_contrast.html, or download and build the Nile system from https://github.com/damelang/nile. https://github.com/damelang/nile.) Apparently the Googlers think that too, because among the 39 authors of the white paper are several shining stars of systems research, including some who may be up for a Turing Award in the next decade or two.
- Fede_V 11y ago
  One small note on the paper itself. I love that the paper's author are in alphabetical orders, without any stupid jostling over first/last authorship. I think that's fantastic and wish academia went that way too (yeah right).
  - jamessb 11y ago
eddd 11y ago
2:21 in the video you can see a link to: corp.google.com I love their UI for single sign on.
- andy_ppp 11y ago
  Hmmm, the video appears to only be 2:17 long now...
  - eddd 11y ago
    1:52 actually, sorry I was mistaken.
fitzwatermellow 11y ago
Great job, guys! The Docker of Neural Compute..
dryginmartini 11y ago
How does TensorFlow compare to Torch7 (http://torch.ch http://torch.ch)? They look very similar to me.
- albertzeyer 11y ago
  Actually, this looks much more like Theano to me. It's all symbolic, it has symbolic differentiation, it's Python and it even seems to have a similar API like Theano. In the Whitepaper (http://download.tensorflow.org/paper/whitepaper2015.pdf http://download.tensorflow.org/paper/whitepaper2015.pdf), it's compared to Theano, Torch and others.
  - nickpsecurity 11y ago
    That might be a strong advantage in its favor, though. Building a better version of a proven concept with similar usage/API's often gets more market uptake. Might get more use.
- mailshanx 11y ago
  TensorFlow seems to emphasise the distributed computing aspects a lot more compared to Torch7. Also, TensorFlow is made for deploying machine learning pipelines in general, whereas Torch7 focuses on deep neural networks specifically.
blazespin 11y ago
GPU support, woohoo!
- varelse 11y ago
  And just 4 years after their insistence that GPUs would never have a role at Google... How 'bout that?
  - pacala 11y ago
    Google is a big place, with diverse opinions. Even better, it encourages updating one's opinion in face of new evidence. GPUs have proven themselves cost-effective in tackling a number of vision problems [and others], Google has those problems, so the opinion was updated and the problems were solved in a cost-effective manner.
    - varelse 11y ago
      That... Was not... My experience there... But kudos to the deep learning guys for overcoming that potential energy barrier NVIDIA couldn't surmount on their own...
      - Florin_Andrei 11y ago
        To me, processing lots and lots of data with "relatively simple" algorithms spells GPU, pretty much. Machine learning seemed destined to bump into GPUs sooner or later.
        varelse 11y ago
        Why yes... https://github.com/BIDData/BIDMach https://github.com/BIDData/BIDMach
  - pilooch 11y ago
    Google is certainly Nvidia's top customer.
RivieraKid 11y ago
What kind of tool can be used to create that animated illustration? Looks really neat.
- croddin 11y ago
  It looks like it is TensorBoard, included with TensorFlow http://tensorflow.org/how_tos/summaries_and_tensorboard/index.md# http://tensorflow.org/how_tos/summaries_and_tensorboard/inde...
syllogism 11y ago
Any reason you wrote the Python wrapper in SWIG instead of Cython? The nice thing about Cython is that you can wire up two C extensions to talk to each other directly, without going through Python. Cython also gives you 2/3 compatibility out of the box.
- pacala 11y ago
  SWIG supports many other languages besides Python.
andyjohnson0 11y ago
"TensorFlow is an Open Source Software Library for Machine Intelligence" and then later "TensorFlow™ is an open source software library for numerical computation using data flow graphs." So it seems to be a dataflow computation library that is being used for AI/learning. Since I know almost nothing about either, I'm wondering if this (dataflow) approach has other applications unrelated to deep learning. Any comments? Also, i wonder if this could be implemented in hardware - say on a FPGA?
- varelse 11y ago
  FPGAs are not measuring up to GPUs yet for these tasks. There's too much floating-point math (and yes, Altera is addressing this) for forward prediction, and way too much intermediate state to save for training (GPU memory controllers are 'da bomb for this). Finally, OpenCL compilation time on FPGAs is measured in hours as compared to seconds for GPUs. That said, address all of the above and maybe NVIDIA's stock won't hit 50 next year.
  - Sanddancer 11y ago
    There are a few people that seem to be working with integer-based deep learning and FPGAs, and even though they're at the beginning of research, things are looking pretty promising. Here's a paper that was released earlier this year that seems to show that at least some algorithms can see a big benefit from FPGAs http://arxiv.org/pdf/1502.02551v1.pdf http://arxiv.org/pdf/1502.02551v1.pdf .
  - dharma1 11y ago
    did you see this? http://arxiv.org/pdf/1510.03009v1.pdf http://arxiv.org/pdf/1510.03009v1.pdf
    - nickpsecurity 11y ago
      Now THAT has some potential. Both for FPGA's and SIMD/MIMD architectures. Thanks for the link.
Fede_V 11y ago
This is awesome, thanks so much for sharing this - and I love how whimsical the docs are: Our pond is a perfect 500 x 500 square, as is the case for most ponds found in nature. A feedback on the PDE aspect: is there any nice way to write your model and grid using TensorFlow, and still obtain the free derivatives/optimized computation graph if you use a different solver to actually integrate the equations? The example shows off a very simple toy first order Euler solver, but in practice this is grossly inadequate. I suspect that it's very difficult if not impossible - especially since most solver libraries are very old Fortran libs, and carrying forward the reverse mode differentiation there is a nightmare. Still, this makes for very nice building blocks. Another question - would you consider adding an example of using this together with Google Ceres - to make the two autodiffs play nice together?
deleted 11y ago
[deleted]
mark_l_watson 11y ago
I can't wait to try this tonight. I have a fun "messing around" project (I am using GnuGo to generate as much training data as I need for playing Go on a very small board, and I am almost to the point of starting to train and test models). BTW, Ruby has always been my scripting language but because of the wealth of libraries for deep learning I am thinking of switching to Python.
- mkorfmann 11y ago
  Why switch? I use Ruby, when I can and Python when I can't. Would really love to see ruby forks of popular ml libs. Or is it because Python is better perfomance-wise?
  - mark_l_watson 11y ago
    Really good question since I really enjoy using Ruby, more so even than Clojure and Haskell. I too would like to see Ruby ports of some of the popular ML libraries. The classifier gem provides an example.
- thomasahle 11y ago
  I believe the big facebook and google Go networks were trained on data from professional games. Probably you'd get better performance if you did that as well.
- mark_l_watson 11y ago
  I have started to experiment with it. Really awesome documentation and setup nstructions (using Ubuntu).
deleted 11y ago
[deleted]
nye2k 11y ago
If we could stop using the term Neural Networks in relation to computing, that would be great. Computers don't have neurons.
- deleted 11y ago
  [deleted]
- BinaryIdiot 11y ago
  There are artificial and biological neural networks. The artificial one is based upon the biological one. Therefore dropping the 'artificial' or 'biological' seems fine to me.
  - nye2k 11y ago
    This is practical, however I strongly disagree with this semantically. The term should always be accompanied with 'artificial' or another name should be invented as they are vastly different fields. Search Neural Network, I assure you that you will no longer find biological information; it is littering.
    - deleted 11y ago
      [deleted]
- ska 11y ago
  If we get hung up on every abuse of terminology we run into, nobody will get anything useful done. At this point the name is well established. If it makes you feel better, just remember to tag "artificial" on to the beginning, nobody will mind.
obstbraende 11y ago
Are there any major conceptual differences to Theano? Not that I wouldn't appreciate a more polished, well funded competitor in the same space. It looks like using TensorFlow from Python will feel quite familiar to a Theano user, starting with the separation of graph building and graph running, but also down into the details of how variables, inputs and 'givens' (called feed dicts in tensorflow) are handled.
- albertzeyer 11y ago
  I'm looking at the RNN implementation right now (https://github.com/tensorflow/tensorflow/blob/master/tensorflow/models/rnn/rnn.py https://github.com/tensorflow/tensorflow/blob/master/tensorf...). It looks like the loop over the time frames is actually in Python itself. for time, input_ in enumerate(inputs): ... This confuses me a bit. Maybe the shape is not symbolic but must be fixed. I also haven't seen some theano.scan equivalent. Which is not needed in many cases when you know the shape in advance.
  - obstbraende 11y ago
    I think this loop actually still only builds the graph -- what `scan` would do. The computation still happens outside of python. That is, in tensorflow they perhaps don't need `scan` because a loop with repeated assignments "just works"... Let's try this: It seems like in TensorFlow you can say: import tensorflow as tf sess = tf.InteractiveSession() # magic incantation state = init_state = tf.Variable(1) # initialise a scalar variable states = [] for step in range(10): # this seems to define a graph that updates `state`: state = tf.add(state,state) states.append(state) sess.run(tf.initialize_all_variables()) at this point, states is a list of symbolic tensors. now if you query for their value: print sess.run(states) >>> [2, 4, 8, 16, 32, 64, 128, 256, 512, 1024] you get what you would naively expect. I don't think that would work in Theano. Cool.
    - benanne 11y ago
      Why wouldn't this work in Theano? >>> import theano >>> import theano.tensor as T >>> state = theano.shared(1.0) >>> states = [] >>> for step in range(10): >>> state = state + state >>> states.append(state) >>> >>> f = theano.function([], states) >>> f() [array(2.0), array(4.0), array(8.0), array(16.0), array(32.0), array(64.0), array(128.0), array(256.0), array(512.0), array(1024.0)]
superfx 11y ago
Does anyone know how this compares to Twitter's recently released torch autograd? https://blog.twitter.com/2015/autograd-for-torch https://blog.twitter.com/2015/autograd-for-torch EDIT: To be a bit more specific, autograd just requires that one writes the forward model, and it can involve any program logic like conditionals, etc, so long as the end result is differentiable. It then takes care of all the gradient calculations.
musesum 11y ago
I'm kinda floored, by this. Have been developing a flow based language to do this. Even started with the approach of using Python and then moving to scripting directly in C++. Maybe I tweak my own project. Anyone playing with low level GPU deployments, like Metal? [EDIT] Ok, so there is some GPU - should probably play some more before asking obvious questions. Really keen on scripting out flow directly.
- pilooch 11y ago
  link to your project ?
  - musesum 11y ago
    Sorry, not ready for prime-time. Closest is a visualization of the previous version of the script: https://www.youtube.com/watch?v=a703TTbxghc https://www.youtube.com/watch?v=a703TTbxghc That version wasn't a NN graph for ML; more of an executable flow graph for visual music. Will post something on https://github.com/musesum https://github.com/musesum in a couple months.
malmsteen 11y ago
So, i have a very simple question.. : I want to start deep learning now, to implement RNNs, autoencoders, Q-learning on a very specific application (not images). I've read a lot of papers, but not done any DL implementations yet (although many classical ML so far), my question is very simple : Where do I start ???? Should I use Torch ? Theano ? Theano + pylearn2 ? Theano + pylearn + Lasagne ? Caffe ? Or should I just switch to this magical new library directly ? I feel confused.. any advice ?
- varelse 11y ago
  Methinks embrace the magical library written by engineers and researchers at Google instead of by a random goulash of machine learning grad students with no investment in the outcome other than graduating.
  - rhaps0dy 11y ago
    > with no investment in the outcome other than graduating Come on. Grad students are motivated by advancing the field. ... are they not? Maybe you can also be saying that the ones who write libraries are motivated only by graduating.
    - varelse 11y ago
      Sure, we all start out that way. But by the 3rd or 4th year, you frequently just want to write that thesis and get on to the next thing because you've already advanced the field as much as you're going to as a grad student. I have a large package of code on sourceforge that consists of my grad school and postdoctoral efforts. If I compare its style to what 2 decades in the industry since has taught me, it's laughable. That said, it is C code, so I occasionally plunder it for algorithms I figured out way back when.
      - benanne 11y ago
        Nobody forced us to open-source Lasagne, so I think that remark was a bit unfair. If we really didn't care about anything but graduating, why would we bother going through the trouble of sharing the code in the first place? But I do see your point. Google obviously has a lot more manpower to spend on this, so it might be a better bet in the long run. It's also worth comparing this to a few similar projects that have been announced recently: MXNet (http://mxnet.readthedocs.org/en/latest/ http://mxnet.readthedocs.org/en/latest/), Chainer (http://chainer.org/ http://chainer.org/) and CGT (http://rll.berkeley.edu/cgt/ http://rll.berkeley.edu/cgt/). And Theano of course, which has been the flag-bearer of the computational graph approach in deep learning for many years.
datashovel 11y ago
Didn't read the paper, but when I saw that a main focus is on distribution I started thinking about how awesome it would be if people could start crowdsourcing compute resources to solve some big problems.
doczoidberg 11y ago
I am a newbie to machine learning. How does it compete to Azure ML besides it is open source?
- dswalter 11y ago
  It's not really directly comparable. This is a computational framework which you can use to implement a variety of ML algorithms (or in general numerical computation), so the interface to this is lower-level than Azure ML. If Azure ML is a bunch of premade sandcastle molds, TensorFlow is a more accurate, faster way to pour sand. You make the mold.
  - mohamedmansour 11y ago
    TensorFlow built a language, AzureML uses a bunch of languages of your choice to build any machine learning on Azure. Try it https://studio.azureml.net/ https://studio.azureml.net/ follow the tutorial
jheriko 11y ago
i really wish people would stop calling multidimensional arrays tensors. there is already well defined language for this. multidimensional array is fine, psuedotensor is fine. tensor is confusing if you have any previous background with tensor calculus before the word became machine learning flavour of the month. still. this does look pretty cool overall. processing large volumes of data is becoming increasingly less specialised and more generic, which can only be good. :)
- xyzzyz 11y ago
  I'd say tensor is exactly as confusing as vector -- vectors are just elements of some vector spaces, and you get a list of numbers by choosing a basis of that vector space. Similarly, tensors are just elements of a tensor product of some vector spaces (or modules), and you get a multidimensional array by choosing bases in the factors.
- cmarschner 11y ago
  Can't agree more. The first time I looked up tensors when they appeared in Torch I got totally confused by the description of tensors used in physics.
  - jordigh 11y ago
    The problem is complicated a bit more because physicists use the word "tensor" to mean a tensor field, i.e. a collection of tensors on which calculus makes sense.
- zer0nes 11y ago
  Tensor is a much better term than multidimensional array: - It's not as verbose - The concept of tensor has linear algebra operations associated with it while multidimensional array is just a programming term without any attached Math operations.
  - jordigh 11y ago
    > The concept of tensor has linear algebra operations associated with it while multidimensional array is just a programming term without any attached Math operations. But that's the precise problem. The multidimensional arrays that programmers call "tensors" do not generally have the operations defined on them that you would expect from a tensor, such as index contraction, tensor products, or differentiation. At the same time, a tensor is different from an array, just like a number is different from its decimal representation. These distinctions are important when you want to change the basis, a very useful and frequent operation for both tensors and numbers. Then again, this battle was already fought and lost for vectors, which programmers specialised to mean "arrays" instead of meaning "elements of a linear space".
irq-1 11y ago
This looks similar to Orange. Orange has a visual designer, easy data access (SQL, arbitrary files) and lots of widgets, but it's a desktop tool not a library and doesn't standardize the data structures. http://orange.biolab.si/ http://orange.biolab.si/ http://docs.orange.biolab.si/widgets/rst/index.html http://docs.orange.biolab.si/widgets/rst/index.html
- infinite8s 11y ago
  The newest version of Orange (version 3) is standardizing on the scientific python stack (numpy, scipy, scikit-learn)
thadd 11y ago
Just found what I'm going to do today
an4rchy 11y ago
I tried going through the site and also the comments but couldn't wrap my head around what this library actually is. It sounds awesome based on the response/comments. Can anyone explain this to a layman?
- gyom 11y ago
  If you want to train neural nets, you can either rewrite everything from scratch and get a bug-ridden sub-optimal implementation, or you can use a kind of off-the-shelf library. The problem is that there are about 3-5 alternatives out there, and none of them are mature enough or convincing enough to dominate. The field changes so fast that it's easy for them to become obsolete. What you're seeing here is the enthusiasm of people who really want to get a good tool with proper support, and be able to stick with it. I'm still not sure if TensorFlow is that tool, but it depends on what will happen to it during the coming years.
- sputknick 11y ago
  It looks like to me the big win here is that this is a ML library just like all the others, except that the code is not written for a specific type of processor (GPU versus CPU), nor is it written for a specific compute level (desktop, server, phone). It's target is to be a general purpose ML library. This should have the affect of getting ML solutions into useable applications more quickly, since you don't have to develop on one platform, then port to another. It just works everywhere. Best guess on my part based on what I'm reading on the site.
- levesque 11y ago
  It is a library for the optimization of machine learning algorithms, similar to Theano. You write a model you want to optimize as a graph of symbolic expressions, and the library will compile this graph to be executed on a target platform (could be CPU, GPU). This is really neat because you essentially wrote your program in Python and now it's going to be running as optimized C++ code (or as optimized kernels on your GPU). The fact that you defined your model as a symbolic graph means you can derive gradient equations automatically, hence ridding you of a rather tedious step in optimization. With such tools, the workflow for a practitioner is much simpler: 1-code models as graphs of symbolic expressions, and 2-define objective functions (ex. for a regression problem, your objective would be the root mean squared error, plus some regularization). Disclaimer: I did not look at the code, but from what I understand it is pretty much the same as Theano (which I've been using a lot lately).
devit 11y ago
Does it support OpenCL and thus Intel integrated and discrete AMD GPUs?
- chimtim 11y ago
  Appears to be only CUDA from the code. Also no distributed systems support released. Also, I cannot get it to work since this morning.
  - happycube 11y ago
    Requires nvidia capability >=3.5 at that (GK11x and Maxwell) which rules out anything I've currently got (a couple of 760's/GK104's)... which is the chip Amazon EC2 uses as well. They're probably using Dynamic Parallelism.
Nitramp 11y ago
Maybe of interest: this is (amongst others) by Jeff Dean, famous for Bigtable, Map Reduce, Spanner, Google Web Search, Protocol Buffers, LevelDB, and many, many other things. He's a Google Senior Fellow, but also doubles as the patron saint of Google Engineering.
- tootie 11y ago
  Did they give him a Turing Award yet?
- zeroxfe 11y ago
  He is also a ninja, an astronaut, a master chef, and a damn good fellow!
samuell 11y ago
Seems like Ben Lorica's call for open source tensor libraries for data science [1], were answered! [1.a] http://radar.oreilly.com/2015/03/lets-build-open-source-tensor-libraries-for-data-science.html http://radar.oreilly.com/2015/03/lets-build-open-source-tens... (March 2015) [1.b] http://radar.oreilly.com/2015/05/the-tensor-renaissance-in-data-science.html http://radar.oreilly.com/2015/05/the-tensor-renaissance-in-d... (May 2015)
fibo 11y ago
For those interested in a minimal data flow JavaScript Engine see my project dflow: https://www.npmjs.com/package/dflow https://www.npmjs.com/package/dflow
cdnsteve 11y ago
AWS has GPU clusters already available FYI: http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using_cluster_computing.html http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using_clu... Not sure how much effort is required to use them with this library.
scott_s 11y ago
The programming model here is very similar to that of the language I work on, Streams Processing Language, normally just called SPL. A paper we have submitted on the language itself: http://www.scott-a-s.com/files/ibm_tr_2014.pdf http://www.scott-a-s.com/files/ibm_tr_2014.pdf Our domain is online stream processing for, generally, the big data space. I do think, however, that describing computations in this manner gives enormous flexibility to runtime systems to actually exploit all available parallelism dynamically.
peterjliu 11y ago
For those looking for more context, here's the Google research blog post: http://googleresearch.blogspot.com/2015/11/tensorflow-googles-latest-machine_9.html http://googleresearch.blogspot.com/2015/11/tensorflow-google...
fscherer 11y ago
can anybody tell me if this can calculate the gradients of a conditional graph? (something that is not implemented in theano which gives me a huge headache right now)
samuell 11y ago
Interesting! I wrote in a draft post back in late 2013 [1], that asked: "What if one could have a fully declarative “matrix language” in which all data transformations ever needed could be declaratively defined in a way that is very easy to comprehend?" I'm now pondering whether TensorFlow isn't quite an answer to this question? [1] Posted the draft now for reference: http://bionics.it/posts/matrix-transformation-as-model-for-data-flow-operations http://bionics.it/posts/matrix-transformation-as-model-for-d...
- davmre 11y ago
  The idea of representing a program by a declarative computation graph of matrix transformations has been big in deep learning research for a few years. Theano [1] is the canonical example, though more recently the space has gotten bigger with other frameworks including CGT [2] and now TensorFlow. The computational graph abstraction is really nice in part because it gives you very straightforward automatic differentiation, which dovetails very nicely with the trend in machine learning of casting a wide range of algorithms as (stochastic) gradient ascent on some loss function. [1] http://deeplearning.net/software/theano/ http://deeplearning.net/software/theano/ [2] http://rll.berkeley.edu/cgt/ http://rll.berkeley.edu/cgt/
  - samuell 11y ago
    Thanks! Yea, though in fact my idea is more about wiring the actual (matrix/tensor) operation with dataflow, rather than just the dataflow between such operations. But it might be bit of a different problem area :)
pvnick 11y ago
François Chollet, the author of keras, made the following tweet this morning: "For those wondering about Keras and TensorFlow: I hope to start working on porting Keras to TensorFlow soon (Theano support will continue)."
sandGorgon 11y ago
what would be really awesome is if Andrew Ng or Norvig build a course around Tensorflow. It is really not useful to a beginner to be learning everything in matlab.
- eob 11y ago
  My first thought when reading the docs was: this is a game changer for the efficiency of applied ML (nlp, vision, speech) phd students. Coming from a school where there's a bit of a libertarian "write it all yourself, from scratch!" ethos, I always marveled at how much mileage other students got from research groups that built off a common codebase. Exciting to start to see glimmers of that possibility across the entire field.
- davmre 11y ago
  There are already a few university ML courses out there using Theano (for which Tensorflow is essentially a drop-in replacement), and I think this will be a much bigger trend over the next few years. IMHO for a first course it's useful to do some work at the Matlab/numpy level just so you get experience with deriving/implementing gradients yourself, but for larger (deep) models automatic differentiation is an amazing productivity boost that should make it possible to cover a lot of interesting topics that you'd otherwise not have space for.
  - sandGorgon 11y ago
    actually numpy would be a brilliant starting point - but I'm not able to find any popular ones that dont use matlab (or some dialect thereof). disclaimer: I have no idea what I'm talking about, but I do know that coursera and stanford courses are the oft cited ones and they use matlab/octave.
  - robert-zaremba 11y ago
    The same goes for Torch: New York, Oxford ...
deleted 11y ago
[deleted]
swah 11y ago
OTOH code completion is still only mediocre for most programming languages! Of course, those things aren't really related, but it feels easier than self-driving cars.
benmathes 11y ago
Found a mistake in the docs: http://tensorflow.org/get_started/basic_usage.md http://tensorflow.org/get_started/basic_usage.md In the "Variables" example, looks like a variable name got changed but not updated everywhere: # at the start: var = tf.Variable(0, name="counter") # then: one = tf.constant(1) new_value = tf.add(state, one) update = tf.assign(state, new_value) The variable "state" should be the variable "var", or vice-versa.
Omnipresent 11y ago
I was going to dive into Theano. Is this much different than Theano? Or better?
mrwilliamchang 11y ago
Google says they are working on a distributed version and will release when it is ready. https://github.com/tensorflow/tensorflow/issues/12#issuecomment-155150681 https://github.com/tensorflow/tensorflow/issues/12#issuecomm...
Dowwie 11y ago
Something tells me that Facebook's AI group will release its React to their Angular
mjw 11y ago
Does anyone know if TensorFlow can apply algebraic simplifications and numerical optimisations to the compute graph, in the way that Theano does with its optimisations? Sounds like it doesn't suffer from the (alleged) slow compile times of Theano, but I wonder if the flipside of that is that you have to implement larger-scale custom Ops (like torch's layers) in order to ensure that a composite compute graph is implemented optimally?
Houshalter 11y ago
This seems really similar to Brain Simulator: http://www.goodai.com/#!brain-simulator/c81c http://www.goodai.com/#!brain-simulator/c81c
nosoynadasinti 11y ago
Nosoynadasinti
nosoynadasinti 11y ago
Vsfd
nosoynadasinti 11y ago
Ndkdo9qmwbdvx
orasis 11y ago
Could this system be used to implement a Gaussian Process?
aminorex 11y ago
Quite an ugly set of dependencies to build. In particular, bootstrapping bazel is excruciating.
ertand 11y ago
Geoffrey Hinton and Jeff Dean will give separate tutorials at NIPS. https://nips.cc/Conferences/2015/Schedule?day=0 https://nips.cc/Conferences/2015/Schedule?day=0
ppoutonnet 11y ago
Great job guys!
auston 11y ago
Many many thanks in advance to anyone who answers: I've futzed around with ML docs (dataquest, random articles, first 3 weeks of andrew ng course on coursera) & what I don't get about this (http://tensorflow.org/tutorials/mnist/beginners/index.md http://tensorflow.org/tutorials/mnist/beginners/index.md) is how to actually make a prediction / classification on something that is not "test" data. Can anyone point me in the right direction of how to "use" a model once I have it validated, like we do at the end of that tutorial?
jefurii 11y ago
The diagram at least looks a lot like Max/MSP and PureData, which are data-flow tools for processing MIDI and audio data. Could this be used to implement something along those lines?
HockeyPlayer 11y ago
Anyone experienced with machine learning want to try this out on some high frequency trading data? I'm in the processing of preparing a data set for more traditional analysis and would be willing to share. We are in Colorado, but I'm happy to work with someone remotely.
- deleted 11y ago
  [deleted]
rcthompson 11y ago
Is there a part of TensorFlow that generates that slick diagram on the front page?
argonaut 11y ago
Honestly, the raving that's going on in this thread is unwarranted. This is a very nice, well put together library, but it does not do anything fundamentally different from what has already been done with Theano / Torch. It is not a "game-changer" or a spectacular moment in history as some people seem to be saying.
vmarkovtsev 11y ago
This is very similar to what we did in Samsung several years ago: https://velesnet.ml https://velesnet.ml
joanfihu 11y ago
Great stuff to see that Google is releasing their own tools for research and production systems. Does anybody have a grasp of the main differences between the triple Ts: Tensorflow, Theano and Torch? As far as I can see Theano and Tensorflow support dataflow-like computations, automatic differentiation and GPU support via CUDA. Finally, why didn't Google, The Lisa Lab and Facebook worked together on building a unique library instead of three?
m00n 11y ago
Minor nitpick: The authors are not the first ones to recognize that 'tensor' makes a great (brand-) name. As someone who is paid to think about tensors this is a bit annoying, because the first thing in my mind when I read a page like OP, is "where are the tensors?". Alas, there are none. For a mathematician, a "multidimensional data array" is not a tensor. It is a tuple. A tensor is a tuple with much more structure, associated to linear actions on its components. Said differently, if you can't tell me, what 7*[Alice] means, you have no right to call [Alice] a tensor.
vasthack 11y ago
i never believed hacking was real until i was helped with my grades by one crazy hacker. i wonder how disappointing it would have been if i didn't graduate. thank you VASTHACK. You may contact him via vasthack@gmail.com
- vasthack 11y ago
  wow! I didn't know he could do that to he helped me moved money from a cheating partner's bank account too. That guy is indeed crazy

1559 points by jmomarty 11y ago

> The TensorFlow Python API requires Python 2.7 The whole scientific Python stack is available for Python 3. This seems like a somewhat backwards thing to do -- or perhaps the requirement is intended to mean at least 2.7? Edit: added context Edited again: More careful wording

jdiez17 11y ago
It's open source, so feel free to port it to Python 3.
vrv 11y ago
We're looking to support Python 3 -- there are a few changes we are aware of that are required, and we welcome contributions to help! Tracking here: https://github.com/tensorflow/tensorflow/issues/1 https://github.com/tensorflow/tensorflow/issues/1
- Schiphol 11y ago
  Thanks for a constructive reply to a (well-meaning, but) not very nice comment! Congrats on the library, which looks awesome.
  - witty_username 11y ago
    Criticism is considered "not very nice"? Your initial comment wasn't impolite or anything.
- cdnsteve 11y ago
  Python 3 is very much needed! <3 because it's issue #1 :)
- deleted 11y ago
  [deleted]
- sandGorgon 11y ago
  is this compatible with pypy (or plan to be). Pypy is such an awesome project, done on such a small budget that it needs some well deserved love !
deleted 11y ago
[deleted]
kragen 11y ago
This "backwards" meme doesn't make any sense. Python 2 and Python 3 are different languages that happen to share the same name, a lot of syntax, and near-source compatibility, but they've been developed concurrently for ten years now. It's kind of like C and C++. If the current Python 2 team (which is part of the Python Software Foundation) abandons it, probably somebody else will take over maintenance, because it seems unlikely that it will fall out of use. If the Python Software Foundation drops the ball, someone else will pick it up.
- nonelement 11y ago
  Let me start by saying I understand what you're saying. I understand why what you're saying is true. However, many people see (and consequently think) this: "Short version: Python 2.x is legacy, Python 3.x is the present and future of the language" ... which was lifted from https://wiki.python.org/moin/Python2orPython3 https://wiki.python.org/moin/Python2orPython3. It follows that some people would think something along the lines of "...so this library was built for legacy Python?"
- asdfologist 11y ago
  The Python Software Foundation has stated explicitly that Python 2.7 will EOL in 2020...
  - vegabook 11y ago
    ...but the science guys have been offered zero incentive to move, this "end of life" argument has been worked to death without success, and so it's perhaps time to shift focus to the science community's real needs (ie: GPU).
    - Nrpf 11y ago
      Check out Dynd.
- 1ris 11y ago
  Python 2 is dead.
vegabook 11y ago
I am really excited about this library. Tensors are the future; matrices are going to look so old in a few years. In fairness, Theano already knew that. I cannot wait to dig into this and am very relieved that Google's best are still using Python for this endeavour...but...using Python 2.7. I have just recently been persuaded by the community that 3.5 is cost free, and here I have this enormous counterexample. For the science guys, clearly, the message is not getting through, and I'm not surprised: 3.x offers them nothing. Hence they're blissfully continuing with 2.7. So I guess I'll have to run two Pythons on my system. Not the ideal situation. Now if Python 3.(x > 5) would give us native GPU, which might require putting (a GPU-enabled) Numpy into the standard library...as opposed to, say, spending 2 years re-inventing Tornado....
- pathsjs 11y ago
  > Tensors are the future; matrices are going to look so old in a few years. I am honestly curious about this point of view. Is there any example where actual multidimensional tensor have any relevance? What I mostly see around is just standard linear algebra operations on matrices and vectors lifted to higher-dimensional tensors point-wise (for instance applying a certain operation to all 2-dimensional subtensors of a given tensor). I never saw the need for modeling matrices in terms of tensors - the latter seem just more complex to use and usually tensor operations (like symmetrization, antisymmetrization, wedge products...) are not even available in libraries (And by the way, both matrices and tensors are now more than one century old...)
  - vegabook 11y ago
    In my domain (finance) correlations between vectors are unstable but (maybe) dependent on cross-sectional relationships in the problem space. Some of the mathematics behind elastic body deformation (car tyres in in mechanical engineering, fluid dynamics in weather forecasting) have high applicability. Tensors are required. It's true that tensors are hard to reason about - they overclock my brain most of the time - but there is no doubt that, just like moving from scalars to vectors massively increases your real-world modelling power, so does the natural extrapolation to matrices, and from there, tensors.
pacala 11y ago
It has a C++ API, so there is no lock-in to Python, either 2.7 or 3.x.
- gbrown 11y ago
  It looks like the C++ API is for executing graphs only, which is kind of odd.

MoOmer 11y ago

I'm really excited to dig in and read the source, actually. Thank you for open sourcing this.

cogware 11y ago

Here's the TensorFlow whitepaper http://download.tensorflow.org/paper/whitepaper2015.pdf http://download.tensorflow.org/paper/whitepaper2015.pdf

aconz2 11y ago

> This open source release supports single machines and mobile devices. Can someone clarify if "single machines" means the Apache license only applies to single machines and not distributed systems? Sidenote: I wish every open source project had a release video.

sudhirj 11y ago
There's a note on the site about being able to move computation to a process on a different machine, but it does mention copying the model. If I understand correctly, I think models tend to be quite small, though. Does it actually make sense to train a model simultaneously on different machines? I can understand running the same model on lots of machines, of course.
- aconz2 11y ago
  I was interested in whether the license only permits a single machine.
  - Sanddancer 11y ago
    Nope, nothing in the license prohibiting it. Seems more a case that they just haven't released the glue for it to run on multiple machines, but hope to have it added one way or another.
- prajit 11y ago
  Yes, there are two types of parallelism: model and data. Data parallelism is simply training the the model on multiple computers with different minibatches, and aggregating the gradients. Model parallelism is hosting different parts of the model on different computers. Of course, these two parallelisms can be combined. A good explanation is the One Weird Trick paper: http://arxiv.org/abs/1404.5997 http://arxiv.org/abs/1404.5997
- dr_zoidberg 11y ago
  If there is randomness in some part of the training process (and most surely there will be), then it makes sense. Different trainings may end up in different local minimums/maximums, so there is value in running multiple trainings simultaneously in different machines.

kragen 11y ago

This is really significant. At this moment in history, the growth of computer power has made a bunch of important signal-processing and statistical tasks just feasible, so we are seeing things like self-driving cars, superhuman image recognition, and so on. But it's been very difficult to take advantage of the available computational power, because it's in the form of GPUs and clusters. TensorFlow is a library designed to make it easy to do exactly these things, and to scale them with your available computing power, along with libraries of the latest tricks in neural networks, machine learning (which is pretty close to "statistics"). As a bonus, it has built-in automatic differentiation so that you can run gradient descent on any algorithm — which means that you can just write a program to evaluate the goodness of a solution and efficiently iterate it to a local maximum. If you do this enough times, hopefully you'll find a global maximum. There are a variety of other numerical optimization algorithms, but gradient descent is very simple and broadly applicable. And it runs in IPython (now Jupyter), which is a really amazingly powerful way to do exploratory software development. If you haven't tried Jupyter/IPython, google up a screencast and take a look. I'm just repeating the stuff that's on the home page in a different form, but this is a really big deal. Most of the most significant software of the next five or ten years is going to be built in TensorFlow or something like it. (Maybe Dan Amelang's Nile; see Bret Victor's amazing video on the dataflow visualization he and Dan built for Nile recently at https://youtu.be/oUaOucZRlmE?t=24m53s https://youtu.be/oUaOucZRlmE?t=24m53s, or try the live demo at http://tinlizzie.org/dbjr/high_contrast.html http://tinlizzie.org/dbjr/high_contrast.html, or download and build the Nile system from https://github.com/damelang/nile. https://github.com/damelang/nile.) Apparently the Googlers think that too, because among the 39 authors of the white paper are several shining stars of systems research, including some who may be up for a Turing Award in the next decade or two.

Fede_V 11y ago
One small note on the paper itself. I love that the paper's author are in alphabetical orders, without any stupid jostling over first/last authorship. I think that's fantastic and wish academia went that way too (yeah right).
- jamessb 11y ago

eddd 11y ago

2:21 in the video you can see a link to: corp.google.com I love their UI for single sign on.

andy_ppp 11y ago
Hmmm, the video appears to only be 2:17 long now...
- eddd 11y ago
  1:52 actually, sorry I was mistaken.

fitzwatermellow 11y ago

Great job, guys! The Docker of Neural Compute..

dryginmartini 11y ago

How does TensorFlow compare to Torch7 (http://torch.ch http://torch.ch)? They look very similar to me.

albertzeyer 11y ago
Actually, this looks much more like Theano to me. It's all symbolic, it has symbolic differentiation, it's Python and it even seems to have a similar API like Theano. In the Whitepaper (http://download.tensorflow.org/paper/whitepaper2015.pdf http://download.tensorflow.org/paper/whitepaper2015.pdf), it's compared to Theano, Torch and others.
- nickpsecurity 11y ago
  That might be a strong advantage in its favor, though. Building a better version of a proven concept with similar usage/API's often gets more market uptake. Might get more use.
mailshanx 11y ago
TensorFlow seems to emphasise the distributed computing aspects a lot more compared to Torch7. Also, TensorFlow is made for deploying machine learning pipelines in general, whereas Torch7 focuses on deep neural networks specifically.

blazespin 11y ago

GPU support, woohoo!

varelse 11y ago
And just 4 years after their insistence that GPUs would never have a role at Google... How 'bout that?
- pacala 11y ago
  Google is a big place, with diverse opinions. Even better, it encourages updating one's opinion in face of new evidence. GPUs have proven themselves cost-effective in tackling a number of vision problems [and others], Google has those problems, so the opinion was updated and the problems were solved in a cost-effective manner.
  - varelse 11y ago
    That... Was not... My experience there... But kudos to the deep learning guys for overcoming that potential energy barrier NVIDIA couldn't surmount on their own...
    - Florin_Andrei 11y ago
      To me, processing lots and lots of data with "relatively simple" algorithms spells GPU, pretty much. Machine learning seemed destined to bump into GPUs sooner or later.
      - varelse 11y ago
        Why yes... https://github.com/BIDData/BIDMach https://github.com/BIDData/BIDMach
- pilooch 11y ago
  Google is certainly Nvidia's top customer.

RivieraKid 11y ago

What kind of tool can be used to create that animated illustration? Looks really neat.

croddin 11y ago
It looks like it is TensorBoard, included with TensorFlow http://tensorflow.org/how_tos/summaries_and_tensorboard/index.md# http://tensorflow.org/how_tos/summaries_and_tensorboard/inde...

syllogism 11y ago

Any reason you wrote the Python wrapper in SWIG instead of Cython? The nice thing about Cython is that you can wire up two C extensions to talk to each other directly, without going through Python. Cython also gives you 2/3 compatibility out of the box.

pacala 11y ago
SWIG supports many other languages besides Python.

andyjohnson0 11y ago

"TensorFlow is an Open Source Software Library for Machine Intelligence" and then later "TensorFlow™ is an open source software library for numerical computation using data flow graphs." So it seems to be a dataflow computation library that is being used for AI/learning. Since I know almost nothing about either, I'm wondering if this (dataflow) approach has other applications unrelated to deep learning. Any comments? Also, i wonder if this could be implemented in hardware - say on a FPGA?

varelse 11y ago
FPGAs are not measuring up to GPUs yet for these tasks. There's too much floating-point math (and yes, Altera is addressing this) for forward prediction, and way too much intermediate state to save for training (GPU memory controllers are 'da bomb for this). Finally, OpenCL compilation time on FPGAs is measured in hours as compared to seconds for GPUs. That said, address all of the above and maybe NVIDIA's stock won't hit 50 next year.
- Sanddancer 11y ago
  There are a few people that seem to be working with integer-based deep learning and FPGAs, and even though they're at the beginning of research, things are looking pretty promising. Here's a paper that was released earlier this year that seems to show that at least some algorithms can see a big benefit from FPGAs http://arxiv.org/pdf/1502.02551v1.pdf http://arxiv.org/pdf/1502.02551v1.pdf .
- dharma1 11y ago
  did you see this? http://arxiv.org/pdf/1510.03009v1.pdf http://arxiv.org/pdf/1510.03009v1.pdf
  - nickpsecurity 11y ago
    Now THAT has some potential. Both for FPGA's and SIMD/MIMD architectures. Thanks for the link.

Fede_V 11y ago

This is awesome, thanks so much for sharing this - and I love how whimsical the docs are: Our pond is a perfect 500 x 500 square, as is the case for most ponds found in nature. A feedback on the PDE aspect: is there any nice way to write your model and grid using TensorFlow, and still obtain the free derivatives/optimized computation graph if you use a different solver to actually integrate the equations? The example shows off a very simple toy first order Euler solver, but in practice this is grossly inadequate. I suspect that it's very difficult if not impossible - especially since most solver libraries are very old Fortran libs, and carrying forward the reverse mode differentiation there is a nightmare. Still, this makes for very nice building blocks. Another question - would you consider adding an example of using this together with Google Ceres - to make the two autodiffs play nice together?

deleted 11y ago

[deleted]

mark_l_watson 11y ago

I can't wait to try this tonight. I have a fun "messing around" project (I am using GnuGo to generate as much training data as I need for playing Go on a very small board, and I am almost to the point of starting to train and test models). BTW, Ruby has always been my scripting language but because of the wealth of libraries for deep learning I am thinking of switching to Python.

mkorfmann 11y ago
Why switch? I use Ruby, when I can and Python when I can't. Would really love to see ruby forks of popular ml libs. Or is it because Python is better perfomance-wise?
- mark_l_watson 11y ago
  Really good question since I really enjoy using Ruby, more so even than Clojure and Haskell. I too would like to see Ruby ports of some of the popular ML libraries. The classifier gem provides an example.
thomasahle 11y ago
I believe the big facebook and google Go networks were trained on data from professional games. Probably you'd get better performance if you did that as well.
mark_l_watson 11y ago
I have started to experiment with it. Really awesome documentation and setup nstructions (using Ubuntu).

deleted 11y ago

[deleted]

nye2k 11y ago

If we could stop using the term Neural Networks in relation to computing, that would be great. Computers don't have neurons.

deleted 11y ago
[deleted]
BinaryIdiot 11y ago
There are artificial and biological neural networks. The artificial one is based upon the biological one. Therefore dropping the 'artificial' or 'biological' seems fine to me.
- nye2k 11y ago
  This is practical, however I strongly disagree with this semantically. The term should always be accompanied with 'artificial' or another name should be invented as they are vastly different fields. Search Neural Network, I assure you that you will no longer find biological information; it is littering.
  - deleted 11y ago
    [deleted]
ska 11y ago
If we get hung up on every abuse of terminology we run into, nobody will get anything useful done. At this point the name is well established. If it makes you feel better, just remember to tag "artificial" on to the beginning, nobody will mind.

obstbraende 11y ago

Are there any major conceptual differences to Theano? Not that I wouldn't appreciate a more polished, well funded competitor in the same space. It looks like using TensorFlow from Python will feel quite familiar to a Theano user, starting with the separation of graph building and graph running, but also down into the details of how variables, inputs and 'givens' (called feed dicts in tensorflow) are handled.

albertzeyer 11y ago
I'm looking at the RNN implementation right now (https://github.com/tensorflow/tensorflow/blob/master/tensorflow/models/rnn/rnn.py https://github.com/tensorflow/tensorflow/blob/master/tensorf...). It looks like the loop over the time frames is actually in Python itself. for time, input_ in enumerate(inputs): ... This confuses me a bit. Maybe the shape is not symbolic but must be fixed. I also haven't seen some theano.scan equivalent. Which is not needed in many cases when you know the shape in advance.
- obstbraende 11y ago
  I think this loop actually still only builds the graph -- what `scan` would do. The computation still happens outside of python. That is, in tensorflow they perhaps don't need `scan` because a loop with repeated assignments "just works"... Let's try this: It seems like in TensorFlow you can say: import tensorflow as tf sess = tf.InteractiveSession() # magic incantation state = init_state = tf.Variable(1) # initialise a scalar variable states = [] for step in range(10): # this seems to define a graph that updates `state`: state = tf.add(state,state) states.append(state) sess.run(tf.initialize_all_variables()) at this point, states is a list of symbolic tensors. now if you query for their value: print sess.run(states) >>> [2, 4, 8, 16, 32, 64, 128, 256, 512, 1024] you get what you would naively expect. I don't think that would work in Theano. Cool.
  - benanne 11y ago
    Why wouldn't this work in Theano? >>> import theano >>> import theano.tensor as T >>> state = theano.shared(1.0) >>> states = [] >>> for step in range(10): >>> state = state + state >>> states.append(state) >>> >>> f = theano.function([], states) >>> f() [array(2.0), array(4.0), array(8.0), array(16.0), array(32.0), array(64.0), array(128.0), array(256.0), array(512.0), array(1024.0)]

superfx 11y ago

Does anyone know how this compares to Twitter's recently released torch autograd? https://blog.twitter.com/2015/autograd-for-torch https://blog.twitter.com/2015/autograd-for-torch EDIT: To be a bit more specific, autograd just requires that one writes the forward model, and it can involve any program logic like conditionals, etc, so long as the end result is differentiable. It then takes care of all the gradient calculations.

musesum 11y ago

I'm kinda floored, by this. Have been developing a flow based language to do this. Even started with the approach of using Python and then moving to scripting directly in C++. Maybe I tweak my own project. Anyone playing with low level GPU deployments, like Metal? [EDIT] Ok, so there is some GPU - should probably play some more before asking obvious questions. Really keen on scripting out flow directly.

pilooch 11y ago
link to your project ?
- musesum 11y ago
  Sorry, not ready for prime-time. Closest is a visualization of the previous version of the script: https://www.youtube.com/watch?v=a703TTbxghc https://www.youtube.com/watch?v=a703TTbxghc That version wasn't a NN graph for ML; more of an executable flow graph for visual music. Will post something on https://github.com/musesum https://github.com/musesum in a couple months.

malmsteen 11y ago

So, i have a very simple question.. : I want to start deep learning now, to implement RNNs, autoencoders, Q-learning on a very specific application (not images). I've read a lot of papers, but not done any DL implementations yet (although many classical ML so far), my question is very simple : Where do I start ???? Should I use Torch ? Theano ? Theano + pylearn2 ? Theano + pylearn + Lasagne ? Caffe ? Or should I just switch to this magical new library directly ? I feel confused.. any advice ?

varelse 11y ago
Methinks embrace the magical library written by engineers and researchers at Google instead of by a random goulash of machine learning grad students with no investment in the outcome other than graduating.
- rhaps0dy 11y ago
  > with no investment in the outcome other than graduating Come on. Grad students are motivated by advancing the field. ... are they not? Maybe you can also be saying that the ones who write libraries are motivated only by graduating.
  - varelse 11y ago
    Sure, we all start out that way. But by the 3rd or 4th year, you frequently just want to write that thesis and get on to the next thing because you've already advanced the field as much as you're going to as a grad student. I have a large package of code on sourceforge that consists of my grad school and postdoctoral efforts. If I compare its style to what 2 decades in the industry since has taught me, it's laughable. That said, it is C code, so I occasionally plunder it for algorithms I figured out way back when.
    - benanne 11y ago
      Nobody forced us to open-source Lasagne, so I think that remark was a bit unfair. If we really didn't care about anything but graduating, why would we bother going through the trouble of sharing the code in the first place? But I do see your point. Google obviously has a lot more manpower to spend on this, so it might be a better bet in the long run. It's also worth comparing this to a few similar projects that have been announced recently: MXNet (http://mxnet.readthedocs.org/en/latest/ http://mxnet.readthedocs.org/en/latest/), Chainer (http://chainer.org/ http://chainer.org/) and CGT (http://rll.berkeley.edu/cgt/ http://rll.berkeley.edu/cgt/). And Theano of course, which has been the flag-bearer of the computational graph approach in deep learning for many years.

datashovel 11y ago

Didn't read the paper, but when I saw that a main focus is on distribution I started thinking about how awesome it would be if people could start crowdsourcing compute resources to solve some big problems.

doczoidberg 11y ago

I am a newbie to machine learning. How does it compete to Azure ML besides it is open source?

dswalter 11y ago
It's not really directly comparable. This is a computational framework which you can use to implement a variety of ML algorithms (or in general numerical computation), so the interface to this is lower-level than Azure ML. If Azure ML is a bunch of premade sandcastle molds, TensorFlow is a more accurate, faster way to pour sand. You make the mold.
- mohamedmansour 11y ago
  TensorFlow built a language, AzureML uses a bunch of languages of your choice to build any machine learning on Azure. Try it https://studio.azureml.net/ https://studio.azureml.net/ follow the tutorial

jheriko 11y ago

i really wish people would stop calling multidimensional arrays tensors. there is already well defined language for this. multidimensional array is fine, psuedotensor is fine. tensor is confusing if you have any previous background with tensor calculus before the word became machine learning flavour of the month. still. this does look pretty cool overall. processing large volumes of data is becoming increasingly less specialised and more generic, which can only be good. :)

xyzzyz 11y ago
I'd say tensor is exactly as confusing as vector -- vectors are just elements of some vector spaces, and you get a list of numbers by choosing a basis of that vector space. Similarly, tensors are just elements of a tensor product of some vector spaces (or modules), and you get a multidimensional array by choosing bases in the factors.
cmarschner 11y ago
Can't agree more. The first time I looked up tensors when they appeared in Torch I got totally confused by the description of tensors used in physics.
- jordigh 11y ago
  The problem is complicated a bit more because physicists use the word "tensor" to mean a tensor field, i.e. a collection of tensors on which calculus makes sense.
zer0nes 11y ago
Tensor is a much better term than multidimensional array: - It's not as verbose - The concept of tensor has linear algebra operations associated with it while multidimensional array is just a programming term without any attached Math operations.
- jordigh 11y ago
  > The concept of tensor has linear algebra operations associated with it while multidimensional array is just a programming term without any attached Math operations. But that's the precise problem. The multidimensional arrays that programmers call "tensors" do not generally have the operations defined on them that you would expect from a tensor, such as index contraction, tensor products, or differentiation. At the same time, a tensor is different from an array, just like a number is different from its decimal representation. These distinctions are important when you want to change the basis, a very useful and frequent operation for both tensors and numbers. Then again, this battle was already fought and lost for vectors, which programmers specialised to mean "arrays" instead of meaning "elements of a linear space".

irq-1 11y ago

This looks similar to Orange. Orange has a visual designer, easy data access (SQL, arbitrary files) and lots of widgets, but it's a desktop tool not a library and doesn't standardize the data structures. http://orange.biolab.si/ http://orange.biolab.si/ http://docs.orange.biolab.si/widgets/rst/index.html http://docs.orange.biolab.si/widgets/rst/index.html

infinite8s 11y ago
The newest version of Orange (version 3) is standardizing on the scientific python stack (numpy, scipy, scikit-learn)

thadd 11y ago

Just found what I'm going to do today

an4rchy 11y ago

I tried going through the site and also the comments but couldn't wrap my head around what this library actually is. It sounds awesome based on the response/comments. Can anyone explain this to a layman?

gyom 11y ago
If you want to train neural nets, you can either rewrite everything from scratch and get a bug-ridden sub-optimal implementation, or you can use a kind of off-the-shelf library. The problem is that there are about 3-5 alternatives out there, and none of them are mature enough or convincing enough to dominate. The field changes so fast that it's easy for them to become obsolete. What you're seeing here is the enthusiasm of people who really want to get a good tool with proper support, and be able to stick with it. I'm still not sure if TensorFlow is that tool, but it depends on what will happen to it during the coming years.
sputknick 11y ago
It looks like to me the big win here is that this is a ML library just like all the others, except that the code is not written for a specific type of processor (GPU versus CPU), nor is it written for a specific compute level (desktop, server, phone). It's target is to be a general purpose ML library. This should have the affect of getting ML solutions into useable applications more quickly, since you don't have to develop on one platform, then port to another. It just works everywhere. Best guess on my part based on what I'm reading on the site.
levesque 11y ago
It is a library for the optimization of machine learning algorithms, similar to Theano. You write a model you want to optimize as a graph of symbolic expressions, and the library will compile this graph to be executed on a target platform (could be CPU, GPU). This is really neat because you essentially wrote your program in Python and now it's going to be running as optimized C++ code (or as optimized kernels on your GPU). The fact that you defined your model as a symbolic graph means you can derive gradient equations automatically, hence ridding you of a rather tedious step in optimization. With such tools, the workflow for a practitioner is much simpler: 1-code models as graphs of symbolic expressions, and 2-define objective functions (ex. for a regression problem, your objective would be the root mean squared error, plus some regularization). Disclaimer: I did not look at the code, but from what I understand it is pretty much the same as Theano (which I've been using a lot lately).

devit 11y ago

Does it support OpenCL and thus Intel integrated and discrete AMD GPUs?

chimtim 11y ago
Appears to be only CUDA from the code. Also no distributed systems support released. Also, I cannot get it to work since this morning.
- happycube 11y ago
  Requires nvidia capability >=3.5 at that (GK11x and Maxwell) which rules out anything I've currently got (a couple of 760's/GK104's)... which is the chip Amazon EC2 uses as well. They're probably using Dynamic Parallelism.

Nitramp 11y ago

Maybe of interest: this is (amongst others) by Jeff Dean, famous for Bigtable, Map Reduce, Spanner, Google Web Search, Protocol Buffers, LevelDB, and many, many other things. He's a Google Senior Fellow, but also doubles as the patron saint of Google Engineering.

tootie 11y ago
Did they give him a Turing Award yet?
zeroxfe 11y ago
He is also a ninja, an astronaut, a master chef, and a damn good fellow!

samuell 11y ago

Seems like Ben Lorica's call for open source tensor libraries for data science [1], were answered! [1.a] http://radar.oreilly.com/2015/03/lets-build-open-source-tensor-libraries-for-data-science.html http://radar.oreilly.com/2015/03/lets-build-open-source-tens... (March 2015) [1.b] http://radar.oreilly.com/2015/05/the-tensor-renaissance-in-data-science.html http://radar.oreilly.com/2015/05/the-tensor-renaissance-in-d... (May 2015)

fibo 11y ago

For those interested in a minimal data flow JavaScript Engine see my project dflow: https://www.npmjs.com/package/dflow https://www.npmjs.com/package/dflow

cdnsteve 11y ago

AWS has GPU clusters already available FYI: http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using_cluster_computing.html http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using_clu... Not sure how much effort is required to use them with this library.

scott_s 11y ago

The programming model here is very similar to that of the language I work on, Streams Processing Language, normally just called SPL. A paper we have submitted on the language itself: http://www.scott-a-s.com/files/ibm_tr_2014.pdf http://www.scott-a-s.com/files/ibm_tr_2014.pdf Our domain is online stream processing for, generally, the big data space. I do think, however, that describing computations in this manner gives enormous flexibility to runtime systems to actually exploit all available parallelism dynamically.

peterjliu 11y ago

For those looking for more context, here's the Google research blog post: http://googleresearch.blogspot.com/2015/11/tensorflow-googles-latest-machine_9.html http://googleresearch.blogspot.com/2015/11/tensorflow-google...

fscherer 11y ago

can anybody tell me if this can calculate the gradients of a conditional graph? (something that is not implemented in theano which gives me a huge headache right now)

samuell 11y ago

Interesting! I wrote in a draft post back in late 2013 [1], that asked: "What if one could have a fully declarative “matrix language” in which all data transformations ever needed could be declaratively defined in a way that is very easy to comprehend?" I'm now pondering whether TensorFlow isn't quite an answer to this question? [1] Posted the draft now for reference: http://bionics.it/posts/matrix-transformation-as-model-for-data-flow-operations http://bionics.it/posts/matrix-transformation-as-model-for-d...

davmre 11y ago
The idea of representing a program by a declarative computation graph of matrix transformations has been big in deep learning research for a few years. Theano [1] is the canonical example, though more recently the space has gotten bigger with other frameworks including CGT [2] and now TensorFlow. The computational graph abstraction is really nice in part because it gives you very straightforward automatic differentiation, which dovetails very nicely with the trend in machine learning of casting a wide range of algorithms as (stochastic) gradient ascent on some loss function. [1] http://deeplearning.net/software/theano/ http://deeplearning.net/software/theano/ [2] http://rll.berkeley.edu/cgt/ http://rll.berkeley.edu/cgt/
- samuell 11y ago
  Thanks! Yea, though in fact my idea is more about wiring the actual (matrix/tensor) operation with dataflow, rather than just the dataflow between such operations. But it might be bit of a different problem area :)

pvnick 11y ago

François Chollet, the author of keras, made the following tweet this morning: "For those wondering about Keras and TensorFlow: I hope to start working on porting Keras to TensorFlow soon (Theano support will continue)."

sandGorgon 11y ago

what would be really awesome is if Andrew Ng or Norvig build a course around Tensorflow. It is really not useful to a beginner to be learning everything in matlab.

eob 11y ago
My first thought when reading the docs was: this is a game changer for the efficiency of applied ML (nlp, vision, speech) phd students. Coming from a school where there's a bit of a libertarian "write it all yourself, from scratch!" ethos, I always marveled at how much mileage other students got from research groups that built off a common codebase. Exciting to start to see glimmers of that possibility across the entire field.
davmre 11y ago
There are already a few university ML courses out there using Theano (for which Tensorflow is essentially a drop-in replacement), and I think this will be a much bigger trend over the next few years. IMHO for a first course it's useful to do some work at the Matlab/numpy level just so you get experience with deriving/implementing gradients yourself, but for larger (deep) models automatic differentiation is an amazing productivity boost that should make it possible to cover a lot of interesting topics that you'd otherwise not have space for.
- sandGorgon 11y ago
  actually numpy would be a brilliant starting point - but I'm not able to find any popular ones that dont use matlab (or some dialect thereof). disclaimer: I have no idea what I'm talking about, but I do know that coursera and stanford courses are the oft cited ones and they use matlab/octave.
- robert-zaremba 11y ago
  The same goes for Torch: New York, Oxford ...

deleted 11y ago

[deleted]

swah 11y ago

OTOH code completion is still only mediocre for most programming languages! Of course, those things aren't really related, but it feels easier than self-driving cars.

benmathes 11y ago

Found a mistake in the docs: http://tensorflow.org/get_started/basic_usage.md http://tensorflow.org/get_started/basic_usage.md In the "Variables" example, looks like a variable name got changed but not updated everywhere: # at the start: var = tf.Variable(0, name="counter") # then: one = tf.constant(1) new_value = tf.add(state, one) update = tf.assign(state, new_value) The variable "state" should be the variable "var", or vice-versa.

Omnipresent 11y ago

I was going to dive into Theano. Is this much different than Theano? Or better?

mrwilliamchang 11y ago

Google says they are working on a distributed version and will release when it is ready. https://github.com/tensorflow/tensorflow/issues/12#issuecomment-155150681 https://github.com/tensorflow/tensorflow/issues/12#issuecomm...

Dowwie 11y ago

Something tells me that Facebook's AI group will release its React to their Angular

mjw 11y ago

Does anyone know if TensorFlow can apply algebraic simplifications and numerical optimisations to the compute graph, in the way that Theano does with its optimisations? Sounds like it doesn't suffer from the (alleged) slow compile times of Theano, but I wonder if the flipside of that is that you have to implement larger-scale custom Ops (like torch's layers) in order to ensure that a composite compute graph is implemented optimally?

Houshalter 11y ago

This seems really similar to Brain Simulator: http://www.goodai.com/#!brain-simulator/c81c http://www.goodai.com/#!brain-simulator/c81c

nosoynadasinti 11y ago

Nosoynadasinti

nosoynadasinti 11y ago

Vsfd

nosoynadasinti 11y ago

Ndkdo9qmwbdvx

orasis 11y ago

Could this system be used to implement a Gaussian Process?

aminorex 11y ago

Quite an ugly set of dependencies to build. In particular, bootstrapping bazel is excruciating.

ertand 11y ago

Geoffrey Hinton and Jeff Dean will give separate tutorials at NIPS. https://nips.cc/Conferences/2015/Schedule?day=0 https://nips.cc/Conferences/2015/Schedule?day=0

ppoutonnet 11y ago

Great job guys!

auston 11y ago

Many many thanks in advance to anyone who answers: I've futzed around with ML docs (dataquest, random articles, first 3 weeks of andrew ng course on coursera) & what I don't get about this (http://tensorflow.org/tutorials/mnist/beginners/index.md http://tensorflow.org/tutorials/mnist/beginners/index.md) is how to actually make a prediction / classification on something that is not "test" data. Can anyone point me in the right direction of how to "use" a model once I have it validated, like we do at the end of that tutorial?

jefurii 11y ago

The diagram at least looks a lot like Max/MSP and PureData, which are data-flow tools for processing MIDI and audio data. Could this be used to implement something along those lines?

HockeyPlayer 11y ago

Anyone experienced with machine learning want to try this out on some high frequency trading data? I'm in the processing of preparing a data set for more traditional analysis and would be willing to share. We are in Colorado, but I'm happy to work with someone remotely.

deleted 11y ago
[deleted]

rcthompson 11y ago

Is there a part of TensorFlow that generates that slick diagram on the front page?

argonaut 11y ago

Honestly, the raving that's going on in this thread is unwarranted. This is a very nice, well put together library, but it does not do anything fundamentally different from what has already been done with Theano / Torch. It is not a "game-changer" or a spectacular moment in history as some people seem to be saying.

vmarkovtsev 11y ago

This is very similar to what we did in Samsung several years ago: https://velesnet.ml https://velesnet.ml

joanfihu 11y ago

Great stuff to see that Google is releasing their own tools for research and production systems. Does anybody have a grasp of the main differences between the triple Ts: Tensorflow, Theano and Torch? As far as I can see Theano and Tensorflow support dataflow-like computations, automatic differentiation and GPU support via CUDA. Finally, why didn't Google, The Lisa Lab and Facebook worked together on building a unique library instead of three?

m00n 11y ago

Minor nitpick: The authors are not the first ones to recognize that 'tensor' makes a great (brand-) name. As someone who is paid to think about tensors this is a bit annoying, because the first thing in my mind when I read a page like OP, is "where are the tensors?". Alas, there are none. For a mathematician, a "multidimensional data array" is not a tensor. It is a tuple. A tensor is a tuple with much more structure, associated to linear actions on its components. Said differently, if you can't tell me, what 7*[Alice] means, you have no right to call [Alice] a tensor.

vasthack 11y ago

i never believed hacking was real until i was helped with my grades by one crazy hacker. i wonder how disappointing it would have been if i didn't graduate. thank you VASTHACK. You may contact him via vasthack@gmail.com

vasthack 11y ago
wow! I didn't know he could do that to he helped me moved money from a cheating partner's bank account too. That guy is indeed crazy