4 ms·

TensorFlow: Large-Scale Machine Learning on Distributed Systems (2015) [pdf]

201 points by max_ 11y ago

struct 11y ago
Not to sound overly critical, but I don't enjoy the tone of this paper: presenting the concepts (e.g. graph operations) and features (e.g. automatic differentiation) of TensorFlow as new and novel when in fact very similar systems like Theano have existed since at least 2010 (which they finally get round to mentioning 14 pages in). They really should have shorn the paper in half and just focused on the distributed bit, which is the really novel and exciting bit. Addendum/Edit: With that said, I don't want to disparage the amazing technical achievement of the Google Brain team and the way TensorFlow really smartly reuses and improves on the concepts that make Theano really powerful. I'm following the project very closely and TensorFlow has a very high chance of being the foundation for my future projects :)
- zeroxfe 11y ago
  Where does it say that its automatic differentiation feature is new and novel? It actually calls out Theano in the paper: "Like Theano and Chainer, TensorFlow supports symbolic differentiation, thus making it easier to define and work with gradient based optimization algorithms." EDIT: Okay, I see that you already saw the reference. I guess I didn't find the tone as bad as you did. :-/
  - emcq 11y ago
    It's confusing but automatic differentiation, symbolic differentiation, and finite differences are all different approaches. "Automatic differentiation" typically refers to reverse order chain rule. Higher level libraries do really neat stuff with it. For example autograd can differentiate much of your numpy calls, even of composed of many complex functions.
- crocal 11y ago
  Even on distribution, I fail to understand where the novel idea is. I must miss something.
  - cinquemb 11y ago
    Sadly, most are only even paying attention because of the 100kg gorilla in the room…
  - deleted 11y ago
    [deleted]
- pacala 11y ago
  Not to sound overly critical ;), but the word "novel" doesn't appear in the paper and the word "new" only appears in tehnical contexts like "quickly experimenting with new models".
brudgers 11y ago
Preliminary White Paper A Google Beta by any other name would smell so sweet.
dekhn 11y ago
it's not a white paper, it's a full scientific publication.
- sp332 11y ago
  The name of the file on the server is "whitepaper2015.pdf"
- deleted 11y ago
  [deleted]
- GFK_of_xmaspast 11y ago
  Literally the third line of the article says: "(Preliminary White Paper, November 9, 2015)"
  - dekhn 11y ago
    The line is wrong.
    - GFK_of_xmaspast 11y ago
      If I'm not supposed to trust that line, how do I know what lines I'm supposed to trust.
    - vrv 11y ago
      The arxiv paper is just a repost of the preliminary paper, which is really more a design outline and rationale -- we may work on a more substantial paper later, we've been busy building the software.
      - dekhn 11y ago
        It would be nice if you changed it from "white paper" to something more accurate, like "provisional". The reason is that "white paper" has a very specific connotation, and this paper is not consistent with the white paper. Personal opinion: in science the word publication now means "to make accessible" rather "to publish in a conference" (the conference = publication thing is mostly CS) or "to publish in a journal" (like biology and a lot of non-arxiv physics). Pre-pub peer review isn't even required any more. arxiv (and scientists putting preprints on web sites) changed this. Of course, this is personal opinion, based mainly my observations and doesn't necessarily represent the mainstream opinion (yet).
nxzero 11y ago
>> "A computation expressed using TensorFlow can be executed with little or no change on a wide variety of hetero- geneous systems, ranging from mobile devices such as phones and tablets up to large-scale distributed systems of hundreds of machines and thousands of computational devices such as GPU cards." Are there other comparable systems that deploy using the same build across all platforms? If not, putting aside other advantages like doing computation across multiple machines, being able to using the same build is a major step forward in my opinion.
- mindcrime 11y ago
  Are there other comparable systems that deploy using the same build across all platforms? SystemML[1] doesn't, AFAIK, claim to scale down to mobile devices, but it does scale from a single desktop machine up to an arbitrarily large distributed cluster, using the same code. If you write code in DML or PyDML, the runtime will take care of distributing it across the cluster. And it's "smart" enough to take the cluster characteristics into account, and can also dynamically re-evaluate stages of the computation as the job is running, so it's always trying to use resources as efficiently as possible. [1]: https://systemml.apache.org/ https://systemml.apache.org/
cheez 11y ago
Are there APIs to do the training in C++? I can't get away with doing what I want to do in the cloud (nor do I really want to). Edit: I see http://mlpack.org/ http://mlpack.org/ - anyone have experience using this?
- covi 11y ago
  More C++ support is on our roadmap: https://github.com/tensorflow/tensorflow/blob/master/tensorflow/g3doc/resources/roadmap.md#improve-support-for-c-only-users https://github.com/tensorflow/tensorflow/blob/master/tensorf...
lucasp82 11y ago
Does anyone know what they used to generate those technical diagrams? Che bella!
- colah3 11y ago
  I drew the diagrams in Inkscape. (Naturally, this doesn't include the screenshots of TensorBoard or EEG.)

201 points by max_ 11y ago

Not to sound overly critical, but I don't enjoy the tone of this paper: presenting the concepts (e.g. graph operations) and features (e.g. automatic differentiation) of TensorFlow as new and novel when in fact very similar systems like Theano have existed since at least 2010 (which they finally get round to mentioning 14 pages in). They really should have shorn the paper in half and just focused on the distributed bit, which is the really novel and exciting bit. Addendum/Edit: With that said, I don't want to disparage the amazing technical achievement of the Google Brain team and the way TensorFlow really smartly reuses and improves on the concepts that make Theano really powerful. I'm following the project very closely and TensorFlow has a very high chance of being the foundation for my future projects :)

zeroxfe 11y ago
Where does it say that its automatic differentiation feature is new and novel? It actually calls out Theano in the paper: "Like Theano and Chainer, TensorFlow supports symbolic differentiation, thus making it easier to define and work with gradient based optimization algorithms." EDIT: Okay, I see that you already saw the reference. I guess I didn't find the tone as bad as you did. :-/
- emcq 11y ago
  It's confusing but automatic differentiation, symbolic differentiation, and finite differences are all different approaches. "Automatic differentiation" typically refers to reverse order chain rule. Higher level libraries do really neat stuff with it. For example autograd can differentiate much of your numpy calls, even of composed of many complex functions.
crocal 11y ago
Even on distribution, I fail to understand where the novel idea is. I must miss something.
- cinquemb 11y ago
  Sadly, most are only even paying attention because of the 100kg gorilla in the room…
- deleted 11y ago
  [deleted]
pacala 11y ago
Not to sound overly critical ;), but the word "novel" doesn't appear in the paper and the word "new" only appears in tehnical contexts like "quickly experimenting with new models".

brudgers 11y ago

Preliminary White Paper A Google Beta by any other name would smell so sweet.

dekhn 11y ago

it's not a white paper, it's a full scientific publication.

sp332 11y ago
The name of the file on the server is "whitepaper2015.pdf"
deleted 11y ago
[deleted]
GFK_of_xmaspast 11y ago
Literally the third line of the article says: "(Preliminary White Paper, November 9, 2015)"
- dekhn 11y ago
  The line is wrong.
  - GFK_of_xmaspast 11y ago
    If I'm not supposed to trust that line, how do I know what lines I'm supposed to trust.
  - vrv 11y ago
    The arxiv paper is just a repost of the preliminary paper, which is really more a design outline and rationale -- we may work on a more substantial paper later, we've been busy building the software.
    - dekhn 11y ago
      It would be nice if you changed it from "white paper" to something more accurate, like "provisional". The reason is that "white paper" has a very specific connotation, and this paper is not consistent with the white paper. Personal opinion: in science the word publication now means "to make accessible" rather "to publish in a conference" (the conference = publication thing is mostly CS) or "to publish in a journal" (like biology and a lot of non-arxiv physics). Pre-pub peer review isn't even required any more. arxiv (and scientists putting preprints on web sites) changed this. Of course, this is personal opinion, based mainly my observations and doesn't necessarily represent the mainstream opinion (yet).

nxzero 11y ago

>> "A computation expressed using TensorFlow can be executed with little or no change on a wide variety of hetero- geneous systems, ranging from mobile devices such as phones and tablets up to large-scale distributed systems of hundreds of machines and thousands of computational devices such as GPU cards." Are there other comparable systems that deploy using the same build across all platforms? If not, putting aside other advantages like doing computation across multiple machines, being able to using the same build is a major step forward in my opinion.

mindcrime 11y ago
Are there other comparable systems that deploy using the same build across all platforms? SystemML[1] doesn't, AFAIK, claim to scale down to mobile devices, but it does scale from a single desktop machine up to an arbitrarily large distributed cluster, using the same code. If you write code in DML or PyDML, the runtime will take care of distributing it across the cluster. And it's "smart" enough to take the cluster characteristics into account, and can also dynamically re-evaluate stages of the computation as the job is running, so it's always trying to use resources as efficiently as possible. [1]: https://systemml.apache.org/ https://systemml.apache.org/

cheez 11y ago

Are there APIs to do the training in C++? I can't get away with doing what I want to do in the cloud (nor do I really want to). Edit: I see http://mlpack.org/ http://mlpack.org/ - anyone have experience using this?

covi 11y ago
More C++ support is on our roadmap: https://github.com/tensorflow/tensorflow/blob/master/tensorflow/g3doc/resources/roadmap.md#improve-support-for-c-only-users https://github.com/tensorflow/tensorflow/blob/master/tensorf...

lucasp82 11y ago

Does anyone know what they used to generate those technical diagrams? Che bella!

colah3 11y ago
I drew the diagrams in Inkscape. (Naturally, this doesn't include the screenshots of TensorBoard or EEG.)