7 ms·

The tensor renaissance in data science

85 points by datascientist 11y ago

clebio 11y ago
Does using tensors for storage yield some benefit over the 'panels' metaphor that Python Pandas affords? It seems like the real power of tensors is in the calculus that can be run over them, rather than just them by themselves. If I want to store lists of numbers, arrays and (linked) lists are basically equivalent. It's when I want to calculate cross- and dot-products that vectors prove especially useful. The discussions I've seen recently in data science about tensors seems to only be about the storage aspect. And, I suppose, if your libraries _don't_ provide such a storage mechanism, then yes, you'll benefit from adding that capability. But I keep waiting for more from the ML community about leveraging differential geometry full-on. I assume it's out there somewhere, in some shops, but I've not found a lot of discussion about it.
- tel 11y ago
  There's actually quite a lot of this under the phrase "information geometry". Offhand, I feel like a lot of this work is coming out of Japan. http://www.amazon.com/Information-Translations-Mathematical-Monographs-Tanslations/dp/0821843028/ref=pd_sim_b_2?ie=UTF8&refRID=0E92317F4Y8R9KQWT7JM http://www.amazon.com/Information-Translations-Mathematical-... http://www.amazon.com/Algebraic-Statistical-Monographs-Computational-Mathematics/dp/0521864674 http://www.amazon.com/Algebraic-Statistical-Monographs-Compu...
  - clebio 11y ago
    Oh, awesome, thanks for those links!
    - tel 11y ago
      No problem! There's also one more book I was trying to find which basically recasts normal linear regression in the geometric projection terms that most people know but that are never emphasized when learning it. The nice part of that method is that then you can extend the geometric interpretation to manifolds and get, I believe, GLM very naturally.
      - clebio 11y ago
        Thanks again. I'd found this text[1], but it's a bit pricey! [1]: http://www.amazon.com/gp/product/0412398605/ http://www.amazon.com/gp/product/0412398605/
chestervonwinch 11y ago
Every time I get hear about tensors: "It's just a multidimensional array!" Ok, simple enough. But then I go to wikipedia get hit with differential geometry, manifolds, multi-linear algebra, etc... I mean these are all topics I've been meaning to learn for a while, but each one seems like it would take a year to properly introduce myself to (for which I never seem to have the time).
- krastanov 11y ago
  Tensors are most definitely not just multidimensional arrays. (this is a pet peeve of mine) Tensors have geometric meaning just like vectors (just like vectors are not just single-axis arrays). It would be too long to delve into the geometry (especially if it is abstract "geometry" like the geometrical space spanned by the songs in the Pandora database or the space spanned by the faces in a face-recognition software). Instead I will try to give an example: You all know what rotating a vector means. All the components change in certain way, but the geometrical meaning of this object is preserved in some sense. This is not true for some arbitrary list of numbers. Well, a tensor is just a more complicated geometrical structure that still has very specific rules governing what its components can be and how they can change. And a counterexample: the Christoffel Symbol is a multi-dimensional array that is not a tensor. P.S. It is true that it is a very common abuse of nomenclature to call all multidimensional arrays "tensors".
  - chestervonwinch 11y ago
    It's true though that vectors are basically 1D lists if we're speaking about elements from the vector space R^n. Is it true that tensors are multidimensional arrays if they come from the space R^{m1 x m2 x ... x mn} ? I'm also confused: do you think about a tensor as a vector from vector space? as a object that maps between vector spaces? or something completely different? I realize there's a LOT of detail here - just trying to flesh out my 500,000,000 mile view a little more.
    - tel 11y ago
      It's not true, though. A list of numbers in R is not a vector until you imbue the position of each of those numbers with a geometric basis, or, alternatively, relate that list of numbers to the set of all lists of such numbers along with some algebraic operations which turn it into a vector space. This is similar with tensors. It's even harder to talk about them, though, without the geometric/algebraic bits.
riemannzeta 11y ago
Although not a small amount of effort, it really helps to learn geometric algebra notation: http://www.amazon.com/Geometric-Algebra-Physicists-Chris-Doran/dp/0521715954 http://www.amazon.com/Geometric-Algebra-Physicists-Chris-Dor... Once you get an intuitive grasp of bivectors, it's not hard to "see" tensors as a linear map of bivector into bivector.
- cschwan 11y ago
  This is a really great book! It covers a lot of physics and gives plenty of insight on the relation of geometry and physics. It even made me program this library: https://github.com/cschwan/hep-ga https://github.com/cschwan/hep-ga (shameless self-advertisement).
dxbydt 11y ago
Luckily, I was at mlconf last week where Dr.Anandkumar spoke - they called her the "tensor lady" :) She's using tensors in machine learning for a bunch of things - Latent Variable Models: Training LVM's using local search methods like EM, gradient descent, variational bayes etc. have a bunch of problems - they get stuck on local minima, the algorithms are hard to parallelize with poor convergence. In these cases, tensors yield guaranteed learning using embarassingly parallel algorithms, so faster convergence & can be run on Spark. Also saw a demo on training 2-layer nets for GMM using tensors, and they learnt the weights rather fast. So using tensors in deep learning shows promise, though the techniques are in their infancy. One of the challenges the professor mentioned was the availability of open source libraries to do tensor decomposition, which the above methods require. It was a very successful talk - https://twitter.com/cdubhland/status/594220061025435649 https://twitter.com/cdubhland/status/594220061025435649 Tensor slides: http://www.slideshare.net/SessionsEvents/animashree-anandkumar-electrical-engineering-and-cs-dept-uc-irvine-at-mlconf-sea-50115 http://www.slideshare.net/SessionsEvents/animashree-anandkum...
- tachim 11y ago
  This isn't quite right. Moment methods (that rely on tensor decompositions) have a few problems: (i) they have convergence bounds, but in practice need more data than we have available (ii) they don't do as well as EM usually, but using them to initialize parameters for EM sometimes does better than EM with random initialization schemes (iii) it turns out variational methods can also be embarrassingly parallelized without losing much accuracy in practice (iv) right now moment methods don't work for arbitrary graphical models
  - dxbydt 11y ago
    I believe while you are right in general, she is looking at a class of problems for which tensors handily triumph other methods. You might be interested in these papers - http://newport.eecs.uci.edu/anandkumar/pubs/powerdynamics.pdf http://newport.eecs.uci.edu/anandkumar/pubs/powerdynamics.pd... http://newport.eecs.uci.edu/anandkumar/pubs/ProvableNN_sparse.pdf http://newport.eecs.uci.edu/anandkumar/pubs/ProvableNN_spars...
jheriko 11y ago
He keeps saying tensor when he means pseudotensor. It's an important distinction Tensors and tensor fields can be even more powerful by expressing non trivial relationships between components that remain unchanged under transformation into other spaces... Although whether that applies here I do not really know.
nether 11y ago
Coworker drives a Toyota Matrix, has a license plate frame saying, "My other car is a tensor."
deleted 11y ago
[deleted]
animakumar 11y ago
It is great to see all the discussion. We also have source code for topic modeling using tensor methods. Look forward to feedback and further code development http://t.co/VvWyTmZLps http://t.co/VvWyTmZLps

85 points by datascientist 11y ago

Does using tensors for storage yield some benefit over the 'panels' metaphor that Python Pandas affords? It seems like the real power of tensors is in the calculus that can be run over them, rather than just them by themselves. If I want to store lists of numbers, arrays and (linked) lists are basically equivalent. It's when I want to calculate cross- and dot-products that vectors prove especially useful. The discussions I've seen recently in data science about tensors seems to only be about the storage aspect. And, I suppose, if your libraries _don't_ provide such a storage mechanism, then yes, you'll benefit from adding that capability. But I keep waiting for more from the ML community about leveraging differential geometry full-on. I assume it's out there somewhere, in some shops, but I've not found a lot of discussion about it.

tel 11y ago
There's actually quite a lot of this under the phrase "information geometry". Offhand, I feel like a lot of this work is coming out of Japan. http://www.amazon.com/Information-Translations-Mathematical-Monographs-Tanslations/dp/0821843028/ref=pd_sim_b_2?ie=UTF8&refRID=0E92317F4Y8R9KQWT7JM http://www.amazon.com/Information-Translations-Mathematical-... http://www.amazon.com/Algebraic-Statistical-Monographs-Computational-Mathematics/dp/0521864674 http://www.amazon.com/Algebraic-Statistical-Monographs-Compu...
- clebio 11y ago
  Oh, awesome, thanks for those links!
  - tel 11y ago
    No problem! There's also one more book I was trying to find which basically recasts normal linear regression in the geometric projection terms that most people know but that are never emphasized when learning it. The nice part of that method is that then you can extend the geometric interpretation to manifolds and get, I believe, GLM very naturally.
    - clebio 11y ago
      Thanks again. I'd found this text[1], but it's a bit pricey! [1]: http://www.amazon.com/gp/product/0412398605/ http://www.amazon.com/gp/product/0412398605/

chestervonwinch 11y ago

Every time I get hear about tensors: "It's just a multidimensional array!" Ok, simple enough. But then I go to wikipedia get hit with differential geometry, manifolds, multi-linear algebra, etc... I mean these are all topics I've been meaning to learn for a while, but each one seems like it would take a year to properly introduce myself to (for which I never seem to have the time).

krastanov 11y ago
Tensors are most definitely not just multidimensional arrays. (this is a pet peeve of mine) Tensors have geometric meaning just like vectors (just like vectors are not just single-axis arrays). It would be too long to delve into the geometry (especially if it is abstract "geometry" like the geometrical space spanned by the songs in the Pandora database or the space spanned by the faces in a face-recognition software). Instead I will try to give an example: You all know what rotating a vector means. All the components change in certain way, but the geometrical meaning of this object is preserved in some sense. This is not true for some arbitrary list of numbers. Well, a tensor is just a more complicated geometrical structure that still has very specific rules governing what its components can be and how they can change. And a counterexample: the Christoffel Symbol is a multi-dimensional array that is not a tensor. P.S. It is true that it is a very common abuse of nomenclature to call all multidimensional arrays "tensors".
- chestervonwinch 11y ago
  It's true though that vectors are basically 1D lists if we're speaking about elements from the vector space R^n. Is it true that tensors are multidimensional arrays if they come from the space R^{m1 x m2 x ... x mn} ? I'm also confused: do you think about a tensor as a vector from vector space? as a object that maps between vector spaces? or something completely different? I realize there's a LOT of detail here - just trying to flesh out my 500,000,000 mile view a little more.
  - tel 11y ago
    It's not true, though. A list of numbers in R is not a vector until you imbue the position of each of those numbers with a geometric basis, or, alternatively, relate that list of numbers to the set of all lists of such numbers along with some algebraic operations which turn it into a vector space. This is similar with tensors. It's even harder to talk about them, though, without the geometric/algebraic bits.

riemannzeta 11y ago

Although not a small amount of effort, it really helps to learn geometric algebra notation: http://www.amazon.com/Geometric-Algebra-Physicists-Chris-Doran/dp/0521715954 http://www.amazon.com/Geometric-Algebra-Physicists-Chris-Dor... Once you get an intuitive grasp of bivectors, it's not hard to "see" tensors as a linear map of bivector into bivector.

cschwan 11y ago
This is a really great book! It covers a lot of physics and gives plenty of insight on the relation of geometry and physics. It even made me program this library: https://github.com/cschwan/hep-ga https://github.com/cschwan/hep-ga (shameless self-advertisement).

dxbydt 11y ago

Luckily, I was at mlconf last week where Dr.Anandkumar spoke - they called her the "tensor lady" :) She's using tensors in machine learning for a bunch of things - Latent Variable Models: Training LVM's using local search methods like EM, gradient descent, variational bayes etc. have a bunch of problems - they get stuck on local minima, the algorithms are hard to parallelize with poor convergence. In these cases, tensors yield guaranteed learning using embarassingly parallel algorithms, so faster convergence & can be run on Spark. Also saw a demo on training 2-layer nets for GMM using tensors, and they learnt the weights rather fast. So using tensors in deep learning shows promise, though the techniques are in their infancy. One of the challenges the professor mentioned was the availability of open source libraries to do tensor decomposition, which the above methods require. It was a very successful talk - https://twitter.com/cdubhland/status/594220061025435649 https://twitter.com/cdubhland/status/594220061025435649 Tensor slides: http://www.slideshare.net/SessionsEvents/animashree-anandkumar-electrical-engineering-and-cs-dept-uc-irvine-at-mlconf-sea-50115 http://www.slideshare.net/SessionsEvents/animashree-anandkum...

tachim 11y ago
This isn't quite right. Moment methods (that rely on tensor decompositions) have a few problems: (i) they have convergence bounds, but in practice need more data than we have available (ii) they don't do as well as EM usually, but using them to initialize parameters for EM sometimes does better than EM with random initialization schemes (iii) it turns out variational methods can also be embarrassingly parallelized without losing much accuracy in practice (iv) right now moment methods don't work for arbitrary graphical models
- dxbydt 11y ago
  I believe while you are right in general, she is looking at a class of problems for which tensors handily triumph other methods. You might be interested in these papers - http://newport.eecs.uci.edu/anandkumar/pubs/powerdynamics.pdf http://newport.eecs.uci.edu/anandkumar/pubs/powerdynamics.pd... http://newport.eecs.uci.edu/anandkumar/pubs/ProvableNN_sparse.pdf http://newport.eecs.uci.edu/anandkumar/pubs/ProvableNN_spars...

jheriko 11y ago

He keeps saying tensor when he means pseudotensor. It's an important distinction Tensors and tensor fields can be even more powerful by expressing non trivial relationships between components that remain unchanged under transformation into other spaces... Although whether that applies here I do not really know.

nether 11y ago

Coworker drives a Toyota Matrix, has a license plate frame saying, "My other car is a tensor."

deleted 11y ago

[deleted]

animakumar 11y ago

It is great to see all the discussion. We also have source code for topic modeling using tensor methods. Look forward to feedback and further code development http://t.co/VvWyTmZLps http://t.co/VvWyTmZLps