3 ms·
> However, although tensors describe the relationship between arbitrary higher-dimensional arrays, in practice the TPU hardware that we will consider is designe
by layer8 3y ago
> However, although tensors describe the relationship between arbitrary higher-dimensional arrays, in practice the TPU hardware that we will consider is designed to perform calculations associated with one and two-dimensional arrays. Or, more specifically, vector and matrix operations.
I still don’t understand why the term “tensor” is used if it’s only vectors and matrices.
- ralusek 3y agoIf nothing else, the term "tensor" is shorter than "vectors and matrices," and then has the added benefit of representing n-dimensional arrays.
- layer8 3y agoHow is that an added benefit if the hardware doesn’t actually support n-dimensional arrays (other the n = 1 and 2)? And, strictly speaking, a vector can be considered a 1xn (or nx1) matrix, so Matrix Processing Unit would have been fine.
- thatguysaguy 3y agoAt the end of the day all the arrays are 1 dimensional and thinking of them as 2 dimensional is just an indexing convenience. A matrix multiply is a bunch of vector dot products in a row. Higher tensor contractions can be built out of lower-dimensional ones, so I don't think it's really fair to say the hardware doesn't support it.
- whimsicalism 3y agoit’s an abstraction, just like 2d arrays
- layer8 3y agoI’d say it’s more like calling an ALU that can perform unary and binary operations (so 1 or 2 inputs) an “array processing unit” because it’s like it can process 1- and 2-element arrays. ;)
- whimsicalism 3y agowhat? the ml framework can support n-dimensional arrays. that’s what i mean by an abstraction
- deleted 3y ago[deleted]
- necroforest 3y agoIt's branding (see: TensorFlow); also, pretty much anything (linear) you would do with an arbitrarily ranked tensor can be expressed in terms of vector ops and matmuls
- nxobject 3y ago"Fixed-Function Matrix Accelerator" just doesn't have the same buzzy ring to it.
- sillysaurusx 3y agoI was confused as hell for a long time when I first got into ML, until I figured out how to think about tensors in a visual way. You're right: fundamentally ML is about vector and matrix operations (1D and 2D). So then why are most ML programs 3D, 4D, and in a transformer sometimes up to 6D (?!) One reasonable guess is that the third dimension is time. Actually not. It turns out that time is pretty rare in ML, and it's only (relatively) recently that it's been introduced into e.g. video models. Another guess is that it's to represent "time" as in, think of how transformers work: they generate a token, then another given the previous, then a third given the first two, etc. That's a certain way of describing "time". But it turns out that transformers don't do this as a 3D or 4D dimension. It only needs to be 2D, because tokens are 1D -- if you're representing tokens over time, you get a 2D output. So even with a cutting edge model like transformers, you still only need plain old 2D matrix operations. The attention layer creates a mask, which ends up being 2D. So then why do models get to 3D and above? Usually batching. You get a certain efficiency boost when you pack a bunch of operations together. And if you pack a bunch of 2D operations together, that third dimension is the batch dimension. For images, you typically end up with 4D, with the convension N,C,H,W, which stands for "Batch, Channel, Height, Width". It can also be N,H,W,C, which is the same thing but it's packed in memory as red green blue, red green blue, etc instead of all the red pixels first, then all the green pixels, then all the blue pixels. This matters in various subtle ways. I have no idea why the batch dimension is called N, but it's probably "number of images". "Vector" wouldn't quite cover all of this, and although "tensor" is confusing, it's fine. It's the ham sandwich of naming conventions: flexible, satisfying to some, and you can make them in a bunch of different varieties. Under the hood, TPUs actually flatten 3D tensors down into 2D matrix multiplications. I was surprised by this, but it makes total sense. The native size for a TPU is 8x128 -- you can think of it a bit like the native width of a CPU, except it's 2D. So if you have a 3x4x256 tensor, it actually gets flattened out to 12x256, then the XLA black box magic figures out how to split that across a certain number of 8x128 vector registers. Note they're called "vector registers" rather than "tensor registers", which is interesting. See https://cloud.google.com/tpu/docs/performance-guide https://cloud.google.com/tpu/docs/performance-guide
- layer8 3y agoThanks for the background! I still don’t think it’s appropriate to call a batch of matrices a tensor.
- shrubble 3y agoTensor is from mathematics and was popularized over a century ago.
- layer8 3y agoI know what a tensor is mathematically. However, as far as I can see, ML isn’t based on tensor calculus as such.
- phkahler 3y agoSomething similar happens on Wikipedia, where topics that use math inevitably get explained in the highest level math possible. It makes topics harder to understand than they need to be.
- jiggawatts 3y agoAs a helpful Wiki editor just trying to make sure that we don't lead people astray, I've made some small changes to clarify your statement: In the virtual compendium of Wikipedia, an extensive repository of human knowledge, there is a discernible proclivity for the hermeneutics of mathematically-infused topics to be articulated through the prism of esoteric and sophisticated mathematical constructs, often employing a panoply of arcane lexemes and syntactic structures of Greek and Latin etymology. This phenomenon, redolent of an academic periphrasis, tends to transmute the exegesis of such subjects into a crucible of abstruse and high-order mathematical discourse. Consequently, this modus operandi obfuscates the intrinsic didactic intent, thereby precipitating an epistemological chasm that challenges the layperson's erudition and obviates the pedagogical utility of the exposition.
- xarope 3y agoscarily, I actually understood this.
- whimsicalism 3y agomultidimensional arrays are multilinear mappings, and that is how they are used in ml usually. it seems fine to me
- thatguysaguy 3y agoWell, in the transformer forward pass there are a bunch of 4-dimensional arrays being used.
- deleted 3y ago[deleted]
- smilekzs 3y agoCame in to say this. The Einsum notation makes it desirable to formulate your model/layer as multi-dimensional arrays connected by (loosely) named axes, without worrying too much about breaking it down to primitives yourself. Once you get used to it, the terseness is liberating.
- jeffhwang 3y ago(I think) technically, all of these mathematical objects are tensors of different ranks: 0. Scalar numbers are tensors of rank 0. 1. Vectors (eg velocity, acceleration in intro high school physics) are tensors of rank 1. 2. Matrices that you learn in intro linear anlgebra are tensors of rank 2. Nested arrays 1 level deep, aka a 2d array. 0. Tensors numbers are tensors of rank 3 or higher. I explain this as ‘nested arrays’ to people with programming backgrounds as nested arrays of arrays with 3dimensions of arrays or higher. But I’m mostly self-taught in math so ymmv.
- a_wild_dandan 3y agoEvery tensor is just a stack of vectors wearing a trench coat.
- WhitneyLand 3y agoIt says: tensors describe the relationship between high-d arrays It does not say: tensors “only” describe the relationship between high-d arrays The term “tensor” is used because it covers all cases: scalars, vectors, matrices, and higher-dimensional arrays. Tensors are still a generalization of vectors and matrices. Note the context: In ML and computer science, they are considered a generalization. From a strict pure math standpoint they can be considered different. As frustrating as it seems one is not really more right and context is the decider. There are lots of definitions across STEM fields that change based on the context or field they’re applied to.
- adrian_b 3y agoThe word tensor has become more ambiguous during the time. Before 1900, the use of the word tensor was consistent with its etymology, because it was used only for symmetric matrices, which correspond to affine transformations that stretch or compress a body in certain directions. The square matrix that corresponds to a general affine transformation can be decomposed into the product of a tensor (a symmetric matrix which stretches) and a versor (a rotation matrix, which is antisymmetric and which rotates). When Ricci-Curbastro and Levi-Civitta have published the first theory of what now are called tensors, they did not define any new word for the concept of a multidimensional array with certain rules of transformation when the coordinate system is changed, which is now called tensor. When Einstein has published the Theory of General Relativity during WWI in which he used what is now called tensor theory, for an unknown reason and without any explanation for this choice he has begun to use the word "tensor" with the current meaning, in contrast with all previous physics publications. Because Einstein has become extremely popular immediately after WWI, his usage of the word "tensor" has spread everywhere, including in mathematics (and including in the American translations of the works of Ricci and Levi-Civita, where the word tensor has been introduced everywhere, despite the fact that it did not exist in the original). Nevertheless, for many years the word "tensor" could not be used for arbitrary multi-dimensional arrays, but only for those which observe the tensor transformation rules with respect to coordinate changes. The use of the word "tensor" as a synonym for the word "array", like in ML/AI, is a recent phenomenon. Previously, e.g. in all early computer literature, the word "array" (or "table" in COBOL literature) was used to cover all cases, from scalars, vectors and matrices to arrays with an arbitrary number of dimensions, so no new words are necessary.
- adrian_b 3y agoI do not know which is the real origin of the fashion to use the word tensor in the context of AI/ML. Nevertheless, I have always interpreted it as a reference to the fact that the optimal method of multiplying matrices is to decompose the matrix multiplication into tensor products of vectors. The other 2 alternative methods, i.e. decomposing the matrix multiplication into scalar products of vectors or into AXPY operations on pairs of vectors, have a much worse ratio between computation operations and transfer operations. Unfortunately, most people learn in school the much less useful definition of the matrix multiplication based on scalar products of vectors, instead of its definition based on tensor products of vectors, which is the one needed in practice. The 3 possible methods for multiplying matrices correspond to the 6 possible orders for the 3 indices of the 3 nested loops that compute a matrix product.