4 ms·

We need also something like Tensorization:Introduction

alfalfasprout 3y ago
Tensor ops are usually implemented using the principles of vectorizing code. After all most tensors are implemented as 1D arrays with strides for each dimension with the innermost dimension always contiguous.
- richardw 3y ago
  ChatGPT interpretation: Sure! Let me break it down for you: When we talk about "tensor ops," we're referring to operations performed on tensors, which are multidimensional arrays of numbers commonly used in mathematics and computer science. Now, these tensor operations are typically implemented using a technique called "vectorizing code." In simple terms, vectorizing code means performing operations on entire arrays of data instead of looping through each element one by one. It's like doing multiple calculations at once, which can be more efficient and faster. Tensors are usually represented as 1D arrays, meaning all the elements are arranged in a single line. Each dimension of the tensor has a concept called "stride," which represents how many elements we need to skip to move to the next element in that dimension. This helps us efficiently access and manipulate the data in the tensor. Additionally, the innermost dimension of a tensor is always "contiguous," which means the elements are stored sequentially without any gaps. This arrangement also aids in efficient processing of the tensor data. So, in summary, tensor ops involve performing operations on multidimensional arrays, and we use vectorized code to do these operations more efficiently. Tensors are represented as 1D arrays with strides for each dimension, and the innermost dimension is always stored sequentially without any gaps.
- sva_ 3y ago
  I think the bigger difficulty lies in how to store and process this stuff on a GPU/other accelerator. If the OP cares about implementation details about how an API like PyTorch is made, I think the MiniTorch 'book' is a pretty good intro: https://minitorch.github.io/ https://minitorch.github.io/
  - lying4fun 3y ago
    The MiniTorch book seems amazing, thanks
- riku_iki 3y ago
  > the principles of vectorizing code. > 1D arrays I think accelerators mostly do computations over small matrices and not vectors, so it is "matricized" code, but I am not an expert in this area.
corsix 3y ago
From a hardware perspective, vector instructions operate on small 1D vectors, whereas tensor instructions operate on small 2D matrices. I say “instructions”, but it’s really only matrix multiply or matrix multiply and accumulate - most other instructions are fine staying as 1D.
- nologic01 3y ago
  If there is matrix multiply at hardware level its fair to have another name than vectorization. For example the dimensions and partitioning of large matrices to fit would be specific to that design and very different from rolling things out on 1D arrays
  - bee_rider 3y ago
    Maybe they should call it a Matrix Multiplication Xcelerator just to mess with people.
amelius 3y ago
We really need to name tensors differently. "Array" would be a more fitting name. Tensors are really different mathematical objects with a far more rich structure than those used in the Deep Learning context. https://en.wikipedia.org/wiki/Tensor https://en.wikipedia.org/wiki/Tensor
- itishappy 3y ago
  Same with vectors! https://en.wikipedia.org/wiki/Vector_(mathematics_and_physics) https://en.wikipedia.org/wiki/Vector_(mathematics_and_physic...
  - antognini 3y ago
    As a transplant to the ML world who migrated from physics I do find it funny that people in ML will insist that gradients are vectors. Once you get to general relativity your physics professors will beat into you the fact that gradients are not vectors. (They are one-forms.) But in the ML context it's a distinction without a difference, unless you're in a really specialized corner of the field like information geometry.
    - ReleaseCandidat 3y ago
      > Once you get to general relativity your physics professors will beat into you the fact that gradients are not vectors. In linear algebra you should learn this as soon as you hear about it. Is the english Wikipedia page correct in that you always use nabla for the gradient? https://en.wikipedia.org/wiki/Gradient#Generalizations https://en.wikipedia.org/wiki/Gradient#Generalizations Because `grad f = ∇f` holds in scalar fields f only (not in vector or tensor fields (tensors that aren't scalars)).
      - antognini 3y ago
        Even if you are taking the gradient of a scalar field, the resulting object is not a vector. The difference is that if you apply some transformation to your coordinate system a gradient transforms differently from a normal vector. In the ML context it generally doesn't matter because you don't do these transformations (and more generally there is no metric for your space). So you can just treat a gradient as an array of numbers. But in a physical context the distinction starts to matter, at least on a manifold with curvature. The one place it does matter in ML (that I'm aware of) is information geometry, where you try to do a transformation from the normal coordinate space to a more "natural" coordinate system that is based on the Fisher information, and this introduces curvature into model.
godelski 3y ago
This is mostly going to be the same. The main difference in tensorization is that your vectors are not contiguous. So you lose some speedup. How you typically optimize this is by flattening your tensor. Then your data is contiguious and all the same rules apply. This is overly simplified though. Things get different when we start talking about {S,M}I{S,M}D (page addresses SIMD) or GPUs. Parallel computing is a whole other game, and CUDA takes it to the next level (lots of people can write CUDA kernels, not a lot of people can write GOOD CUDA kernels (I'm definitely not a pro, but know some)). Parallelism adds another level of complexity due to being able to access different memory locations simultaneously. If you think concurrent optimization is magic, parallel optimization is wizardry (I still believe this is true)
quickthrower2 3y ago
PyTorch internals: http://blog.ezyang.com/2019/05/pytorch-internals/ http://blog.ezyang.com/2019/05/pytorch-internals/ A good basic thing to know is a tensor is a single dimensional array under the hood. At least as far as I know in my mental model - I have not yet looked at the code :-) Therefore "converting" a 2 * 5 * 10 tensor into a 2 * 50 one is an immediate operation, it doesn't have to scan. Just change some metadata.
- jabl 3y ago
  That's pretty much how every serious dense matrix (or more generally, multi-dimensional array) representation has been done since, well, forever. It's only C programmers writing "My First Matrix Program" that insist on allocating each row separately due to the lack of native multidimensional arrays in that language.
  - quickthrower2 3y ago
    Yes! I was surprised how simple it was. Obvious once you know how. I never really considered multidimensional storage until looking into Machine Learning recently. My only brush with it is relational database tuning, but that is very different, as the number of "dimensions" are small and they are usually not homogenous.
  - ReleaseCandidat 3y ago
    Fortran did it: A 2-dimensional array A will, in the object program, be stored sequentially in the order Ai,i, A2,i Am l , A| i2 , A2f2 Am, 2> , Am,„. Thus it is stored “columnwise”, with the first of its subscripts varying most rapidly, and the last varying least rapidly. The same is true of 3-dimensional arrays. 1 -dimensional arrays are of course simply stored sequentially. All arrays are stored backwards in storage; i.e. the above sequence is in the order of decreasing absolute location. https://ia800807.us.archive.org/0/items/history-of-fortran/Image072217151026_text.pdf https://ia800807.us.archive.org/0/items/history-of-fortran/I...

Tensor ops are usually implemented using the principles of vectorizing code. After all most tensors are implemented as 1D arrays with strides for each dimension with the innermost dimension always contiguous.

richardw 3y ago
ChatGPT interpretation: Sure! Let me break it down for you: When we talk about "tensor ops," we're referring to operations performed on tensors, which are multidimensional arrays of numbers commonly used in mathematics and computer science. Now, these tensor operations are typically implemented using a technique called "vectorizing code." In simple terms, vectorizing code means performing operations on entire arrays of data instead of looping through each element one by one. It's like doing multiple calculations at once, which can be more efficient and faster. Tensors are usually represented as 1D arrays, meaning all the elements are arranged in a single line. Each dimension of the tensor has a concept called "stride," which represents how many elements we need to skip to move to the next element in that dimension. This helps us efficiently access and manipulate the data in the tensor. Additionally, the innermost dimension of a tensor is always "contiguous," which means the elements are stored sequentially without any gaps. This arrangement also aids in efficient processing of the tensor data. So, in summary, tensor ops involve performing operations on multidimensional arrays, and we use vectorized code to do these operations more efficiently. Tensors are represented as 1D arrays with strides for each dimension, and the innermost dimension is always stored sequentially without any gaps.
sva_ 3y ago
I think the bigger difficulty lies in how to store and process this stuff on a GPU/other accelerator. If the OP cares about implementation details about how an API like PyTorch is made, I think the MiniTorch 'book' is a pretty good intro: https://minitorch.github.io/ https://minitorch.github.io/
- lying4fun 3y ago
  The MiniTorch book seems amazing, thanks
riku_iki 3y ago
> the principles of vectorizing code. > 1D arrays I think accelerators mostly do computations over small matrices and not vectors, so it is "matricized" code, but I am not an expert in this area.

corsix 3y ago

From a hardware perspective, vector instructions operate on small 1D vectors, whereas tensor instructions operate on small 2D matrices. I say “instructions”, but it’s really only matrix multiply or matrix multiply and accumulate - most other instructions are fine staying as 1D.

nologic01 3y ago
If there is matrix multiply at hardware level its fair to have another name than vectorization. For example the dimensions and partitioning of large matrices to fit would be specific to that design and very different from rolling things out on 1D arrays
- bee_rider 3y ago
  Maybe they should call it a Matrix Multiplication Xcelerator just to mess with people.

amelius 3y ago

We really need to name tensors differently. "Array" would be a more fitting name. Tensors are really different mathematical objects with a far more rich structure than those used in the Deep Learning context. https://en.wikipedia.org/wiki/Tensor https://en.wikipedia.org/wiki/Tensor

itishappy 3y ago
Same with vectors! https://en.wikipedia.org/wiki/Vector_(mathematics_and_physics) https://en.wikipedia.org/wiki/Vector_(mathematics_and_physic...
- antognini 3y ago
  As a transplant to the ML world who migrated from physics I do find it funny that people in ML will insist that gradients are vectors. Once you get to general relativity your physics professors will beat into you the fact that gradients are not vectors. (They are one-forms.) But in the ML context it's a distinction without a difference, unless you're in a really specialized corner of the field like information geometry.
  - ReleaseCandidat 3y ago
    > Once you get to general relativity your physics professors will beat into you the fact that gradients are not vectors. In linear algebra you should learn this as soon as you hear about it. Is the english Wikipedia page correct in that you always use nabla for the gradient? https://en.wikipedia.org/wiki/Gradient#Generalizations https://en.wikipedia.org/wiki/Gradient#Generalizations Because `grad f = ∇f` holds in scalar fields f only (not in vector or tensor fields (tensors that aren't scalars)).
    - antognini 3y ago
      Even if you are taking the gradient of a scalar field, the resulting object is not a vector. The difference is that if you apply some transformation to your coordinate system a gradient transforms differently from a normal vector. In the ML context it generally doesn't matter because you don't do these transformations (and more generally there is no metric for your space). So you can just treat a gradient as an array of numbers. But in a physical context the distinction starts to matter, at least on a manifold with curvature. The one place it does matter in ML (that I'm aware of) is information geometry, where you try to do a transformation from the normal coordinate space to a more "natural" coordinate system that is based on the Fisher information, and this introduces curvature into model.

godelski 3y ago

This is mostly going to be the same. The main difference in tensorization is that your vectors are not contiguous. So you lose some speedup. How you typically optimize this is by flattening your tensor. Then your data is contiguious and all the same rules apply. This is overly simplified though. Things get different when we start talking about {S,M}I{S,M}D (page addresses SIMD) or GPUs. Parallel computing is a whole other game, and CUDA takes it to the next level (lots of people can write CUDA kernels, not a lot of people can write GOOD CUDA kernels (I'm definitely not a pro, but know some)). Parallelism adds another level of complexity due to being able to access different memory locations simultaneously. If you think concurrent optimization is magic, parallel optimization is wizardry (I still believe this is true)

quickthrower2 3y ago

PyTorch internals: http://blog.ezyang.com/2019/05/pytorch-internals/ http://blog.ezyang.com/2019/05/pytorch-internals/ A good basic thing to know is a tensor is a single dimensional array under the hood. At least as far as I know in my mental model - I have not yet looked at the code :-) Therefore "converting" a 2 * 5 * 10 tensor into a 2 * 50 one is an immediate operation, it doesn't have to scan. Just change some metadata.

jabl 3y ago
That's pretty much how every serious dense matrix (or more generally, multi-dimensional array) representation has been done since, well, forever. It's only C programmers writing "My First Matrix Program" that insist on allocating each row separately due to the lack of native multidimensional arrays in that language.
- quickthrower2 3y ago
  Yes! I was surprised how simple it was. Obvious once you know how. I never really considered multidimensional storage until looking into Machine Learning recently. My only brush with it is relational database tuning, but that is very different, as the number of "dimensions" are small and they are usually not homogenous.
- ReleaseCandidat 3y ago
  Fortran did it: A 2-dimensional array A will, in the object program, be stored sequentially in the order Ai,i, A2,i Am l , A| i2 , A2f2 Am, 2> , Am,„. Thus it is stored “columnwise”, with the first of its subscripts varying most rapidly, and the last varying least rapidly. The same is true of 3-dimensional arrays. 1 -dimensional arrays are of course simply stored sequentially. All arrays are stored backwards in storage; i.e. the above sequence is in the order of decreasing absolute location. https://ia800807.us.archive.org/0/items/history-of-fortran/Image072217151026_text.pdf https://ia800807.us.archive.org/0/items/history-of-fortran/I...