4 ms·
Tensor Is the Might
- srean 2mo ago> A tensor is nothing but a flat array of numbers, plus some metadata telling you how to interpret those numbers as a multi-dimensional object. Erm... many would disagree. I think what he means is just a multidimensional array.
- zserge 2mo agoNot in a general mathematical sense, of course. Rather carthesian tensors. Seems like for most libraries it's `numpy.ndarray`: https://docs.tinygrad.org/tensor/ https://docs.tinygrad.org/tensor/ or https://docs.pytorch.org/docs/main/tensors.html https://docs.pytorch.org/docs/main/tensors.html
- srean 2mo agoTo add to the confusion, Mathematicians and Physicists often disagree on what that word means. A Physicist's tensor is often a mathematicians tensor field. An appropriate tensor can certainly be represented by a multidimensional array but not all multidimensional arrays are tensors. It depends on how that multidimensional array participates with others. For example, (lat, long) is not a 2d vector although it can be represented as a 2 dimensional array.
- srean 2mo agoIf one wants to add the capability to reason about shape and shape compatibility, Barry Jay's FiSh would be an interesting detour. https://web.archive.org/web/20111015133833/http://www-staff.it.uts.edu.au/~cbj/Publications/shapes.html https://web.archive.org/web/20111015133833/http://www-staff.... This was used in his shape aware language FiSh, for dealing with multidimensional arrays. Shape compatibilities were statically type checked, if I recall correctly. Shapes were also used to optimize the loops. [Programming in FISh] https://link.springer.com/article/10.1007/s100090050037 https://link.springer.com/article/10.1007/s100090050037 [Towards Dynamic Shaping] https://www.researchgate.net/publication/265975794_Towards_D https://www.researchgate.net/publication/265975794_Towards_D...
- zserge 2mo agoI wish we used something else but Python as the default ML language, so that some advanced type system would work for us to ensure tensor shapes and compatibility.
- srean 2mo agoHonestly, I quite like Python most of the time. The times when I don't, argghh type errors. The worst I have had in a while is when I was dealing with a sparse array data type and getting some processing done in a C function to speed things up. Infuriatingly enough, the index type used to change after initialising the array. It would check if Int32 could accomodate the size of the array. If it could it would downgrade to Int32 at runtime. My C counterpart had no clue. That was so not fun to debug because the errors were intermittent (depending on the fill factor of the sparse array)
- weiserwx 2mo ago[dead]
- churchsub1 2mo agoHere's a Python library for static type checking of tensor shapes: https://pypie.dev/ https://pypie.dev/
- srean 2mo agoThanks. Did not know about it. Looks a little FiShy. In a good way [0]. [0] https://link.springer.com/article/10.1007/s100090050037 https://link.springer.com/article/10.1007/s100090050037
- bssrdf 2mo agoI found [ggml] is really easy, educational and fun to use. It is written in C and the backbone for llama.cpp. [ggml]https://github.com/ggml-org/ggml https://github.com/ggml-org/ggml
- ux266478 2mo ago
- khalic 2mo agoWhy does does the diagram say tensors are 3D?
- zserge 2mo agoI guess I miscommunicated it. I meant I'd call everything a tensor but for some smaller dimensions we have specific names. Of course, 1D can still be a tensor and so is 6D or nD.
- reerdna 2mo agoCool, but I find rather than just shapes and indexes, tensors with labels are much easier to use and reason about. E.g.: { {user:bob, movie:"Heat"}:0.1, {user:alice, movie:"Frozen"}:0.9, {user:carol, movie:"Top Gun"}:0.3, } https://docs.vespa.ai/en/ranking/tensor-user-guide.html https://docs.vespa.ai/en/ranking/tensor-user-guide.html
- hasteg 2mo agoI just recently watched some (not all) of this video "coding a machine learning library in c from scratch" and seems like he's going through a similar process in this blog as this video. I would recommend watching the video to get an idea of what the fundamentals of a ML library look like. From someone who has recently been getting interested in actually writing ML code and trying to make sense of it myself (from the perspective of just a typical backend engineer) it was very interesting to see. Previously my experience with ML libs (PyTorch specific) was writing my own Mini-GPT and training it on a small dataset using my own GPU (5090). Cool to see the behind the scenes and took away some o the handwaveyness... https://www.youtube.com/watch?v=hL_n_GljC0I https://www.youtube.com/watch?v=hL_n_GljC0I
- terminalbraid 2mo ago> A tensor is nothing but a flat array of numbers I'm so very, very tired of tech coopting rigorous mathematical terms.
- mamcx 2mo agoI read it as "tensors in programming*. This also happens to "numbers", "integers" and such, because in the computer universe don't exist out math, only a subset(?) variation(?) of it
- terminalbraid 2mo agoYes, this is what I'm complaining about. > coopt: to appropriate as one's own; preempt. The term existed for some time before programming if that's your source of confusion. Now it is a twisted and ambiguous version of itself.
- nathan_compton 2mo agoI know there are different contexts, but a tensor is not a collection of numbers, in a mathematical sense. A vector is not a list of numbers. Such collections of numbers are representations of objects with very specific kinds of properties under coordinate transformations. I think it genuinely damages people's ability to digest the mathematics to tell them first and foremost that these objects are collections of numbers.
- ksd482 2mo ago> A tensor is nothing but a flat array of numbers, plus some metadata telling you how to interpret those numbers as a multi-dimensional object. Yikes! No. I mean even for the intents and purposes of using this definition in ML, this might not be right. I am trying not to be pedantic, so I will not go with the official/mathematical definition of a tensor as that could be incredibly confusing (look it up!!!). But a tensor is a LOT more than that. Essentially it's a multilinear map that transforms a set of basis vectors in a certain way, and is coordinate agnostic. This is not even half its definition so you can see how much the author left out. Having said that, this is still a good way to start getting intuition into it and I urge the author to continue refining the definition as he/she learns more. Disclaimer: MS in Math with concentration of GR. EDIT: Also tensor aren't simply "flat" array of numbers. They are multidimensional. A grounded example, a rank 3 tensor is a collection of 2d matrices. Think of it as a bunch of 2d matrices stacked on top of each other. You need 3 indices to keep track of numbers --- sure in a programming language, it can be represented as a 1d array as well with 0s filling up empty spaces, but you get the idea.
- prideout 2mo agoThey were speaking from an implementation perspective.