26 ms·
Functions are vectors
- johnthescott 3y agoimmutable functions are also relations. but i digress.
- ssivark 3y agoMeditating on the converse statement is also an interesting thought exercise: A vector is (just) a (cached) function (evaluation).
- jesuslop 3y agoyep, because of this one can do O(1) sigmoidals in float16 nnets with a 64K-word table.
- User23 3y agoCaching and evaluation doesn’t make any kind of sense for mathematical functions. They’re just mappings.
- slavapestov 3y agoYeah, but it’s true in a sense because an element of a finite-dimensional vector space can be thought of as a function from a finite set into a field.
- xigoi 3y agoOnly if the space has a canonical basis.
- harerazer 3y agoIndeed, many linear algebra textbooks define a tuple of real numbers as a function f: {1,...,n} -> R.
- matteoraso 3y agoThis touches on the actual definition of a function, which is a mapping between sets where every element of the first set maps to exactly one element of the second set. The problem with using vectors is that vectors aren't as general as sets, so there's functions that can't be expressed using vectors. For example, vectors can't be used to handle undefined values or non-numeric elements.
- adammarples 3y agoThat's not the definition of a function, what you're describing would be called a bijective function. A simple function that is not bijective and maps to two distinct values would be sqrt(x)
- tlb 3y ago"In mathematics, a function from a set X to a set Y assigns to each element of X exactly one element of Y." [0] [0] https://en.wikipedia.org/wiki/Function_(mathematics) https://en.wikipedia.org/wiki/Function_(mathematics)
- eternityforest 3y agoWhich doesn't mean each element of Y has exactly one matching counterpart in X, two elements in X can share one in Y
- ElFitz 3y agoe.g. hash collisions.
- slavik81 3y agoYou may be confused because a bijective function must be one-to-one, while in general, a function may be many-to-one. However, the standard definition of a function excludes one-to-many mappings. > A function f from a set X to a set Y, denoted f: X -> Y, is a relation from X, the domain, to Y, the co-domain, that satisfies two properties: (1) every element in X is related to some element in Y, and (2) no element in X is related to more than one element in Y. Susanna S. Epp. 2010. Discrete Mathematics with Applications (4th. ed.). p.384
- jesuslop 3y agoI always liked this viewpoint a lot. I'm enjoying with abandon some dusty lectures that Vito Volterra gave in Madrid on differential and integrodifferential equations, while helping also to create Functional Analysis (a Functional being the analogue of a dual vector). He is constantly exploiting this analogy method from finite variable constructions to infinite, also uncountable variables. Even up to showing some embarrassment of being too repetitive with the idea! People in teaching should join and take a peek. https://searchworks.stanford.edu/view/526111 https://searchworks.stanford.edu/view/526111
- selimthegrim 3y agoGivental uses this viewpoint too in his differential equations class notes. It may soothe students who notice a disjunction between linear independence of functions and that of vectors.
- User23 3y agoThere’s probably a PhD or two waiting to be earned exploring the relation between Dijkstra/Scholten boolean structures and vector spaces.
- gatane 3y agoThat or Lagrangian mechanics and Dijkstra
- noduerme 3y agoThis is a fascinating take as far as I can follow it, which unfortunately is not that far. But does any of this formal logic help with deriving a function that describes a vector? Because it seems like the greatest inefficiencies and bottlenecks in big data analysis e.g. training networks still boils down to how to find functions that approximate output comparable to expected vectors, whether that's done by symbolic regression or layers of transformations. It would be "magic" if you could operate only on the vectors-as-functions without needing to distill or compress the relationships between their inputs and outputs somehow.
- formally 3y agoYou should look into the pigeonhole principle.
- noduerme 3y agoLooked it up. How does that apply to this, or am I lacking the imagination to see it?
- formally 3y agoIf the data set is large enough then there is no way to represent it as a functional relationship between finite dimensional vector spaces. In fact, this problem already is visible in existing large neural networks because they can only work with data that conforms to the dimensional constraints of the input space. It's why image transformers trained on NxM images don't work on any other grid size.
- noduerme 3y agoah. I mean, the dataset can be infinitely large and still be covered perfectly by a function, if it was generated from a function. That's why absurd functions can be found that overfit for some subset of basic real world results. (Ask anyone who built a funky physics system for a videogame in the 90s). I think what's more interesting is the question of what that essential function is for a stop sign or a pedestrian, as opposed to the function for finding it in a 512 grid or something.
- thorel 3y agoThe realization that functions can be treated as elements in an abstract vector space (with infinitely many dimensions) is a turning point in the history of mathematics that led to the emergence of the sub-field known as functional analysis. The significance of this paradigm shift is that it allowed mathematicians to apply some of the geometric intuition developed from the study of finite-dimensional spaces (such as the 3D Euclidean space) to difficult questions involving functions, such as the existence of solutions to certain differential equations. The history of this change of perspective is absolutely fascinating and can be traced back to the end of the 19th century and beginning of the 20th century. At the time, work on axiomatic foundations of mathematics was driving a systematization of the study of mathematical objects by capturing their structure with a concise list of axioms. This is for example how the concept of an abstract vector space was born, encompassing not only Euclidean spaces but also infinite-dimensional spaces of functions. An early reference already demonstrating this change of perspective, albeit in a primitive form, is a memoir by Vito Volterra from 1889 [1]. The PhD thesis of Maurice Fréchet from 1906 [2] is arguably the work that was most influential in crystalizing the new paradigm and presenting it in a modern form that served as a key reference for the first half of the 19th century. Of course, these are only two among a multitude of works around that time. Looking at later developments in the 19th century, it is hard not to also mention the book by Stefan Banach from 1932 [3]. [1] https://projecteuclid.org/journals/acta-mathematica/volume-12/issue-none/Sur-une-gen%c3%a9ralisation-de-la-th%c3%a9orie-des-fonctions-dune-variable/10.1007/BF02592183.full https://projecteuclid.org/journals/acta-mathematica/volume-1... [2] https://zenodo.org/record/1428464/files/article.pdf https://zenodo.org/record/1428464/files/article.pdf [3] http://kielich.amu.edu.pl/Stefan_Banach/pdf/teoria-operacji-fr/banach-teorie-des-operations-lineaires.pdf http://kielich.amu.edu.pl/Stefan_Banach/pdf/teoria-operacji-...
- t-vi 3y agoNot saying that the vector space bit isn't neat, but it's called functional analysis because you can take limits of various forms and define (semi-) continuity, have completions of spaces, and all that has nice properties. So to me, a crucial thing is that these vector spaces are indeed topological.
- hayasaki 3y ago
- formally 3y agoThis is only true if the codomain has the relevant structure for vector operations. Functions are more general than vectors.
- civilized 3y agoIt's easier to see functions as vectors if you first see vectors as functions.
- opportune 3y agoI’d probably have titled this something including either the term “linear” or “functional analysis”. Because submitted here, we will first interpret “functions” in the context of a function in computer programming, where the statement is more provocative and thus clickbaity. The problem is many real world functions and problems are nonlinear. But they may have linear components. For example, a dog can be recognized by its outline and the texture of the fur, pattern of the face, etc within that outline. Deep neural nets solve this by composing vector operations, hence the universal approximation theorem and existence of NNs that can recognize dogs (though I could have picked a better example as dog recognition is not continuous I think). In the context of computer programming, it is not really a helpful statement to say that functions are vectors. But because of the universal approximation theorem and its relatives you could say that “functions are (can be approximated as) compositions of vector operations”
- HeavyStorm 3y agoYep, went there to learn how to use vectors to replace methods...
- eigenket 3y agoFor some reason computer programmers took the mathematical definition of "functions", used it for their things which are emphatically not mathematical functions, and are now complaining that they get confused by people talking about the old math definition.
- xigoi 3y agoSame with “vector”.
- oasisaimlessly 3y agoNothing in this article assumed that the functions in question were linear and/or being approximated.
- fanpu 3y ago
- lanstin 3y agoI have never seen these index functions used as a transfinite basis for a vector space. And it seems like the function is not a limit point of finite sequences of basis functions, but some weird transfinite sum with mostly zero entries? Clearly there is no Fourier transformation possible on all functions? I think diagonalization methods would be easy to disprove any useful result. Even Hilbert spaces are usually just indexed by the ints. And such a basis gives you zero continuity or differential conditions. All the functional analysis I have seen uses some continuity conditions and has some countable basis. Other than that, it is a very useful perspective on functions, and kind of the required start to understanding quantum mechanics formalism.
- xigoi 3y agoMathematicians HATE this weird trick! Learn how he constructed a basis for the vector space of real functions without the axiom of choice.
- movpasd 3y agoThis is my major gripe with the article. While useful for intuition, that "..." after the sum is mathematically meaningless. This is also a common issue with quantum mechanics as taught at an introductory level. But it seems the article, similarly to intro QM courses, is more about motivating functional analysis concepts, which is useful as exposition even if not rigorous.
- rrobukef 3y agoThe article is some summary of a book with chapters. At some point they limit the space to the subspace of functions periodic over (b-a) and change the basis (with proof) from dirac delta to sines of frequency 2pi*k/(b-a) [with k in N]. In this subspace all functions have Fourier transformations.
- lanstin 3y agoBig changes there.
- constantcrying 3y ago
- ubj 3y agoI wish I could upvote this twice. This is the best basic introduction to concepts in functional analysis that I've seen. Another great overview that goes deeper into the math is [1]. Another fantastic application that the website doesn't mention is the composition / Koopman operator. In control theory (e.g. autonomous drones, cars, robot arms, etc.), most real-world systems are described by nonlinear dynamics which are very difficult to work with (e.g. safety/stability guarantees, optimizing over forward horizons using NMPC, state estimation, etc.) The Koopman operator however gives a globally relevant linear approximation of non-linear systems. In other words, you can treat a nonlinear system as a linear system with fairly high accuracy. This greatly simplifies control and estimation from a computational perspective. You can also learn these linearizations from data. Steve Brunton has some good materials on Koopman theory [2][3], and there are some great applications to control of systems such as soft robots [4]. [1]: https://arxiv.org/abs/1904.02539 https://arxiv.org/abs/1904.02539 [2]: https://youtube.com/playlist?list=PLMrJAkhIeNNSVXUvppZTYNHKQUD-oWys9 https://youtube.com/playlist?list=PLMrJAkhIeNNSVXUvppZTYNHKQ... [3]: https://arxiv.org/abs/2102.12086 https://arxiv.org/abs/2102.12086 [4]: https://arxiv.org/abs/1902.02827 https://arxiv.org/abs/1902.02827
- gauddasa 3y agoI upvoted this post and your comment, that is equivalent to upvoting the post twice.
- sheepscreek 3y agoMy first thought was about its striking conceptual similarity to Fourier transforms. Truly fascinating. Going to explore this a bit more. Thanks for sharing.
- cmehdy 3y agoI am so thankful for Steve Brunton's content. Had I been able to access his content a decade ago during my M.Sc, I would have probably pursued a PhD out of the passion and quality he brings to this domain - instead I just felt done with academia, strugglign to find grants, reading yet again terse books all alone, and just moved on. Solid YouTube educators are creating immense future opportunities and we'll all benefit from that. Control theory connects the dots between all sorts of things and can be a joy for those of us who love seeing patterns and structures everywhere (iirc Steve has a recent video about control theory for social models also).
- tantalor 3y ago> Now, a vector really is just an arbitrary function Not really grokking this. Seems to come out of nowhere.
- xcv123 3y agoIn mathematics, a function is a mapping from input values to output values. A vector is a mapping from a set of integers to a value at the specified index, therefore it is a function. e.g. float vector[4]{1.0, 5.0, 3.0, 2.5}; vector[0] == 1.0; vector[1] == 5.0; vector[2] == 3.0; vector[3] == 2.5;
- drtgh 3y agotry to look at it as a kind of format were you are storing equations (by the moment) You process the operations between such equations -sum, inner, outer- by "unpacking" them into Matrix equations format (populating the Matrix with the vector values), doing the operations, and returning back to the vector format. So in essence you are working with Matrix equations, but with a more compact format as vector. For being able to work with such equations' compact format, its needed to follow some rules restricted to the purpose, case contrary, as in reality they are vectors per se, the geometries could mix dimensions were the equations' proprieties are different.
- bionhoward 3y agoWhat makes the author say that functions are infinite dimensional? Seems like the space of functions might be infinite dimensional but one function is usually not. “AND” is 0001 for 00 01 10 11. 2^4=16 binary Boolean functions, in ternary it blows up, but it’s not infinite.
- Bjartr 3y agoI think I understand it, let's see if I can explain it. Hopefully I'll say something useful. Take a vector for normal space, [x, y, z]. We say each component of this vector is one dimension, so this one is 3D, and each of its three components can vary. Two such vectors are different if one or more components differ between them. Treating a function as a vector means treating each distinct possible input to the function as a distinct component. For example, consider the integer function f(x) = x^2. This can be represented as the vector [..., 16, 9, 4, 0, 4, 9, 16, ...] Where the complete vector has as many components as integers. Since there's infinitely many integers, there's infinite components, so instead of 3D like the three component vector above, this vector is ∞D. Any single function is representable in this way, so each distinct function has its own unique infinitely long vector. So each different function is a different "point" in an infinite dimensional vector space.
- dreamcompiler 3y agoA function on the reals maps any real to another [or maybe the same] real. Given some systematic way to order the inputs, you could describe the function as a vector lookup table with an infinite number of elements -- one output for each possible input. That vector describes a single point in an infinite-dimensional space. Thus every function from R to R is a single point in an infinite-dimensional space. Now you can use linear algebra to move these points around in the infinite-dimensional space, measure how far two points [functions] are from each other, etc. That's functional analysis. The linear operators that do this moving around and measuring are called functionals to indicate that they take functions as arguments. (Like higher-order functions in a programming language.) "Functional Analysis" is thus "The analysis of the objects known as functionals". Differentiation is an example of a functional.
- rrobukef 3y ago
- capn_duck 3y agodoesn't is suffice to say that functions meet all the prerequisites of a vector space? f + g = g + f f + (g + h) = (f + g) + h f + -f = 0 (a * b) * f = a * bf a (f + g) = af + ag (a + b) * f = af + bf
- keithalewis 3y ago"Given these definitions, we can now prove all necessary vector space axioms." And that is just the first howler. This person never bothered to learn the subject they are expounding on.
- eigenket 3y agoThe article is by no means perfect, but that "howler" sees completely fine to me. If you want to prove something is a vector space the standard way would be to prove that all the vector space axioms hold for it.
- keithalewis 3y agoYes. Prove the axioms hold for a space of functions. Very sloppily written article.
- eigenket 3y agoI wouldn't describe that as sloppy. If it was written in isolation it would be a bit weird but given the context I think its completely fine.
- GolDDranks 3y agoI haven't read the article yet, but I've known that functions are (infinite) vectors for some years. However, there's something that has been bothering me: most of my understanding of linear algebra comes from 2D and 3D spaces, and then in different context of machine learning, datasets that have from tens to even millions of dimensions. In the former, geometric context, the connection between the dimensions are clear: they are orthogonal, but conceptually exactly similar. They are just a 90 degree rotation away from each other. On the other hand, in ML datasets some dimensions are conceptually very similar, and some are totally different. Some are correlated (nearby pixels of an image), some are not, but represent the same unit of quality, and some represent totally different, unrelated things. And as we go toward the mid-layer representations, it becomes very unclear and fuzzy what they represent. In the case of functions, there's usually a clear connection between the dimensions: they are of the same unit (the domain and the image (the outputs and inputs) of the function are sets, and those tend to be made of similar-ish – or same type of – things, at least in well-behaved math). And there's often a similarity metric between the elements of the sets. The 2D/3D linear algebra that I know doesn't bother with the "connectedness" of the input dimensions; it only cares the connections from the inputs to the outputs. But surely there is a lot of interesting math that is concerned with the connectedness of the input and output spaces themselves, in context of there still existing a mapping between the input and output. What is that field of math called? What are the interesting results, theorems and such? I love learning more so I'm kind of just looking for some pointers/keywords.
- znkr 3y agoI am not sure I fully understand your question, but I’ll try to answer the question I understood. Very generally, every field of math I know concerns itself with mappings and what they do (e.g. matrixes are mappings, as are metrics and norms). The differences between the different areas of math (oversimplified) is that they concern themselves with different kinds of mappings. The area I specialized in was partial differential equations, here the mappings of interest are solutions to partial differential equations (usually generalized functions). One of the important questions to ask is of these functions are continuous, differentiable, or any other of the many variations. All of these properties are defined over how a function maps an input to an output. BTW: There’s usually an infinite number of solutions to a differential equation, and IIRC (it’s been a while) those solutions too form a vector space.
- LudwigNagasena 3y agoI think the article has it backwards and provides bad intuition. It is not input that makes functions form a vector space, it is the output. Functions from any set X to a field F can form a vector space, even if X is unordered.
- atemerev 3y agoAnd with this, you basically learned all pre-QFT quantum mechanics.
- javajosh 3y agoThis looks absolutely fantastic and I want to go over it in more detail later. You'll go over most of this in a typical physics degree. But like a good movie or book, the concepts are interesting enough to go over more than once. I will say that as a programmer some of these techniques look a lot like hacks. You start off with a perfectly reasonable integer index. And then you realize you can generalize the index, effectively cramming more information into the index than was originally intended. The really shocking thing is that these stupid, abusive ideas seem to always lead to something really insightful and useful down the road. It's a little bit magical.
- fritzo 3y agoPlug for the Funsor library, written by Eli Bingham and me for use in the Pyro and NumPyro probabilistic programming languages. We tried to take the "functions are tensors" perspective and make a numpy-like library for functions, aimed mostly at the log-density functions of probability distributions. Paper: "Functional Tensors for Probabilistic Programming" (2019) https://arxiv.org/abs/1910.10775 https://arxiv.org/abs/1910.10775 Code: https://github.com/pyro-ppl/funsor https://github.com/pyro-ppl/funsor