4 ms·

This is a fascinating take as far as I can follow it, which unfortunately is not that far. But does any of this formal logic help with deriving a function that

This is a fascinating take as far as I can follow it, which unfortunately is not that far. But does any of this formal logic help with deriving a function that describes a vector? Because it seems like the greatest inefficiencies and bottlenecks in big data analysis e.g. training networks still boils down to how to find functions that approximate output comparable to expected vectors, whether that's done by symbolic regression or layers of transformations. It would be "magic" if you could operate only on the vectors-as-functions without needing to distill or compress the relationships between their inputs and outputs somehow.

formally 3y ago
You should look into the pigeonhole principle.
- noduerme 3y ago
  Looked it up. How does that apply to this, or am I lacking the imagination to see it?
  - formally 3y ago
    If the data set is large enough then there is no way to represent it as a functional relationship between finite dimensional vector spaces. In fact, this problem already is visible in existing large neural networks because they can only work with data that conforms to the dimensional constraints of the input space. It's why image transformers trained on NxM images don't work on any other grid size.
    - noduerme 3y ago
      ah. I mean, the dataset can be infinitely large and still be covered perfectly by a function, if it was generated from a function. That's why absurd functions can be found that overfit for some subset of basic real world results. (Ask anyone who built a funky physics system for a videogame in the 90s). I think what's more interesting is the question of what that essential function is for a stop sign or a pedestrian, as opposed to the function for finding it in a 512 grid or something.
  - nine_k 3y ago
    I suspect the idea is that there can be many functions that fit a particular vector, because there are more functions than vectors, but I nay be wrong.
Sharlin 3y ago
Sure. You can Fourier transform a vector, drop some number of the least contributing terms (frequencies) and store only the remaining coefficients. Which is essentially the basic idea behind MP3 and JPEG compression. You're trading space for time, of course, as now to get an approximation of the original vector you have to apply reverse Fourier transform first.
chongli 3y ago
You may be thinking of a vector as a concrete collection of values, like a vector in R^3: [x y z]. This piece is about abstract vector spaces, their properties (vector addition, scalar multiplication, etc.) and specifically how functions meet the definition, giving you vector spaces of functions (function spaces). So the idea is that if you two functions, f and g, and a scalar b, then you can do stuff like: f + g = g + f b(f + g) = bf + bg The existence of (-f) so you have: f + (-f) = 0 Where 0 is the zero function (which also must exist for function spaces).
- noduerme 3y ago
  Isn't this just describing a tautology? I was reading here earlier today about the naming of the constant for light as c, and I had a question which I was too embarrassed to ask. It is this: In e=mc^2, what are the units, and if the units aren't defined and it's just a relationship, then why specify c^2? What's the point of squaring a constant, since it's just another constant? Not that I understand a damn thing about that equation. But the idea that two functions can be thought of operating additively on a vector - or space - seems... trivial. This is where I fear I am too stupid to understand the value of this.
  - chongli 3y ago
    But the idea that two functions can be thought of operating additively on a vector - or space - seems... trivial. No, the functions are not operating on a vector, the functions themselves are vectors, which means you can do things like define a linear transformation L and then you have things like this: L(af + bg) = aL(f) + bL(g) You may also define a norm on a function space (such as an L^P space [1]), ||f||_p, which maps functions to non-negative real numbers and obeys the properties of norms that we expect, such as the triangle inequality: ||f + g||_p <= ||f||_p + ||g||_p [1] https://en.wikipedia.org/wiki/Lp_space https://en.wikipedia.org/wiki/Lp_space
    - noduerme 3y ago
      Thanks. I'm trying to understand this. So take a function that returns a random shuffled deck of cards with 52! possible real number vector outputs. What is the generalized insight from treating this function itself as a vector space... is there a shortcut to monte carlo-ing a million random shuffles if you can "divide" that function by one which produces a straight flush?

formally 3y ago

You should look into the pigeonhole principle.

noduerme 3y ago
Looked it up. How does that apply to this, or am I lacking the imagination to see it?
- formally 3y ago
  If the data set is large enough then there is no way to represent it as a functional relationship between finite dimensional vector spaces. In fact, this problem already is visible in existing large neural networks because they can only work with data that conforms to the dimensional constraints of the input space. It's why image transformers trained on NxM images don't work on any other grid size.
  - noduerme 3y ago
    ah. I mean, the dataset can be infinitely large and still be covered perfectly by a function, if it was generated from a function. That's why absurd functions can be found that overfit for some subset of basic real world results. (Ask anyone who built a funky physics system for a videogame in the 90s). I think what's more interesting is the question of what that essential function is for a stop sign or a pedestrian, as opposed to the function for finding it in a 512 grid or something.
- nine_k 3y ago
  I suspect the idea is that there can be many functions that fit a particular vector, because there are more functions than vectors, but I nay be wrong.

Sharlin 3y ago

Sure. You can Fourier transform a vector, drop some number of the least contributing terms (frequencies) and store only the remaining coefficients. Which is essentially the basic idea behind MP3 and JPEG compression. You're trading space for time, of course, as now to get an approximation of the original vector you have to apply reverse Fourier transform first.

chongli 3y ago

You may be thinking of a vector as a concrete collection of values, like a vector in R^3: [x y z]. This piece is about abstract vector spaces, their properties (vector addition, scalar multiplication, etc.) and specifically how functions meet the definition, giving you vector spaces of functions (function spaces). So the idea is that if you two functions, f and g, and a scalar b, then you can do stuff like: f + g = g + f b(f + g) = bf + bg The existence of (-f) so you have: f + (-f) = 0 Where 0 is the zero function (which also must exist for function spaces).

noduerme 3y ago
Isn't this just describing a tautology? I was reading here earlier today about the naming of the constant for light as c, and I had a question which I was too embarrassed to ask. It is this: In e=mc^2, what are the units, and if the units aren't defined and it's just a relationship, then why specify c^2? What's the point of squaring a constant, since it's just another constant? Not that I understand a damn thing about that equation. But the idea that two functions can be thought of operating additively on a vector - or space - seems... trivial. This is where I fear I am too stupid to understand the value of this.
- chongli 3y ago
  But the idea that two functions can be thought of operating additively on a vector - or space - seems... trivial. No, the functions are not operating on a vector, the functions themselves are vectors, which means you can do things like define a linear transformation L and then you have things like this: L(af + bg) = aL(f) + bL(g) You may also define a norm on a function space (such as an L^P space [1]), ||f||_p, which maps functions to non-negative real numbers and obeys the properties of norms that we expect, such as the triangle inequality: ||f + g||_p <= ||f||_p + ||g||_p [1] https://en.wikipedia.org/wiki/Lp_space https://en.wikipedia.org/wiki/Lp_space
  - noduerme 3y ago
    Thanks. I'm trying to understand this. So take a function that returns a random shuffled deck of cards with 52! possible real number vector outputs. What is the generalized insight from treating this function itself as a vector space... is there a shortcut to monte carlo-ing a million random shuffles if you can "divide" that function by one which produces a straight flush?