3 ms·

I haven't read the article yet, but I've known that functions are (infinite) vectors for some years. However, there's something that has been bothering me: mos

I haven't read the article yet, but I've known that functions are (infinite) vectors for some years. However, there's something that has been bothering me: most of my understanding of linear algebra comes from 2D and 3D spaces, and then in different context of machine learning, datasets that have from tens to even millions of dimensions. In the former, geometric context, the connection between the dimensions are clear: they are orthogonal, but conceptually exactly similar. They are just a 90 degree rotation away from each other. On the other hand, in ML datasets some dimensions are conceptually very similar, and some are totally different. Some are correlated (nearby pixels of an image), some are not, but represent the same unit of quality, and some represent totally different, unrelated things. And as we go toward the mid-layer representations, it becomes very unclear and fuzzy what they represent. In the case of functions, there's usually a clear connection between the dimensions: they are of the same unit (the domain and the image (the outputs and inputs) of the function are sets, and those tend to be made of similar-ish – or same type of – things, at least in well-behaved math). And there's often a similarity metric between the elements of the sets. The 2D/3D linear algebra that I know doesn't bother with the "connectedness" of the input dimensions; it only cares the connections from the inputs to the outputs. But surely there is a lot of interesting math that is concerned with the connectedness of the input and output spaces themselves, in context of there still existing a mapping between the input and output. What is that field of math called? What are the interesting results, theorems and such? I love learning more so I'm kind of just looking for some pointers/keywords.

znkr 3y ago
I am not sure I fully understand your question, but I’ll try to answer the question I understood. Very generally, every field of math I know concerns itself with mappings and what they do (e.g. matrixes are mappings, as are metrics and norms). The differences between the different areas of math (oversimplified) is that they concern themselves with different kinds of mappings. The area I specialized in was partial differential equations, here the mappings of interest are solutions to partial differential equations (usually generalized functions). One of the important questions to ask is of these functions are continuous, differentiable, or any other of the many variations. All of these properties are defined over how a function maps an input to an output. BTW: There’s usually an infinite number of solutions to a differential equation, and IIRC (it’s been a while) those solutions too form a vector space.
nonameiguess 3y ago
I think what you're getting at is two things. You have a dataset. This consists of some set of observations, each of which contains features that have been observed. These may be things like education level, age, income, favorite color, gender, and height. Three of these variables are continuous. One is ordinal. One is nominal. One is binary. Can you do any kind of meaningful matrix operations with this data? Traditionally, in regression analysis, you'd use dummy variables for the nominal/binary data, turning all of them into binary {0, 1}. This is GF(2), a bonified vector space. Education level you've got some choices. You can also encode these as binary, potentially losing some information given they're ordered. But if you choose to map them to integers or real numbers, what is the justification for the distance function you're defining? Is MD > PhD? How much greater is PhD than BA than high school diploma? If you stick to one-hot encoding, there's justification in performing regression analysis still, because you end up in a case where most of the values are 0 and drop out and you get multiple models, one for each case, so the matrix operations being performed to generate the model only operate on the remaining real-valued vectors, and the one case that is 1 becomes the constant term. This is just basic ANCOVA. You have another issue, though, right? Are "height" and "income" and "age" really from the same vector space? Obviously not. Some modelers choose to ignore this. Some will assume they're all normally distributed and tranform the individual values to a Z-score or something and then they're all from the same vector space. But this still may be dubious. For one thing, they're not actually normally distributed. Height and age have a strictly zero lower-bound. They have some upper-bound, whatever it may be. Income is severely right-skewed. But they're probably all at least approximately normal over some restricted range of the most common values. We're starting to see the issue. Modeling in this way might work reasonably well within some restricted range of common values, but extreme cases are left out. We need some other techniques that don't rely on matrix math, because once we have to admit all of our elements from not from the same field, we admit we don't really have a matrix, even if a computer will gladly let us pretend anything we can encode as a number is actually a number. I think your question is how can we encode data such that it all ends up in the same vector space? It's easy if your data is imagery. Color channel intensity of each pixel is all real-valued, possibly within different ranges, but easily normed. If all you have is text, representing the corpus by an incidence matrix puts all of your elements in GF(2). Plenty of other word embeddings are well-justified. But when you start looking at longitudinal data from epidemology, the shit they do in econometrics, it starts to get less math and more black magic. By the time you're a data scientist at Netflix, I'm sure they have reason to believe whatever they're doing works, but it may be hard to reason about why it works. I don't really know if there any field of mathematics studying ways of encoding data such that doing things like adding and multipying height by favorite color actually means anything, but it's an interesting question.
smokel 3y ago
This is bothering me a little as well. Perhaps someone more knowledgeable could guide us into the light. As a concrete example, I am toying with a reinforcement learning to solve the game of Yahtzee. When trying to formulate the state space, I have to lump in together the states of the dice (1..6), the current state of the score card (13 boolean values, and 13 integer scores in the range 0..50), and the current turn (first attempt, second attempt). This could be formulated as a 5+13+13+2 = 34 dimensional vector, but I might as well use one-hot encoding for some aspects, and even go all the way up to 56+13+56+13+30+30+1+1+1+1+30 = 180 dimensions. Which formulation would be the most "natural"? Then again, what may seem (un)natural to a human, might be the other way around for a neural network. (And, yes, I am aware of papers out there describing how to solve Yahtzee with reinforcement learning, but I'm still at a stage where avoiding those seems to increase the amount of fun.)
- Someone 3y ago
  In general, you want to do one-hot encoding whenever your number doesn’t really represent a quantity. For example, for the current state of a rolled die in Yahtzee, ‘2’ mostly (see below) isn’t smaller than ‘3’, isn’t between ‘1’ and ‘3’, and isn’t closer to ‘3’ than it is to ‘4’. So, ‘2’ isn’t a quantity there; it’s just one of the markings on the dice. About that ‘mostly’: the straights require the ability to order the values on the dice, but even then, it’s just an order, not anything about them being equidistant, or about (2,6) showing more eyes than (3,4) (again, there’s an exception there, with the ‘Chance’ score)
zmgsabst 3y ago
> But surely there is a lot of interesting math that is concerned with the connectedness of the input and output spaces themselves, in context of there still existing a mapping between the input and output. Topology. Topology studies neighborhoods in your space and how they relate to things like functions, limits, etc. Since you brought up images: https://en.wikipedia.org/wiki/Digital_topology https://en.wikipedia.org/wiki/Digital_topology And data science: https://en.wikipedia.org/wiki/Topological_data_analysis https://en.wikipedia.org/wiki/Topological_data_analysis
- GolDDranks 3y ago
  Yeah, topology seems to have a lot to do with it! However, I hesitated to jump into that conclusion, because my impression of topology is that it's concerned of "whether" something is connected (and often about the directedness of surfaces/spaces), because topology allows "stretchiness" of its study matter. The connections between things are then kind of binary things - either something is connected or not. However, in the world of "connected dimensions", there seems to be degrees of connectedness. Linear correlation being from the simplest end, but more generally, shared entropy. So it feels like just "topology" doesn't quite catch the field and its pecularities. Thanks for the links!! Much appreciated.

znkr 3y ago

I am not sure I fully understand your question, but I’ll try to answer the question I understood. Very generally, every field of math I know concerns itself with mappings and what they do (e.g. matrixes are mappings, as are metrics and norms). The differences between the different areas of math (oversimplified) is that they concern themselves with different kinds of mappings. The area I specialized in was partial differential equations, here the mappings of interest are solutions to partial differential equations (usually generalized functions). One of the important questions to ask is of these functions are continuous, differentiable, or any other of the many variations. All of these properties are defined over how a function maps an input to an output. BTW: There’s usually an infinite number of solutions to a differential equation, and IIRC (it’s been a while) those solutions too form a vector space.

nonameiguess 3y ago

I think what you're getting at is two things. You have a dataset. This consists of some set of observations, each of which contains features that have been observed. These may be things like education level, age, income, favorite color, gender, and height. Three of these variables are continuous. One is ordinal. One is nominal. One is binary. Can you do any kind of meaningful matrix operations with this data? Traditionally, in regression analysis, you'd use dummy variables for the nominal/binary data, turning all of them into binary {0, 1}. This is GF(2), a bonified vector space. Education level you've got some choices. You can also encode these as binary, potentially losing some information given they're ordered. But if you choose to map them to integers or real numbers, what is the justification for the distance function you're defining? Is MD > PhD? How much greater is PhD than BA than high school diploma? If you stick to one-hot encoding, there's justification in performing regression analysis still, because you end up in a case where most of the values are 0 and drop out and you get multiple models, one for each case, so the matrix operations being performed to generate the model only operate on the remaining real-valued vectors, and the one case that is 1 becomes the constant term. This is just basic ANCOVA. You have another issue, though, right? Are "height" and "income" and "age" really from the same vector space? Obviously not. Some modelers choose to ignore this. Some will assume they're all normally distributed and tranform the individual values to a Z-score or something and then they're all from the same vector space. But this still may be dubious. For one thing, they're not actually normally distributed. Height and age have a strictly zero lower-bound. They have some upper-bound, whatever it may be. Income is severely right-skewed. But they're probably all at least approximately normal over some restricted range of the most common values. We're starting to see the issue. Modeling in this way might work reasonably well within some restricted range of common values, but extreme cases are left out. We need some other techniques that don't rely on matrix math, because once we have to admit all of our elements from not from the same field, we admit we don't really have a matrix, even if a computer will gladly let us pretend anything we can encode as a number is actually a number. I think your question is how can we encode data such that it all ends up in the same vector space? It's easy if your data is imagery. Color channel intensity of each pixel is all real-valued, possibly within different ranges, but easily normed. If all you have is text, representing the corpus by an incidence matrix puts all of your elements in GF(2). Plenty of other word embeddings are well-justified. But when you start looking at longitudinal data from epidemology, the shit they do in econometrics, it starts to get less math and more black magic. By the time you're a data scientist at Netflix, I'm sure they have reason to believe whatever they're doing works, but it may be hard to reason about why it works. I don't really know if there any field of mathematics studying ways of encoding data such that doing things like adding and multipying height by favorite color actually means anything, but it's an interesting question.

smokel 3y ago

This is bothering me a little as well. Perhaps someone more knowledgeable could guide us into the light. As a concrete example, I am toying with a reinforcement learning to solve the game of Yahtzee. When trying to formulate the state space, I have to lump in together the states of the dice (1..6), the current state of the score card (13 boolean values, and 13 integer scores in the range 0..50), and the current turn (first attempt, second attempt). This could be formulated as a 5+13+13+2 = 34 dimensional vector, but I might as well use one-hot encoding for some aspects, and even go all the way up to 56+13+56+13+30+30+1+1+1+1+30 = 180 dimensions. Which formulation would be the most "natural"? Then again, what may seem (un)natural to a human, might be the other way around for a neural network. (And, yes, I am aware of papers out there describing how to solve Yahtzee with reinforcement learning, but I'm still at a stage where avoiding those seems to increase the amount of fun.)

Someone 3y ago
In general, you want to do one-hot encoding whenever your number doesn’t really represent a quantity. For example, for the current state of a rolled die in Yahtzee, ‘2’ mostly (see below) isn’t smaller than ‘3’, isn’t between ‘1’ and ‘3’, and isn’t closer to ‘3’ than it is to ‘4’. So, ‘2’ isn’t a quantity there; it’s just one of the markings on the dice. About that ‘mostly’: the straights require the ability to order the values on the dice, but even then, it’s just an order, not anything about them being equidistant, or about (2,6) showing more eyes than (3,4) (again, there’s an exception there, with the ‘Chance’ score)

zmgsabst 3y ago

> But surely there is a lot of interesting math that is concerned with the connectedness of the input and output spaces themselves, in context of there still existing a mapping between the input and output. Topology. Topology studies neighborhoods in your space and how they relate to things like functions, limits, etc. Since you brought up images: https://en.wikipedia.org/wiki/Digital_topology https://en.wikipedia.org/wiki/Digital_topology And data science: https://en.wikipedia.org/wiki/Topological_data_analysis https://en.wikipedia.org/wiki/Topological_data_analysis

GolDDranks 3y ago
Yeah, topology seems to have a lot to do with it! However, I hesitated to jump into that conclusion, because my impression of topology is that it's concerned of "whether" something is connected (and often about the directedness of surfaces/spaces), because topology allows "stretchiness" of its study matter. The connections between things are then kind of binary things - either something is connected or not. However, in the world of "connected dimensions", there seems to be degrees of connectedness. Linear correlation being from the simplest end, but more generally, shared entropy. So it feels like just "topology" doesn't quite catch the field and its pecularities. Thanks for the links!! Much appreciated.