7 ms·

I've always found the bipartite graph conceptualization of matrix multiplication the most intuitive, and especially so if you're familiar with neural networks:

I've always found the bipartite graph conceptualization of matrix multiplication the most intuitive, and especially so if you're familiar with neural networks: https://www.math3ma.com/blog/matrices-probability-graphs https://www.math3ma.com/blog/matrices-probability-graphs

crdrost 6y ago
That is genuinely a really fun way to look at it, thank you for linking this! Especially this fits nicely with Markov matrices where you have N input nodes and N output nodes and the sum of all of the probabilities coming out of one of the nodes needs to equal 1. What I might find a little more difficult to teach to people through this lens is the phenomenon of eigenvectors, but I suppose that's to be expected—nothing will work well for all purposes.
- layoutIfNeeded 6y ago
  >What I might find a little more difficult to teach to people through this lens is the phenomenon of eigenvectors Why? Eigenvectors are simply inputs to the network where the output keeps its shape, that is, at most it gets rescaled, as if you had applied a uniform gain to the components, but otherwise it will be the same as the input.
  - tonyarkles 6y ago
    Oh. Oh my. I have studied linear algebra theory some. I have used it a ton. I have used eigenvectors to solve problems. I have never grokked them. This description changed that. Thank you!
  - crdrost 6y ago
    So for example let me ask for your knee-jerk opinions based on this idea: - does a matrix have the same left-eigenvectors as its right-eigenvectors? - what is the relationship between the left-eigenvalues and right-eigenvalues? - is there always an eigenvector? when is there a complete set? how do you generalize your notion of eigenvectors so that matrices always have a complete set of them? I am not sure any of these are intuitive here.
    - pmiller2 6y ago
      If A is a square matrix, then a left eigenvector v is a vector such that vA = \lambda_v v for some \lambda_v. Likewise, if u is a right eigenvector of A, Au = \lambda_u u. Notice that u and v cannot be equal, because they are not the same shape. However, if v is a right eigenvector with eigenvalue \lambda, then v^T is a left eigenvector with eigenvalue \lambda, as well. More or less what this means is that we tend to just ignore left eigenvalues and only use the right eigenvalues, because the math is exactly the same up to a transpose operation. A matrix does not necessarily have nontrivial eigenvectors. Think about the 0 matrix here. But, if A is nonzero, then it must have at least one nonzero eigenvalue, hence one nontrivial eigenvector. This is because a nonzero matrix must have at least one nonzero row and column; however, if you construct the other rows (columns) to be multiples of the first column, you end up with a matrix with only one nontrivial eigenvalue. This also illustrates the conditions necessary for an nxn matrix A to have n linearly independent eigenvectors: the rows of A must be linearly independent. All of this is covered in a decent undergrad linear algebra course. I would suggest either finding a video course, or getting a good book and working through it, if you want to understand these things better.
      - philip-b 6y ago
        >But, if A is nonzero, then it must have at least one nonzero eigenvalue, hence one nontrivial eigenvector. How about this matrix? [[0, -1], [[1, 0]] >However, if v is a right eigenvector with eigenvalue \lambda, then v^T is a left eigenvector with eigenvalue \lambda, as well. A snippet of code producing a counterexample: import numpy as np import scipy.linalg as spla A = np.random.randn(3, 3) right_eigenvector = spla.eig(A)[1][:,0] right_eigenvalue = ((A @ right_eigenvector) / right_eigenvector)[0] potential_left_eigenvector = right_eigenvector[np.newaxis, :] # if all components of the following are not the same, then it's not # a left eigenvector print((potential_left_eigenvector @ A) / potential_left_eigenvector) # prints [[-1.1327836 -0.j -0.14850693-0.j -1.84397691+0.j]]
        crdrost 6y ago
        Your first matrix doesn't count, but the theorem is bogus: see my counterexample above. The correct theorem is that every complex square matrix has an eigenvector. If we interpret your matrix as a complex matrix, then it does have two eigenvectors, namely [1; ±i]. Hence why I’d say it kind of “doesn’t count.”
- ssivark 6y ago
  An eigenvector would correspond to the "stationary distribution" of the Markov transition matrix represented by the graph. (think "page rank")
ssivark 6y ago
Was just coming in to point to this. While the geometric intuition in the post being discussed is quite visual, thinking of matrix elements as directed arrows generalizes easily to high-dimensional vector spaces, and also makes it very easy to understand the behavior of sparse transformation matrices, and motivates a nice correspondence between linear algebra and graph algorithms (ref. graphblas)
bee_rider 6y ago
This seems more accurate than the original post, which seems quite tied to this code/data analogy. OTOH, it is just vectors in a many-dimensional space. We're neigh-supernatural machines for describing vectors, that's how we throw things better than any other animal, I don't really know why we need analogies here. Why do we want x'x? It maps to a distance...
deleted 6y ago
[deleted]
tsjq 6y ago
Thanks for linking to this.

crdrost 6y ago

That is genuinely a really fun way to look at it, thank you for linking this! Especially this fits nicely with Markov matrices where you have N input nodes and N output nodes and the sum of all of the probabilities coming out of one of the nodes needs to equal 1. What I might find a little more difficult to teach to people through this lens is the phenomenon of eigenvectors, but I suppose that's to be expected—nothing will work well for all purposes.

layoutIfNeeded 6y ago
>What I might find a little more difficult to teach to people through this lens is the phenomenon of eigenvectors Why? Eigenvectors are simply inputs to the network where the output keeps its shape, that is, at most it gets rescaled, as if you had applied a uniform gain to the components, but otherwise it will be the same as the input.
- tonyarkles 6y ago
  Oh. Oh my. I have studied linear algebra theory some. I have used it a ton. I have used eigenvectors to solve problems. I have never grokked them. This description changed that. Thank you!
- crdrost 6y ago
  So for example let me ask for your knee-jerk opinions based on this idea: - does a matrix have the same left-eigenvectors as its right-eigenvectors? - what is the relationship between the left-eigenvalues and right-eigenvalues? - is there always an eigenvector? when is there a complete set? how do you generalize your notion of eigenvectors so that matrices always have a complete set of them? I am not sure any of these are intuitive here.
  - pmiller2 6y ago
    If A is a square matrix, then a left eigenvector v is a vector such that vA = \lambda_v v for some \lambda_v. Likewise, if u is a right eigenvector of A, Au = \lambda_u u. Notice that u and v cannot be equal, because they are not the same shape. However, if v is a right eigenvector with eigenvalue \lambda, then v^T is a left eigenvector with eigenvalue \lambda, as well. More or less what this means is that we tend to just ignore left eigenvalues and only use the right eigenvalues, because the math is exactly the same up to a transpose operation. A matrix does not necessarily have nontrivial eigenvectors. Think about the 0 matrix here. But, if A is nonzero, then it must have at least one nonzero eigenvalue, hence one nontrivial eigenvector. This is because a nonzero matrix must have at least one nonzero row and column; however, if you construct the other rows (columns) to be multiples of the first column, you end up with a matrix with only one nontrivial eigenvalue. This also illustrates the conditions necessary for an nxn matrix A to have n linearly independent eigenvectors: the rows of A must be linearly independent. All of this is covered in a decent undergrad linear algebra course. I would suggest either finding a video course, or getting a good book and working through it, if you want to understand these things better.
    - philip-b 6y ago
      >But, if A is nonzero, then it must have at least one nonzero eigenvalue, hence one nontrivial eigenvector. How about this matrix? [[0, -1], [[1, 0]] >However, if v is a right eigenvector with eigenvalue \lambda, then v^T is a left eigenvector with eigenvalue \lambda, as well. A snippet of code producing a counterexample: import numpy as np import scipy.linalg as spla A = np.random.randn(3, 3) right_eigenvector = spla.eig(A)[1][:,0] right_eigenvalue = ((A @ right_eigenvector) / right_eigenvector)[0] potential_left_eigenvector = right_eigenvector[np.newaxis, :] # if all components of the following are not the same, then it's not # a left eigenvector print((potential_left_eigenvector @ A) / potential_left_eigenvector) # prints [[-1.1327836 -0.j -0.14850693-0.j -1.84397691+0.j]]
      - crdrost 6y ago
        Your first matrix doesn't count, but the theorem is bogus: see my counterexample above. The correct theorem is that every complex square matrix has an eigenvector. If we interpret your matrix as a complex matrix, then it does have two eigenvectors, namely [1; ±i]. Hence why I’d say it kind of “doesn’t count.”
ssivark 6y ago
An eigenvector would correspond to the "stationary distribution" of the Markov transition matrix represented by the graph. (think "page rank")

ssivark 6y ago

Was just coming in to point to this. While the geometric intuition in the post being discussed is quite visual, thinking of matrix elements as directed arrows generalizes easily to high-dimensional vector spaces, and also makes it very easy to understand the behavior of sparse transformation matrices, and motivates a nice correspondence between linear algebra and graph algorithms (ref. graphblas)

bee_rider 6y ago

This seems more accurate than the original post, which seems quite tied to this code/data analogy. OTOH, it is just vectors in a many-dimensional space. We're neigh-supernatural machines for describing vectors, that's how we throw things better than any other animal, I don't really know why we need analogies here. Why do we want x'x? It maps to a distance...

deleted 6y ago

[deleted]

tsjq 6y ago

Thanks for linking to this.