5 ms·

I studied physics, and I did a bit of 3D graphics back in the day, and I could never remember the difference between column and row vectors (or co- and contrava

I studied physics, and I did a bit of 3D graphics back in the day, and I could never remember the difference between column and row vectors (or co- and contravariant vectors, bra and ket vectors, ...) and when to use which. My guilty secret is that I don't care. For me a vector is just abstractly three or four numbers together. (Maybe with some defined transformation properties if I'm doing physics.) Whether a matrix is stored row-major or column-major is an implementation detail I try to hide. In fact I used to make indexing errors all the time. But now when I need to deal with vectors and matrices, I look up or decide on the "correct" way, hide everything behind Vector and Matrix classes, implement dot and cross product once, and never fall back to the raw vector elements if possible.

chombier 4y ago
If one thinks of vectors as the matrix of some linear map over the scalars (and similarly covectors as linear maps to the scalars), and assuming the usual composition order of maps then it seems there is no choice as to the column/row representation: vectors have to be (n x 1) and covectors have to be (1 x n) This doesn't say anything regarding their practical storage, but at least conceptually vectors/covectors are a bit more than just a collection of numbers. This distinction is pretty useful in practice when encoded in types to avoid implicitly mixing things that are conceptually different.
IIAOPSW 4y ago
I was with you until you said bra and ket didn't do it for you. Its right there in the name bra(c)ket. <bra|, |ket>. Logically equiv to any other notation, but impossible to fudge up. When they are multiplied in a way that lines up like <bra|ket>, you got an inner product (scalar measure of overlap). When they are multiplied wrong way around, you got a ketbra |i><j| (which is basically a neat way to represent a matrix element at indices i and j). Is the bra the row and the ket the column or vice versa? Don't matter. Just keep things facing whichever way they were originally facing and nothing will go wrong. In fact, you don't even need to think in terms of columns at all if you don't want. That's it. That's the whole system. For the engineers who never seen this, |i> |j> |k> work exactly like unit vectors i,j,k, but you aren't limited to the usual 3 dimensions and 3 letters. Writing |x> in a ket like this just means "I am using x as a label for a degree of freedom in some vector space, not as the literal value x". This lets you do cool shit, like if you got n dimensions you can write a vector as sum v_i |i> ranging from 0 to n. You can also do super useful stuff like merging two vector spaces together to represent a larger system (aka a tensor product). Eg if I have a vector v = sum v_i |i> representing one subsystem and another vector w = sum w_j |j> representing some other subsystem, I can construct the vector u = sum v_i w_j |i, j> representing the whole system (sum is over both i and j). (I know you already learned all this, I wrote it for everyone else's benefit). Unrelated, given the webs origin at cern, HTML should have used bras as start tags and kets as end tags. <HTML| <head|head> <body| ... |body> |HTML>.
- azalemeth 4y ago
  Fun anecdote: when I was a teenager learning LaTeX, I'd got in the habit of google image searching the symbols – "Latex infinity" would turn up a pic of an infinity symbol with "\infty" below it; "latex contour integral symbol" would show "\oint", and so on. As a nineteen year old first-year university student, I genuinely typed "latex bra" into google image search, pressed enter, and then had a second or two of shock as my brain was genuinely surprised to not see quantum mechanics in front of me with a nifty little "\langle" involved, but rather lots of women in decidedly unique looking underwear...
  - sam-s 4y ago
    I learned that "garbage collection" is not just a memory management technique but also refers to big trucks removing the content of trash cans when I took a "nerd test".
    - dbcurtis 4y ago
      One of the architects of the Pentium Pro, Intel’s first out-of-order execution engine, told of walking up to a Coke machine with an “out of order” sign on it — “Oh, out-of-order has other meanings.”
      - IIAOPSW 4y ago
        The vending machine at intel drops a can of coke. A CPU architect walking by picks it up then inserts his money. The machine is out of order.
- sfpotter 4y ago
  If you define u to be a column vector, and u' to be its (conjugate) transpose, and define matrix multiplication as usual, then u'v is a dot product and uv' is an outer product. What you mentioned about the standard basis vectors in R^3 works for standard basis vectors e_i in R^n using this notation; e.g., e_i e_j' is an all zero matrix with a one in the ith row and jth column. If you have a hard time remembering whether u should be a column vector, you can simply identify things that look like u with vectors (|ket>), and things that look like u' with linear functionals (<bra|). Then u'u is just a dot product. There are other notations, too... Sometimes (u, v) and u \cdot v are inner products, sometimes u \otimes v is an outer product. Generally linear algebra-style notation is terser and has less line noise. IMO, some manipulations are simpler to think about using this notation than with the bra-ket notation, especially when you're doing numerical optimization or numerical linear algebra. Maybe another reason to prefer it is that modern notation for operator algebra follows this notation and not the bra-ket notation (although I think bra-ket notation was invented for this purpose...). If L is a linear operator, L^* is its adjoint, Lu is the application of L to u, etc... Of course, what notation you should use is determined by what community you belong to. If you're publishing papers on numerical linear algebra, it would be very strange to decide to use the bra-ket notation instead. The purpose of notation is communication. The goal is to communicate ideas to other researchers. It's good to develop the ability to be flexible with notation and go along with what other people are doing.
ogogmad 4y ago
I imagine that's the right way to do it. Treating a linear map as a table of numbers is too concrete, and there are far too many conventions for how to do it. Abstractly, it's a type-(1,1) tensor, which is naturally isomorphic to a linear map. (I'm slightly abusing how I use the term "naturally isomorphic"). There are many notations for tensors that don't mention indices, including the somewhat surprising Abstract Index Notation by Roger Penrose. For the sake of completion: A vector is a type-(1,0) tensor. A linear form (or "covector" if you prefer this term) is a type-(0,1) tensor. A bilinear form is a type-(0,2) tensor. The point of saying the type of a tensor, is to clarify what geometric object an array of numbers represents. A curse (and a blessing) of matrix notation is that it doesn't tell you this. Both a type-(0,2) and type-(1,1) tensor can be represented by matrices, but the first is used for (let's say) dot products, and the second is used for linear transformations. The only problem with type-(m,n) notation is remembering whether "m" is the contravariant order or the covariant order. It turns out that it's contravariant, which is the opposite to what the naming would suggest. In programming, we would do {cov: m, con: n}, so we wouldn't have to remember.
geokon 4y ago
Unfortunately this physics-guy/spherical-cow thinking is really common in people doing linear algebra on computers and the results are poor performance. Abstractions are great for general programming, but when you are doing tight-loop math it's really not applicable. I've seen this a lot recently with "clever" NDArray style math libraries (not to mention the utility of more than 2 dimensions is .. very limited) In my not-super-extensive experience, if you're programming and dealing with linear algebra problems - and you're past the MATLAB prototyping stage - then I really suggest using the BLAS/LAPACK API. You're not gunna beat decades of nerds programming ballistic missiles on punchcards. They're kinda weird and unwieldy and don't match to textbook math - but they've had a ton of thought put into them. And are made with as few footguns as possible. At least re-implementing an algo with them a few times is really educational. You'll see that it forces you to think about your memory layout and you'll realize that the back of napkin complexity calculations are actually tricky to massage into a good algorithms that use memory well. The final result with something like the Intel MKL will be way way faster than anything your abstraction can achieve (Unfortunately I haven't found a good primer on using the BLAS well)
- halotrope 4y ago
  "You're not gunna beat decades of nerds programming ballistic missiles on punchcards" funny and true
- sfpotter 4y ago
  MATLAB uses BLAS and LAPACK under the hood. You can write reasonably fast code in MATLAB if most of your time is spent calling out to faster low level implementations. There are also many different implementations of BLAS, with varying performance. The API itself is not the reason for its performance. Actually, I would argue that it’s a pretty crufty API which should probably be done away with at this point and replaced with something less awkward. I don’t know how many times I’ve had to remind myself what a “leading dimension” was… A good example is BLIS, which does provide the BLAS API but has a more modern “object-style” C API which is significantly easier to work with than the BLAS API with no performance hit. Also, depending on what you’re doing, abstractions can be very helpful indeed to keep around. Even at the BLAS level, groups of bits are thought of as singles, doubles, real or complex… a matrix has a size and shape… etc. Having this information is useful. Linear algebra is full of type information which can be used to dispatch different algorithms. Lots of other alternatives to BLAS which are a bit higher level but still very useful: Eigen and Armadillo in C++, Julia, etc. Anyway, I would say (based on my actual, significant experience ;-) ) people are beating the guys with punch cards all the time, no reason to stay in the 70s…
pfortuny 4y ago
They should not be calles vectors both. Horizontal vectors are forms (maps from R^n to R). Vertical vectors are vectors (elements of R^n). Forms applied to vectors are “rows multiplied by columns”, and so: numbers. Same thing as bras and kets, by the way.
- lupire 4y ago
  This is a great example of how math is generally weak at types (constantly doing implicit casts and conversions with ambiguous notation, all the way back to Liebniz notation for calculus, and probably further), and the computer scientists add clarity. It's also an example of how "making it simpler" by dropping types (old Python, Bash, typeless vectors and matrices) ends up making it harder to get work done right when the ideas are non-trivial.
  - pfortuny 4y ago
    Well, yes. But it is another way of saying that we think in analogies. At the same time, forms are also vectors… and vectors are forms of the dual… So it is not that easy.
- captainmuon 4y ago
  If you have two pointy arrows, and want to know if they are perpendicular, then you use the scalar product u⋅v and check if it is 0. There is no difference between u and v in this case. You could say v is a vector, and u a function taking a vector and returning a scalar (a form, a dual vector, ...). But often that is unneccessary artifice. You can just say they are both vectors in the same vector space, which is imbued with an inner product. Declaring that some objects are forms (duals, covariant vectors, ...) is only neccessary when you want to keep track of transformation properties, for example.

chombier 4y ago

If one thinks of vectors as the matrix of some linear map over the scalars (and similarly covectors as linear maps to the scalars), and assuming the usual composition order of maps then it seems there is no choice as to the column/row representation: vectors have to be (n x 1) and covectors have to be (1 x n) This doesn't say anything regarding their practical storage, but at least conceptually vectors/covectors are a bit more than just a collection of numbers. This distinction is pretty useful in practice when encoded in types to avoid implicitly mixing things that are conceptually different.

IIAOPSW 4y ago

I was with you until you said bra and ket didn't do it for you. Its right there in the name bra(c)ket. <bra|, |ket>. Logically equiv to any other notation, but impossible to fudge up. When they are multiplied in a way that lines up like <bra|ket>, you got an inner product (scalar measure of overlap). When they are multiplied wrong way around, you got a ketbra |i><j| (which is basically a neat way to represent a matrix element at indices i and j). Is the bra the row and the ket the column or vice versa? Don't matter. Just keep things facing whichever way they were originally facing and nothing will go wrong. In fact, you don't even need to think in terms of columns at all if you don't want. That's it. That's the whole system. For the engineers who never seen this, |i> |j> |k> work exactly like unit vectors i,j,k, but you aren't limited to the usual 3 dimensions and 3 letters. Writing |x> in a ket like this just means "I am using x as a label for a degree of freedom in some vector space, not as the literal value x". This lets you do cool shit, like if you got n dimensions you can write a vector as sum v_i |i> ranging from 0 to n. You can also do super useful stuff like merging two vector spaces together to represent a larger system (aka a tensor product). Eg if I have a vector v = sum v_i |i> representing one subsystem and another vector w = sum w_j |j> representing some other subsystem, I can construct the vector u = sum v_i w_j |i, j> representing the whole system (sum is over both i and j). (I know you already learned all this, I wrote it for everyone else's benefit). Unrelated, given the webs origin at cern, HTML should have used bras as start tags and kets as end tags. <HTML| <head|head> <body| ... |body> |HTML>.

azalemeth 4y ago
Fun anecdote: when I was a teenager learning LaTeX, I'd got in the habit of google image searching the symbols – "Latex infinity" would turn up a pic of an infinity symbol with "\infty" below it; "latex contour integral symbol" would show "\oint", and so on. As a nineteen year old first-year university student, I genuinely typed "latex bra" into google image search, pressed enter, and then had a second or two of shock as my brain was genuinely surprised to not see quantum mechanics in front of me with a nifty little "\langle" involved, but rather lots of women in decidedly unique looking underwear...
- sam-s 4y ago
  I learned that "garbage collection" is not just a memory management technique but also refers to big trucks removing the content of trash cans when I took a "nerd test".
  - dbcurtis 4y ago
    One of the architects of the Pentium Pro, Intel’s first out-of-order execution engine, told of walking up to a Coke machine with an “out of order” sign on it — “Oh, out-of-order has other meanings.”
    - IIAOPSW 4y ago
      The vending machine at intel drops a can of coke. A CPU architect walking by picks it up then inserts his money. The machine is out of order.
sfpotter 4y ago
If you define u to be a column vector, and u' to be its (conjugate) transpose, and define matrix multiplication as usual, then u'v is a dot product and uv' is an outer product. What you mentioned about the standard basis vectors in R^3 works for standard basis vectors e_i in R^n using this notation; e.g., e_i e_j' is an all zero matrix with a one in the ith row and jth column. If you have a hard time remembering whether u should be a column vector, you can simply identify things that look like u with vectors (|ket>), and things that look like u' with linear functionals (<bra|). Then u'u is just a dot product. There are other notations, too... Sometimes (u, v) and u \cdot v are inner products, sometimes u \otimes v is an outer product. Generally linear algebra-style notation is terser and has less line noise. IMO, some manipulations are simpler to think about using this notation than with the bra-ket notation, especially when you're doing numerical optimization or numerical linear algebra. Maybe another reason to prefer it is that modern notation for operator algebra follows this notation and not the bra-ket notation (although I think bra-ket notation was invented for this purpose...). If L is a linear operator, L^* is its adjoint, Lu is the application of L to u, etc... Of course, what notation you should use is determined by what community you belong to. If you're publishing papers on numerical linear algebra, it would be very strange to decide to use the bra-ket notation instead. The purpose of notation is communication. The goal is to communicate ideas to other researchers. It's good to develop the ability to be flexible with notation and go along with what other people are doing.

ogogmad 4y ago

I imagine that's the right way to do it. Treating a linear map as a table of numbers is too concrete, and there are far too many conventions for how to do it. Abstractly, it's a type-(1,1) tensor, which is naturally isomorphic to a linear map. (I'm slightly abusing how I use the term "naturally isomorphic"). There are many notations for tensors that don't mention indices, including the somewhat surprising Abstract Index Notation by Roger Penrose. For the sake of completion: A vector is a type-(1,0) tensor. A linear form (or "covector" if you prefer this term) is a type-(0,1) tensor. A bilinear form is a type-(0,2) tensor. The point of saying the type of a tensor, is to clarify what geometric object an array of numbers represents. A curse (and a blessing) of matrix notation is that it doesn't tell you this. Both a type-(0,2) and type-(1,1) tensor can be represented by matrices, but the first is used for (let's say) dot products, and the second is used for linear transformations. The only problem with type-(m,n) notation is remembering whether "m" is the contravariant order or the covariant order. It turns out that it's contravariant, which is the opposite to what the naming would suggest. In programming, we would do {cov: m, con: n}, so we wouldn't have to remember.

geokon 4y ago

Unfortunately this physics-guy/spherical-cow thinking is really common in people doing linear algebra on computers and the results are poor performance. Abstractions are great for general programming, but when you are doing tight-loop math it's really not applicable. I've seen this a lot recently with "clever" NDArray style math libraries (not to mention the utility of more than 2 dimensions is .. very limited) In my not-super-extensive experience, if you're programming and dealing with linear algebra problems - and you're past the MATLAB prototyping stage - then I really suggest using the BLAS/LAPACK API. You're not gunna beat decades of nerds programming ballistic missiles on punchcards. They're kinda weird and unwieldy and don't match to textbook math - but they've had a ton of thought put into them. And are made with as few footguns as possible. At least re-implementing an algo with them a few times is really educational. You'll see that it forces you to think about your memory layout and you'll realize that the back of napkin complexity calculations are actually tricky to massage into a good algorithms that use memory well. The final result with something like the Intel MKL will be way way faster than anything your abstraction can achieve (Unfortunately I haven't found a good primer on using the BLAS well)

halotrope 4y ago
"You're not gunna beat decades of nerds programming ballistic missiles on punchcards" funny and true
sfpotter 4y ago
MATLAB uses BLAS and LAPACK under the hood. You can write reasonably fast code in MATLAB if most of your time is spent calling out to faster low level implementations. There are also many different implementations of BLAS, with varying performance. The API itself is not the reason for its performance. Actually, I would argue that it’s a pretty crufty API which should probably be done away with at this point and replaced with something less awkward. I don’t know how many times I’ve had to remind myself what a “leading dimension” was… A good example is BLIS, which does provide the BLAS API but has a more modern “object-style” C API which is significantly easier to work with than the BLAS API with no performance hit. Also, depending on what you’re doing, abstractions can be very helpful indeed to keep around. Even at the BLAS level, groups of bits are thought of as singles, doubles, real or complex… a matrix has a size and shape… etc. Having this information is useful. Linear algebra is full of type information which can be used to dispatch different algorithms. Lots of other alternatives to BLAS which are a bit higher level but still very useful: Eigen and Armadillo in C++, Julia, etc. Anyway, I would say (based on my actual, significant experience ;-) ) people are beating the guys with punch cards all the time, no reason to stay in the 70s…

pfortuny 4y ago

They should not be calles vectors both. Horizontal vectors are forms (maps from R^n to R). Vertical vectors are vectors (elements of R^n). Forms applied to vectors are “rows multiplied by columns”, and so: numbers. Same thing as bras and kets, by the way.

lupire 4y ago
This is a great example of how math is generally weak at types (constantly doing implicit casts and conversions with ambiguous notation, all the way back to Liebniz notation for calculus, and probably further), and the computer scientists add clarity. It's also an example of how "making it simpler" by dropping types (old Python, Bash, typeless vectors and matrices) ends up making it harder to get work done right when the ideas are non-trivial.
- pfortuny 4y ago
  Well, yes. But it is another way of saying that we think in analogies. At the same time, forms are also vectors… and vectors are forms of the dual… So it is not that easy.
captainmuon 4y ago
If you have two pointy arrows, and want to know if they are perpendicular, then you use the scalar product u⋅v and check if it is 0. There is no difference between u and v in this case. You could say v is a vector, and u a function taking a vector and returning a scalar (a form, a dual vector, ...). But often that is unneccessary artifice. You can just say they are both vectors in the same vector space, which is imbued with an inner product. Declaring that some objects are forms (duals, covariant vectors, ...) is only neccessary when you want to keep track of transformation properties, for example.